Francesco Romani has uploaded a new change for review.
Change subject: vm: always held confLock when updating pauseCode ......................................................................
vm: always held confLock when updating pauseCode
Updating pauseCode can create a new field in Vm.conf. If a client calls the getVMList() API, or for any other API which needs to loop over all the VM conf, we can end up with
RuntimeError: dictionary changed size during iteration
This patch make sure that when we update Vm.conf with pauseCode-related data, we always have Vm.confLock held.
Change-Id: Ie84e2442165fb524f2433fab7faa3b049b340038 Backport-To: 3.6 Backport-To: 3.5 Signed-off-by: Francesco Romani fromani@redhat.com --- M vdsm/virt/vm.py 1 file changed, 14 insertions(+), 9 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/05/46005/1
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index df8debc..cee448d 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -714,7 +714,8 @@
self.lastStatus = vmstatus.UP if self._initTimePauseCode: - self.conf['pauseCode'] = self._initTimePauseCode + with self._confLock: + self.conf['pauseCode'] = self._initTimePauseCode if self._initTimePauseCode == 'ENOSPC': self.cont() else: @@ -1371,8 +1372,9 @@ 'clientIp': self.conf.get('clientIp', ''), }
- if 'pauseCode' in self.conf: - stats['pauseCode'] = self.conf['pauseCode'] + with self._confLock: + if 'pauseCode' in self.conf: + stats['pauseCode'] = self.conf['pauseCode'] if self.isMigrating(): stats['migrationProgress'] = self.migrateStatus()['progress']
@@ -1757,7 +1759,8 @@ vmstatus.RESTORING_STATE): self._initTimePauseCode = self._readPauseCode() if not self.recovering and self._initTimePauseCode: - self.conf['pauseCode'] = self._initTimePauseCode + with self._confLock: + self.conf['pauseCode'] = self._initTimePauseCode if self._initTimePauseCode == 'ENOSPC': self.cont()
@@ -1866,10 +1869,11 @@ self.log.info(domxml)
flags = libvirt.VIR_DOMAIN_NONE - if 'launchPaused' in self.conf: - flags |= libvirt.VIR_DOMAIN_START_PAUSED - self.conf['pauseCode'] = 'NOERR' - del self.conf['launchPaused'] + with self._confLock: + if 'launchPaused' in self.conf: + flags |= libvirt.VIR_DOMAIN_START_PAUSED + self.conf['pauseCode'] = 'NOERR' + del self.conf['launchPaused'] self._dom = virdomain.Notifying( self._connection.createXML(domxml, flags), self._timeoutExperienced) @@ -3609,7 +3613,8 @@ if action == libvirt.VIR_DOMAIN_EVENT_IO_ERROR_PAUSE: self.log.info('abnormal vm stop device %s error %s', blockDevAlias, err) - self.conf['pauseCode'] = reason + with self._confLock: + self.conf['pauseCode'] = reason self._setGuestCpuRunning(False) self._logGuestCpuStatus('onIOError') if reason == 'ENOSPC':
automation@ovirt.org has posted comments on this change.
Change subject: vm: always held confLock when updating pauseCode ......................................................................
Patch Set 1:
* Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])
automation@ovirt.org has posted comments on this change.
Change subject: vm: always held confLock when updating pauseCode ......................................................................
Patch Set 2:
* Update tracker::#1261918::OK * Check Bug-Url::OK * Check Public Bug::#1261918::OK, public bug * Check Product::#1261918::OK, Correct product Red Hat Enterprise Virtualization Manager * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])
automation@ovirt.org has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 3:
* Update tracker::#1261918::OK * Check Bug-Url::OK * Check Public Bug::#1261918::OK, public bug * Check Product::#1261918::OK, Correct product Red Hat Enterprise Virtualization Manager * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])
Michal Skrivanek has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 3: Code-Review+1
Martin Polednik has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 3: Code-Review+1
Francesco Romani has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 3: Verified+1
verified carefully inspecting the flows to avoid double lock (seems OK) and doing various mass start of VMs with no RuntimeError spotted; being a race, the second test proves little, but still.
Dan Kenigsberg has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 3: Code-Review+2
Dan Kenigsberg has submitted this change and it was merged.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
vm: always hold confLock when updating pauseCode
An update of pauseCode can create a new field in Vm.conf. If a client calls the getVMList() API, or for any other API which needs to loop over all the VM conf, we can end up with
RuntimeError: dictionary changed size during iteration
This patch make sure that when we update Vm.conf with pauseCode-related data, we always have Vm.confLock held.
Change-Id: Ie84e2442165fb524f2433fab7faa3b049b340038 Bug-Url: https://bugzilla.redhat.com/1261918 Backport-To: 3.6 Backport-To: 3.5 Signed-off-by: Francesco Romani fromani@redhat.com Reviewed-on: https://gerrit.ovirt.org/46005 Continuous-Integration: Jenkins CI Reviewed-by: Michal Skrivanek michal.skrivanek@redhat.com Reviewed-by: Martin Polednik mpolednik@redhat.com Reviewed-by: Dan Kenigsberg danken@redhat.com --- M vdsm/virt/vm.py 1 file changed, 14 insertions(+), 9 deletions(-)
Approvals: Jenkins CI: Passed CI tests Dan Kenigsberg: Looks good to me, approved Francesco Romani: Verified Michal Skrivanek: Looks good to me, but someone else must approve Martin Polednik: Looks good to me, but someone else must approve
automation@ovirt.org has posted comments on this change.
Change subject: vm: always hold confLock when updating pauseCode ......................................................................
Patch Set 4:
* Update tracker::#1261918::OK * Set MODIFIED::bug 1261918::::#1261918::::IGNORE, not oVirt prod but Red Hat Enterprise Virtualization Manager
vdsm-patches@lists.fedorahosted.org