April 2016 - vdsm-patches - Fedora mailing-lists

Change in vdsm[master]: [WIP]ABRT integration
by ykaplan＠redhat.com 26 Mar '17

26 Mar '17

Yeela Kaplan has uploaded a new change for review. Change subject: [WIP]ABRT integration ...................................................................... [WIP]ABRT integration Change-Id: I1ca5e66c9f029be75483b86414e328d074c7e454 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=917062 Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M init/systemd/vdsmd.service.in M init/vdsmd_init_common.sh.in M lib/vdsm/tool/configurators/Makefile.am A lib/vdsm/tool/configurators/abrt.py M vdsm.spec.in M vdsm/vdsm-logrotate 6 files changed, 138 insertions(+), 28 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/87/44287/1 diff --git a/init/systemd/vdsmd.service.in b/init/systemd/vdsmd.service.in index d7d025f..ccacb2e 100644 --- a/init/systemd/vdsmd.service.in +++ b/init/systemd/vdsmd.service.in @@ -2,9 +2,10 @@ Description=Virtual Desktop Server Manager Requires=multipathd.service libvirtd.service time-sync.target \ iscsid.service rpcbind.service supervdsmd.service sanlock.service \ - vdsm-network.service + vdsm-network.service abrtd.service After=multipathd.service libvirtd.service iscsid.service rpcbind.service \ - supervdsmd.service sanlock.service vdsm-network.service + supervdsmd.service sanlock.service vdsm-network.service \ + abrtd.service Conflicts=libvirt-guests.service Wants=mom-vdsm.service diff --git a/init/vdsmd_init_common.sh.in b/init/vdsmd_init_common.sh.in index 00e4054..df2878c 100644 --- a/init/vdsmd_init_common.sh.in +++ b/init/vdsmd_init_common.sh.in @@ -83,16 +83,6 @@ return 0 } -task_configure_coredump() { - local conf_file="@CONFDIR@/vdsm.conf" - local getconfitem="@VDSMDIR@/get-conf-item" - - if "${getconfitem}" "${conf_file}" vars core_dump_enable false | - tr A-Z a-z | grep -q true; then - echo "/var/log/core/core.%p.%t.dump" > /proc/sys/kernel/core_pattern - fi -} - task_configure_vdsm_logs() { local vdsm_logs=" @VDSMLOGDIR@/connectivity.log @@ -296,7 +286,6 @@ --pre-start) run_tasks " \ mkdirs \ - configure_coredump \ configure_vdsm_logs \ wait_for_network \ run_init_hooks \ diff --git a/lib/vdsm/tool/configurators/Makefile.am b/lib/vdsm/tool/configurators/Makefile.am index 9fc7e57..f70b4bb 100644 --- a/lib/vdsm/tool/configurators/Makefile.am +++ b/lib/vdsm/tool/configurators/Makefile.am @@ -21,6 +21,7 @@ dist_configurators_PYTHON = \ __init__.py \ + abrt.py \ certificates.py \ libvirt.py \ multipath.py \ diff --git a/lib/vdsm/tool/configurators/abrt.py b/lib/vdsm/tool/configurators/abrt.py new file mode 100644 index 0000000..778f505 --- /dev/null +++ b/lib/vdsm/tool/configurators/abrt.py @@ -0,0 +1,132 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import sys +import os + +from .import \ + MAYBE, \ + NO +from .. import service +from ... import utils +from ... import constants + +from .. import conf_utils + +CONF_VERSION = '4.17.0' + + +services = ("abrtd",) + + +def configure(): + + conf_utils.remove_conf(FILES, CONF_VERSION) + + for conf_file, content in FILES.items(): + content['configure'](content, CONF_VERSION) + + +def isconfigured(): + """ + Check if abrt is already configured for vdsm + """ + ret = MAYBE + + for path in (conf_utils.get_persisted_files(FILES)): + if not conf_utils.open_config(path, CONF_VERSION).hasConf(): + ret = NO + + if ret == MAYBE: + sys.stdout.write("abrt is already configured for vdsm\n") + else: + sys.stdout.write("abrt is not configured for vdsm\n") + return ret + + +FILES = { + 'ABRT_CONF': { + 'path': os.path.join( + constants.SYSCONF_PATH, + 'abrt/abrt.conf' + ), + 'configure': conf_utils.add_section, + 'removeConf': conf_utils.remove_section, + 'persisted': True, + 'fragments': [ + { + 'conditions': {}, + 'content': { + 'DumpLocation': '/var/tmp/abrt' + }, + }, + ] + }, + 'CCPP_CONF': { + 'path': os.path.join( + constants.SYSCONF_PATH, + 'abrt/plugins/CCpp.conf' + ), + 'configure': conf_utils.add_section, + 'removeConf': conf_utils.remove_section, + 'persisted': True, + 'fragments': [ + { + 'conditions': {}, + 'content': { + 'Backtrace': 'yes' + }, + }, + ] + }, + 'PYTHON_CONF': { + 'path': os.path.join( + constants.SYSCONF_PATH, + 'abrt/plugins/python.conf' + ), + 'configure': conf_utils.add_section, + 'removeConf': conf_utils.remove_section, + 'persisted': True, + 'fragments': [ + { + 'conditions': {}, + 'content': { + 'Backtrace': 'yes' + }, + }, + ] + }, + 'PKG_CONF': { + 'path': os.path.join( + constants.SYSCONF_PATH, + 'abrt/abrt-action-save-package-data.conf' + ), + 'configure': conf_utils.add_section, + 'removeConf': conf_utils.remove_section, + 'persisted': True, + 'fragments': [ + { + 'conditions': {}, + 'content': { + 'openGPGCheck': 'no' + }, + }, + ] + } +} diff --git a/vdsm.spec.in b/vdsm.spec.in index 21e1b50..6e524c5 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -132,6 +132,7 @@ Requires: nfs-utils Requires: m2crypto Requires: libnl3 +Requires: abrt-desktop Requires: curl Requires: %{name}-xmlrpc = %{version}-%{release} Requires: %{name}-jsonrpc = %{version}-%{release} @@ -709,13 +710,6 @@ /etc/sysctl.conf fi -# hack until we replace core dump with abrt -if /usr/sbin/selinuxenabled; then - /usr/sbin/semanage fcontext -a -t virt_cache_t '/var/log/core(/.*)?' -fi -/sbin/restorecon -R /var/log/core >/dev/null 2>&1 -# hack until we replace core dump with abrt - # VDSM installs vdsm-modules-load.d.conf file - the following command will # refresh vdsm kernel modules requirements to start on boot /bin/systemctl restart systemd-modules-load.service >/dev/null 2>&1 || : @@ -1097,6 +1091,7 @@ %{python_sitelib}/%{vdsm_name}/tool/conf_utils.py* %{python_sitelib}/%{vdsm_name}/tool/configurator.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/__init__* +%{python_sitelib}/%{vdsm_name}/tool/configurators/abrt.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/certificates.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/libvirt.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/passwd.py* diff --git a/vdsm/vdsm-logrotate b/vdsm/vdsm-logrotate index e2027ac..17d26ab 100755 --- a/vdsm/vdsm-logrotate +++ b/vdsm/vdsm-logrotate @@ -7,12 +7,4 @@ /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi -if [ -d /var/log/core ] ; then - /usr/bin/find /var/log/core -type f -name '*xz' -mtime +7 -exec /bin/rm -f '{}' \; - EXITVALUE=$? - if [ $EXITVALUE != 0 ]; then - /usr/bin/logger -t logrotate "ALERT clean old core files exited abnormally with [$EXITVALUE]" - fi -fi - exit $EXITVALUE -- To view, visit https://gerrit.ovirt.org/44287 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1ca5e66c9f029be75483b86414e328d074c7e454 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

3 8

Change in vdsm[master]: [WIP]configurators: move configurator functions into conf_utils
by ykaplan＠redhat.com 26 Mar '17

26 Mar '17

Yeela Kaplan has uploaded a new change for review. Change subject: [WIP]configurators: move configurator functions into conf_utils ...................................................................... [WIP]configurators: move configurator functions into conf_utils Change-Id: Id67a0f51adb2d543c0542a48c020d34a13d1f2ed Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M lib/vdsm/tool/Makefile.am A lib/vdsm/tool/conf_utils.py M vdsm.spec.in 3 files changed, 110 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/86/44286/1 diff --git a/lib/vdsm/tool/Makefile.am b/lib/vdsm/tool/Makefile.am index c8979ca..c3ed0cc 100644 --- a/lib/vdsm/tool/Makefile.am +++ b/lib/vdsm/tool/Makefile.am @@ -35,7 +35,8 @@ dump_bonding_defaults.py \ dump_volume_chains.py \ nwfilter.py \ - configfile.py \ + conf_utils.py \ + configfile.py \ configurator.py \ register.py \ restore_nets.py \ diff --git a/lib/vdsm/tool/conf_utils.py b/lib/vdsm/tool/conf_utils.py new file mode 100644 index 0000000..1308097 --- /dev/null +++ b/lib/vdsm/tool/conf_utils.py @@ -0,0 +1,107 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +import os + +from .. configfile import ( + ConfigFile, +) + +from ... import utils + +if utils.isOvirtNode(): + from ovirt.node.utils.fs import Config as NodeCfg + + +def get_file(fname, files): + return files[fname]['path'] + + +def remove_conf(files, version): + for cfile, content in files.items(): + content['removeConf'](content['path'], version) + + +def add_section(content, version, vdsmConfiguration={}): + """ + Add a 'configuration section by vdsm' part to a config file. + This section contains only keys not originally defined + The section headers will include the current configuration version. + """ + configuration = {} + for fragment in content['fragments']: + if vdsmConfiguration: + if is_applicable(fragment, vdsmConfiguration): + configuration.update(fragment['content']) + else: + configuration.update(fragment['content']) + if configuration: + with open_config(content['path'], version) as conff: + for key, val in configuration.items(): + conff.addEntry(key, val) + + +def remove_section(path, version): + """ + remove entire 'configuration section by vdsm' section. + section is removed regardless of it's version. + """ + if os.path.exists(path): + with open_config(path, version) as conff: + conff.removeConf() + + +def remove_file(content, vdsmConfiguration): + """ + delete a file if it exists. + """ + if utils.isOvirtNode(): + NodeCfg().delete(content['path']) + else: + try: + os.unlink(content['path']) + except OSError as e: + if e.errno != errno.ENOENT: + raise + + +def get_persisted_files(files): + """ + get files where vdsm is expected to add a section. + """ + return [ + cfile['path'] for cfile in files.values() + if cfile['persisted'] + ] + + +def open_config(path, conf_version): + return ConfigFile(path, conf_version) + + +def is_applicable(fragment, vdsmConfiguration): + """ + Return true if 'fragment' should be included for current + configuration. An applicable fragment is a fragment who's list + of conditions are met according to vdsmConfiguration. + """ + applyFragment = True + for key, booleanValue in fragment['conditions'].items(): + if vdsmConfiguration[key] != booleanValue: + applyFragment = False + return applyFragment diff --git a/vdsm.spec.in b/vdsm.spec.in index 5020d8b..21e1b50 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -1094,6 +1094,7 @@ %{python_sitelib}/%{vdsm_name}/tool/dummybr.py* %{python_sitelib}/%{vdsm_name}/tool/dump_bonding_defaults.py* %{python_sitelib}/%{vdsm_name}/tool/nwfilter.py* +%{python_sitelib}/%{vdsm_name}/tool/conf_utils.py* %{python_sitelib}/%{vdsm_name}/tool/configurator.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/__init__* %{python_sitelib}/%{vdsm_name}/tool/configurators/certificates.py* -- To view, visit https://gerrit.ovirt.org/44286 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id67a0f51adb2d543c0542a48c020d34a13d1f2ed Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

3 6

Change in vdsm[master]: contrib: Simple jsonrpc client
by Nir Soffer 30 Nov '16

30 Nov '16

Nir Soffer has uploaded a new change for review. Change subject: contrib: Simple jsonrpc client ...................................................................... contrib: Simple jsonrpc client This is a simple jsonrpc client for communicating with the jsonrpc server from the command line. Arguments method one of the mehtods described in json schema params optionl json object with message parameters Examples Calling method without arguements: # jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] } Calling method with arguements: # jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] } Requires stomp.py library: https://pypi.python.org/pypi/stomp.py Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A contrib/jsonrpc 1 file changed, 104 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/81/35181/1 diff --git a/contrib/jsonrpc b/contrib/jsonrpc new file mode 100755 index 0000000..3080193 --- /dev/null +++ b/contrib/jsonrpc @@ -0,0 +1,104 @@ +#!/usr/bin/python +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +""" +jsonrpc-cli - Vdsm jsonrpc client +""" + +import json +import os +import signal +import sys +import time +import uuid +import optparse + +import stomp + +# Copied from lib/vdsm/vdscli.py +PKIDIR = '/etc/pki/vdsm' +KEYFILE = os.path.join(PKIDIR, 'keys/vdsmkey.pem') +CERTFILE = os.path.join(PKIDIR, 'certs/vdsmcert.pem') +CACERT = os.path.join(PKIDIR, 'certs/cacert.pem') + +PORT = 54321 +DESTINATION = "/queue/_local/vdsm/requests" + + +class Listener(stomp.ConnectionListener): + + def on_error(self, headers, message): + print 'Error: %s' % message + terminate() + + def on_message(self, headers, message): + msg = json.loads(message) + print json.dumps(msg, indent=4) + terminate() + + +def main(args): + parser = option_parser() + options, args = parser.parse_args(args) + if not args: + parser.error("method required") + + msg = { + "id": str(uuid.uuid4()), + "jsonrpc": "2.0", + "method": args[0] + } + + if len(args) > 1: + msg["params"] = json.loads(args[1]) + + conn = stomp.Connection10( + host_and_ports=((options.host, PORT),), + use_ssl=True, + ssl_key_file=KEYFILE, + ssl_cert_file=CERTFILE, + ssl_ca_certs=CACERT) + + conn.set_listener("", Listener()) + conn.start() + conn.send(body=json.dumps(msg), destination=DESTINATION) + + try: + signal.pause() + except KeyboardInterrupt: + pass + + conn.disconnect() + + +def option_parser(): + parser = optparse.OptionParser(usage='%prog [options] method [params]') + parser.add_option("-a", "--host", dest="host", + help="host address (default localhost)") + parser.set_defaults(host="localhost") + return parser + + +def terminate(): + os.kill(os.getpid(), signal.SIGINT) + + +if __name__ == "__main__": + main(sys.argv[1:]) -- To view, visit http://gerrit.ovirt.org/35181 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 53

Change in vdsm[master]: spbackends: simplify over-general evaluation
by Dan Kenigsberg 27 Nov '16

27 Nov '16

Dan Kenigsberg has uploaded a new change for review. Change subject: spbackends: simplify over-general evaluation ...................................................................... spbackends: simplify over-general evaluation The removed code is too clever for the use case of only two values. Change-Id: I775c4a6003a03c6b65bee3abc8559a88e3eaec01 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M vdsm/storage/spbackends.py 1 file changed, 5 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/41253/1 diff --git a/vdsm/storage/spbackends.py b/vdsm/storage/spbackends.py index 953e01e..f6ce02b 100644 --- a/vdsm/storage/spbackends.py +++ b/vdsm/storage/spbackends.py @@ -229,8 +229,11 @@ def setSpmStatus(self, lVer=None, spmId=None): self.invalidateMetadata() - metaParams = dict(filter(lambda kv: kv[1] is not None, - ((PMDK_LVER, lVer), (PMDK_SPM_ID, spmId)))) + metaParams = {} + if lVer is not None: + metaParams[PMDK_LVER] = lVer + if spmId is not None: + metaParams[PMDK_SPM_ID] = spmId self._metadata.update(metaParams) @unsecured -- To view, visit https://gerrit.ovirt.org/41253 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I775c4a6003a03c6b65bee3abc8559a88e3eaec01 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

5 6

Change in vdsm[master]: virt: enable libgfapi
by ahino＠redhat.com 21 Oct '16

21 Oct '16

Ala Hino has uploaded a new change for review. Change subject: virt: enable libgfapi ...................................................................... virt: enable libgfapi This change is based on Federico's changes: https://gerrit.ovirt.org/33768/ Change-Id: I54b81e87b959b0b49c0f06810f88410e7c75de1d Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> Signed-off-by: Ala Hino <ahino(a)redhat.com> --- M vdsm/storage/glusterVolume.py M vdsm/storage/hsm.py M vdsm/virt/vm.py 3 files changed, 22 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/61/44061/1 diff --git a/vdsm/storage/glusterVolume.py b/vdsm/storage/glusterVolume.py index 8f701ba..109d1c9 100644 --- a/vdsm/storage/glusterVolume.py +++ b/vdsm/storage/glusterVolume.py @@ -54,4 +54,5 @@ return {'volType': VmVolumeInfo.TYPE_NETWORK, 'path': glusterPath, 'protocol': 'gluster', 'volPort': volPort, 'volTransport': volTrans, - 'volfileServer': volfileServer} + 'volfileServer': volfileServer, + 'bricks': volInfo[volname]['bricks']} diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index f68d3bb..109cdfa 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3209,8 +3209,7 @@ path = os.path.join(dom.domaindir, sd.DOMAIN_IMAGES, imgUUID, volUUID) volInfo = {'domainID': sdUUID, 'imageID': imgUUID, - 'volumeID': volUUID, 'path': path, - 'volType': "path"} + 'volumeID': volUUID, 'path': path} leasePath, leaseOffset = dom.getVolumeLease(imgUUID, volUUID) @@ -3221,8 +3220,8 @@ }) imgVolumesInfo.append(volInfo) - if volUUID == leafUUID: - leafInfo = volInfo + + leafInfo = dom.produceVolume(imgUUID, leafUUID).getVmVolumeInfo() return {'path': leafPath, 'info': leafInfo, 'imgVolumesInfo': imgVolumesInfo} diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 71a74b3..2976f3b 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -805,6 +805,23 @@ # A destroy request has been issued, exit early break drive['path'] = self.cif.prepareVolumePath(drive, self.id) + if drive.get('diskType') == DISK_TYPE.NETWORK: + volinfo = drive.get('volumeInfo') + drive['path'] = volinfo['path'] + drive['protocol'] = volinfo['protocol'] + if drive.get('hosts') is None: + """ + THIS CODE COMMENTED OUT DUE TO A BUG IN LIBVIRT, + CANNOT WORK WITH MULTIPLE HOSTS + """ + """ + hosts = [dict(name=brick.split(":")[0], + port='0', transport='tcp') + for brick in volinfo['bricks']] + """ + hosts = [dict(name=volinfo['bricks'][0].split(":")[0], + port='0', transport='tcp')] + drive['hosts'] = hosts else: # Now we got all the resources we needed -- To view, visit https://gerrit.ovirt.org/44061 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54b81e87b959b0b49c0f06810f88410e7c75de1d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ala Hino <ahino(a)redhat.com>

5 45

Change in vdsm[master]: tests: Add basic tests for RWLock
by alitke＠redhat.com 16 Oct '16

16 Oct '16

Adam Litke has uploaded a new change for review. Change subject: tests: Add basic tests for RWLock ...................................................................... tests: Add basic tests for RWLock Change-Id: I77b9ea1cd6d378738fbf14de119c88bf0ebc94e2 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/miscTests.py 1 file changed, 18 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/42772/1 diff --git a/tests/miscTests.py b/tests/miscTests.py index 976b503..c6d9690 100644 --- a/tests/miscTests.py +++ b/tests/miscTests.py @@ -1268,3 +1268,21 @@ def _run(self): self.result = self._func() + + +class RWLockTests(TestCaseBase): + + def test_multiple_acquire(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(exclusive=True)) + self.assertTrue(lock.acquire(exclusive=True)) + + def test_demote(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(exclusive=True)) + self.assertTrue(lock.acquire(exclusive=False)) + + def test_promote(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(False)) + self.assertRaises(RuntimeError, lock.acquire, True) -- To view, visit https://gerrit.ovirt.org/42772 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I77b9ea1cd6d378738fbf14de119c88bf0ebc94e2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 9

Change in vdsm[master]: lvm: Fail loudly if called with unexptected input.
by Nir Soffer 16 Oct '16

16 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: lvm: Fail loudly if called with unexptected input. ...................................................................... lvm: Fail loudly if called with unexptected input. When creating pvs with the force option, we are very carefull to accept only True. When using the jsonrpc transport, engine was sending "true" and "false", causing the call to fail misteiously. Now we are also carefull about rejecting invlid input, making debugging easier. Change-Id: If9e6754d4aa2efaf894a9309cfaa4595d710063b Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/lvm.py 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/29/37329/1 diff --git a/vdsm/storage/lvm.py b/vdsm/storage/lvm.py index aa3c04b..549839a 100644 --- a/vdsm/storage/lvm.py +++ b/vdsm/storage/lvm.py @@ -724,6 +724,11 @@ else: raise + # We must be very carefull here; any value execpt True or False is a user + # error. + if type(force) != bool: + raise ValueError("Invalid value for 'force': %r" % force) + if force is True: options = ("-y", "-ff") _initpvs_removeHolders() -- To view, visit http://gerrit.ovirt.org/37329 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If9e6754d4aa2efaf894a9309cfaa4595d710063b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 19

Change in vdsm[master]: hsm: Report vg name in getDeviceList
by Nir Soffer 09 Oct '16

09 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: hsm: Report vg name in getDeviceList ...................................................................... hsm: Report vg name in getDeviceList Hosted engine needs the iscsi session info used by the hosted engine storage domain. getDeviceList() seems to include the needed info, but it does not report the vg name for each device, making it hard to match the iscsi session info and the hosted engine storage domain. We return now the vg name of each device, which seems to be useful info regardless of hosted engine needs, and can be used on the engine side for reconstructing host state or validating engine view vs host view. Change-Id: I116714cb5143ea92f5cb54c3f80f895c07ada536 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/rpc/vdsmapi-schema.json M vdsm/storage/hsm.py 2 files changed, 11 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/23/45823/1 diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index e0e95b9..1b2a171 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -1473,6 +1473,10 @@ # # @status: The device status (free/used/unknown) # +# @vgname: The LVM volume group name, if this device is used as +# a physical volume. This is typically a storage domain +# UUID. +# # Since: 4.10.0 # # Notes: The value of @serial may be dependent on the current host so this @@ -1490,7 +1494,8 @@ 'pathstatus': ['BlockDevicePathInfo'], 'pathlist': ['IscsiSessionInfo'], 'logicalblocksize': 'uint', 'physicalblocksize': 'uint', 'partitioned': 'bool', - 'pvsize': 'uint', 'status': 'BlockDeviceStatus'}} + 'pvsize': 'uint', 'status': 'BlockDeviceStatus', + 'vgname': 'str'}} ## # @Host.getDeviceList: diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 1b8c064..32e16c3 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -2019,14 +2019,18 @@ pvuuid = pv.uuid pvsize = pv.size vguuid = pv.vg_uuid + vgname = pv.vg_name else: pvuuid = "" pvsize = "" vguuid = "" + vgname = "" devInfo = {'GUID': dev.get("guid", ""), 'pvUUID': pvuuid, 'pvsize': str(pvsize), - 'vgUUID': vguuid, 'vendorID': dev.get("vendor", ""), + 'vgUUID': vguuid, + 'vgname': vgname, + 'vendorID': dev.get("vendor", ""), 'productID': dev.get("product", ""), 'fwrev': dev.get("fwrev", ""), "serial": dev.get("serial", ""), -- To view, visit https://gerrit.ovirt.org/45823 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I116714cb5143ea92f5cb54c3f80f895c07ada536 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 22

Change in vdsm[master]: utils: Consider sleep time in deadline calculation
by Nir Soffer 09 Oct '16

09 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: utils: Consider sleep time in deadline calculation ...................................................................... utils: Consider sleep time in deadline calculation This patches fixes 2 utils.retry() broken tests: - The special case when deadline has reached when an operation was done. Previously we use to sleep and perform another retry, now we bail out. - The special case when deadline was not reached when an operation was done, but we don't have time for sleep (sleeping will reach or exceed the deadline). Previously we used to sleep and perform another retry, now we bail out. Change-Id: I7f41c6b21e3432159c13d46cfe75d1f6236cbb8c Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/utils.py M tests/utilsTests.py 2 files changed, 3 insertions(+), 8 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/00/46400/1 diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py index caf4cc2..4a4b153 100644 --- a/lib/vdsm/utils.py +++ b/lib/vdsm/utils.py @@ -938,10 +938,7 @@ if tries in [0, None]: tries = -1 - if timeout in [0, None]: - timeout = -1 - - startTime = monotonic_time() + deadline = monotonic_time() + timeout if timeout else None while True: tries -= 1 @@ -951,7 +948,7 @@ if tries == 0: raise - if (timeout > 0) and ((monotonic_time() - startTime) > timeout): + if deadline and monotonic_time() + sleep >= deadline: raise if stopCallback is not None and stopCallback(): diff --git a/tests/utilsTests.py b/tests/utilsTests.py index d213de5..fd4f42e 100644 --- a/tests/utilsTests.py +++ b/tests/utilsTests.py @@ -44,7 +44,7 @@ from testlib import permutations, expandPermutations from testlib import VdsmTestCase as TestCaseBase from testValidation import checkSudo -from testValidation import brokentest, stresstest +from testValidation import stresstest from multiprocessing import Process EXT_SLEEP = "sleep" @@ -86,7 +86,6 @@ # Make sure we had the proper amount of iterations before failing self.assertEquals(counter[0], limit) - @brokentest("deadline is not respected") @MonkeyPatch(utils, 'monotonic_time', FakeTime(0)) @MonkeyPatch(time, 'sleep', fake_sleep) def testTimeoutDeadlineReached(self): @@ -104,7 +103,6 @@ timeout=3, sleep=1) self.assertEqual(utils.monotonic_time.now, 3) - @brokentest("sleep is not considered in deadline calculation") @MonkeyPatch(utils, 'monotonic_time', FakeTime(0)) @MonkeyPatch(time, 'sleep', fake_sleep) def testTimeoutNoTimeForSleep(self): -- To view, visit https://gerrit.ovirt.org/46400 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7f41c6b21e3432159c13d46cfe75d1f6236cbb8c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 10

Change in vdsm[master]: hsm: Pep8ize function name
by Nir Soffer 09 Oct '16

09 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: hsm: Pep8ize function name ...................................................................... hsm: Pep8ize function name Remove unneeded _private prefix for inner function and use lowercase name. Change-Id: I54715971378319a8501a49cb89b24f27e50a07f4 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/06/39306/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index a4bb858..8f75a39 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -2070,7 +2070,7 @@ boolean :rtype: dict """ - def _isVisible(guid): + def is_visible(guid): path = os.path.join('/dev/mapper', guid) try: st = os.stat(path) @@ -2081,10 +2081,10 @@ else: return (st.st_mode & stat.S_IRUSR) != 0 - visibility = [_isVisible(guid) for guid in guids] + visibility = [is_visible(guid) for guid in guids] if not all(visibility): multipath.rescan() - visibility = [_isVisible(guid) for guid in guids] + visibility = [is_visible(guid) for guid in guids] visibility = dict(zip(guids, visibility)) # After multipath.rescan, existing devices may disapper, and new -- To view, visit https://gerrit.ovirt.org/39306 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54715971378319a8501a49cb89b24f27e50a07f4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 9

Change in vdsm[master]: Live Merge: Restore watermark tracking
by alitke＠redhat.com 09 Oct '16

09 Oct '16

Adam Litke has uploaded a new change for review. Change subject: Live Merge: Restore watermark tracking ...................................................................... Live Merge: Restore watermark tracking Change-Id: I632f31e7795ec5d8c6f52a480116b14470c3163f Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 108 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/36924/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index f22610d..09080b9 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1512,12 +1512,94 @@ with self._confLock: self.conf['timeOffset'] = newTimeOffset + def _getWriteWatermarks(self): + def pathToVolID(drive, path): + for vol in drive.volumeChain: + if os.path.realpath(vol['path']) == os.path.realpath(path): + return vol['volumeID'] + raise LookupError("Unable to find VolumeID for path '%s'", path) + + volAllocMap = {} + statsFlags = self._libvirtBackingChainStatsFlag() + conn = libvirtconnection.get() + blkStats = conn.domainListGetStats([self._dom._dom], + libvirt.VIR_DOMAIN_STATS_BLOCK, + statsFlags)[0][1] + for i in xrange(0, blkStats['block.count']): + name = blkStats['block.%i.name' % i] + try: + drive = self._findDriveByName(name) + except LookupError: + continue + if not drive.blockDev or drive.format != 'cow': + continue + + try: + path = blkStats['block.%i.path' % i] + alloc = blkStats['block.%i.allocation' % i] + except KeyError as e: + self.log.debug("Block stats are missing expected key '%s', " + "skipping volume", e.args[0]) + continue + volID = pathToVolID(drive, path) + volAllocMap[volID] = alloc + return volAllocMap + + def _getLiveMergeExtendCandidates(self): + # The common case is that there are no active jobs. + if not self.conf['_blockJobs'].values(): + return {} + + candidates = {} + watermarks = self._getWriteWatermarks() + for job in self.conf['_blockJobs'].values(): + try: + drive = self._findDriveByUUIDs(job['disk']) + except LookupError: + # After an active layer merge completes the vdsm metadata will + # be out of sync for a brief period. If we cannot find the old + # disk then it's safe to skip it. + continue + + if not drive.blockDev: + continue + + if job['strategy'] == 'commit': + volumeID = job['baseVolume'] + else: + self.log.debug("Unrecognized merge strategy '%s'", + job['strategy']) + continue + res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, + drive.imageID, volumeID) + if res['status']['code'] != 0: + self.log.error("Unable to get the info of volume %s (domain: " + "%s image: %s)", volumeID, drive.domainID, + drive.imageID) + continue + volInfo = res['info'] + + if volInfo['format'].lower() != 'cow': + continue + + if volumeID in watermarks: + self.log.debug("Adding live merge extension candidate: " + "volume=%s allocation=%i", volumeID, + watermarks[volumeID]) + candidates[drive.imageID] = { + 'alloc': watermarks[volumeID], + 'physical': int(volInfo['truesize']), + 'capacity': int(volInfo['apparentsize']), + 'volumeID': volumeID} + else: + self.log.warning("No watermark info available for %s", + volumeID) + return candidates + def _getExtendCandidates(self): ret = [] - # FIXME: mergeCandidates should be a dictionary of candidate volumes - # once libvirt starts reporting watermark information for all volumes. - mergeCandidates = {} + mergeCandidates = self._getLiveMergeExtendCandidates() for drive in self._devices[hwclass.DISK]: if not drive.blockDev or drive.format != 'cow': continue @@ -4771,6 +4853,14 @@ jobsRet[jobID] = entry return jobsRet + def _libvirtBackingChainStatsFlag(self): + # Since libvirt 1.2.13, the virConnectGetAllDomainStats API will return + # block statistics for all volumes in the chain when using a new flag. + try: + return libvirt.VIR_CONNECT_GET_ALL_DOMAINS_STATS_BACKING + except AttributeError: + return 0 + def merge(self, driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID): if not caps.getLiveMergeSupport(): self.log.error("Live merge is not supported on this host") @@ -4815,6 +4905,8 @@ if res['info']['voltype'] == 'SHARED': self.log.error("merge: Refusing to merge into a shared volume") return errCode['mergeErr'] + baseSize = int(res['info']['apparentsize']) + baseCow = bool(res['info']['format'].lower() == 'cow') # Indicate that we expect libvirt to maintain the relative paths of # backing files. This is necessary to ensure that a volume chain is @@ -4865,13 +4957,19 @@ # blockCommit will cause data to be written into the base volume. # Perform an initial extension to ensure there is enough space to - # copy all the required data. Normally we'd use monitoring to extend - # the volume on-demand but internal watermark information is not being - # reported by libvirt so we must do the full extension up front. In - # the worst case, we'll need to extend 'base' to the same size as 'top' - # plus a bit more to accomodate additional writes to 'top' during the - # live merge operation. - self.extendDriveVolume(drive, baseVolUUID, topSize) + # copy all the required data. If libvirt supports monitoring of + # backing chain volumes, just extend by one chunk now and monitor + # during the rest of the operation. Otherwise, extend now to + # accomodate the worst case scenario: no intersection between the + # allocated blocks in the base volume and the top volume. + if drive.blockDev and baseCow: + if self._libvirtBackingChainStatsFlag(): + self.extendDrivesIfNeeded() + else: + extendSize = baseSize + topSize + self.log.debug("Preemptively extending volume %s with size %i" + "(job: %s)", baseVolUUID, extendSize, jobUUID) + self.extendDriveVolume(drive, baseVolUUID, extendCurSize) # Trigger the collection of stats before returning so that callers # of getVmStats after this returns will see the new job -- To view, visit http://gerrit.ovirt.org/36924 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I632f31e7795ec5d8c6f52a480116b14470c3163f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

6 24

Change in vdsm[master]: GuestIF Refactoring
by Vinzenz Feenstra 09 Oct '16

09 Oct '16

Vinzenz Feenstra has uploaded a new change for review. Change subject: GuestIF Refactoring ...................................................................... GuestIF Refactoring Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d Signed-off-by: Vinzenz Feenstra <vfeenstr(a)redhat.com> --- M vdsm/guestIF.py 1 file changed, 114 insertions(+), 76 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/18/24618/1 diff --git a/vdsm/guestIF.py b/vdsm/guestIF.py index 229a55d..96ad68c 100644 --- a/vdsm/guestIF.py +++ b/vdsm/guestIF.py @@ -39,6 +39,115 @@ union(set(range(0x86, 0x9F + 1))) +class UnknownMessageError(Exception): + def __init__(self, message, args): + Exception.__init__(self, 'Unknown or unsupported guest agent message ' + '"%s" received with args "%s"' % (message, + str(args))) + + +class MessageHandler(object): + def __init__(self, agent): + self.log = agent.log + self._agent = agent + + def __call__(self, message, args): + handler = self.getattr(self, message.replace('-', '_'), None) + if handler: + handler(args) + else: + raise UnknownMessageError(message, args) + + def applications(self, args): + self._agent.guestInfo['appsList'] = args['applications'] + + def fqdn(self, args): + self._agent.guestInfo['guestFQDN'] = args['fqdn'] + + def host_name(self, args): + self._agent.guestInfo['guestName'] = args['name'] + + def os_version(self, args): + self._agent.guestInfo['guestOs'] = args['version'] + + def session_lock(self, args): + self.agent.guestInfo['session'] = 'Locked' + + def session_logoff(self, args): + self.agent.guestInfo['session'] = 'LoggedOff' + + def session_logon(self, args): + self.agent.guestInfo['session'] = 'UserLoggedOn' + + def session_unlock(self, args): + self.agent.guestInfo['session'] = 'Active' + + def session_shutdown(self, args): + self.log.debug('Guest system shuts down') + + def session_startup(self, args): + self.log.debug('Guest system started or restarted') + + def uninstalled(self, args): + self.log.debug('Guest agent was uninstalled') + self._agent.guestInfo['appsList'] = [] + + def heartbeat(self, args): + self._agent.guestStatus = 'Up' + self._agent.guestInfo['memUsage'] = int(args['free-ram']) + # ovirt-guest-agent reports the following fields in 'memory-stat': + # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out', + # 'pageflt' and 'majflt' + if 'memory-stat' in args: + for (k, v) in args['memory-stat'].iteritems(): + # Convert the value to string since 64-bit integer is not + # supported in XMLRPC + self._agent.guestInfo['memoryStats'][k] = str(v) + + if 'apiVersion' in args: + # The guest agent supports API Versioning + self._agent._handleAPIVersion(args['apiVersion']) + elif self._agent.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO: + # Older versions of the guest agent (before the introduction + # of API versioning) do not report this field + # Disable the API if not already disabled (e.g. after + # downgrade of the guest agent) + self.log.debug("API versioning no longer reported by guest.") + self._agent.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO + + def network_interfaces(self, args): + interfaces = [] + old_ips = '' + for iface in args['interfaces']: + iface['inet'] = iface.get('inet', []) + iface['inet6'] = iface.get('inet6', []) + interfaces.append(iface) + # Provide the old information which includes + # only the IP addresses. + old_ips += ' '.join(iface['inet']) + ' ' + self._agent.guestInfo['netIfaces'] = interfaces + self._agent.guestInfo['guestIPs'] = old_ips.strip() + + def active_user(self, args): + currentUser = args['name'] + if ((currentUser != self._agent.guestInfo['username']) and + not (currentUser == 'Unknown' and + self._agent.guestInfo['username'] == 'None')): + self._agent.guestInfo['username'] = currentUser + self._agent.guestInfo['lastLogin'] = time.time() + self.log.debug("username: %s", repr(self.guestInfo['username'])) + + def disks_usage(self, args): + disks = [] + for disk in args['disks']: + # Converting to string because XML-RPC doesn't support 64-bit + # integers. + disk['total'] = str(disk['total']) + disk['used'] = str(disk['used']) + disks.append(disk) + self._agent.guestInfo['disksUsage'] = disks + + def _filterXmlChars(u): """ The set of characters allowed in XML documents is described in @@ -109,6 +218,7 @@ def __init__(self, socketName, channelListener, log, user='Unknown', ips='', connect=True): + self.handler = MessageHandler(self) self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO self.log = log self._socketName = socketName @@ -223,82 +333,10 @@ self.log.log(logging.TRACE, "Guest's message %s: %s", message, args) if self.guestStatus is None: self.guestStatus = 'Up' - if message == 'heartbeat': - self.guestStatus = 'Up' - self.guestInfo['memUsage'] = int(args['free-ram']) - # ovirt-guest-agent reports the following fields in 'memory-stat': - # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out', - # 'pageflt' and 'majflt' - if 'memory-stat' in args: - for (k, v) in args['memory-stat'].iteritems(): - # Convert the value to string since 64-bit integer is not - # supported in XMLRPC - self.guestInfo['memoryStats'][k] = str(v) - - if 'apiVersion' in args: - # The guest agent supports API Versioning - self._handleAPIVersion(args['apiVersion']) - elif self.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO: - # Older versions of the guest agent (before the introduction - # of API versioning) do not report this field - # Disable the API if not already disabled (e.g. after - # downgrade of the guest agent) - self.log.debug("API versioning no longer reported by guest.") - self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO - elif message == 'host-name': - self.guestInfo['guestName'] = args['name'] - elif message == 'os-version': - self.guestInfo['guestOs'] = args['version'] - elif message == 'network-interfaces': - interfaces = [] - old_ips = '' - for iface in args['interfaces']: - iface['inet'] = iface.get('inet', []) - iface['inet6'] = iface.get('inet6', []) - interfaces.append(iface) - # Provide the old information which includes - # only the IP addresses. - old_ips += ' '.join(iface['inet']) + ' ' - self.guestInfo['netIfaces'] = interfaces - self.guestInfo['guestIPs'] = old_ips.strip() - elif message == 'applications': - self.guestInfo['appsList'] = args['applications'] - elif message == 'active-user': - currentUser = args['name'] - if ((currentUser != self.guestInfo['username']) and - not (currentUser == 'Unknown' and - self.guestInfo['username'] == 'None')): - self.guestInfo['username'] = currentUser - self.guestInfo['lastLogin'] = time.time() - self.log.debug("username: %s", repr(self.guestInfo['username'])) - elif message == 'session-logon': - self.guestInfo['session'] = "UserLoggedOn" - elif message == 'session-lock': - self.guestInfo['session'] = "Locked" - elif message == 'session-unlock': - self.guestInfo['session'] = "Active" - elif message == 'session-logoff': - self.guestInfo['session'] = "LoggedOff" - elif message == 'uninstalled': - self.log.debug("RHEV agent was uninstalled.") - self.guestInfo['appsList'] = [] - elif message == 'session-startup': - self.log.debug("Guest system is started or restarted.") - elif message == 'fqdn': - self.guestInfo['guestFQDN'] = args['fqdn'] - elif message == 'session-shutdown': - self.log.debug("Guest system shuts down.") - elif message == 'disks-usage': - disks = [] - for disk in args['disks']: - # Converting to string because XML-RPC doesn't support 64-bit - # integers. - disk['total'] = str(disk['total']) - disk['used'] = str(disk['used']) - disks.append(disk) - self.guestInfo['disksUsage'] = disks - else: - self.log.error('Unknown message type %s', message) + try: + self.handler(message, args) + except UnknownMessageError as e: + self.log.error(e) def stop(self): self._stopped = True -- To view, visit http://gerrit.ovirt.org/24618 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Vinzenz Feenstra <vfeenstr(a)redhat.com>

5 13

Change in vdsm[master]: network: Use new concurrent.thread() utility
by Nir Soffer 25 Sep '16

25 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: network: Use new concurrent.thread() utility ...................................................................... network: Use new concurrent.thread() utility This patch updates the networking subsystem to use the new utility. Behavior changes: - dhclient.DhcpClient threads are protected from silent failures - configurators/ifcfg._ifup threads are proected from silent failures. Change-Id: I62e80bbbb9354d3173cce631ed5579532cf7cdcb Relates-To: https://bugzilla.redhat.com/1141422 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/network/configurators/dhclient.py M vdsm/network/configurators/ifcfg.py M vdsm/network/sourceroutethread.py 3 files changed, 8 insertions(+), 14 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/45553/1 diff --git a/vdsm/network/configurators/dhclient.py b/vdsm/network/configurators/dhclient.py index 181c302..a25ae8f 100644 --- a/vdsm/network/configurators/dhclient.py +++ b/vdsm/network/configurators/dhclient.py @@ -23,8 +23,8 @@ import logging import os import signal -import threading +from vdsm import concurrent from vdsm import cmdutils from vdsm import ipwrapper from vdsm import netinfo @@ -76,9 +76,8 @@ rc, _, _ = self._dhclient() return rc else: - t = threading.Thread(target=self._dhclient, name='vdsm-dhclient-%s' - % self.iface) - t.daemon = True + t = concurrent.thread(self._dhclient, + name='vdsm-dhclient-%s' % self.iface) t.start() def shutdown(self): diff --git a/vdsm/network/configurators/ifcfg.py b/vdsm/network/configurators/ifcfg.py index e1d3e94..f676f83 100644 --- a/vdsm/network/configurators/ifcfg.py +++ b/vdsm/network/configurators/ifcfg.py @@ -28,11 +28,11 @@ import re import selinux import shutil -import threading from libvirt import libvirtError, VIR_ERR_NO_NETWORK from vdsm.config import config +from vdsm import concurrent from vdsm import cmdutils from vdsm import constants from vdsm import ipwrapper @@ -782,9 +782,8 @@ if not iface.blockingdhcp and (iface.ipv4.bootproto == 'dhcp' or iface.ipv6.dhcpv6): # wait for dhcp in another thread, so vdsm won't get stuck (BZ#498940) - t = threading.Thread(target=_exec_ifup, name='ifup-waiting-on-dhcp', - args=(iface.name, cgroup)) - t.daemon = True + t = concurrent.thread(_exec_ifup, name='ifup-waiting-on-dhcp', + args=(iface.name, cgroup)) t.start() else: _exec_ifup(iface.name, cgroup) diff --git a/vdsm/network/sourceroutethread.py b/vdsm/network/sourceroutethread.py index 0a49760..042e5bd 100644 --- a/vdsm/network/sourceroutethread.py +++ b/vdsm/network/sourceroutethread.py @@ -19,12 +19,11 @@ from __future__ import absolute_import import logging import os -import threading import pyinotify from vdsm.constants import P_VDSM_RUN -from vdsm import utils +from vdsm import concurrent from .configurators.iproute2 import Iproute2 from .sourceroute import DynamicSourceRoute @@ -68,13 +67,10 @@ def start(): - thread = threading.Thread(target=_subscribeToInotifyLoop, - name='sourceRoute') - thread.daemon = True + thread = concurrent.thread(_subscribeToInotifyLoop, name='sourceRoute') thread.start() -(a)utils.traceback() def _subscribeToInotifyLoop(): logging.debug("sourceRouteThread.subscribeToInotifyLoop started") -- To view, visit https://gerrit.ovirt.org/45553 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I62e80bbbb9354d3173cce631ed5579532cf7cdcb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 9

Change in vdsm[master]: libvirtconnection: Replace assert with AssertionError
by Nir Soffer 25 Sep '16

25 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: libvirtconnection: Replace assert with AssertionError ...................................................................... libvirtconnection: Replace assert with AssertionError The code wrongly assumed that assert always exists. When running in optimized mode, the check would be skipped, and instead of getting an AssertionError, which is the expected error for programmer error (starting the eventloop twice), we could get a confusing RuntimeException or RuntimeError from Thread.start (depending on Python version). RuntimeError misused in the standard library for all kinds of errors that do not have builtin errors. It is particularry bad option when used for usage error. Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/libvirtconnection.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/64/34364/1 diff --git a/lib/vdsm/libvirtconnection.py b/lib/vdsm/libvirtconnection.py index 5430c82..009f8b7 100644 --- a/lib/vdsm/libvirtconnection.py +++ b/lib/vdsm/libvirtconnection.py @@ -37,7 +37,8 @@ self.__thread = None def start(self): - assert not self.run + if self.run: + raise AssertionError("EventLoop is running") self.__thread = threading.Thread(target=self.__run, name="libvirtEventLoop") self.__thread.setDaemon(True) -- To view, visit http://gerrit.ovirt.org/34364 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 8

Change in vdsm[master]: cache: Add caching decorator with invalidation
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: cache: Add caching decorator with invalidation ...................................................................... cache: Add caching decorator with invalidation The new cache.memoized extends utils.memoized, adding invalidation support. Features added: - An optional "validate" argument. This is a callable invoked each time the memoized function is called. When the callable returns False, the cache is invalidated. - Memoized functions have an "invalidate" method, used to invalidate the cache during testing. - file_validator - invalidates the cache when a file changes. Example usage: from vdsm.cache import memoized, file_validator @memoized(file_validator('/bigfile')) def parse_bigfile(): # Expensive code processing '/bigfile' contents Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/Makefile.am A lib/vdsm/cache.py M tests/Makefile.am A tests/cacheTests.py 4 files changed, 366 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/34709/1 diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am index b862e71..6f0040d 100644 --- a/lib/vdsm/Makefile.am +++ b/lib/vdsm/Makefile.am @@ -23,6 +23,7 @@ dist_vdsmpylib_PYTHON = \ __init__.py \ + cache.py \ compat.py \ define.py \ exception.py \ diff --git a/lib/vdsm/cache.py b/lib/vdsm/cache.py new file mode 100644 index 0000000..9806e40 --- /dev/null +++ b/lib/vdsm/cache.py @@ -0,0 +1,98 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import errno +import os +import functools + + +def memoized(validate=None): + """ + Return a caching decorator supporting invalidation. + + The decorator accepts an optional validate callable, called each time the + memoized function is called. If the validate callable return True, the + memoized function will use the cache. If the validate callable return + False, the memoized cache is cleared. + + The memoized function may accept multiple positional arguments. The + cache store the result for each combination of arguments. Functions with + kwargs are not supported. + + Memoized functions have an "invalidate" method, used to invalidate the + memoized cache during testing. + + To invalidate the cache when a file changes, use the file_validator from + this module. + + Example usage: + + from vdsm.cache import memoized, file_validator + + @memoized(file_validator('/bigfile')) + def parse_bigfile(): + # Expensive code processing '/bigfile' contents + + """ + def decorator(f): + cache = {} + + @functools.wraps(f) + def wrapper(*args): + if validate is not None and not validate(): + cache.clear() + try: + value = cache[args] + except KeyError: + value = cache[args] = f(*args) + return value + + wrapper.invalidate = cache.clear + return wrapper + + return decorator + + +class file_validator(object): + """ + I'm a validator returning False when a file has changed since the last + validation. + """ + + UNKNOWN = 0 + MISSING = 1 + + def __init__(self, path): + self.path = path + self.stats = self.UNKNOWN + + def __call__(self): + try: + stats = os.stat(self.path) + except OSError as e: + if e.errno != errno.ENOENT: + raise + stats = self.MISSING + else: + stats = stats.st_ino, stats.st_size, stats.st_mtime + if stats != self.stats: + self.stats = stats + return False + return True diff --git a/tests/Makefile.am b/tests/Makefile.am index 36a1cdd..6fa7e64 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -26,6 +26,7 @@ alignmentScanTests.py \ blocksdTests.py \ bridgeTests.py \ + cacheTests.py \ cPopenTests.py \ capsTests.py \ clientifTests.py \ diff --git a/tests/cacheTests.py b/tests/cacheTests.py new file mode 100644 index 0000000..8927b39 --- /dev/null +++ b/tests/cacheTests.py @@ -0,0 +1,266 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA +# 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import os +from vdsm.cache import memoized +from vdsm.cache import file_validator +from testlib import VdsmTestCase +from testlib import namedTemporaryDir + + +class Validator(object): + """ I'm a callable returning a boolean value (self.valid) """ + + def __init__(self): + self.valid = True + self.count = 0 + + def __call__(self): + self.count += 1 + return self.valid + + +class Accessor(object): + """ I'm recording how many times a dict was accessed. """ + + def __init__(self, d): + self.d = d + self.count = 0 + + def get(self, key): + self.count += 1 + return self.d[key] + + +class MemoizedTests(VdsmTestCase): + + def setUp(self): + self.values = {'a': 0, 'b': 10, ('a',): 20, ('a', 'b'): 30} + + def test_no_args(self): + accessor = Accessor(self.values) + + @memoized() + def func(key): + return accessor.get(key) + + # Fill the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + + def test_validation(self): + accessor = Accessor(self.values) + validator = Validator() + + @memoized(validator) + def func(key): + return accessor.get(key) + + # Fill the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 4) + + # Values has changed + self.values['a'] += 1 + self.values['b'] += 1 + + # Next call should clear the cache + validator.valid = False + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 3) + self.assertEqual(validator.count, 5) + + # Next call should add next value to cache + validator.valid = True + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 4) + self.assertEqual(validator.count, 6) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 4) + self.assertEqual(validator.count, 8) + + def test_raise_errors_in_memoized_func(self): + accessor = Accessor(self.values) + validator = Validator() + + @memoized(validator) + def func(key): + return accessor.get(key) + + # First run should fail, second shold fill the cache + self.assertRaises(KeyError, func, 'no such key') + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 2) + + def test_multiple_args(self): + accessor = Accessor(self.values) + + @memoized() + def func(*args): + return accessor.get(args) + + # Fill the cache + self.assertEqual(func('a'), self.values[('a',)]) + self.assertEqual(func('a', 'b'), self.values[('a', 'b')]) + self.assertEqual(accessor.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values[('a',)]) + self.assertEqual(func('a', 'b'), self.values[('a', 'b')]) + self.assertEqual(accessor.count, 2) + + def test_kwargs_not_supported(self): + @memoized() + def func(a=None, b=None): + pass + self.assertRaises(TypeError, func, a=1, b=2) + + def test_invalidate(self): + accessor = Accessor(self.values) + + @memoized() + def func(key): + return accessor.get(key) + + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 1) + + func.invalidate() + + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 2) + + +class FileValidatorTests(VdsmTestCase): + + def test_no_file(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + # Must be False so memoise call the decorated function + self.assertEqual(validator(), False) + + # Since file state did not change, must remain True + self.assertEqual(validator(), True) + + def test_file_created(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + with open(path, 'w') as f: + f.write('data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_file_removed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + with open(path, 'w') as f: + f.write('data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + os.unlink(path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_size_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + with open(path, 'w') as f: + f.write(data + ' new data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_mtime_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + # Fake timestamp change, as timestamp resolution may not be good + # enough when comparing changes during the test. + atime = mtime = os.path.getmtime(path) + 1 + os.utime(path, (atime, mtime)) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_ino_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + tmp = path + '.tmp' + with open(tmp, 'w') as f: + f.write(data) + os.rename(tmp, path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) -- To view, visit http://gerrit.ovirt.org/34709 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 40

Change in vdsm[master]: fencing: Make getHostLeaseStatus API public
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: fencing: Make getHostLeaseStatus API public ...................................................................... fencing: Make getHostLeaseStatus API public Getting host lease status will allow engine to make smarter decisions when a host is non-responsive by using a proxy host to get the non-responsive host status. See http://pastebin.com/KqqeAdSu for example output from this API. Change-Id: I415c1fee6256bf8d4e03ee542cc58e193162e9b8 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py M vdsm/rpc/Bridge.py M vdsm/rpc/vdsmapi-schema.json 5 files changed, 62 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/29157/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 2c09b28..aea1503 100644 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1760,6 +1760,18 @@ status = self.s.stopMonitoringDomain(sdUUID) return status['status']['code'], status['status']['message'] + def getHostLeaseStatus(self, args): + domains = {} + for pair in args: + sdUUID, hostId = pair.split('=', 1) + domains[sdUUID] = int(hostId) + response = self.s.getHostLeaseStatus(domains) + if response['status']['code']: + print "Cannot get host storage liveliness" + return response['status']['code'], response['status']['message'] + pp.pprint(response['domains']) + return 0, '' + def snapshot(self, args): vmUUID, sdUUID, imgUUID, baseVolUUID, volUUID = args @@ -2579,6 +2591,11 @@ ('<sdUUID>', 'Stop monitoring SD: sdUUID' )), + 'getHostLeaseStatus': (serv.getHostLeaseStatus, + ('<sdUUID>=<hostId> [<sdUUID>=<hostId>] ...', + 'Returns host lease status for hostId on ' + 'each domain.' + )), 'snapshot': (serv.snapshot, ('<vmId> <sdUUID> <imgUUID> <baseVolUUID> <volUUID>', 'Take a live snapshot' diff --git a/vdsm/API.py b/vdsm/API.py index e739294..0b44459 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1497,6 +1497,9 @@ def stopMonitoringDomain(self, sdUUID): return self._irs.stopMonitoringDomain(sdUUID) + def getHostLeaseStatus(self, domains): + return self._irs.getHostLeaseStatus(domains) + def getLVMVolumeGroups(self, storageType=None): return self._irs.getVGList(storageType) diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index c1c7490..a06a3b4 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -917,6 +917,10 @@ api = API.Global() return api.stopMonitoringDomain(sdUUID) + def getHostLeaseStatus(self, domains, options=None): + api = API.Global() + return api.getHostLeaseStatus(domains) + def vgsGetList(self, storageType=None, options=None): api = API.Global() return api.getLVMVolumeGroups(storageType) @@ -1070,6 +1074,7 @@ (self.storageRepoGetStats, 'repoStats'), (self.startMonitoringDomain, 'startMonitoringDomain'), (self.stopMonitoringDomain, 'stopMonitoringDomain'), + (self.getHostLeaseStatus, 'getHostLeaseStatus'), (self.vgsGetList, 'getVGList'), (self.devicesGetList, 'getDeviceList'), (self.devicesGetVisibility, 'getDevicesVisibility'), diff --git a/vdsm/rpc/Bridge.py b/vdsm/rpc/Bridge.py index 7e898de..ba700d1 100644 --- a/vdsm/rpc/Bridge.py +++ b/vdsm/rpc/Bridge.py @@ -349,6 +349,7 @@ 'Host_getStorageRepoStats': {'ret': Host_getStorageRepoStats_Ret}, 'Host_startMonitoringDomain': {}, 'Host_stopMonitoringDomain': {}, + 'Host_getHostLeaseStatus': {'ret': 'domains'}, 'Host_getVMList': {'call': Host_getVMList_Call, 'ret': Host_getVMList_Ret}, 'Host_getVMFullList': {'call': Host_getVMFullList_Call, 'ret': 'vmList'}, 'Host_getAllVmStats': {'ret': 'statsList'}, diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 0c8a6f6..7617185 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -2052,6 +2052,42 @@ 'returns': ''} ## +# @HostIdMap: +# +# A mapping of hostId indexed by domain UUID. +# +# Since: 4.15.0 +## +{'map': 'HostIdMap', + 'key': 'UUID', 'value': 'int'} + +## +# @HostLeaseStatusMap: +# +# A mapping of status codes indexed by domain UUID. +# +# Since: 4.15.0 +## +{'map': 'HostLeaseStatusMap', + 'key': 'UUID', 'value': 'str'} + +## +# @Host.getHostLeaseStatus: +# +# Returns host status for for specified domains +# +# @domains: A mapping of hostId indexed by domain UUID +# +# Returns: +# Host status code for each domain +# +# Since: 4.15.0 +## +{'command': {'class': 'Host', 'name': 'getHostLeaseStatus'}, + 'data': {'domains': 'HostIdMap'} + 'returns': {'domains': 'HostLeaseStatusMap'}} + +## # @VmStatus: # # An enumeration of possible virtual machine statuses. -- To view, visit http://gerrit.ovirt.org/29157 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I415c1fee6256bf8d4e03ee542cc58e193162e9b8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 14

Change in vdsm[master]: sdc: Rename method to make it less confusing
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: sdc: Rename method to make it less confusing ...................................................................... sdc: Rename method to make it less confusing StorageCache.refresh() does not do any refresh, but clearing the domain cahce and invlidating lvm cache (actually clearing it). Rename to clear() to reflect what it does. Change-Id: I2c67ae0ddc98857e406fec62be0cbcf817213236 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/sdc.py M vdsm/storage/sp.py 2 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/16/47916/1 diff --git a/vdsm/storage/sdc.py b/vdsm/storage/sdc.py index ecb9708..a26c05a 100644 --- a/vdsm/storage/sdc.py +++ b/vdsm/storage/sdc.py @@ -181,7 +181,7 @@ return uuids - def refresh(self): + def clear(self): with self._syncroot: lvm.invalidateCache() self.__domainCache.clear() diff --git a/vdsm/storage/sp.py b/vdsm/storage/sp.py index b8fd8f3..6ecaf8d 100644 --- a/vdsm/storage/sp.py +++ b/vdsm/storage/sp.py @@ -627,7 +627,7 @@ self.id = hostID # Make sure SDCache doesn't have stale data (it can be in case of FC) sdCache.invalidateStorage() - sdCache.refresh() + sdCache.clear() # Rebuild whole Pool self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) self.__createMailboxMonitor() @@ -1245,7 +1245,7 @@ 'msdUUID' - master storage domain UUID """ sdCache.invalidateStorage() - sdCache.refresh() + sdCache.clear() self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) def updateVM(self, vmList, sdUUID): -- To view, visit https://gerrit.ovirt.org/47916 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2c67ae0ddc98857e406fec62be0cbcf817213236 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 7

Change in vdsm[master]: scsi: Scan only the required domain type
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: scsi: Scan only the required domain type ...................................................................... scsi: Scan only the required domain type We used to perform both iSCSI and FCP rescan when creating or editing a storage domain, connecting to storage server, getting vg and storage domain list and more. The unneeded rescan is typically fast, but if a storage server or device is not accessible, a SCSI rescan may block for couple of minutes, leading to unwanted blocking of unrelated storage threads. This is particularly bad when you are interested only in one domain type, but the host get stuck scanning the other type. To improve storage domain isolation, we use the specified storage type to perform a rescan only of the relevant type. If storage type was not specified, we scan both ISCSI and FCP keeping the old behavior. Change-Id: Ic32cd683020e94df016dd77b19ae3eb7317c5554 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/hsm.py M vdsm/storage/multipath.py M vdsm/storage/sdc.py 3 files changed, 25 insertions(+), 17 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/45824/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 1b8c064..541b699 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -1984,15 +1984,16 @@ return dict(devList=devices) def _getDeviceList(self, storageType=None, guids=(), checkStatus=True): - sdCache.refreshStorage() - typeFilter = lambda dev: True - if storageType: - if sd.storageType(storageType) == sd.type2name(sd.ISCSI_DOMAIN): - typeFilter = \ - lambda dev: multipath.devIsiSCSI(dev.get("devtype")) - elif sd.storageType(storageType) == sd.type2name(sd.FCP_DOMAIN): - typeFilter = \ - lambda dev: multipath.devIsFCP(dev.get("devtype")) + domType = sd.storageType(storageType) if storageType else None + + sdCache.refreshStorage(domType) + + if domType == sd.ISCSI_DOMAIN: + typeFilter = lambda dev: multipath.devIsiSCSI(dev.get("devtype")) + elif domType == sd.FCP_DOMAIN: + typeFilter = lambda dev: multipath.devIsFCP(dev.get("devtype")) + else: + typeFilter = lambda dev: True devices = [] pvs = {} @@ -2470,7 +2471,7 @@ # while the VDSM was not connected, we need to # call refreshStorage. if domType in (sd.FCP_DOMAIN, sd.ISCSI_DOMAIN): - sdCache.refreshStorage() + sdCache.refreshStorage(domType) try: doms = self.__prefetchDomains(domType, conObj) except: @@ -2864,7 +2865,8 @@ """ vars.task.setDefaultException( se.StorageDomainActionError("spUUID: %s" % spUUID)) - sdCache.refreshStorage() + domType = sd.storageType(storageType) if storageType else None + sdCache.refreshStorage(domType) if spUUID and spUUID != volume.BLANK_UUID: domList = self.getPool(spUUID).getDomains() domains = domList.keys() @@ -2925,7 +2927,8 @@ :rtype: dict """ vars.task.setDefaultException(se.VolumeGroupActionError()) - sdCache.refreshStorage() + domType = sd.storageType(storageType) if storageType else None + sdCache.refreshStorage(domType) # getSharedLock(connectionsResource...) vglist = [] vgs = self.__getVGsInfo() diff --git a/vdsm/storage/multipath.py b/vdsm/storage/multipath.py index ad81d2d..32deb98 100644 --- a/vdsm/storage/multipath.py +++ b/vdsm/storage/multipath.py @@ -39,6 +39,7 @@ import misc import iscsi import devicemapper +import sd DEV_ISCSI = "iSCSI" DEV_FCP = "FCP" @@ -61,7 +62,7 @@ """ multipath operation failed """ -def rescan(): +def rescan(domType=None): """ Forces multipath daemon to rescan the list of available devices and refresh the mapping table. New devices can be found under /dev/mapper @@ -70,8 +71,12 @@ """ # First rescan iSCSI and FCP connections - iscsi.rescan() - hba.rescan() + + if domType in (None, sd.ISCSI_DOMAIN): + iscsi.rescan() + + if domType in (None, sd.FCP_DOMAIN): + hba.rescan() # Now let multipath daemon pick up new devices misc.execCmd([constants.EXT_MULTIPATH], sudo=True) diff --git a/vdsm/storage/sdc.py b/vdsm/storage/sdc.py index ecb9708..273c5c0 100644 --- a/vdsm/storage/sdc.py +++ b/vdsm/storage/sdc.py @@ -77,10 +77,10 @@ self.__staleStatus = self.STORAGE_STALE @misc.samplingmethod - def refreshStorage(self): + def refreshStorage(self, domType=None): self.__staleStatus = self.STORAGE_REFRESHING - multipath.rescan() + multipath.rescan(domType) multipath.resize_devices() lvm.invalidateCache() -- To view, visit https://gerrit.ovirt.org/45824 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic32cd683020e94df016dd77b19ae3eb7317c5554 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 7

Change in vdsm[master]: cache: Replace utils.memoized with cache.memoized()
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: cache: Replace utils.memoized with cache.memoized() ...................................................................... cache: Replace utils.memoized with cache.memoized() Repalce users of utils.memoized with cache.memoized() and remove utils.memoized. No behavior change is expected. Change-Id: I12e2f2919cf92ff7d0758d70e2ed40523d66174f Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/netinfo.py M lib/vdsm/utils.py M tests/vmApiTests.py M vdsm/caps.py M vdsm/dmidecodeUtil.py M vdsm/ppc64HardwareInfo.py M vdsm/supervdsmServer M vdsm/virt/sampling.py 8 files changed, 30 insertions(+), 51 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/34710/1 diff --git a/lib/vdsm/netinfo.py b/lib/vdsm/netinfo.py index d2ef90b..8c56863 100644 --- a/lib/vdsm/netinfo.py +++ b/lib/vdsm/netinfo.py @@ -43,7 +43,7 @@ from .ipwrapper import routeShowGateways from . import libvirtconnection from .netconfpersistence import RunningConfig -from .utils import memoized +from .cache import memoized from .netlink import link as nl_link from .netlink import addr as nl_addr from .netlink import route as nl_route @@ -395,7 +395,7 @@ return paddr -@memoized +@memoized() def _getAllDefaultBondingOptions(): """ Return default options per mode, in a dictionary of dictionaries. All keys @@ -405,7 +405,7 @@ return json.loads(defaults.read()) -@memoized +@memoized() def _getDefaultBondingOptions(mode=None): """ Return default options for the given mode. If it is None, return options diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py index feb4079..131145c 100644 --- a/lib/vdsm/utils.py +++ b/lib/vdsm/utils.py @@ -51,6 +51,7 @@ import vdsm.infra.zombiereaper as zombiereaper from cpopen import CPopen +from . import cache from . import constants # Buffsize is 1K because I tested it on some use cases and 1K was fastest. If @@ -828,32 +829,6 @@ fcntl.fcntl(fd, fcntl.F_SETFD, old | fcntl.FD_CLOEXEC) -class memoized(object): - """ - Decorator that caches a function's return value each time it is called. - If called later with the same arguments, the cached value is returned, and - not re-evaluated. There is no support for uncachable arguments. - - Adaptation from http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize - """ - def __init__(self, func): - self.func = func - self.cache = {} - functools.update_wrapper(self, func) - - def __call__(self, *args): - try: - return self.cache[args] - except KeyError: - value = self.func(*args) - self.cache[args] = value - return value - - def __get__(self, obj, objtype): - """Support instance methods.""" - return functools.partial(self.__call__, obj) - - def validateMinimalKeySet(dictionary, reqParams): if not all(key in dictionary for key in reqParams): raise ValueError @@ -993,7 +968,7 @@ sys.exit(-3) -@memoized +(a)cache.memoized() def isOvirtNode(): return (os.path.exists('/etc/rhev-hypervisor-release') or bool(glob.glob('/etc/ovirt-node-*-release'))) diff --git a/tests/vmApiTests.py b/tests/vmApiTests.py index eb89fb1..4f940cf 100644 --- a/tests/vmApiTests.py +++ b/tests/vmApiTests.py @@ -25,14 +25,14 @@ from virt import vmexitreason from vdsm import define from testlib import VdsmTestCase as TestCaseBase -from vdsm import utils +from vdsm import cache from rpc import vdsmapi import vmfakelib as fake class TestSchemaCompliancyBase(TestCaseBase): - @utils.memoized + @cache.memoized() def _getAPI(self): testPath = os.path.realpath(__file__) dirName = os.path.split(testPath)[0] diff --git a/vdsm/caps.py b/vdsm/caps.py index dba9b5b..6e261d8 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -40,6 +40,7 @@ import dsaversion from vdsm import netinfo import hooks +from vdsm import cache from vdsm import utils import storage.hba from network.configurators import qos @@ -218,12 +219,12 @@ ENABLED = 1 -(a)utils.memoized +(a)cache.memoized() def _getCapsXMLStr(): return libvirtconnection.get().getCapabilities() -(a)utils.memoized +(a)cache.memoized() def _getCpuTopology(capabilities): if capabilities is None: capabilities = _getCapsXMLStr() @@ -277,7 +278,7 @@ return None -(a)utils.memoized +(a)cache.memoized() def _getLiveSnapshotSupport(arch, capabilities=None): if capabilities is None: capabilities = _getCapsXMLStr() @@ -294,7 +295,7 @@ return None -(a)utils.memoized +(a)cache.memoized() def getLiveMergeSupport(): """ Determine if libvirt provides the necessary features to enable live merge. @@ -318,7 +319,7 @@ return True -(a)utils.memoized +(a)cache.memoized() def getNumaTopology(): capabilities = _getCapsXMLStr() caps = minidom.parseString(capabilities) @@ -369,7 +370,7 @@ return memDict -(a)utils.memoized +(a)cache.memoized() def getNumaNodeDistance(): nodeDistance = {} retcode, out, err = utils.execCmd(['numactl', '--hardware']) @@ -385,7 +386,7 @@ return nodeDistance -(a)utils.memoized +(a)cache.memoized() def getAutoNumaBalancingInfo(): retcode, out, err = utils.execCmd(['sysctl', '-n', '-e', 'kernel.numa_balancing']) @@ -399,7 +400,7 @@ return AutoNumaBalancingStatus.UNKNOWN -(a)utils.memoized +(a)cache.memoized() def _getEmulatedMachines(arch, capabilities=None): if capabilities is None: capabilities = _getCapsXMLStr() @@ -461,7 +462,7 @@ return allModels -(a)utils.memoized +(a)cache.memoized() def _getCompatibleCpuModels(): c = libvirtconnection.get() allModels = _getAllCpuModels() @@ -532,7 +533,7 @@ return kdumpStatus -(a)utils.memoized +(a)cache.memoized() def getos(): if os.path.exists('/etc/rhev-hypervisor-release'): return OSName.RHEVH @@ -550,7 +551,7 @@ return OSName.UNKNOWN -(a)utils.memoized +(a)cache.memoized() def osversion(): version = release = '' @@ -700,7 +701,7 @@ return info -(a)utils.memoized +(a)cache.memoized() def _getVersionInfo(): if not hasattr(libvirt, 'VIR_MIGRATE_ABORT_ON_ERROR'): return _dropVersion('3.4', diff --git a/vdsm/dmidecodeUtil.py b/vdsm/dmidecodeUtil.py index add2526..f0c72fa 100644 --- a/vdsm/dmidecodeUtil.py +++ b/vdsm/dmidecodeUtil.py @@ -18,7 +18,7 @@ # Refer to the README and COPYING files for full details of the license # -from vdsm import utils +from vdsm import cache # This function gets dict and returns new dict that includes only string @@ -35,7 +35,7 @@ return ret -(a)utils.memoized +(a)cache.memoized() def getAllDmidecodeInfo(): import dmidecode @@ -45,7 +45,7 @@ return myLeafDict -(a)utils.memoized +(a)cache.memoized() def getHardwareInfoStructure(): dmiInfo = getAllDmidecodeInfo() sysStruct = {} diff --git a/vdsm/ppc64HardwareInfo.py b/vdsm/ppc64HardwareInfo.py index 23f7388..1a89639 100644 --- a/vdsm/ppc64HardwareInfo.py +++ b/vdsm/ppc64HardwareInfo.py @@ -16,6 +16,7 @@ # Refer to the README and COPYING files for full details of the license # +from vdsm import cache from vdsm import utils import os @@ -31,7 +32,7 @@ return 'unavailable' -(a)utils.memoized +(a)cache.memoized() def getHardwareInfoStructure(): infoStructure = {'systemSerialNumber': 'unavailable', 'systemFamily': 'unavailable', @@ -59,7 +60,7 @@ return infoStructure -(a)utils.memoized +(a)cache.memoized() def getCpuTopology(capabilities): topology = {} diff --git a/vdsm/supervdsmServer b/vdsm/supervdsmServer index c6c139b..aba32fb 100755 --- a/vdsm/supervdsmServer +++ b/vdsm/supervdsmServer @@ -51,6 +51,7 @@ except ImportError: _glusterEnabled = False +from vdsm import cache from vdsm import utils from vdsm.tool import restore_nets from parted_utils import getDevicePartedInfo as _getDevicePartedInfo @@ -378,7 +379,7 @@ raise OSError(errno.EINVAL, "Could not reload-rules for device " "%s" % guid) - @utils.memoized + @cache.memoized() def __udevVersion(self): cmd = [EXT_UDEVADM, '--version'] rc, out, err = utils.execCmd(cmd) diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 6bb4211..b1c3db6 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -30,6 +30,7 @@ import errno import re +from vdsm import cache from vdsm import utils from vdsm import netinfo from vdsm.ipwrapper import getLinks @@ -524,7 +525,7 @@ if not self._stopEvent.isSet(): self._log.error("Error while sampling stats", exc_info=True) - @utils.memoized + @cache.memoized() def _boot_time(self): # Try to get boot time only once, if N/A just log the error and never # include it in the response. -- To view, visit http://gerrit.ovirt.org/34710 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I12e2f2919cf92ff7d0758d70e2ed40523d66174f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 18

Change in vdsm[master]: [WIP]fencenode: split fenceNode into its own module
by ykaplan＠redhat.com 18 Sep '16

18 Sep '16

Yeela Kaplan has uploaded a new change for review. Change subject: [WIP]fencenode: split fenceNode into its own module ...................................................................... [WIP]fencenode: split fenceNode into its own module Change-Id: I7bd5e7246cf6da21e355849014a8fc71d5edbde6 Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M debian/vdsm-python.install M lib/vdsm/Makefile.am A lib/vdsm/fencenode.py M vdsm.spec.in M vdsm/API.py 5 files changed, 120 insertions(+), 80 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/97/43597/1 diff --git a/debian/vdsm-python.install b/debian/vdsm-python.install index d9402c3..901edb6 100644 --- a/debian/vdsm-python.install +++ b/debian/vdsm-python.install @@ -10,6 +10,7 @@ ./usr/lib/python2.7/dist-packages/vdsm/exception.py ./usr/lib/python2.7/dist-packages/vdsm/executor.py ./usr/lib/python2.7/dist-packages/vdsm/ipwrapper.py +./usr/lib/python2.7/dist-packages/vdsm/fencenode.py ./usr/lib/python2.7/dist-packages/vdsm/jsonrpcvdscli.py ./usr/lib/python2.7/dist-packages/vdsm/libvirtconnection.py ./usr/lib/python2.7/dist-packages/vdsm/netconfpersistence.py diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am index 95e236f..875507c 100644 --- a/lib/vdsm/Makefile.am +++ b/lib/vdsm/Makefile.am @@ -30,6 +30,7 @@ exception.py \ executor.py \ ipwrapper.py \ + fencenode.py \ jsonrpcvdscli.py \ libvirtconnection.py \ netconfpersistence.py \ diff --git a/lib/vdsm/fencenode.py b/lib/vdsm/fencenode.py new file mode 100644 index 0000000..3456b2f --- /dev/null +++ b/lib/vdsm/fencenode.py @@ -0,0 +1,100 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import logging +import os +import signal + +import utils +import constants +from define import doneCode, errCode + + +(a)utils.traceback(on=logging.name) +def fence(script, data): + # non-status actions are sent asyncronously. deathSignal is set to + # make sure that no stray fencing scripts are left behind if Vdsm + # crashes. + rc, out, err = utils.execCmd( + [script], deathSignal=signal.SIGTERM, data=data) + logging.debug('rc %s data %s out %s err %s', rc, + hidePasswd(data), out, err) + return rc, out, err + + +def hidePasswd(text): + cleantext = '' + for line in text.splitlines(True): + if line.startswith('passwd='): + line = 'passwd=XXXX\n' + cleantext += line + return cleantext + + +def fenceNode(addr, port, agent, username, password, action, + secure=False, options='', policy=None): + + logging.debug('fenceNode(addr=%s,port=%s,agent=%s,user=%s,passwd=%s,' + 'action=%s,secure=%s,options=%s,policy=%s)', + addr, port, agent, username, password, action, secure, + options, policy) + + if action not in ('status', 'on', 'off', 'reboot'): + raise ValueError('illegal action ' + action) + + script = constants.EXT_FENCE_PREFIX + agent + + data = ('agent=fence_%s\nipaddr=%s\nlogin=%s\naction=%s\n' + 'passwd=%s\n') % (agent, addr, username, action, password.value) + if port != '': + data += 'port=%s\n' % (port,) + if utils.tobool(secure): + data += 'secure=yes\n' + data += options + + try: + rc, out, err = fence(script, data) + except OSError as e: + if e.errno == os.errno.ENOENT: + return errCode['fenceAgent'] + raise + + logging.debug('rc %s data %s out %s err %s', + rc, hidePasswd(data), out, err) + + if not 0 <= rc <= 2: + return {'status': {'code': 1, + 'message': out + err}} + + message = doneCode['message'] + if action == 'status': + if rc == 0: + power = 'on' + elif rc == 2: + power = 'off' + else: + power = 'unknown' + message = out + err + return {'status': {'code': 0, 'message': message}, + 'power': power} + if rc != 0: + message = out + err + return {'status': {'code': rc, 'message': message}, + 'power': 'unknown', 'operationStatus': 'initiated'} diff --git a/vdsm.spec.in b/vdsm.spec.in index 966a974..9e69162 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -1071,6 +1071,7 @@ %{python_sitelib}/%{vdsm_name}/exception.py* %{python_sitelib}/%{vdsm_name}/executor.py* %{python_sitelib}/%{vdsm_name}/ipwrapper.py* +%{python_sitelib}/%{vdsm_name}/fencenode.py* %{python_sitelib}/%{vdsm_name}/jsonrpcvdscli.py* %{python_sitelib}/%{vdsm_name}/libvirtconnection.py* %{python_sitelib}/%{vdsm_name}/netinfo.py* diff --git a/vdsm/API.py b/vdsm/API.py index 0a93d2a..0719926 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -21,8 +21,6 @@ # pylint: disable=R0904 from contextlib import contextmanager -import os -import signal import six import sys import time @@ -38,6 +36,7 @@ from clientIF import clientIF from vdsm import netinfo from vdsm import constants +from vdsm import fencenode import storage.misc import storage.clusterlock import storage.volume @@ -1187,6 +1186,11 @@ APIBase.__init__(self) # General Host functions + def ping(self): + "Ping the server. Useful for tests" + updateTimestamp() + return {'status': doneCode} + def fenceNode(self, addr, port, agent, username, password, action, secure=False, options='', policy=None): """Send a fencing command to a remote node. @@ -1194,44 +1198,24 @@ agent is one of (rsa, ilo, drac5, ipmilan, etc) action can be one of (status, on, off, reboot).""" - @utils.traceback(on=self.log.name) - def fence(script, inp): - # non-status actions are sent asyncronously. deathSignal is set to - # make sure that no stray fencing scripts are left behind if Vdsm - # crashes. - rc, out, err = utils.execCmd( - [script], deathSignal=signal.SIGTERM, - data=inp) - self.log.debug('rc %s inp %s out %s err %s', rc, - hidePasswd(inp), out, err) - return rc, out, err - - def hidePasswd(text): - cleantext = '' - for line in text.splitlines(True): - if line.startswith('passwd='): - line = 'passwd=XXXX\n' - cleantext += line - return cleantext - def should_fence(policy): # skip fence execution if map of storage domains with host id is # entered and at least one storage domain connection from host is # alive if policy is None: - self.log.debug('No policy specified') + logging.debug('No policy specified') return True hostIdMap = policy.get('storageDomainHostIdMap') if not hostIdMap: - self.log.warning('Invalid policy specified') + logging.warning('Invalid policy specified') return True result = self._irs.getHostLeaseStatus(hostIdMap) - if result['status']['code'] != 0: - self.log.error( - "Error getting host lease status, error code '%s'", - result['status']['code']) + rc = result['status']['code'] + if rc != 0: + logging.error( + "Error getting host lease status, error code '%s'", rc) return True # HOST_STATUS_LIVE means that host renewed its lease in last 80 @@ -1239,65 +1223,18 @@ # fencing, even when it's unreachable from engine for sd, status in result['domains'].iteritems(): if status == storage.clusterlock.HOST_STATUS_LIVE: - self.log.debug("Host has live lease on '%s'", sd) + logging.debug("Host has live lease on '%s'", sd) return False - self.log.debug("Host doesn't have any live lease") + logging.debug("Host doesn't have any live lease") return True - self.log.debug('fenceNode(addr=%s,port=%s,agent=%s,user=%s,passwd=%s,' - 'action=%s,secure=%s,options=%s,policy=%s)', - addr, port, agent, username, password, action, secure, - options, policy) - - if action not in ('status', 'on', 'off', 'reboot'): - raise ValueError('illegal action ' + action) - if action != 'status' and not should_fence(policy): - self.log.debug("Skipping execution of action '%s'", action) + logging.debug("Skipping execution of action '%s'", action) return {'status': doneCode, 'operationStatus': 'skipped'} - script = constants.EXT_FENCE_PREFIX + agent - - inp = ('agent=fence_%s\nipaddr=%s\nlogin=%s\naction=%s\n' - 'passwd=%s\n') % (agent, addr, username, action, password.value) - if port != '': - inp += 'port=%s\n' % (port,) - if utils.tobool(secure): - inp += 'secure=yes\n' - inp += options - - try: - rc, out, err = fence(script, inp) - except OSError as e: - if e.errno == os.errno.ENOENT: - return errCode['fenceAgent'] - raise - self.log.debug('rc %s in %s out %s err %s', rc, - hidePasswd(inp), out, err) - if not 0 <= rc <= 2: - return {'status': {'code': 1, - 'message': out + err}} - message = doneCode['message'] - if action == 'status': - if rc == 0: - power = 'on' - elif rc == 2: - power = 'off' - else: - power = 'unknown' - message = out + err - return {'status': {'code': 0, 'message': message}, - 'power': power} - if rc != 0: - message = out + err - return {'status': {'code': rc, 'message': message}, - 'power': 'unknown', 'operationStatus': 'initiated'} - - def ping(self): - "Ping the server. Useful for tests" - updateTimestamp() - return {'status': doneCode} + return fencenode.fenceNode(addr, port, agent, username, password, + action, secure, options, policy) def getCapabilities(self): """ -- To view, visit https://gerrit.ovirt.org/43597 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7bd5e7246cf6da21e355849014a8fc71d5edbde6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

4 5

Change in vdsm[master]: qemuimg: Memoize _supports_qcow2_compat
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: qemuimg: Memoize _supports_qcow2_compat ...................................................................... qemuimg: Memoize _supports_qcow2_compat We used to run qemu-img twice when creating or converting qcow2 images. The first run check if qemu-img supports the qcow2 "comapt" option, and the second run uses the result to format the qemu-img command. Now we run qemu-img once for "create" and "convert" to learn about its capabilities, and use the cached value on the next runs. If qemu-img executable is modified, we drop the cache, in case a new version was installed with different capabilities. Change-Id: Ic63f5e8c06993df8e4066bf7ac2dabfb4b4bdbfb Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/qemuimg.py M tests/qemuimgTests.py 2 files changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/34711/1 diff --git a/lib/vdsm/qemuimg.py b/lib/vdsm/qemuimg.py index cf428b2..f295acc 100644 --- a/lib/vdsm/qemuimg.py +++ b/lib/vdsm/qemuimg.py @@ -23,6 +23,7 @@ import signal from . import utils +from . import cache _qemuimg = utils.CommandPath("qemu-img", "/usr/bin/qemu-img",) # Fedora, EL6 @@ -221,6 +222,7 @@ raise QImgError(rc, out, err) +(a)cache.memoized(cache.file_validator(_qemuimg.cmd)) def _supports_qcow2_compat(command): """ qemu-img "create" and "convert" commands support a "compat" option in diff --git a/tests/qemuimgTests.py b/tests/qemuimgTests.py index 813a497..a642374 100644 --- a/tests/qemuimgTests.py +++ b/tests/qemuimgTests.py @@ -118,6 +118,9 @@ class CreateTests(TestCaseBase): + def setUp(self): + qemuimg._supports_qcow2_compat.invalidate() + def test_no_format(self): def create(cmd, **kw): expected = [QEMU_IMG, 'create', 'image'] @@ -161,6 +164,9 @@ class ConvertTests(TestCaseBase): + def setUp(self): + qemuimg._supports_qcow2_compat.invalidate() + def test_no_format(self): def convert(cmd, **kw): expected = [QEMU_IMG, 'convert', '-t', 'none', 'src', 'dst'] -- To view, visit http://gerrit.ovirt.org/34711 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic63f5e8c06993df8e4066bf7ac2dabfb4b4bdbfb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 14

Change in vdsm[master]: vdsm: add eventfd and EventFile synchronization
by Federico Simoncelli 13 Sep '16

13 Sep '16

Federico Simoncelli has uploaded a new change for review. Change subject: vdsm: add eventfd and EventFile synchronization ...................................................................... vdsm: add eventfd and EventFile synchronization Change-Id: I0d237f13c42b1f4505c90d30c6d3c3ecbd1e9fa7 Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> --- M lib/vdsm/Makefile.am A lib/vdsm/eventfd.py M tests/Makefile.am A tests/eventfdTests.py 4 files changed, 251 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/87/33687/1 diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am index 4bebf28..e712cad 100644 --- a/lib/vdsm/Makefile.am +++ b/lib/vdsm/Makefile.am @@ -25,6 +25,7 @@ __init__.py \ compat.py \ define.py \ + eventfd.py \ exception.py \ ipwrapper.py \ libvirtconnection.py \ diff --git a/lib/vdsm/eventfd.py b/lib/vdsm/eventfd.py new file mode 100644 index 0000000..b2a7084 --- /dev/null +++ b/lib/vdsm/eventfd.py @@ -0,0 +1,140 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +"""\ +This module provides the support for eventfd(2). + +More information about eventfd and usage examples can be found in the +eventfd(2) man page. + +The EventFile class provides a single synchronization object exposing +the python Event interface and associated eventfds. + +The eventfd() context manager returns a file descriptor that can be +used to provide the event notice to select, poll and epoll, e.g. + + import os + import sys + import select + import threading + import time + from vdsm.eventfd import EventFile, DATASIZE + + e = EventFile() + p = select.epoll() + + threading.Timer(5, e.set).start() + + with e.eventfd() as efd: + p.register(efd, select.EPOLLIN) + p.register(sys.stdin.fileno(), select.EPOLLIN) + + print "Echoing lines until event is received" + event_received = False + + while not event_received: + for fileno, event in p.poll(): + if not event & select.EPOLLIN: + continue + + if fileno == efd: + os.read(efd, DATASIZE) + event_received = True + elif fileno == sys.stdin.fileno(): + print os.read(sys.stdin.fileno(), 1024), + + print "Event received!" + + +The Event set() semantic is preserved in the eventfd context manager: +if the event is set then the eventfd already contains the notification. +This is both to maintain the semantic and to avoid possible races as: + + if not e.is_set(): + with e.eventfd() as efd: + ... +""" + +import os +import ctypes +import threading + +from contextlib import contextmanager + +_libc = ctypes.CDLL('libc.so.6', use_errno=True) + +EFD_NONBLOCK = os.O_NONBLOCK +EFD_CLOEXEC = 02000000 # os.O_CLOEXEC in python 3.3 +EFD_SEMAPHORE = 00000001 + +DATASIZE = ctypes.sizeof(ctypes.c_ulonglong) + + +def eventfd(initval, flags): + return _libc.eventfd(initval, flags) + + +class EventFile(object): + def __init__(self, event=None): + self.__lock = threading.Lock() + self.__fds = set() + self.__event = event or threading.Event() + + @staticmethod + def __fire_event(fd): + os.write(fd, ctypes.c_ulonglong(1)) + + def open_eventfd(self): + with self.__lock: + fd = eventfd(0, 0) + + self.__fds.add(fd) + + if self.__event.is_set(): + self.__fire_event(fd) + + return fd + + @contextmanager + def eventfd(self): + fd = self.open_eventfd() + + yield fd + + with self.__lock: + self.__fds.remove(fd) + os.close(fd) + + def isSet(self): + return self.__event.isSet() + + is_set = isSet + + def set(self): + with self.__lock: + self.__event.set() + for fd in self.__fds: + self.__fire_event(fd) + + def clear(self): + self.__event.clear() + + def wait(self, timeout=None, balancing=True): + self.__event.wait(timeout) diff --git a/tests/Makefile.am b/tests/Makefile.am index 449d7b1..120712e 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -31,6 +31,7 @@ clientifTests.py \ configNetworkTests.py \ domainMonitorTests.py \ + eventfdTests.py \ fileVolumeTests.py \ fileUtilTests.py \ fuserTests.py \ diff --git a/tests/eventfdTests.py b/tests/eventfdTests.py new file mode 100644 index 0000000..be15248 --- /dev/null +++ b/tests/eventfdTests.py @@ -0,0 +1,109 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import os +import select +from vdsm.eventfd import EventFile, DATASIZE +from nose.tools import timed, raises, TimeExpired + +TEST_TIMEOUT = 1 +WAIT_TIMEOUT = 2 + + +def test_set(): + e = EventFile() + e.set() + assert e.is_set() + assert e.isSet() + + +def text_clear(): + e = EventFile() + e.set() + assert e.is_set() + e.clear() + assert not e.is_set() + + +@timed(TEST_TIMEOUT) +def test_wait_set(): + e = EventFile() + e.set() + e.wait(WAIT_TIMEOUT) + + +@raises(TimeExpired) +@timed(TEST_TIMEOUT) +def test_wait_noset(): + e = EventFile() + e.wait(WAIT_TIMEOUT) + + +@timed(TEST_TIMEOUT) +def test_eventfd_earlyset(): + e = EventFile() + e.set() + with e.eventfd() as fd: + assert len(__select_and_read(fd)) == DATASIZE + + +@timed(TEST_TIMEOUT) +def test_eventfd_lateset(): + e = EventFile() + with e.eventfd() as fd: + e.set() + assert len(__select_and_read(fd)) == DATASIZE + + +@raises(TimeExpired) +@timed(TEST_TIMEOUT) +def test_eventfd_noset(): + e = EventFile() + with e.eventfd() as fd: + assert len(__select_and_read(fd)) != DATASIZE + + +@timed(TEST_TIMEOUT) +def test_eventfd_multiple(): + e = EventFile() + e.set() + with e.eventfd() as fd1: + assert len(__select_and_read(fd1)) == DATASIZE + with e.eventfd() as fd2: + assert len(__select_and_read(fd2)) == DATASIZE + with e.eventfd() as fd3: + assert len(__select_and_read(fd3)) == DATASIZE + + +@raises(TimeExpired) +@timed(TEST_TIMEOUT) +def test_eventfd_clear(): + e = EventFile() + e.set() + e.clear() + with e.eventfd() as fd: + assert len(__select_and_read(fd)) != DATASIZE + + +def __select_and_read(fd): + rd, wr, ex = select.select((fd,), (), (), WAIT_TIMEOUT) + if fd in rd: + return os.read(fd, DATASIZE) + return '' -- To view, visit http://gerrit.ovirt.org/33687 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0d237f13c42b1f4505c90d30c6d3c3ecbd1e9fa7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>

4 6

Change in vdsm[master]: debug: Add virt-alignment-scan.debug tool
by Nir Soffer 11 Sep '16

11 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: debug: Add virt-alignment-scan.debug tool ...................................................................... debug: Add virt-alignment-scan.debug tool Change-Id: I2ec4d0219ea039d57ee8809339a7e03d71980bf8 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A vdsm/virt-alignment-scan.debug 1 file changed, 24 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/96/40896/1 diff --git a/vdsm/virt-alignment-scan.debug b/vdsm/virt-alignment-scan.debug new file mode 100755 index 0000000..e904b45 --- /dev/null +++ b/vdsm/virt-alignment-scan.debug @@ -0,0 +1,24 @@ +#!/bin/sh +# +# Debug tool for virt-alignmnet-scan +# +# Installation: +# +# mv /usr/bin/virt-alignment-scan /usr/bin/virt-alignment-scan.real +# ln -s /usr/share/vdsm/virt-alignment-scan.debug /usr/bin/virt-alignment-scan +# +# Uinstall +# +# mv /usr/bin/virt-alignment-scan.real /usr/bin/virt-alignment-scan + +log="/var/log/vdsm/virt-alignmnet-scan.debug.log" + +/usr/bin/virt-alignment-scan "$@" +rc=$? +if [ $rc -ne 0 ]; then + # See https://bugzilla.redhat.com/show_bug.cgi?id=1151838#c28 + sleep 30 + /usr/bin/libguestfs-test-tool >> $log 2>&1 +fi + +exit $rc -- To view, visit https://gerrit.ovirt.org/40896 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2ec4d0219ea039d57ee8809339a7e03d71980bf8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 9

Change in vdsm[master]: profiling: memory: port from dowser to tracemalloc
by fromani＠redhat.com 31 Aug '16

31 Aug '16

Francesco Romani has uploaded a new change for review. Change subject: profiling: memory: port from dowser to tracemalloc ...................................................................... profiling: memory: port from dowser to tracemalloc ** WORK IN PROGRESS ** This patch implements integration with tracemalloc (http://pytracemalloc.readthedocs.org/), replacing dowser module. tracemalloc is - included in python 3.4, thus received good amount of peer review - more comprehensive, as it plugs inside cpython - augmented with nice UI to analyze memory profile snapshot It, however, requires a python VM compliant with PEP 445, and an additional extension module installed in the system Change-Id: If00a3af13ea48bb75d40ed11e56de04d90452ff7 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M lib/vdsm/profiling/memory.py 1 file changed, 32 insertions(+), 30 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/47624/1 diff --git a/lib/vdsm/profiling/memory.py b/lib/vdsm/profiling/memory.py index 31f833c..80a07ac 100755 --- a/lib/vdsm/profiling/memory.py +++ b/lib/vdsm/profiling/memory.py @@ -24,19 +24,20 @@ """ import logging +import os import threading +import time +from vdsm import constants from vdsm.config import config -from vdsm.utils import traceback from .errors import UsageError -# Import modules lazily when profile is started -dowser = None -cherrypy = None +# Import tracemalloc lazily when profile is started +tracemalloc = None + _lock = threading.Lock() -_thread = None def start(): @@ -48,7 +49,7 @@ def stop(): """ Stops application memory profiling """ if is_enabled(): - _stop_profiling() + _stop_profiling(_make_snapshot_name()) def is_enabled(): @@ -56,48 +57,49 @@ def is_running(): - return _thread is not None + return tracemalloc and tracemalloc.is_tracing() -@traceback() -def _memory_viewer(): - cherrypy.tree.mount(dowser.Root()) +def snapshot(filename=None): + with _lock: + if not is_running(): + raise UsageError('Memory profiler must be running ' + 'to take snapshots') + snapshot_name = filename or _make_snapshot_name() + with open(snapshot_name, 'wb') as snap: + snap.write(tracemalloc.take_snapshot()) + return snapshot_name - cherrypy.config.update({ - 'server.socket_host': '0.0.0.0', - 'server.socket_port': config.getint('devel', 'memory_profile_port')}) - cherrypy.engine.start() +def _make_snapshot_name(): + return os.path.join(constants.P_VDSM_RUN, + 'vdsm_memory_%s.pickle' % _make_timestamp()) + + +def _make_timestamp(): + return time.strftime('%Y%m%d_%H%M%S') def _start_profiling(): - global cherrypy - global dowser - global _thread + global tracemalloc logging.debug("Starting memory profiling") - import cherrypy - import dowser + import tracemalloc # this nonsense makes pyflakes happy - cherrypy - dowser + tracemalloc with _lock: if is_running(): raise UsageError('Memory profiler is already running') - _thread = threading.Thread(name='memprofile', - target=_memory_viewer) - _thread.daemon = True - _thread.start() + tracemalloc.start() -def _stop_profiling(): +def _stop_profiling(filename): global _thread logging.debug("Stopping memory profiling") with _lock: if is_running(): - cherrypy.engine.exit() - cherrypy.engine.block() - _thread.join() - _thread = None + snapshot(filename) + tracemalloc.clear_traces() + tracemalloc.stop() -- To view, visit https://gerrit.ovirt.org/47624 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If00a3af13ea48bb75d40ed11e56de04d90452ff7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 7

Change in vdsm[master]: profiling: memory: profile snapshots in vdsClient
by fromani＠redhat.com 31 Aug '16

31 Aug '16

Francesco Romani has uploaded a new change for review. Change subject: profiling: memory: profile snapshots in vdsClient ...................................................................... profiling: memory: profile snapshots in vdsClient ** WORK IN PROGRESS ** Add the ability to take memory snapshots in vdsClient. Change-Id: I8c7c5a074953bd54a8c2704d8aa2b074e277fdde Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/bindingxmlrpc.py 3 files changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/25/47625/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index fe263f2..739374b 100755 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1596,6 +1596,13 @@ return stats['status']['code'], stats['status']['message'] return 0, '' + def do_memoryProfileSnapshot(self): + stats = self.s.memoryProfileSnapshot() + if stats['status']['code']: + return stats['status']['code'], stats['status']['message'] + print 'snapshot saved on host on %s' % stats['name'] + return 0, '' + def do_setMOMPolicy(self, policyFile): stats = self.s.setMOMPolicy(policyFile) if stats['status']['code']: @@ -2635,6 +2642,11 @@ ('<level> [logName][,logName]...', 'set log verbosity' ' level (10=DEBUG, 50=CRITICAL' )), + 'memoryProfileSnapshot': (serv.do_memoryProfileSnapshot, + ('', + 'save memory profile snapshot for later ' + 'inspection')), + 'setMOMPolicy': (serv.do_setMOMPolicy, ('<policyfile>', 'set MOM policy')), 'setMOMPolicyParameters': (serv.do_setMOMPolicyParameters, diff --git a/vdsm/API.py b/vdsm/API.py index 0b5c8f7..1345dd8 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -51,6 +51,7 @@ from virt.vmdevices import hwclass from vdsm.compat import pickle from vdsm.define import doneCode, errCode, Kbytes, Mbytes +from vdsm.profiling import memory import caps from vdsm.config import config import hooks @@ -1409,6 +1410,11 @@ return dict(status=doneCode) + def memoryProfileSnapshot(self): + name = memory.snapshot() + logging.debug('saved memory profiling snapshot on %s', name) + return {'status': doneCode, 'snapshot': name} + # VM-related functions def getVMList(self, fullStatus=False, vmList=(), onlyUUID=False): """ return a list of known VMs with full (or partial) config each """ diff --git a/vdsm/rpc/bindingxmlrpc.py b/vdsm/rpc/bindingxmlrpc.py index c521cd8..c0212df 100644 --- a/vdsm/rpc/bindingxmlrpc.py +++ b/vdsm/rpc/bindingxmlrpc.py @@ -649,6 +649,10 @@ api = API.Global() return api.setLogLevel(level) + def memoryProfileSnapshot(self): + api = API.Global() + return api.memoryProfileSnapshot() + def setMOMPolicy(self, policy): api = API.Global() return api.setMOMPolicy(policy) @@ -1088,6 +1092,7 @@ (self.addNetwork, 'addNetwork'), (self.delNetwork, 'delNetwork'), (self.editNetwork, 'editNetwork'), + (self.memoryProfileSnapshot, 'memoryProfileSnapshot'), (self.setupNetworks, 'setupNetworks'), (self.ping, 'ping'), (self.setSafeNetworkConfig, 'setSafeNetworkConfig'), -- To view, visit https://gerrit.ovirt.org/47625 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8c7c5a074953bd54a8c2704d8aa2b074e277fdde Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 6

Change in vdsm[master]: WIP: add simple tracemalloc inspection tool
by fromani＠redhat.com 31 Aug '16

31 Aug '16

Francesco Romani has uploaded a new change for review. Change subject: WIP: add simple tracemalloc inspection tool ...................................................................... WIP: add simple tracemalloc inspection tool Change-Id: Ideb2652f345edb6f4a4d66c5299b601e53e85d33 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- A contrib/memory-stats A contrib/tracemalloc.py 2 files changed, 592 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/15/48115/1 diff --git a/contrib/memory-stats b/contrib/memory-stats new file mode 100755 index 0000000..5625ad1 --- /dev/null +++ b/contrib/memory-stats @@ -0,0 +1,102 @@ +#!/usr/bin/env python +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import argparse +import os +import sys +import time + +import tracemalloc + + +def eprint(s): + sys.stderr.write('%s\n' % s) + + +def _load(filename): + if filename: + eprint('Loading snapshot: "%s"' % filename) + before = time.time() + snap = tracemalloc.Snapshot.load(filename) + eprint('Loaded snapshot: "%s" in %.3f seconds' % ( + filename, time.time() - before)) + return snap + return None + + +def display_top(snapshot1, snapshot2=None, group_by='lineno', limit=10): + if snapshot2 is None: + top_stats = snapshot1.statistics(group_by) + else: + top_stats = snapshot2.compare_to(snapshot1, group_by) + print_stats(top_stats, limit) + + +def _print_sep(width=52): + print('-' * width) + + +def print_stats(top_stats, limit=10): + total = float(sum(stat.size for stat in top_stats)) + print("* Total allocated size: %.2f KiB" % (total / 1024.)) + print("* Top %s lines:" % limit) + _print_sep() + for index, stat in enumerate(top_stats[:limit], 1): + frame = stat.traceback[0] + # replace "/path/to/module/file.py" with "module/file.py" + filename = os.sep.join(frame.filename.split(os.sep)[-2:]) + where = "%s:%s" % (filename, frame.lineno) + print("#%3s: %-32s: %.2f KiB (%.2f%%)" + % (index, where, + stat.size / 1024., + (100. * stat.size) / total)) + + other = top_stats[limit:] + if other: + size = sum(stat.size for stat in other) + _print_sep() + where = "%i entries" % len(other) + print("rest: %-32s: %.2f KiB (%.2f%%)" + % (where, size / 1024., (100. * size) / total)) + + +ap = argparse.ArgumentParser(description='analyze tracemalloc snapshot') +ap.add_argument('snapshot', nargs='?', help='memory snapshot to analyze') +ap.add_argument('-c', '--compare-to', metavar='OS', dest='to_compare', + help='memory snapshot to compare') +ap.add_argument('-g', '--group-by', metavar='ATTR', dest='groupby', + default='lineno', + help='group samples by either: lineno, filename, traceback') +ap.add_argument('-l', '--limit', metavar='L', dest='limit', default=10, + type=int, + help='limit output to L top entries') + + +args = ap.parse_args() + +if not args.snapshot: + ap.print_help() + sys.exit(1) + +display_top(_load(args.snapshot), + _load(args.to_compare), + args.groupby, + args.limit) diff --git a/contrib/tracemalloc.py b/contrib/tracemalloc.py new file mode 100644 index 0000000..47b633e --- /dev/null +++ b/contrib/tracemalloc.py @@ -0,0 +1,490 @@ +# Copy of tracemalloc.py from pytracemalloc without _tracemalloc, +# to be able to read snapshot without having to install the _tracemaloc +# module. +from collections import Sequence, Iterable +import fnmatch +import linecache +import os.path +import pickle + + +try: + from functools import total_ordering +except ImportError: + # Python 2.6 + def total_ordering(cls): + # Function backported from Python 2.7 + convert = { + '__lt__': [('__gt__', lambda self, other: _not_op_and_not_eq(self.__lt__, self, other)), + ('__le__', lambda self, other: _op_or_eq(self.__lt__, self, other)), + ('__ge__', lambda self, other: _not_op(self.__lt__, other))], + '__le__': [('__ge__', lambda self, other: _not_op_or_eq(self.__le__, self, other)), + ('__lt__', lambda self, other: _op_and_not_eq(self.__le__, self, other)), + ('__gt__', lambda self, other: _not_op(self.__le__, other))], + '__gt__': [('__lt__', lambda self, other: _not_op_and_not_eq(self.__gt__, self, other)), + ('__ge__', lambda self, other: _op_or_eq(self.__gt__, self, other)), + ('__le__', lambda self, other: _not_op(self.__gt__, other))], + '__ge__': [('__le__', lambda self, other: _not_op_or_eq(self.__ge__, self, other)), + ('__gt__', lambda self, other: _op_and_not_eq(self.__ge__, self, other)), + ('__lt__', lambda self, other: _not_op(self.__ge__, other))] + } + roots = [op for op in convert if getattr(cls, op, None) is not getattr(object, op, None)] + if not roots: + raise ValueError('must define at least one ordering operation: < > <= >=') + root = max(roots) + for opname, opfunc in convert[root]: + if opname not in roots: + opfunc.__name__ = opname + opfunc.__doc__ = getattr(int, opname).__doc__ + setattr(cls, opname, opfunc) + return cls + + +def _format_size(size, sign): + for unit in ('B', 'KiB', 'MiB', 'GiB', 'TiB'): + if abs(size) < 100 and unit != 'B': + # 3 digits (xx.x UNIT) + if sign: + return "%+.1f %s" % (size, unit) + else: + return "%.1f %s" % (size, unit) + if abs(size) < 10 * 1024 or unit == 'TiB': + # 4 or 5 digits (xxxx UNIT) + if sign: + return "%+.0f %s" % (size, unit) + else: + return "%.0f %s" % (size, unit) + size /= 1024 + + +class Statistic(object): + """ + Statistic difference on memory allocations between two Snapshot instance. + """ + + __slots__ = ('traceback', 'size', 'count') + + def __init__(self, traceback, size, count): + self.traceback = traceback + self.size = size + self.count = count + + def __hash__(self): + return hash((self.traceback, self.size, self.count)) + + def __eq__(self, other): + return (self.traceback == other.traceback + and self.size == other.size + and self.count == other.count) + + def __str__(self): + text = ("%s: size=%s, count=%i" + % (self.traceback, + _format_size(self.size, False), + self.count)) + if self.count: + average = self.size / self.count + text += ", average=%s" % _format_size(average, False) + return text + + def __repr__(self): + return ('<Statistic traceback=%r size=%i count=%i>' + % (self.traceback, self.size, self.count)) + + def _sort_key(self): + return (self.size, self.count, self.traceback) + + +class StatisticDiff(object): + """ + Statistic difference on memory allocations between an old and a new + Snapshot instance. + """ + __slots__ = ('traceback', 'size', 'size_diff', 'count', 'count_diff') + + def __init__(self, traceback, size, size_diff, count, count_diff): + self.traceback = traceback + self.size = size + self.size_diff = size_diff + self.count = count + self.count_diff = count_diff + + def __hash__(self): + return hash((self.traceback, self.size, self.size_diff, + self.count, self.count_diff)) + + def __eq__(self, other): + return (self.traceback == other.traceback + and self.size == other.size + and self.size_diff == other.size_diff + and self.count == other.count + and self.count_diff == other.count_diff) + + def __str__(self): + text = ("%s: size=%s (%s), count=%i (%+i)" + % (self.traceback, + _format_size(self.size, False), + _format_size(self.size_diff, True), + self.count, + self.count_diff)) + if self.count: + average = self.size / self.count + text += ", average=%s" % _format_size(average, False) + return text + + def __repr__(self): + return ('<StatisticDiff traceback=%r size=%i (%+i) count=%i (%+i)>' + % (self.traceback, self.size, self.size_diff, + self.count, self.count_diff)) + + def _sort_key(self): + return (abs(self.size_diff), self.size, + abs(self.count_diff), self.count, + self.traceback) + + +def _compare_grouped_stats(old_group, new_group): + statistics = [] + for traceback, stat in new_group.items(): + previous = old_group.pop(traceback, None) + if previous is not None: + stat = StatisticDiff(traceback, + stat.size, stat.size - previous.size, + stat.count, stat.count - previous.count) + else: + stat = StatisticDiff(traceback, + stat.size, stat.size, + stat.count, stat.count) + statistics.append(stat) + + for traceback, stat in old_group.items(): + stat = StatisticDiff(traceback, 0, -stat.size, 0, -stat.count) + statistics.append(stat) + return statistics + + +@total_ordering +class Frame(object): + """ + Frame of a traceback. + """ + __slots__ = ("_frame",) + + def __init__(self, frame): + # frame is a tuple: (filename: str, lineno: int) + self._frame = frame + + @property + def filename(self): + return self._frame[0] + + @property + def lineno(self): + return self._frame[1] + + def __eq__(self, other): + return (self._frame == other._frame) + + def __lt__(self, other): + return (self._frame < other._frame) + + def __hash__(self): + return hash(self._frame) + + def __str__(self): + return "%s:%s" % (self.filename, self.lineno) + + def __repr__(self): + return "<Frame filename=%r lineno=%r>" % (self.filename, self.lineno) + + +@total_ordering +class Traceback(Sequence): + """ + Sequence of Frame instances sorted from the most recent frame + to the oldest frame. + """ + __slots__ = ("_frames",) + + def __init__(self, frames): + Sequence.__init__(self) + # frames is a tuple of frame tuples: see Frame constructor for the + # format of a frame tuple + self._frames = frames + + def __len__(self): + return len(self._frames) + + def __getitem__(self, index): + if isinstance(index, slice): + return tuple(Frame(trace) for trace in self._frames[index]) + else: + return Frame(self._frames[index]) + + def __contains__(self, frame): + return frame._frame in self._frames + + def __hash__(self): + return hash(self._frames) + + def __eq__(self, other): + return (self._frames == other._frames) + + def __lt__(self, other): + return (self._frames < other._frames) + + def __str__(self): + return str(self[0]) + + def __repr__(self): + return "<Traceback %r>" % (tuple(self),) + + def format(self, limit=None): + lines = [] + if limit is not None and limit < 0: + return lines + for frame in self[:limit]: + lines.append(' File "%s", line %s' + % (frame.filename, frame.lineno)) + line = linecache.getline(frame.filename, frame.lineno).strip() + if line: + lines.append(' %s' % line) + return lines + + +class Trace(object): + """ + Trace of a memory block. + """ + __slots__ = ("_trace",) + + def __init__(self, trace): + # trace is a tuple: (size, traceback), see Traceback constructor + # for the format of the traceback tuple + self._trace = trace + + @property + def size(self): + return self._trace[0] + + @property + def traceback(self): + return Traceback(self._trace[1]) + + def __eq__(self, other): + return (self._trace == other._trace) + + def __hash__(self): + return hash(self._trace) + + def __str__(self): + return "%s: %s" % (self.traceback, _format_size(self.size, False)) + + def __repr__(self): + return ("<Trace size=%s, traceback=%r>" + % (_format_size(self.size, False), self.traceback)) + + +class _Traces(Sequence): + def __init__(self, traces): + Sequence.__init__(self) + # traces is a tuple of trace tuples: see Trace constructor + self._traces = traces + + def __len__(self): + return len(self._traces) + + def __getitem__(self, index): + if isinstance(index, slice): + return tuple(Trace(trace) for trace in self._traces[index]) + else: + return Trace(self._traces[index]) + + def __contains__(self, trace): + return trace._trace in self._traces + + def __eq__(self, other): + return (self._traces == other._traces) + + def __repr__(self): + return "<Traces len=%s>" % len(self) + + +def _normalize_filename(filename): + filename = os.path.normcase(filename) + if filename.endswith(('.pyc', '.pyo')): + filename = filename[:-1] + return filename + + +class Filter(object): + def __init__(self, inclusive, filename_pattern, + lineno=None, all_frames=False): + self.inclusive = inclusive + self._filename_pattern = _normalize_filename(filename_pattern) + self.lineno = lineno + self.all_frames = all_frames + + @property + def filename_pattern(self): + return self._filename_pattern + + def __match_frame(self, filename, lineno): + filename = _normalize_filename(filename) + if not fnmatch.fnmatch(filename, self._filename_pattern): + return False + if self.lineno is None: + return True + else: + return (lineno == self.lineno) + + def _match_frame(self, filename, lineno): + return self.__match_frame(filename, lineno) ^ (not self.inclusive) + + def _match_traceback(self, traceback): + if self.all_frames: + if any(self.__match_frame(filename, lineno) + for filename, lineno in traceback): + return self.inclusive + else: + return (not self.inclusive) + else: + filename, lineno = traceback[0] + return self._match_frame(filename, lineno) + + +class Snapshot(object): + """ + Snapshot of traces of memory blocks allocated by Python. + """ + + def __init__(self, traces, traceback_limit): + # traces is a tuple of trace tuples: see _Traces constructor for + # the exact format + self.traces = _Traces(traces) + self.traceback_limit = traceback_limit + + def dump(self, filename): + """ + Write the snapshot into a file. + """ + with open(filename, "wb") as fp: + pickle.dump(self, fp, pickle.HIGHEST_PROTOCOL) + + @staticmethod + def load(filename): + """ + Load a snapshot from a file. + """ + with open(filename, "rb") as fp: + return pickle.load(fp) + + def _filter_trace(self, include_filters, exclude_filters, trace): + traceback = trace[1] + if include_filters: + if not any(trace_filter._match_traceback(traceback) + for trace_filter in include_filters): + return False + if exclude_filters: + if any(not trace_filter._match_traceback(traceback) + for trace_filter in exclude_filters): + return False + return True + + def filter_traces(self, filters): + """ + Create a new Snapshot instance with a filtered traces sequence, filters + is a list of Filter instances. If filters is an empty list, return a + new Snapshot instance with a copy of the traces. + """ + if not isinstance(filters, Iterable): + raise TypeError("filters must be a list of filters, not %s" + % type(filters).__name__) + if filters: + include_filters = [] + exclude_filters = [] + for trace_filter in filters: + if trace_filter.inclusive: + include_filters.append(trace_filter) + else: + exclude_filters.append(trace_filter) + new_traces = [trace for trace in self.traces._traces + if self._filter_trace(include_filters, + exclude_filters, + trace)] + else: + new_traces = self.traces._traces[:] + return Snapshot(new_traces, self.traceback_limit) + + def _group_by(self, key_type, cumulative): + if key_type not in ('traceback', 'filename', 'lineno'): + raise ValueError("unknown key_type: %r" % (key_type,)) + if cumulative and key_type not in ('lineno', 'filename'): + raise ValueError("cumulative mode cannot by used " + "with key type %r" % key_type) + + stats = {} + tracebacks = {} + if not cumulative: + for trace in self.traces._traces: + size, trace_traceback = trace + try: + traceback = tracebacks[trace_traceback] + except KeyError: + if key_type == 'traceback': + frames = trace_traceback + elif key_type == 'lineno': + frames = trace_traceback[:1] + else: # key_type == 'filename': + frames = ((trace_traceback[0][0], 0),) + traceback = Traceback(frames) + tracebacks[trace_traceback] = traceback + try: + stat = stats[traceback] + stat.size += size + stat.count += 1 + except KeyError: + stats[traceback] = Statistic(traceback, size, 1) + else: + # cumulative statistics + for trace in self.traces._traces: + size, trace_traceback = trace + for frame in trace_traceback: + try: + traceback = tracebacks[frame] + except KeyError: + if key_type == 'lineno': + frames = (frame,) + else: # key_type == 'filename': + frames = ((frame[0], 0),) + traceback = Traceback(frames) + tracebacks[frame] = traceback + try: + stat = stats[traceback] + stat.size += size + stat.count += 1 + except KeyError: + stats[traceback] = Statistic(traceback, size, 1) + return stats + + def statistics(self, key_type, cumulative=False): + """ + Group statistics by key_type. Return a sorted list of Statistic + instances. + """ + grouped = self._group_by(key_type, cumulative) + statistics = list(grouped.values()) + statistics.sort(reverse=True, key=Statistic._sort_key) + return statistics + + def compare_to(self, old_snapshot, key_type, cumulative=False): + """ + Compute the differences with an old snapshot old_snapshot. Get + statistics as a sorted list of StatisticDiff instances, grouped by + group_by. + """ + new_group = self._group_by(key_type, cumulative) + old_group = old_snapshot._group_by(key_type, cumulative) + statistics = _compare_grouped_stats(old_group, new_group) + statistics.sort(reverse=True, key=StatisticDiff._sort_key) + return statistics -- To view, visit https://gerrit.ovirt.org/48115 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ideb2652f345edb6f4a4d66c5299b601e53e85d33 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 4

Change in vdsm[master]: utils: Add changehash function for change detection
by Nir Soffer 28 Aug '16

28 Aug '16

Nir Soffer has uploaded a new change for review. Change subject: utils: Add changehash function for change detection ...................................................................... utils: Add changehash function for change detection We use Python built-in hash to detect changes in vm state without sending the state in each response. This function is not suitable for this usage. Now we use generic utils.changehash(), implemented using md5 hexdigest. Change-Id: I2242a594383e2d2fe64e3a581f18b8ac662648b0 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/utils.py M vdsm/virt/vm.py 2 files changed, 13 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/45/33045/1 diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py index 23c63e8..1b4a9d5 100644 --- a/lib/vdsm/utils.py +++ b/lib/vdsm/utils.py @@ -37,6 +37,7 @@ import glob import io import itertools +import hashlib import logging import re import sys @@ -1133,3 +1134,13 @@ flags = fcntl.fcntl(fd, fcntl.F_GETFL) flags |= os.O_NONBLOCK fcntl.fcntl(fd, fcntl.F_SETFL, flags) + + +def changehash(s): + """ + Returns a hash of string s, suitable for change detection. + + Tipically changehash(s) is sent to client frequently. When a client detect + that changehash(s) changed, it ask for s itself, which may be much bigger. + """ + return hashlib.md5(s).hexdigest() diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 941f283..b1567f9 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1500,7 +1500,7 @@ self.guestAgent = guestagent.GuestAgent( self._guestSocketFile, self.cif.channelListener, self.log) self._lastXMLDesc = '<domain><uuid>%s</uuid></domain>' % self.id - self._devXmlHash = '0' + self._devXmlHash = utils.changehash('') self._released = False self._releaseLock = threading.Lock() self.saveState() @@ -4495,7 +4495,7 @@ self._lastXMLDesc = self._dom.XMLDesc(0) devxml = _domParseStr(self._lastXMLDesc).childNodes[0]. \ getElementsByTagName('devices')[0] - self._devXmlHash = str(hash(devxml.toxml())) + self._devXmlHash = utils.changehash(devxml.toxml()) return self._lastXMLDesc -- To view, visit http://gerrit.ovirt.org/33045 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2242a594383e2d2fe64e3a581f18b8ac662648b0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 26

Change in vdsm[master]: Adding toolTests.py to python3 check modules
by ybronhei＠redhat.com 23 Aug '16

23 Aug '16

Yaniv Bronhaim has uploaded a new change for review. Change subject: Adding toolTests.py to python3 check modules ...................................................................... Adding toolTests.py to python3 check modules Change-Id: I52f11a1a10cae46773d05d0c09da80ad8eb3b772 Signed-off-by: Yaniv Bronhaim <ybronhei(a)redhat.com> --- M lib/vdsm/tool/configfile.py M tests/Makefile.am 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/52/48052/1 diff --git a/lib/vdsm/tool/configfile.py b/lib/vdsm/tool/configfile.py index 26ec114..0c58205 100644 --- a/lib/vdsm/tool/configfile.py +++ b/lib/vdsm/tool/configfile.py @@ -18,12 +18,12 @@ # from __future__ import absolute_import -import ConfigParser import functools import os import tempfile import re import selinux +from six.moves import configparser as ConfigParser import io from .. import utils @@ -263,5 +263,5 @@ def read(self, path): with open(path, 'r') as f: return self.wrapped.readfp( - io.StringIO(u'[root]\n' + f.read().decode()) + io.StringIO(u'[root]\n' + f.read()) ) diff --git a/tests/Makefile.am b/tests/Makefile.am index 205289e..3b1f8bf 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -30,6 +30,7 @@ apiData.py \ cmdutilsTests.py \ concurrentTests.py \ + toolTests.py \ $(NULL) device_modules = \ -- To view, visit https://gerrit.ovirt.org/48052 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I52f11a1a10cae46773d05d0c09da80ad8eb3b772 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yaniv Bronhaim <ybronhei(a)redhat.com>

5 15

Change in vdsm[master]: clusterLock: Acquire, release, and inquire volume leases
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: clusterLock: Acquire, release, and inquire volume leases ...................................................................... clusterLock: Acquire, release, and inquire volume leases Although we have been creating volume leases on storage for awhile now, there have been no APIs to manage them. Add this support to the clusterLock implementations. Change-Id: Icca901ccd27358767c023cd55b7a3823531d2a5a Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/clusterlock.py M vdsm/storage/sd.py 2 files changed, 80 insertions(+), 23 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/22/38622/1 diff --git a/vdsm/storage/clusterlock.py b/vdsm/storage/clusterlock.py index 51d2906..4077068 100644 --- a/vdsm/storage/clusterlock.py +++ b/vdsm/storage/clusterlock.py @@ -73,6 +73,10 @@ """Raised when the clusterlock class is not supporting inquire""" +class ResourceLeasesNotSupportedError(Exception): + """Raised when the clusterlock class does not support resource leases""" + + class SafeLease(object): log = logging.getLogger("Storage.SafeLease") @@ -145,7 +149,13 @@ raise se.AcquireLockFailure(self._sdUUID, rc, out, err) self.log.debug("Clustered lock acquired successfully") + def acquireResource(self, resource, lockDisk, shared=False): + raise ResourceLeasesNotSupportedError() + def inquire(self): + raise InquireNotSupportedError() + + def inquireResource(self, resource, lockDisk): raise InquireNotSupportedError() def getLockUtilFullPath(self): @@ -165,6 +175,8 @@ self.log.debug("Cluster lock released successfully") + def releaseResource(self, resource, lockDisk): + raise ResourceLeasesNotSupportedError() initSANLockLog = logging.getLogger("Storage.initSANLock") @@ -286,13 +298,9 @@ # HOST_FAIL. return HOST_STATUS_FREE - # The hostId parameter is maintained here only for compatibility with - # ClusterLock. We could consider to remove it in the future but keeping it - # for logging purpose is desirable. - def acquire(self, hostId): + def _acquire(self, resource, lockDisk, shared=False): with nested(self._lock, SANLock._sanlock_lock): - self.log.info("Acquiring cluster lock for domain %s (id: %s)", - self._sdUUID, hostId) + self.log.info("Acquiring resource %s, shared=%s", resource, shared) while True: if SANLock._sanlock_fd is None: @@ -304,27 +312,42 @@ "Cannot register to sanlock", str(e)) try: - sanlock.acquire(self._sdUUID, SDM_LEASE_NAME, - self.getLockDisk(), - slkfd=SANLock._sanlock_fd) + sanlock.acquire(self._sdUUID, resource, lockDisk, + slkfd=SANLock._sanlock_fd, shared=shared) except sanlock.SanlockException as e: if e.errno != os.errno.EPIPE: raise se.AcquireLockFailure( self._sdUUID, e.errno, - "Cannot acquire cluster lock", str(e)) + "Cannot acquire sanlock resource", str(e)) SANLock._sanlock_fd = None continue break - self.log.debug("Cluster lock for domain %s successfully acquired " - "(id: %s)", self._sdUUID, hostId) + self.log.debug("Resource %s successfully acquired", resource) + + # The hostId parameter is maintained here only for compatibility with + # ClusterLock. We could consider to remove it in the future but keeping it + # for logging purpose is desirable. + def acquire(self, hostId): + self.log.info("Acquiring cluster lock for domain %s (id: %s)", + self._sdUUID, hostId) + self._acquire(SDM_LEASE_NAME, self.getLockDisk()) + self.log.debug("Cluster lock for domain %s successfully acquired " + "(id: %s)", self._sdUUID, hostId) + + def acquireResource(self, resource, lockDisk, shared=False): + self._acquire(resource, lockDisk, shared) + res, owners = self._inquire(resource, lockDisk) + self.log.debug("ALITKE: acquire: res:%s owners:%s", res, owners) + + def _inquire(self, resource, lockDisk): + res = sanlock.read_resource(*lockDisk[0]) + owners = sanlock.read_resource_owners(self._sdUUID, resource, lockDisk) + return res, owners def inquire(self): - resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) - owners = sanlock.read_resource_owners(self._sdUUID, SDM_LEASE_NAME, - self.getLockDisk()) - + resource, owners = self._inquire(SDM_LEASE_NAME, self.getLockDisk()) if len(owners) == 1: return resource.get("version"), owners[0].get("host_id") elif len(owners) > 1: @@ -334,19 +357,32 @@ return None, None - def release(self): + def inquireResource(self, resource, lockDisk): + resource, owners = self._inquire(SDM_LEASE_NAME, self.getLockDisk()) + return (resource.get("version"), + [owner.get("host_id") for owner in owners]) + + def _release(self, resource, lockDisk): with self._lock: - self.log.info("Releasing cluster lock for domain %s", self._sdUUID) + self.log.info("Releasing resource %s", resource) try: - sanlock.release(self._sdUUID, SDM_LEASE_NAME, - self.getLockDisk(), slkfd=SANLock._sanlock_fd) + sanlock.release(self._sdUUID, resource, lockDisk, + slkfd=SANLock._sanlock_fd) except sanlock.SanlockException as e: - raise se.ReleaseLockFailure(self._sdUUID, e) + raise se.ReleaseLockFailure(resource, e) self._sanlockfd = None - self.log.debug("Cluster lock for domain %s successfully released", - self._sdUUID) + self.log.debug("Resource %s successfully released", resource) + + def release(self): + self.log.info("Releasing cluster lock for domain %s", self._sdUUID) + self._release(SDM_LEASE_NAME, self.getLockDisk()) + self.log.debug("Cluster lock for domain %s successfully released", + self._sdUUID) + + def releaseResource(self, resource, lockDisk): + self._release(resource, lockDisk) class LocalLock(object): @@ -462,10 +498,16 @@ self.log.debug("Local lock for domain %s successfully acquired " "(id: %s)", self._sdUUID, hostId) + def acquireResource(self, resource, lockDisk, shared=False): + raise ResourceLeasesNotSupportedError() + def inquire(self): with self._globalLockMapSync: hostId, lockFile = self._getLease() return self.LVER, hostId if lockFile else None, None + + def inquireResource(self, resource, lockDisk): + raise InquireNotSupportedError() def release(self): with self._globalLockMapSync: @@ -483,3 +525,6 @@ self.log.debug("Local lock for domain %s successfully released", self._sdUUID) + + def releaseResource(self, resource, lockDisk): + raise ResourceLeasesNotSupportedError() \ No newline at end of file diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index a771abf..9e6f281 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -514,6 +514,18 @@ def inquireClusterLock(self): return self._clusterLock.inquire() + def acquireVolumeLease(self, imgUUID, volUUID, shared=False): + lockDisk = self.getVolumeLease(imgUUID, volUUID) + self._clusterLock.acquireResource(volUUID, [lockDisk], shared) + + def releaseVolumeLease(self, imgUUID, volUUID): + lockDisk = self.getVolumeLease(imgUUID, volUUID) + self._clusterLock.releaseResource(volUUID, [lockDisk]) + + def inquireVolumeLease(self, imgUUID, volUUID): + lockDisk = self.getVolumeLease(imgUUID, volUUID) + return self._clusterLock.inquireResource(volUUID, [lockDisk]) + def attach(self, spUUID): self.invalidateMetadata() pools = self.getPools() -- To view, visit https://gerrit.ovirt.org/38622 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icca901ccd27358767c023cd55b7a3823531d2a5a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 47

Change in vdsm[master]: storage: Factor out getAllVolumes
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Factor out getAllVolumes ...................................................................... storage: Factor out getAllVolumes The SDM garbage collector needs to a mapping of volumes by imageID. Factor out this logic from getAllVolumes so it can be reused later. Change-Id: If8c55edf6416dddc4a4f30b91422b3e968df2b99 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/fileSD.py 1 file changed, 17 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/77/40377/1 diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index 1220939..042f836 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -18,7 +18,6 @@ # Refer to the README and COPYING files for full details of the license # -import collections import os import errno import logging @@ -405,6 +404,21 @@ """ pass + def getImageToVolumesMap(self): + volMetaPattern = os.path.join(self.mountpoint, self.sdUUID, + sd.DOMAIN_IMAGES, "*", "*.meta") + volMetaPaths = self.oop.glob.glob(volMetaPattern) + images = {} + for metaPath in volMetaPaths: + head, tail = os.path.split(metaPath) + volUUID, volExt = os.path.splitext(tail) + imgUUID = os.path.basename(head) + try: + images[imgUUID].append(volUUID) + except KeyError: + images[imgUUID] = [volUUID] + return images + def getAllVolumes(self): """ Return dict {volUUID: ((imgUUIDs,), parentUUID)} of the domain. @@ -422,17 +436,8 @@ Template volumes have no parent, and thus we report BLANK_UUID as their parentUUID. """ - volMetaPattern = os.path.join(self.mountpoint, self.sdUUID, - sd.DOMAIN_IMAGES, "*", "*.meta") - volMetaPaths = self.oop.glob.glob(volMetaPattern) - - # First create mapping from images to volumes - images = collections.defaultdict(list) - for metaPath in volMetaPaths: - head, tail = os.path.split(metaPath) - volUUID, volExt = os.path.splitext(tail) - imgUUID = os.path.basename(head) - images[imgUUID].append(volUUID) + # First get the mapping from images to volumes + images = self.getImageToVolumesMap() # Using images to volumes mapping, we can create volumes to images # mapping, detecting template volumes and template images, based on -- To view, visit https://gerrit.ovirt.org/40377 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If8c55edf6416dddc4a4f30b91422b3e968df2b99 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 18

Change in vdsm[master]: clusterlock: Add reference counting
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: clusterlock: Add reference counting ...................................................................... clusterlock: Add reference counting The clusterLock can now be acquired by multiple threads of execution since it is used by SDM verbs now. We need reference counting to ensure that one thread does not release the clusterLock while another thread still needs it. Change-Id: I846116ae16e88a51bdce20f97ddf22859dea3086 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/clusterlock.py 1 file changed, 55 insertions(+), 45 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/78/40378/1 diff --git a/vdsm/storage/clusterlock.py b/vdsm/storage/clusterlock.py index 86d47ad..96bba2b 100644 --- a/vdsm/storage/clusterlock.py +++ b/vdsm/storage/clusterlock.py @@ -211,6 +211,7 @@ def __init__(self, sdUUID, idsPath, leasesPath, *args): self._lock = threading.Lock() + self._clusterLockUsers = 0 self._sdUUID = sdUUID self._idsPath = idsPath self._leasesPath = leasesPath @@ -302,16 +303,22 @@ # ClusterLock. We could consider to remove it in the future but keeping it # for logging purpose is desirable. def acquireClusterLock(self, hostId): - self.log.info("Acquiring cluster lock for domain %s (id: %s)", - self._sdUUID, hostId) - self._acquire(SDM_LEASE_NAME, self.getLockDisk()) - self.log.debug("Cluster lock for domain %s successfully acquired " - "(id: %s)", self._sdUUID, hostId) + with nested(self._lock, SANLock._sanlock_lock): + self.log.info("Acquiring cluster lock for domain %s (id: %s)", + self._sdUUID, hostId) + if self._clusterLockUsers == 0: + self._acquire(SDM_LEASE_NAME, self.getLockDisk()) + self._clusterLockUsers = self._clusterLockUsers + 1 + self.log.debug("Cluster lock for domain %s successfully acquired " + "(id: %s, users: %i)", self._sdUUID, hostId, + self._clusterLockUsers) def acquireResource(self, resource, lockDisk, shared=False): - self.log.info("Acquiring resource lock for %s", resource) - self._acquire(resource, lockDisk, shared) - self.log.debug("Resource lock for %s successfully acquired", resource) + with nested(self._lock, SANLock._sanlock_lock): + self.log.info("Acquiring resource lock for %s", resource) + self._acquire(resource, lockDisk, shared) + self.log.debug("Resource lock for %s successfully acquired", + resource) def inquireClusterLock(self): resource, owners = self._inquire(SDM_LEASE_NAME, self.getLockDisk()) @@ -330,43 +337,47 @@ [owner.get("host_id") for owner in owners]) def releaseClusterLock(self): - self.log.info("Releasing cluster lock for domain %s", self._sdUUID) - self._release(SDM_LEASE_NAME, self.getLockDisk()) - self.log.debug("Cluster lock for domain %s successfully released", - self._sdUUID) + with self._lock: + self.log.info("Releasing cluster lock for domain %s", self._sdUUID) + if self._clusterLockUsers == 1: + self._release(SDM_LEASE_NAME, self.getLockDisk()) + self._clusterLockUsers = self._clusterLockUsers - 1 + self.log.debug("Cluster lock for domain %s successfully released " + "(users: %i)", self._sdUUID, self._clusterLockUsers) def releaseResource(self, resource, lockDisk): - self.log.info("Releasing resource lock for %s", resource) - self._release(resource, lockDisk) - self.log.debug("Resource lock for %s successfully released", resource) + with self._lock: + self.log.info("Releasing resource lock for %s", resource) + self._release(resource, lockDisk) + self.log.debug("Resource lock for %s successfully released", + resource) def _acquire(self, resource, lockDisk, shared=False): - with nested(self._lock, SANLock._sanlock_lock): - self.log.info("Acquiring resource %s, shared=%s", resource, shared) + self.log.info("Acquiring resource %s, shared=%s", resource, shared) - while True: - if SANLock._sanlock_fd is None: - try: - SANLock._sanlock_fd = sanlock.register() - except sanlock.SanlockException as e: - raise se.AcquireLockFailure( - self._sdUUID, e.errno, - "Cannot register to sanlock", str(e)) - + while True: + if SANLock._sanlock_fd is None: try: - sanlock.acquire(self._sdUUID, resource, lockDisk, - slkfd=SANLock._sanlock_fd, shared=shared) + SANLock._sanlock_fd = sanlock.register() except sanlock.SanlockException as e: - if e.errno != os.errno.EPIPE: - raise se.AcquireLockFailure( - self._sdUUID, e.errno, - "Cannot acquire sanlock resource", str(e)) - SANLock._sanlock_fd = None - continue + raise se.AcquireLockFailure( + self._sdUUID, e.errno, "Cannot register to sanlock", + str(e)) - break + try: + sanlock.acquire(self._sdUUID, resource, lockDisk, + slkfd=SANLock._sanlock_fd, shared=shared) + except sanlock.SanlockException as e: + if e.errno != os.errno.EPIPE: + raise se.AcquireLockFailure( + self._sdUUID, e.errno, + "Cannot acquire sanlock resource", str(e)) + SANLock._sanlock_fd = None + continue - self.log.debug("Resource %s successfully acquired", resource) + break + + self.log.debug("Resource %s successfully acquired", resource) def _inquire(self, resource, lockDisk): res = sanlock.read_resource(*lockDisk[0]) @@ -374,17 +385,16 @@ return res, owners def _release(self, resource, lockDisk): - with self._lock: - self.log.info("Releasing resource %s", resource) + self.log.info("Releasing resource %s", resource) - try: - sanlock.release(self._sdUUID, resource, lockDisk, - slkfd=SANLock._sanlock_fd) - except sanlock.SanlockException as e: - raise se.ReleaseLockFailure(resource, e) + try: + sanlock.release(self._sdUUID, resource, lockDisk, + slkfd=SANLock._sanlock_fd) + except sanlock.SanlockException as e: + raise se.ReleaseLockFailure(resource, e) - self._sanlockfd = None - self.log.debug("Resource %s successfully released", resource) + self._sanlockfd = None + self.log.debug("Resource %s successfully released", resource) class LocalLock(object): -- To view, visit https://gerrit.ovirt.org/40378 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I846116ae16e88a51bdce20f97ddf22859dea3086 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 16

Change in vdsm[master]: XXX: Insert sleep when creating block SD
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: XXX: Insert sleep when creating block SD ...................................................................... XXX: Insert sleep when creating block SD Change-Id: I10ed63d747f1353da824af8ec56ac8a6f66d666b Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/43553/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 0b8bdc7..0d5e403 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -896,6 +896,10 @@ # Create VMS file system _createVMSfs(os.path.join("/dev", vgName, MASTERLV)) + # XXX: When running an iSCSI server and initiator on the same host we + # need to wait a bit for IO to settle before deactivating LVs + import time + time.sleep(10) lvm.deactivateLVs(vgName, MASTERLV) path = lvm.lvPath(vgName, sd.METADATA) -- To view, visit https://gerrit.ovirt.org/43553 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I10ed63d747f1353da824af8ec56ac8a6f66d666b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 12

Change in vdsm[master]: sdm: Add find_domain_manifest helpers
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: sdm: Add find_domain_manifest helpers ...................................................................... sdm: Add find_domain_manifest helpers Change-Id: Ia5d139bde2349c8231141fe842d9ada96fa279fe Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/Makefile.am A tests/domainmanifestfactory_tests.py M tests/manifest_tests.py M tests/storagefakelib.py M tests/storagetestutils.py M vdsm.spec.in M vdsm/storage/Makefile.am M vdsm/storage/blockSD.py A vdsm/storage/domainManifestFactory.py M vdsm/storage/glusterSD.py M vdsm/storage/localFsSD.py M vdsm/storage/nfsSD.py 12 files changed, 214 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/55/43555/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 9cc3692..f116fc4 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -43,6 +43,7 @@ cpuProfileTests.py \ deviceTests.py \ domainDescriptorTests.py \ + domainmanifestfactory_tests.py \ encodingTests.py \ executorTests.py \ fileSDTests.py \ diff --git a/tests/domainmanifestfactory_tests.py b/tests/domainmanifestfactory_tests.py new file mode 100644 index 0000000..4c3f63e --- /dev/null +++ b/tests/domainmanifestfactory_tests.py @@ -0,0 +1,124 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import os +import uuid + +from testlib import VdsmTestCase, namedTemporaryDir +from monkeypatch import MonkeyPatchScope +from storagefakelib import FakeLVM +from storagetestutils import make_blocksd_manifest, make_vg, \ + make_filesd_manifest + +from storage import sd, blockSD, fileSD, localFsSD, mount +from storage import storage_exception as se +from storage import domainManifestFactory + + +class DomainManifestFactoryTests(VdsmTestCase): + def get_patches(self, tmpdir, lvm): + return [(blockSD, 'lvm', lvm), + (blockSD, 'selectMetadata', lambda x: {}), + (sd.StorageDomain, 'storage_repository', tmpdir), + (mount, 'isMounted', lambda x: True)] + + def make_mnt(self, tmpdir, link_name): + mntdir = os.path.join(tmpdir, sd.DOMAIN_MNT_POINT) + os.mkdir(mntdir) + os.symlink(tmpdir, os.path.join(mntdir, link_name)) + + def test_localfs_layout(self): + with namedTemporaryDir() as tmpdir: + self.make_mnt(tmpdir, '_localfssd') + lvm = FakeLVM(tmpdir) + with MonkeyPatchScope(self.get_patches(tmpdir, lvm)): + manifest = make_filesd_manifest(tmpdir) + found_manifest = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + self.assertEquals(manifest.sdUUID, found_manifest.sdUUID) + self.assertIsInstance(found_manifest, + fileSD.FileStorageDomainManifest) + + def test_nfs_layout(self): + with namedTemporaryDir() as tmpdir: + self.make_mnt(tmpdir, '192.168.1.1:_localfssd') + lvm = FakeLVM(tmpdir) + with MonkeyPatchScope(self.get_patches(tmpdir, lvm)): + manifest = make_filesd_manifest(tmpdir) + found_manifest = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + self.assertEquals(manifest.sdUUID, found_manifest.sdUUID) + self.assertIsInstance(found_manifest, + fileSD.FileStorageDomainManifest) + + def test_glusterfs_layout(self): + with namedTemporaryDir() as tmpdir: + mntdir = os.path.join(tmpdir, sd.DOMAIN_MNT_POINT) + glusterdir = os.path.join(mntdir, sd.GLUSTERSD_DIR) + os.makedirs(glusterdir) + os.symlink(tmpdir, os.path.join(glusterdir, 'host:_glustersd')) + + lvm = FakeLVM(tmpdir) + with MonkeyPatchScope(self.get_patches(tmpdir, lvm)): + manifest = make_filesd_manifest(tmpdir) + found_manifest = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + self.assertEquals(manifest.sdUUID, found_manifest.sdUUID) + self.assertEquals(manifest.sdUUID, found_manifest.sdUUID) + self.assertIsInstance(found_manifest, + fileSD.FileStorageDomainManifest) + + def test_block_layout(self): + with namedTemporaryDir() as tmpdir: + lvm = FakeLVM(tmpdir) + with MonkeyPatchScope(self.get_patches(tmpdir, lvm)): + manifest = make_blocksd_manifest() + vg_name = make_vg(lvm, manifest) + lvm.vgmd[vg_name]['tags'] = [blockSD.STORAGE_DOMAIN_TAG] + found_manifest = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + self.assertEquals(manifest.sdUUID, found_manifest.sdUUID) + self.assertIsInstance(found_manifest, + blockSD.BlockStorageDomainManifest) + + def test_cached_finder(self): + def _trap(unused): + raise RuntimeError() + + with namedTemporaryDir() as tmpdir: + self.make_mnt(tmpdir, '_localfssd') + lvm = FakeLVM(tmpdir) + patches = self.get_patches(tmpdir, lvm) + with MonkeyPatchScope(patches): + manifest = make_filesd_manifest(tmpdir) + res1 = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + patches.append((localFsSD, 'findDomainManifest', _trap)) + with MonkeyPatchScope(patches): + res2 = domainManifestFactory.produce( + manifest.sdUUID, manifest._metadata) + self.assertEquals(res1.sdUUID, res2.sdUUID) + + def test_not_found(self): + sdUUID = str(uuid.uuid4()) + with namedTemporaryDir() as tmpdir: + lvm = FakeLVM(tmpdir) + with MonkeyPatchScope(self.get_patches(tmpdir, lvm)): + self.assertRaises(se.StorageDomainDoesNotExist, + domainManifestFactory.produce, sdUUID) \ No newline at end of file diff --git a/tests/manifest_tests.py b/tests/manifest_tests.py index a22d7cf..967b663 100644 --- a/tests/manifest_tests.py +++ b/tests/manifest_tests.py @@ -161,7 +161,6 @@ imguuid = str(uuid.uuid4()) imagepath = manifest.getImagePath(imguuid) - os.makedirs(os.path.dirname(imagepath)) with open(imagepath, 'w') as f: f.truncate(0) self.assertEquals(set(), manifest.getAllImages()) diff --git a/tests/storagefakelib.py b/tests/storagefakelib.py index cb40fbb..bf79d91 100644 --- a/tests/storagefakelib.py +++ b/tests/storagefakelib.py @@ -23,6 +23,7 @@ from copy import deepcopy from storage import lvm as real_lvm +from storage import storage_exception as se class FakeLVM(object): @@ -154,7 +155,11 @@ return real_lvm.PV(**md) def getVG(self, vgName): - vg_md = deepcopy(self.vgmd[vgName]) + try: + vg_md = deepcopy(self.vgmd[vgName]) + except KeyError: + raise se.VolumeGroupDoesNotExist(vgName) + vg_attr = real_lvm.VG_ATTR(**vg_md['attr']) vg_md['attr'] = vg_attr return real_lvm.VG(**vg_md) diff --git a/tests/storagetestutils.py b/tests/storagetestutils.py index ed052af..eae803a 100644 --- a/tests/storagetestutils.py +++ b/tests/storagetestutils.py @@ -79,7 +79,7 @@ if metadata is None: metadata = {sd.DMDK_VERSION: 3} manifest = fileSD.FileStorageDomainManifest(domain_path, metadata) - os.makedirs(os.path.join(manifest.domaindir, sduuid, sd.DOMAIN_IMAGES)) + os.makedirs(os.path.join(manifest.domaindir, sd.DOMAIN_IMAGES)) return manifest diff --git a/vdsm.spec.in b/vdsm.spec.in index 4ba9812..57e673e 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -906,6 +906,7 @@ %{_datadir}/%{vdsm_name}/storage/curlImgWrap.py* %{_datadir}/%{vdsm_name}/storage/devicemapper.py* %{_datadir}/%{vdsm_name}/storage/dispatcher.py* +%{_datadir}/%{vdsm_name}/storage/domainManifestFactory.py %{_datadir}/%{vdsm_name}/storage/monitor.py* %{_datadir}/%{vdsm_name}/storage/fileSD.py* %{_datadir}/%{vdsm_name}/storage/fileUtils.py* diff --git a/vdsm/storage/Makefile.am b/vdsm/storage/Makefile.am index a4ec93a..e460518 100644 --- a/vdsm/storage/Makefile.am +++ b/vdsm/storage/Makefile.am @@ -31,6 +31,7 @@ curlImgWrap.py \ devicemapper.py \ dispatcher.py \ + domainManifestFactory.py \ fileSD.py \ fileUtils.py \ fileVolume.py \ diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 121cc6b..6de2737 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -1426,5 +1426,10 @@ return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) +def findDomainManifest(sdUUID, metadata=None): + return BlockStorageDomainManifest( + BlockStorageDomain.findDomainPath(sdUUID), metadata) + + def getStorageDomainsList(): return [vg.name for vg in lvm.getAllVGs() if _isSD(vg)] diff --git a/vdsm/storage/domainManifestFactory.py b/vdsm/storage/domainManifestFactory.py new file mode 100644 index 0000000..fced9de --- /dev/null +++ b/vdsm/storage/domainManifestFactory.py @@ -0,0 +1,60 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import logging + +import blockSD, glusterSD, localFsSD, nfsSD +import storage_exception as se + + +log = logging.getLogger('Storage.domainManifestFactory') + +domain_manifest_producers = {} + + +def produce(sdUUID, metadata=None): + log.error("looking for domain manifest %s", sdUUID) + + # Check our cache of finder functions first. Since a domain manifest + # cannot change its type we don't have to worry about invalidation. + try: + producer = domain_manifest_producers[sdUUID] + except KeyError: + pass + else: + return producer(sdUUID, metadata) + + # The order is somewhat important, it's ordered + # by how quickly get can find the domain. For instance + # if an nfs mount is unavailable we will get stuck + # until it times out, this should affect fetching + # of block\local domains. If for any case in the future + # this changes, please update the order. + for mod in (blockSD, glusterSD, localFsSD, nfsSD): + try: + manifest = mod.findDomainManifest(sdUUID, metadata) + domain_manifest_producers[sdUUID] = mod.findDomainManifest + return manifest + except se.StorageDomainDoesNotExist: + pass + except Exception: + log.exception("Error while looking for domain `%s`", sdUUID) + + raise se.StorageDomainDoesNotExist(sdUUID) diff --git a/vdsm/storage/glusterSD.py b/vdsm/storage/glusterSD.py index 2931083..a62e242 100644 --- a/vdsm/storage/glusterSD.py +++ b/vdsm/storage/glusterSD.py @@ -30,3 +30,8 @@ def findDomain(sdUUID): return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) + + +def findDomainManifest(sdUUID, metadata=None): + return fileSD.FileStorageDomainManifest( + GlusterStorageDomain.findDomainPath(sdUUID), metadata) diff --git a/vdsm/storage/localFsSD.py b/vdsm/storage/localFsSD.py index 7e11330..b4e0ce0 100644 --- a/vdsm/storage/localFsSD.py +++ b/vdsm/storage/localFsSD.py @@ -119,3 +119,8 @@ def findDomain(sdUUID): return LocalFsStorageDomain(LocalFsStorageDomain.findDomainPath(sdUUID)) + + +def findDomainManifest(sdUUID, metadata=None): + return fileSD.FileStorageDomainManifest( + LocalFsStorageDomain.findDomainPath(sdUUID), metadata) \ No newline at end of file diff --git a/vdsm/storage/nfsSD.py b/vdsm/storage/nfsSD.py index 1f94439..739efc8 100644 --- a/vdsm/storage/nfsSD.py +++ b/vdsm/storage/nfsSD.py @@ -120,3 +120,8 @@ def findDomain(sdUUID): return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) + + +def findDomainManifest(sdUUID, metadata=None): + return fileSD.FileStorageDomainManifest( + NfsStorageDomain.findDomainPath(sdUUID), metadata) -- To view, visit https://gerrit.ovirt.org/43555 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia5d139bde2349c8231141fe842d9ada96fa279fe Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 12

Change in vdsm[master]: sdm: Add decorators for sdm-only functions and their callers
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: sdm: Add decorators for sdm-only functions and their callers ...................................................................... sdm: Add decorators for sdm-only functions and their callers Change-Id: Iaf736144e5640519851dc9175b5f17539d0ce23e Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/Makefile.am A tests/sdm_tests.py M vdsm.spec.in M vdsm/storage/Makefile.am A vdsm/storage/sdm.py 5 files changed, 102 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/56/43556/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index f116fc4..5907a0d 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -92,6 +92,7 @@ scheduleTests.py \ schemaTests.py \ schemaValidationTest.py \ + sdm_tests.py \ securableTests.py \ sourceroutingTests.py \ sslhelper.py \ diff --git a/tests/sdm_tests.py b/tests/sdm_tests.py new file mode 100644 index 0000000..95ae126 --- /dev/null +++ b/tests/sdm_tests.py @@ -0,0 +1,43 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from testlib import VdsmTestCase + +from storage import sdm +from storage.threadLocal import vars + + +class RequireSDMTests(VdsmTestCase): + def setUp(self): + try: + delattr(vars, '__sdm__') + except AttributeError: + pass + + @sdm.sdm_verb + def test_allowed(self): + self.assertTrue(self.sdm_fn()) + + def test_denied(self): + self.assertRaises(sdm.SDMFunctionNotCallable, self.sdm_fn) + + @sdm.require_sdm + def sdm_fn(self): + return True \ No newline at end of file diff --git a/vdsm.spec.in b/vdsm.spec.in index 57e673e..24e0ab7 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -936,6 +936,7 @@ %{_datadir}/%{vdsm_name}/storage/clusterlock.py* %{_datadir}/%{vdsm_name}/storage/sdc.py* %{_datadir}/%{vdsm_name}/storage/sd.py* +%{_datadir}/%{vdsm_name}/storage/sdm.py* %{_datadir}/%{vdsm_name}/storage/securable.py* %{_datadir}/%{vdsm_name}/storage/sp.py* %{_datadir}/%{vdsm_name}/storage/spbackends.py* diff --git a/vdsm/storage/Makefile.am b/vdsm/storage/Makefile.am index e460518..51768ce 100644 --- a/vdsm/storage/Makefile.am +++ b/vdsm/storage/Makefile.am @@ -59,6 +59,7 @@ resourceManager.py \ sdc.py \ sd.py \ + sdm.py \ securable.py \ sp.py \ spbackends.py \ diff --git a/vdsm/storage/sdm.py b/vdsm/storage/sdm.py new file mode 100644 index 0000000..97e599f --- /dev/null +++ b/vdsm/storage/sdm.py @@ -0,0 +1,56 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from functools import wraps +from storage.threadLocal import vars + + +class SDMFunctionNotCallable(Exception): + pass + + +def require_sdm(f): + """ + SDM method decorator. + + This decorator is used to mark methods that can only be called when the + storage is operating in SDM mode (ie. without SPM). + """ + @wraps(f) + def wrapper(*args, **kwds): + if getattr(vars, '__sdm__', False): + return f(*args, **kwds) + else: + raise SDMFunctionNotCallable(f.__name__) + return wrapper + + +def sdm_verb(f): + """ + SDM verb decorator + + This decorator indicates that a function is designed to work without SPM + and is approved to access SDM-only functions. + """ + @wraps(f) + def wrapper(*args, **kwds): + vars.__sdm__ = True + return f(*args, **kwds) + return wrapper -- To view, visit https://gerrit.ovirt.org/43556 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf736144e5640519851dc9175b5f17539d0ce23e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 11

Change in vdsm[master]: storage: Factor apparentsize adjustment into a reusable func...
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Factor apparentsize adjustment into a reusable function ...................................................................... storage: Factor apparentsize adjustment into a reusable function When creating a new volume sometimes the actual size is larger than the requested size due to underlying storage limitations (such as physical extent granularity in LVM). In these cases, the new volume's size must be adjusted to match the actual size. The SDM volume creation code would also like to use this function so extract it into the VolumeMetadata class. Change-Id: I7395850e127b941fc50ae8d5659868a2499b5076 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/volume.py 1 file changed, 28 insertions(+), 18 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/44/44044/1 diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index f7e5bc0..723b3d9 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -495,6 +495,31 @@ puuid) return None + @classmethod + def adjust_new_volume_size(cls, dom_manifest, img_id, vol_id, req_size, + vol_format): + # When the volume format is raw what the guest sees is the apparent + # size of the file/device therefore if the requested size doesn't + # match the apparent size (eg: physical extent granularity in LVM) + # we need to update the size value so that the metadata reflects + # the correct state. + if vol_format == RAW_FORMAT: + apparentSize = int(dom_manifest.getVSize(img_id, vol_id) / + BLOCK_SIZE) + if apparentSize < req_size: + cls.log.error("The volume %s apparent size %s is smaller " + "than the requested size %s", + vol_id, apparentSize, req_size) + raise se.VolumeCreationError() + if apparentSize > req_size: + cls.log.info("The requested size for volume %s doesn't " + "match the granularity on domain %s, " + "updating the volume size from %s to %s", + vol_id, dom_manifest.sdUUID, req_size, + apparentSize) + return apparentSize + return req_size + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -850,24 +875,9 @@ cls.log.error("Failed to create volume %s: %s", volPath, e) vars.task.popRecovery() raise - # When the volume format is raw what the guest sees is the apparent - # size of the file/device therefore if the requested size doesn't - # match the apparent size (eg: physical extent granularity in LVM) - # we need to update the size value so that the metadata reflects - # the correct state. - if volFormat == RAW_FORMAT: - apparentSize = int(dom.getVSize(imgUUID, volUUID) / BLOCK_SIZE) - if apparentSize < size: - cls.log.error("The volume %s apparent size %s is smaller " - "than the requested size %s", - volUUID, apparentSize, size) - raise se.VolumeCreationError() - if apparentSize > size: - cls.log.info("The requested size for volume %s doesn't " - "match the granularity on domain %s, " - "updating the volume size from %s to %s", - volUUID, sdUUID, size, apparentSize) - size = apparentSize + + size = VolumeMetadata.adjust_new_volume_size(dom, imgUUID, volUUID, + size, volFormat) vars.task.pushRecovery( task.Recovery("Create volume metadata rollback", clsModule, -- To view, visit https://gerrit.ovirt.org/44044 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7395850e127b941fc50ae8d5659868a2499b5076 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: storage: Rename _share to share_to_image
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Rename _share to share_to_image ...................................................................... storage: Rename _share to share_to_image When a new volume is created whose parent is a template volume, that template must be "shared" into the new volume's image since volume chains only link volumes in the same image. This logic is reusable by the SDM create volume flow but _share should be renamed to share_to_image to remove the _ which designates this as a protected method. Change-Id: I44c7fe43ada38aa97bbd9c08438a0ee686680330 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/46/44046/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 47336b9..23e636e 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -314,7 +314,7 @@ def refreshVolume(self): lvm.refreshLVs(self.sdUUID, (self.volUUID,)) - def _share(self, dstImgPath): + def share_to_image(self, dstImgPath): """ Share this volume to dstImgPath """ diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 15ca05b..c5b8667 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -300,7 +300,7 @@ os.path.join(dstImgPath, self.volUUID)) self.oop.utils.forceLink(self._getLeaseVolumePath(), dstLeasePath) - def _share(self, dstImgPath): + def share_to_image(self, dstImgPath): """ Share this volume to dstImgPath, including the metadata and the lease """ diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 0d61643..951efe0 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -598,7 +598,7 @@ self.md.removeMetadata() def _share(self, dstImgPath): - return self.md._share(dstImgPath) + return self.md.share_to_image(dstImgPath) @classmethod def _getModuleAndClass(cls): -- To view, visit https://gerrit.ovirt.org/44046 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I44c7fe43ada38aa97bbd9c08438a0ee686680330 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: storage: Rename getParent to getParentId
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Rename getParent to getParentId ...................................................................... storage: Rename getParent to getParentId The Volume and VolumeMetadata classes have functions to return the parent UUID and parent volume instance. It's a bit unclear that getParent returns the ID so let's make this more explicit by renaming getParent to getParentId. Change-Id: Ie310d731709ebf1ba3589e92a7ae506ccdc8ae22 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/image.py M vdsm/storage/sp.py M vdsm/storage/volume.py 5 files changed, 20 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/47/44047/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 23e636e..0168d64 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -167,7 +167,7 @@ def getParentMeta(self): return self.getMetaParam(volume.PUUID) - def getParent(self): + def getParentId(self): """ Return parent volume UUID """ @@ -490,7 +490,7 @@ try: # We need to blank parent record in our metadata # for parent to become leaf successfully. - puuid = self.getParent() + puuid = self.getParentId() self.setParent(volume.BLANK_UUID) if puuid and puuid != volume.BLANK_UUID: pvol = BlockVolume(self.repoPath, self.sdUUID, self.imgUUID, diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index c5b8667..2dad91f 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -150,7 +150,7 @@ return out - def getParent(self): + def getParentId(self): """ Return parent volume UUID """ @@ -452,7 +452,7 @@ try: # We need to blank parent record in our metadata # for parent to become leaf successfully. - puuid = self.getParent() + puuid = self.getParentId() self.setParent(volume.BLANK_UUID) if puuid and puuid != volume.BLANK_UUID: pvol = FileVolume(self.repoPath, self.sdUUID, diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index fa0770c..43e08e5 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -232,7 +232,7 @@ chain.insert(0, srcVol) seen.add(srcVol.volUUID) - parentUUID = srcVol.getParent() + parentUUID = srcVol.getParentId() if parentUUID == volume.BLANK_UUID: break @@ -578,7 +578,7 @@ raise se.VolumeNotSparse() srcVolume = self._getSparsifyVolume(tmpSdUUID, tmpImgUUID, - tmpVolume.getParent()) + tmpVolume.getParentId()) tmpVolume.prepare() try: @@ -898,7 +898,7 @@ successor = chain[-1] tmpVol = volclass(self.repoPath, sdDom.sdUUID, imgUUID, successor) dstParent = volclass(self.repoPath, sdDom.sdUUID, imgUUID, - ancestor).getParent() + ancestor).getParentId() # Mark all volumes as illegal while tmpVol and dstParent != tmpVol.volUUID: @@ -926,7 +926,7 @@ successor = chain[-1] srcVol = volclass(self.repoPath, sdUUID, imgUUID, successor) dstParent = volclass(self.repoPath, sdUUID, imgUUID, - ancestor).getParent() + ancestor).getParentId() while srcVol and dstParent != srcVol.volUUID: try: @@ -953,7 +953,7 @@ successor = chain[-1] srcVol = volclass(self.repoPath, sdDom.sdUUID, imgUUID, successor) dstParent = volclass(self.repoPath, sdDom.sdUUID, imgUUID, - ancestor).getParent() + ancestor).getParentId() while srcVol and dstParent != srcVol.volUUID: self.log.info("Remove volume %s from image %s", srcVol.volUUID, @@ -1143,12 +1143,12 @@ """ chain = [] accumulatedChainSize = 0 - endVolName = vols[ancestor].getParent() # TemplateVolName or None + endVolName = vols[ancestor].getParentId() # TemplateVolName or None currVolName = successor while (currVolName != endVolName): chain.insert(0, currVolName) accumulatedChainSize += vols[currVolName].getVolumeSize() - currVolName = vols[currVolName].getParent() + currVolName = vols[currVolName].getParentId() return accumulatedChainSize, chain @@ -1169,7 +1169,7 @@ self.log.debug("unlinking subchain: %s" % subChain) sdDom = sdCache.produce(sdUUID=sdUUID) - dstParent = sdDom.produceVolume(imgUUID, subChain[0]).getParent() + dstParent = sdDom.produceVolume(imgUUID, subChain[0]).getParentId() subChainTailVol = sdDom.produceVolume(imgUUID, subChain[-1]) if subChainTailVol.isLeaf(): self.log.debug("Leaf volume is being removed from the chain. " @@ -1247,10 +1247,10 @@ srcVolParams = srcVol.getVolumeParams() srcVolParams['children'] = [] for vName, vol in vols.iteritems(): - if vol.getParent() == successor: + if vol.getParentId() == successor: srcVolParams['children'].append(vol) dstVol = vols[ancestor] - dstParentUUID = dstVol.getParent() + dstParentUUID = dstVol.getParentId() if dstParentUUID != sd.BLANK_UUID: volParams = vols[dstParentUUID].getVolumeParams() else: diff --git a/vdsm/storage/sp.py b/vdsm/storage/sp.py index 8574bac..b92f107 100644 --- a/vdsm/storage/sp.py +++ b/vdsm/storage/sp.py @@ -1838,7 +1838,7 @@ volUUID=srcVolUUID) if not srcVol.isShared(): - if srcVol.getParent() == volume.BLANK_UUID: + if srcVol.getParentId() == volume.BLANK_UUID: with rmanager.acquireResource(imageResourcesNamespace, srcImgUUID, rm.LockType.exclusive): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 951efe0..691d3b9 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -250,7 +250,7 @@ "disktype": meta.get(DISKTYPE, ""), "voltype": meta.get(VOLTYPE, ""), "size": int(meta.get(SIZE, "0")), - "parent": self.getParent(), + "parent": self.getParentId(), "description": meta.get(DESCRIPTION, ""), "pool": meta.get(sd.DMDK_POOLS, ""), "domain": meta.get(DOMAIN, ""), @@ -310,7 +310,7 @@ volParams['size'] = self.getSize() volParams['apparentsize'] = self.getVolumeSize(bs=bs) volParams['truesize'] = self.getVolumeTrueSize(bs=bs) - volParams['parent'] = self.getParent() + volParams['parent'] = self.getParentId() volParams['descr'] = self.getDescription() volParams['legality'] = self.getLegality() return volParams @@ -494,7 +494,7 @@ """ Return parent volume object """ - puuid = self.getParent() + puuid = self.getParentId() if puuid and puuid != BLANK_UUID: return sdCache.produce(self.sdUUID).produceVolume(self.imgUUID, puuid) @@ -571,11 +571,11 @@ """ return self.md.getMetadata(metaId) - def getParent(self): + def getParentId(self): """ Return parent volume UUID """ - return self.md.getParent() + return self.md.getParentId() def getChildren(self): """ Return children volume UUIDs. -- To view, visit https://gerrit.ovirt.org/44047 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie310d731709ebf1ba3589e92a7ae506ccdc8ae22 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: storage: Rename getParentVolume* to produceParent
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Rename getParentVolume* to produceParent ...................................................................... storage: Rename getParentVolume* to produceParent The Volume object provides getParentVolume and the VolumeMetadata object provides getParentVolumeMetadata. Both of these functions return an instance of their same type which represents this volume's parent. Since the logic is the same and both object types implement compatible interfaces let's unify around the name produceParent. Once complete, consumers of these objects can use either type without additional modifications. Change-Id: I4553e5bcc8a576abc6fb472148f744c66e5012ef Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/image.py M vdsm/storage/resourceFactories.py M vdsm/storage/volume.py 3 files changed, 16 insertions(+), 16 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/48/44048/1 diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 43e08e5..bd79f6e 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -168,7 +168,7 @@ """ chain = self.getChain(sdUUID, imgUUID, volUUID) newsize = 0 - template = chain[0].getParentVolume() + template = chain[0].produceParent() if template: newsize = template.getVolumeSize() for vol in chain: @@ -241,7 +241,7 @@ imgUUID, srcVol.volUUID, parentUUID) raise se.ImageIsNotLegalChain(imgUUID) - srcVol = srcVol.getParentVolume() + srcVol = srcVol.produceParent() self.log.info("sdUUID=%s imgUUID=%s chain=%s ", sdUUID, imgUUID, chain) return chain @@ -254,7 +254,7 @@ # Find all volumes of image (excluding template) chain = self.getChain(sdUUID, imgUUID) # check if the chain is build above a template, or it is a standalone - pvol = chain[0].getParentVolume() + pvol = chain[0].produceParent() if pvol: tmpl = pvol elif chain[0].isShared(): @@ -351,7 +351,7 @@ fakeTemplate = False pimg = volume.BLANK_UUID # standalone chain # check if the chain is build above a template, or it is a standalone - pvol = srcChain[0].getParentVolume() + pvol = srcChain[0].produceParent() if pvol: # find out parent volume parameters volParams = pvol.getVolumeParams() @@ -454,7 +454,7 @@ srcFormat = volume.fmt2str(srcVol.getFormat()) dstFormat = volume.fmt2str(dstVol.getFormat()) - parentVol = dstVol.getParentVolume() + parentVol = dstVol.produceParent() if parentVol is not None: backing = volume.getBackingVolumePath( @@ -731,7 +731,7 @@ raise se.ImageIsNotLegalChain(imgUUID) chain = self.getChain(sdUUID, imgUUID) # check if the chain is build above a template, or it is a standalone - pvol = chain[0].getParentVolume() + pvol = chain[0].produceParent() if pvol: if not pvol.isLegal() or pvol.isFake(): raise se.ImageIsNotLegalChain(imgUUID) @@ -902,7 +902,7 @@ # Mark all volumes as illegal while tmpVol and dstParent != tmpVol.volUUID: - vol = tmpVol.getParentVolume() + vol = tmpVol.produceParent() tmpVol.setLegality(volume.ILLEGAL_VOL) tmpVol = vol @@ -932,7 +932,7 @@ try: self.log.info("Teardown volume %s from image %s", srcVol.volUUID, imgUUID) - vol = srcVol.getParentVolume() + vol = srcVol.produceParent() srcVol.teardown(sdUUID=srcVol.sdUUID, volUUID=srcVol.volUUID, justme=True) srcVol = vol @@ -958,7 +958,7 @@ while srcVol and dstParent != srcVol.volUUID: self.log.info("Remove volume %s from image %s", srcVol.volUUID, imgUUID) - vol = srcVol.getParentVolume() + vol = srcVol.produceParent() srcVol.delete(postZero=postZero, force=True) chain.remove(srcVol.volUUID) srcVol = vol @@ -1321,7 +1321,7 @@ def _activateVolumeForImportExport(self, domain, imgUUID, volUUID=None): chain = self.getChain(domain.sdUUID, imgUUID, volUUID) - template = chain[0].getParentVolume() + template = chain[0].produceParent() if template or len(chain) > 1: self.log.error("Importing and exporting an image with more " diff --git a/vdsm/storage/resourceFactories.py b/vdsm/storage/resourceFactories.py index 98c362b..fb2522d 100644 --- a/vdsm/storage/resourceFactories.py +++ b/vdsm/storage/resourceFactories.py @@ -126,7 +126,7 @@ return [] # check if the chain is build above a template, or it is a standalone - pvol = chain[0].getParentVolume() + pvol = chain[0].produceParent() if pvol: template = pvol.volUUID elif chain[0].isShared(): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 691d3b9..a098eb7 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -490,7 +490,7 @@ """ pass # Do not remove this method or the V3 upgrade will fail. - def getParentVolume(self): + def produceParent(self): """ Return parent volume object """ @@ -659,7 +659,7 @@ Rebase volume on top of new backing volume """ if rollback: - pvol = self.getParentVolume() + pvol = self.produceParent() if not pvol: self.log.warn("Can't rebase volume %s, parent missing", self.volUUID) @@ -787,7 +787,7 @@ "volUUID=%s imageDir=%s" % (repoPath, sdUUID, imgUUID, volUUID, imageDir)) vol = sdCache.produce(sdUUID).produceVolume(imgUUID, volUUID) - pvol = vol.getParentVolume() + pvol = vol.produceParent() # Remove volume vol.delete(postZero=False, force=True) if len(cls.getImageVolumes(repoPath, sdUUID, imgUUID)): @@ -1120,7 +1120,7 @@ try: if justme: return True - pvol = self.getParentVolume() + pvol = self.produceParent() if pvol: pvol.prepare(rw=chainrw, justme=False, chainrw=chainrw, setrw=setrw) @@ -1149,7 +1149,7 @@ def getInfo(self): return self.md.getInfo() - def getParentVolume(self): + def produceParent(self): """ Return parent volume object """ -- To view, visit https://gerrit.ovirt.org/44048 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4553e5bcc8a576abc6fb472148f744c66e5012ef Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: ImageManifest: move getChain
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: ImageManifest: move getChain ...................................................................... ImageManifest: move getChain The implementation of getChain instantiates storagedomain and volume objects. This must be fixed. We must fix all callers of getChain to work with VolumeMetadata objects instead of Volume objects. Change-Id: Iffd24be3ba74ccdea8dfc744b579c0cf63771cdb Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/image.py 1 file changed, 71 insertions(+), 66 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/49/44049/1 diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index bd79f6e..f852f3c 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -89,6 +89,8 @@ class ImageManifest(object): + log = logging.getLogger('Storage.ImageManifest') + def __init__(self, repoPath): self.repoPath = repoPath @@ -105,6 +107,74 @@ log.info("Create placeholder %s for image's volumes", img_dir) os.mkdir(img_dir) return img_dir + + def getChain(self, sdUUID, imgUUID, volUUID=None): + """ + Return the chain of volumes of image as a sorted list + (not including a shared base (template) if any) + """ + chain = [] + volclass = sdCache.produce(sdUUID).getVolumeClass() + + # Use volUUID when provided + if volUUID: + srcVol = volclass(self.repoPath, sdUUID, imgUUID, volUUID) + + # For template images include only one volume (the template itself) + # NOTE: this relies on the fact that in a template there is only + # one volume + if srcVol.isShared(): + return [srcVol] + + # Find all the volumes when volUUID is not provided + else: + # Find all volumes of image + uuidlist = volclass.getImageVolumes(self.repoPath, sdUUID, imgUUID) + + if not uuidlist: + raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) + + srcVol = volclass(self.repoPath, sdUUID, imgUUID, uuidlist[0]) + + # For template images include only one volume (the template itself) + if len(uuidlist) == 1 and srcVol.isShared(): + return [srcVol] + + # Searching for the leaf + for vol in uuidlist: + srcVol = volclass(self.repoPath, sdUUID, imgUUID, vol) + + if srcVol.isLeaf(): + break + + srcVol = None + + if not srcVol: + self.log.error("There is no leaf in the image %s", imgUUID) + raise se.ImageIsNotLegalChain(imgUUID) + + # We have seen corrupted chains that cause endless loops here. + # https://bugzilla.redhat.com/1125197 + seen = set() + + # Build up the sorted parent -> child chain + while not srcVol.isShared(): + chain.insert(0, srcVol) + seen.add(srcVol.volUUID) + + parentUUID = srcVol.getParentId() + if parentUUID == volume.BLANK_UUID: + break + + if parentUUID in seen: + self.log.error("Image %s volume %s has invalid parent UUID %s", + imgUUID, srcVol.volUUID, parentUUID) + raise se.ImageIsNotLegalChain(imgUUID) + + srcVol = srcVol.produceParent() + + self.log.info("sdUUID=%s imgUUID=%s chain=%s ", sdUUID, imgUUID, chain) + return chain class Image: @@ -179,72 +249,7 @@ return newsize def getChain(self, sdUUID, imgUUID, volUUID=None): - """ - Return the chain of volumes of image as a sorted list - (not including a shared base (template) if any) - """ - chain = [] - volclass = sdCache.produce(sdUUID).getVolumeClass() - - # Use volUUID when provided - if volUUID: - srcVol = volclass(self.repoPath, sdUUID, imgUUID, volUUID) - - # For template images include only one volume (the template itself) - # NOTE: this relies on the fact that in a template there is only - # one volume - if srcVol.isShared(): - return [srcVol] - - # Find all the volumes when volUUID is not provided - else: - # Find all volumes of image - uuidlist = volclass.getImageVolumes(self.repoPath, sdUUID, imgUUID) - - if not uuidlist: - raise se.ImageDoesNotExistInSD(imgUUID, sdUUID) - - srcVol = volclass(self.repoPath, sdUUID, imgUUID, uuidlist[0]) - - # For template images include only one volume (the template itself) - if len(uuidlist) == 1 and srcVol.isShared(): - return [srcVol] - - # Searching for the leaf - for vol in uuidlist: - srcVol = volclass(self.repoPath, sdUUID, imgUUID, vol) - - if srcVol.isLeaf(): - break - - srcVol = None - - if not srcVol: - self.log.error("There is no leaf in the image %s", imgUUID) - raise se.ImageIsNotLegalChain(imgUUID) - - # We have seen corrupted chains that cause endless loops here. - # https://bugzilla.redhat.com/1125197 - seen = set() - - # Build up the sorted parent -> child chain - while not srcVol.isShared(): - chain.insert(0, srcVol) - seen.add(srcVol.volUUID) - - parentUUID = srcVol.getParentId() - if parentUUID == volume.BLANK_UUID: - break - - if parentUUID in seen: - self.log.error("Image %s volume %s has invalid parent UUID %s", - imgUUID, srcVol.volUUID, parentUUID) - raise se.ImageIsNotLegalChain(imgUUID) - - srcVol = srcVol.produceParent() - - self.log.info("sdUUID=%s imgUUID=%s chain=%s ", sdUUID, imgUUID, chain) - return chain + return self.manifest.getChain(sdUUID, imgUUID, volUUID) def getTemplate(self, sdUUID, imgUUID): """ -- To view, visit https://gerrit.ovirt.org/44049 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iffd24be3ba74ccdea8dfc744b579c0cf63771cdb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: fix: validate volumemetadata objects
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: fix: validate volumemetadata objects ...................................................................... fix: validate volumemetadata objects Change-Id: I8330bb23c8109cee472904bde63caa75d10c0240 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/volume.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/65/44565/1 diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index a8211a2..d8cd3fb 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -142,6 +142,7 @@ raise se.InvalidParameterException("imgUUID", imgUUID) if not volUUID or volUUID == BLANK_UUID: raise se.InvalidParameterException("volUUID", volUUID) + self.validate() @property def imagePath(self): @@ -176,7 +177,6 @@ def __init__(self, md): self.md = md - self.md.validate() @property def sdUUID(self): -- To view, visit https://gerrit.ovirt.org/44565 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8330bb23c8109cee472904bde63caa75d10c0240 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 6

Change in vdsm[master]: storage: Add garbage indicator to BlockSDVol namedtuple
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Add garbage indicator to BlockSDVol namedtuple ...................................................................... storage: Add garbage indicator to BlockSDVol namedtuple Change-Id: I3e71b3d0ff7c76b815f84d895cd0cbddff251791 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py 1 file changed, 27 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/44568/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 8d571a6..77e162b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -60,7 +60,7 @@ SPECIAL_LVS = (sd.METADATA, sd.LEASES, sd.IDS, sd.INBOX, sd.OUTBOX, MASTERLV) MASTERLV_SIZE = "1024" # In MiB = 2 ** 20 = 1024 ** 2 => 1GiB -BlockSDVol = namedtuple("BlockSDVol", "name, image, parent") +BlockSDVol = namedtuple("BlockSDVol", "name, image, parent, garbage_meta_id") log = logging.getLogger("Storage.BlockSD") @@ -144,6 +144,24 @@ return f.tell() +def _get_garbage_vol_meta_id(sdUUID, volUUID, tags): + """ + If the LV contains TAG_VOL_GARBAGE then treat this volume as garbage. In + order to garbage collect the LV metadata associated with this volume we + need the metadata ID. We cannot call getMetadataId when the LV contains + TAG_VOL_UNINIT so we handle this case specially. + """ + if blockVolume.TAG_VOL_GARBAGE not in tags: + return None + for tag in tags: + if tag.startswith(blockVolume.TAG_PREFIX_MD): + meta_offset = tag[len(blockVolume.TAG_PREFIX_MD):] + return sdUUID, meta_offset + log.error("missing offset tag on volume %s/%s", sdUUID, volUUID) + raise se.VolumeMetadataReadError("missing offset tag on volume %s/%s" % + (sdUUID, volUUID)) + + def _getVolsTree(sdUUID): lvs = lvm.getLV(sdUUID) vols = {} @@ -156,7 +174,8 @@ elif tag.startswith(blockVolume.TAG_PREFIX_PARENT): parent = tag[len(blockVolume.TAG_PREFIX_PARENT):] if parent and image: - vols[lv.name] = BlockSDVol(lv.name, image, parent) + gc_meta_id = _get_garbage_vol_meta_id(sdUUID, lv.name, lv.tags) + vols[lv.name] = BlockSDVol(lv.name, image, parent, gc_meta_id) break else: if lv.name not in SPECIAL_LVS: @@ -175,6 +194,12 @@ Template self image is the 1st term in template volume entry images. """ vols = _getVolsTree(sdUUID) + + # Remove volumes awaiting garbage collection from the result + for vol_id in vols.iterkeys(): + if vols[vol_id].garbage_meta_id is not None: + del vols[vol_id] + res = {} for volName in vols.iterkeys(): res[volName] = {'imgs': [], 'parent': None} -- To view, visit https://gerrit.ovirt.org/44568 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3e71b3d0ff7c76b815f84d895cd0cbddff251791 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 6

Change in vdsm[master]: StorageDomainManifest: Move imageGarbageCollector
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: Move imageGarbageCollector ...................................................................... StorageDomainManifest: Move imageGarbageCollector Change-Id: I6d608bed1a7add239aca3ffe96d9e9a118799511 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/fileSD.py M vdsm/storage/sd.py 2 files changed, 23 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/70/44570/1 diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index 2508216..2e828c3 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -325,6 +325,20 @@ return leasePath, fileVolume.LEASE_FILEOFFSET return None, None + def imageGarbageCollector(self): + """ + Image Garbage Collector + remove the remnants of the removed images (they could be left sometimes + (on NFS mostly) due to lazy file removal + """ + removedPattern = os.path.join(self.domaindir, sd.DOMAIN_IMAGES, + sd.REMOVED_IMAGE_PREFIX + '*') + removedImages = self.oop.glob.glob(removedPattern) + self.log.debug("Removing remnants of deleted images %s" % + removedImages) + for imageDir in removedImages: + self.oop.fileUtils.cleanupdir(imageDir) + class FileStorageDomain(sd.StorageDomain): manifestClass = FileStorageDomainManifest @@ -639,20 +653,6 @@ mount.getMountFromTarget(self.mountpoint).umount() raise se.FileStorageDomainStaleNFSHandle() raise - - def imageGarbageCollector(self): - """ - Image Garbage Collector - remove the remnants of the removed images (they could be left sometimes - (on NFS mostly) due to lazy file removal - """ - removedPattern = os.path.join(self.domaindir, sd.DOMAIN_IMAGES, - sd.REMOVED_IMAGE_PREFIX + '*') - removedImages = self.oop.glob.glob(removedPattern) - self.log.debug("Removing remnants of deleted images %s" % - removedImages) - for imageDir in removedImages: - self.oop.fileUtils.cleanupdir(imageDir) def templateRelink(self, imgUUID, volUUID): """ diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 5635e46..1ddcd45 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -470,6 +470,14 @@ if preallocate is not None and preallocate not in volume.VOL_TYPE: raise se.IncorrectType(preallocate) + def imageGarbageCollector(self): + """ + Image Garbage Collector + remove the remnants of the removed images (they could be left sometimes + (on NFS mostly) due to lazy file removal + """ + pass + class StorageDomain(object): log = logging.getLogger("Storage.StorageDomain") @@ -964,9 +972,4 @@ return self._manifest.isData() def imageGarbageCollector(self): - """ - Image Garbage Collector - remove the remnants of the removed images (they could be left sometimes - (on NFS mostly) due to lazy file removal - """ - pass + self._manifest.imageGarbageCollector() -- To view, visit https://gerrit.ovirt.org/44570 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6d608bed1a7add239aca3ffe96d9e9a118799511 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 6

Change in vdsm[master]: storage: block: getChildren: Do not count uninit LVs as chil...
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: block: getChildren: Do not count uninit LVs as children ...................................................................... storage: block: getChildren: Do not count uninit LVs as children Any LV with the tag TAG_VOL_UNINIT is not a real volume yet. It is either in the process of being created or deleted. It should not appear in the children list of its parent until the tag is removed. Change-Id: I03e489df8478de26b01ce34f41e87aecc04512ed Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/44571/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index ee531f3..51f60ea 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -183,7 +183,8 @@ """ lvs = lvm.lvsByTag(self.sdUUID, "%s%s" % (TAG_PREFIX_PARENT, self.volUUID)) - return tuple(lv.name for lv in lvs) + + return tuple(lv.name for lv in lvs if TAG_VOL_UNINIT not in lv.tags) def getImage(self): """ -- To view, visit https://gerrit.ovirt.org/44571 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I03e489df8478de26b01ce34f41e87aecc04512ed Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 6

Change in vdsm[master]: Garbage volumes are not part of an image
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: Garbage volumes are not part of an image ...................................................................... Garbage volumes are not part of an image Change-Id: I8776a4b48cc8d16f420426fb28d4696b6f2bf357 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/30/44830/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 9baa3d2..af11725 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -334,7 +334,7 @@ (template) """ lvs = lvm.lvsByTag(sdUUID, "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)) - return [lv.name for lv in lvs] + return [lv.name for lv in lvs if TAG_VOL_GARBAGE not in lv.tags] @logskip("ResourceManager") def llPrepare(self, rw=False, setrw=False): -- To view, visit https://gerrit.ovirt.org/44830 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8776a4b48cc8d16f420426fb28d4696b6f2bf357 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 5

Change in vdsm[master]: testing: Add support for failure injection
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: testing: Add support for failure injection ...................................................................... testing: Add support for failure injection Change-Id: I30e562383fb08c4eeeeb91cca33d44a80bbff7f1 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/misc.py M vdsm/storage/storage_exception.py 2 files changed, 10 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/31/44831/1 diff --git a/vdsm/storage/misc.py b/vdsm/storage/misc.py index b14fd26..c53f46d 100644 --- a/vdsm/storage/misc.py +++ b/vdsm/storage/misc.py @@ -1007,3 +1007,8 @@ def deprecated(f): """Used to mark exported methods as deprecated""" return f + + +def maybe_fail(chance=0.1): + if chance != 0 and random.random() <= chance: + raise se.InjectedFailure() diff --git a/vdsm/storage/storage_exception.py b/vdsm/storage/storage_exception.py index 3a4d33f..4ca0b56 100644 --- a/vdsm/storage/storage_exception.py +++ b/vdsm/storage/storage_exception.py @@ -1724,3 +1724,8 @@ code = 855 message = ("Could not acquire resource. " "Probably resource factory threw an exception.") + + +class InjectedFailure(GeneralException): + code = 999 + message = "Injected Failure" -- To view, visit https://gerrit.ovirt.org/44831 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I30e562383fb08c4eeeeb91cca33d44a80bbff7f1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 6

Change in vdsm[master]: inject failures into create container
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: inject failures into create container ...................................................................... inject failures into create container Change-Id: I1fb5f18be6434ee21ac7550fe14c166b5e82b11e Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/sdm.py 1 file changed, 20 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/32/44832/1 diff --git a/vdsm/storage/sdm.py b/vdsm/storage/sdm.py index 7a53e04..af4d479 100644 --- a/vdsm/storage/sdm.py +++ b/vdsm/storage/sdm.py @@ -31,6 +31,8 @@ from vdsm import qemuimg +from misc import maybe_fail + log = logging.getLogger('sdm') rmanager = rm.ResourceManager.getInstance() @@ -43,40 +45,58 @@ def create_volume_container(dom_manifest, img_id, size, vol_format, disk_type, vol_id, desc, src_img_id, src_vol_id): + maybe_fail(0.005) hostId = get_domain_host_id(dom_manifest.sdUUID) + maybe_fail(0.02) dom_manifest.acquireDomainLock(hostId) imageResourcesNamespace = sd.getNamespace(dom_manifest.sdUUID, IMAGE_NAMESPACE) try: + maybe_fail(0.02) with rmanager.acquireResource(imageResourcesNamespace, img_id, rm.LockType.exclusive): + maybe_fail(0.02) image_manifest = ImageManifest(dom_manifest.getRepoPath()) + maybe_fail(0.02) img_path = image_manifest.create_image_dir(dom_manifest.sdUUID, img_id) + maybe_fail(0.02) vol_parent_md = None + maybe_fail(0.02) if src_vol_id != volume.BLANK_UUID: + maybe_fail(0.02) # When the src_img_id isn't specified we assume it's the same # as img_id if src_img_id == volume.BLANK_UUID: + maybe_fail(0.02) src_img_id = img_id + maybe_fail(0.02) vol_parent_md = dom_manifest.produceVolume(src_img_id, src_vol_id) + maybe_fail(0.02) size = vol_parent_md.getSize() + maybe_fail(0.02) dom_manifest.create_volume_artifacts(img_id, vol_id, size, vol_format, disk_type, desc, src_vol_id) + maybe_fail(0.02) if src_vol_id != volume.BLANK_UUID: + maybe_fail(0.02) _prepare_volume_for_parenthood(vol_parent_md, src_img_id, src_vol_id, img_id, img_path) + maybe_fail(0.02) # TODO: get actual size from create_volume_artifacts retval size = volume.VolumeMetadata.adjust_new_volume_size( dom_manifest, img_id, vol_id, size, vol_format) + maybe_fail(0.02) vol_path = os.path.join(img_path, vol_id) + maybe_fail(0.02) _initialize_volume_contents(img_id, vol_id, vol_path, vol_format, size, vol_parent_md) + maybe_fail(0.02) dom_manifest.commit_volume_artifacts(img_path, img_id, vol_id) finally: dom_manifest.releaseDomainLock() -- To view, visit https://gerrit.ovirt.org/44832 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1fb5f18be6434ee21ac7550fe14c166b5e82b11e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 5

Change in vdsm[master]: blockVolume: Add classmethod helper to clear a metadata slot
by alitke＠redhat.com 21 Aug '16

21 Aug '16

Adam Litke has uploaded a new change for review. Change subject: blockVolume: Add classmethod helper to clear a metadata slot ...................................................................... blockVolume: Add classmethod helper to clear a metadata slot Change-Id: I338f3fe5f232b1e6e7e8a2474efabcf56333e1cc Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py 1 file changed, 7 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/33/44833/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 79061e5..c6c3938 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -296,16 +296,17 @@ # tags self.setMetaParam(volume.IMAGE, imgUUID) - def removeMetadata(self, metaId): - """ - Just wipe meta. - """ + @classmethod + def clearMetadataSlot(cls, metaId): try: - self._putMetadata(metaId, {"NONE": "#" * (sd.METASIZE - 10)}) + cls._putMetadata(metaId, {"NONE": "#" * (sd.METASIZE - 10)}) except Exception as e: - self.log.error(e, exc_info=True) + cls.log.exception(e) raise se.VolumeMetadataWriteError("%s: %s" % (metaId, e)) + def removeMetadata(self, metaId): + self.clearMetadataSlot(metaId) + @classmethod def newVolumeLease(cls, metaId, sdUUID, volUUID): cls.log.debug("Initializing volume lease volUUID=%s sdUUID=%s, " -- To view, visit https://gerrit.ovirt.org/44833 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I338f3fe5f232b1e6e7e8a2474efabcf56333e1cc Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 5

Change in vdsm[master]: vm: Cleaner vm stats thread initialization
by Nir Soffer 21 Aug '16

21 Aug '16

Nir Soffer has uploaded a new change for review. Change subject: vm: Cleaner vm stats thread initialization ...................................................................... vm: Cleaner vm stats thread initialization The vm stats thread was initialized too late, leading to unneeded checks and unclear try except blocks when trying to stop the thread. This patch changes AdvancedStatsThread so it keeps a thread instance instead of inheriting from Thread, and initialize it in Vm.__init__, so it is always available when we access it. Change-Id: I2917de42b76ee3dc8b27bdc23b33f3c984a7cc68 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/sampling.py M vdsm/virt/vm.py 2 files changed, 11 insertions(+), 18 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/99/39299/1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 3206b97..848bb36 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -401,7 +401,7 @@ return self._samples.last() -class AdvancedStatsThread(threading.Thread): +class AdvancedStatsThread(object): """ A thread that runs the registered AdvancedStatsFunction objects for statistic and monitoring purpose. @@ -412,9 +412,8 @@ """ Initialize an AdvancedStatsThread object """ - threading.Thread.__init__(self) self.daemon = daemon - + self._thread = None self._log = log self._stopEvent = threading.Event() self._contEvent = threading.Event() @@ -426,7 +425,7 @@ """ Register the functions listed as arguments """ - if self.isAlive(): + if self._thread is not None: raise RuntimeError("AdvancedStatsThread is started") for statsFunction in args: @@ -437,7 +436,9 @@ Start the execution of the thread and exit """ self._log.debug("Start statistics collection") - threading.Thread.start(self) + self._thread = threading.Thread(target=self._run) + self._thread.daemon = self.daemon + self._thread.start() def stop(self): """ @@ -464,7 +465,7 @@ def getLastSampleTime(self): return self._statsTime - def run(self): + def _run(self): self._log.debug("Stats thread started") self._contEvent.set() diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 28054bf..1bc04e7 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -779,7 +779,7 @@ self._initTimeRTC = long(self.conf.get('timeOffset', 0)) self._guestEvent = vmstatus.POWERING_UP self._guestEventTime = 0 - self._vmStats = None + self._vmStats = VmStatsThread(self) self._guestCpuRunning = False self._guestCpuLock = threading.Lock() self._startTime = time.time() - \ @@ -1269,7 +1269,7 @@ return toSave = self.status() toSave['startTime'] = self._startTime - if self.lastStatus != vmstatus.DOWN and self._vmStats: + if self.lastStatus != vmstatus.DOWN: guestInfo = self.guestAgent.getGuestInfo() toSave['username'] = guestInfo['username'] toSave['guestIPs'] = guestInfo['guestIPs'] @@ -1758,7 +1758,7 @@ decStats = {} try: - if self._vmStats and self._vmStats.getLastSampleTime() is not None: + if self._vmStats.getLastSampleTime() is not None: decStats = self._vmStats.get() self._setUnresponsiveIfTimeout( stats, @@ -2001,19 +2001,11 @@ return domxml.toxml() def startVmStats(self): - self._vmStats = VmStatsThread(self) self._vmStats.start() self._guestEventTime = self._startTime def stopVmStats(self): - # this is less clean that it could be, but we can get here from - # many flows and with various locks held - # (_releaseLock, _shutdownLock) - # _vmStats may be None already, and we're good with that. - try: - self._vmStats.stop() - except AttributeError: - pass + self._vmStats.stop() @staticmethod def _guestSockCleanup(sock): -- To view, visit https://gerrit.ovirt.org/39299 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2917de42b76ee3dc8b27bdc23b33f3c984a7cc68 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 8

Change in vdsm[master]: xmlrpc: retry when flushing socket
by Piotr Kliczewski 16 Aug '16

16 Aug '16

Piotr Kliczewski has uploaded a new change for review. Change subject: xmlrpc: retry when flushing socket ...................................................................... xmlrpc: retry when flushing socket m2crypto requires retry of write and we need to make sure that we do it once flushing data before closing a socket. In order to do so we need to move original implementation to our code base. Bug-Url: https://bugzilla.redhat.com/1261255 Change-Id: I52dd6ad304a82ff6c8d3dff12a38269684abf055 Signed-off-by: pkliczewski <piotr.kliczewski(a)gmail.com> --- M vdsm/rpc/bindingxmlrpc.py 1 file changed, 19 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/37/47637/2 diff --git a/vdsm/rpc/bindingxmlrpc.py b/vdsm/rpc/bindingxmlrpc.py index c521cd8..9f77457 100644 --- a/vdsm/rpc/bindingxmlrpc.py +++ b/vdsm/rpc/bindingxmlrpc.py @@ -25,6 +25,7 @@ import libvirt import threading import re +import socket import sys from vdsm.password import (ProtectedPassword, @@ -33,6 +34,7 @@ from vdsm import utils from vdsm import xmlrpc from vdsm.define import doneCode, errCode +from vdsm.m2cutils import SSL from vdsm.netinfo import getDeviceByIP import API from vdsm.exception import VdsmException @@ -288,7 +290,23 @@ return r def finish(self): - xmlrpc.IPXMLRPCRequestHandler.finish(self) + if not self.wfile.closed: + while True: + try: + self.wfile.flush() + break + except socket.error: + # An final socket error may have occurred here, + # such as the local error ECONNABORTED. + break + except SSL.SSLError as e: + if e.message == 'bad write retry': + continue + else: + break + + self.wfile.close() + self.rfile.close() threadLocal.client = None threadLocal.server = None threadLocal.flowID = None -- To view, visit https://gerrit.ovirt.org/47637 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I52dd6ad304a82ff6c8d3dff12a38269684abf055 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com> Gerrit-Reviewer: automation(a)ovirt.org

4 9

Change in vdsm[master]: fix remove: iterate while del
by alitke＠redhat.com 14 Aug '16

14 Aug '16

Adam Litke has uploaded a new change for review. Change subject: fix remove: iterate while del ...................................................................... fix remove: iterate while del Change-Id: I594c98cf54ed93d045b76df1c6faa5122277b19e Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/44572/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 74bf82d..832fb8b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -201,8 +201,8 @@ vols = _getVolsTree(sdUUID) # Remove volumes awaiting garbage collection from the result - for vol_id in vols.iterkeys(): - if vols[vol_id].garbage_meta_id is not None: + for vol_id, vol_info in vols.items(): + if vol_info.garbage_meta_id is not None: del vols[vol_id] res = {} -- To view, visit https://gerrit.ovirt.org/44572 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I594c98cf54ed93d045b76df1c6faa5122277b19e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 3

Change in vdsm[master]: storage: Do not count garbage volumes as children
by alitke＠redhat.com 14 Aug '16

14 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: Do not count garbage volumes as children ...................................................................... storage: Do not count garbage volumes as children Change-Id: Ice3c70249cbc3577b239bd11d224955f2e29b211 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/73/44573/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 51f60ea..9baa3d2 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -184,7 +184,7 @@ lvs = lvm.lvsByTag(self.sdUUID, "%s%s" % (TAG_PREFIX_PARENT, self.volUUID)) - return tuple(lv.name for lv in lvs if TAG_VOL_UNINIT not in lv.tags) + return tuple(lv.name for lv in lvs if TAG_VOL_GARBAGE not in lv.tags) def getImage(self): """ -- To view, visit https://gerrit.ovirt.org/44573 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ice3c70249cbc3577b239bd11d224955f2e29b211 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 3

Change in vdsm[master]: storage: export volume lease state in getVolumeInfo
by alitke＠redhat.com 12 Aug '16

12 Aug '16

Adam Litke has uploaded a new change for review. Change subject: storage: export volume lease state in getVolumeInfo ...................................................................... storage: export volume lease state in getVolumeInfo In order to support an entity-based polling methodology (ie. in the new SDM verbs) we must know whether a volume is currently locked by a long running operation. Extend the getVolumeInfo API to report whether the lease is free or held. Change-Id: I55f062a4be15593fdc98518fd0a113976cbe0ae7 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/rpc/vdsmapi-schema.json M vdsm/storage/volume.py 2 files changed, 31 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/23/38623/1 diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 988ce64..cba1008 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -7897,6 +7897,19 @@ {'enum': 'VolumeStatus', 'data': ['OK', 'INVALID', 'ILLEGAL']} ## +# @VolumeLeaseState: +# +# An enumeration of Volume lease states. +# +# @FREE: The lease is free +# +# @HELD: The lease is held +# +# Since: 4.10.0 +## +{'enum': 'VolumeLeaseState', 'data': ['FREE', 'HELD']} + +## # @VolumeInfo: # # Information about a Volume. @@ -7937,6 +7950,8 @@ # # @children: A list of decendent Volumes that depend on this Volume # +# @leaseState: #optional The state of the volume lease +# # Since: 4.10.0 ## {'type': 'VolumeInfo', @@ -7946,7 +7961,8 @@ 'description': 'str', 'pool': 'UUID', 'domain': 'UUID', 'image': 'UUID', 'ctime': 'int', '*mtime': 'uint', 'legality': 'VolumeLegality', 'apparentsize': 'uint', - 'truesize': 'uint', 'status': 'VolumeStatus', 'children': ['UUID']}} + 'truesize': 'uint', 'status': 'VolumeStatus', 'children': ['UUID'], + '*leaseState': 'VolumeLeaseState'}} ## # @Volume.getInfo: diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 980b6e3..2e16efc 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -33,6 +33,7 @@ import fileUtils import task from threadLocal import vars +from clusterlock import InquireNotSupportedError import resourceFactories import resourceManager as rm rmanager = rm.ResourceManager.getInstance() @@ -85,6 +86,10 @@ ILLEGAL_VOL = "ILLEGAL" LEGAL_VOL = "LEGAL" FAKE_VOL = "FAKE" + +# Volume lease states +LEASE_FREE = "FREE" +LEASE_HELD = "HELD" log = logging.getLogger('Storage.Volume') @@ -839,6 +844,11 @@ cls.createMetadata(metaId, meta) return meta + def getLeaseState(self): + ver, owners = sdCache.produce(self.sdUUID).inquireVolumeLease( + self.imgUUID, self.volUUID) + return LEASE_HELD if owners else LEASE_FREE + def getInfo(self): """ Get volume info @@ -857,6 +867,10 @@ info['apparentsize'] = str(vsize) info['truesize'] = str(avsize) info['status'] = "OK" + try: + info['leaseState'] = self.getLeaseState() + except InquireNotSupportedError: + pass except se.StorageException as e: self.log.debug("exception: %s:%s" % (str(e.message), str(e.value))) info['apparentsize'] = "0" -- To view, visit https://gerrit.ovirt.org/38623 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I55f062a4be15593fdc98518fd0a113976cbe0ae7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

6 40

Change in vdsm[master]: misc: remove cp parameter
by ykaplan＠redhat.com 07 Aug '16

07 Aug '16

Yeela Kaplan has uploaded a new change for review. Change subject: misc: remove cp parameter ...................................................................... misc: remove cp parameter All usages of cp parameter have been extinguished. Therefore, no longer needed. Change-Id: I0906bfd7dfa128c323aa399810bbd75883618434 Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M vdsm/storage/misc.py 1 file changed, 5 insertions(+), 13 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/13/45613/1 diff --git a/vdsm/storage/misc.py b/vdsm/storage/misc.py index b14fd26..b0146c8 100644 --- a/vdsm/storage/misc.py +++ b/vdsm/storage/misc.py @@ -449,7 +449,7 @@ return n -def rotateFiles(directory, prefixName, gen, cp=False, persist=False): +def rotateFiles(directory, prefixName, gen, persist=False): log.debug("dir: %s, prefixName: %s, versions: %s" % (directory, prefixName, gen)) gen = int(gen) @@ -477,24 +477,16 @@ for key in keys: oldName = os.path.join(directory, fd[key]['old']) newName = os.path.join(directory, fd[key]['new']) - if utils.isOvirtNode() and persist and not cp: + if utils.isOvirtNode() and persist: try: utils.unpersist(oldName) utils.unpersist(newName) except: pass - try: - if cp: - execCmd([constants.EXT_CP, oldName, newName], sudo=True) - if (utils.isOvirtNode() and - persist and not os.path.exists(newName)): - utils.persist(newName) - else: - os.rename(oldName, newName) - except: - pass - if utils.isOvirtNode() and persist and not cp: + os.rename(oldName, newName) + + if utils.isOvirtNode() and persist: try: utils.persist(newName) except: -- To view, visit https://gerrit.ovirt.org/45613 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0906bfd7dfa128c323aa399810bbd75883618434 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

5 17

Change in vdsm[master]: freeze: Freeze guest even when creating memory snapshot
by Nir Soffer 31 Jul '16

31 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: freeze: Freeze guest even when creating memory snapshot ...................................................................... freeze: Freeze guest even when creating memory snapshot We used to skip freezing the guest if creating memory snapshot. This was probably done because qemu is pausing the vm for creating memory snapshot. However, this is not consistent with snapshots of external disks such as network disks, where we always freeze the vm before taking the snapshot. Also, it is probably safer to freeze even when creating memory snapshot, giving applications on the guest chance to pause in consistent state. This patch removes the check for memory snapshot, and freeze the guest unless it is already frozen. Change-Id: I8aa7ac0dea8690ca33df8067f84734d788da8bf8 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 3 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/99/43299/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 332281b..ecbbe1d 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -3093,9 +3093,6 @@ else: snapFlags |= libvirt.VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY - # When creating memory snapshot libvirt will pause the vm - should_freeze = not (memoryParams or frozen) - snapxml = snap.toprettyxml() # TODO: this is debug information. For 3.6.x we still need to # see the XML even with 'info' as default level. @@ -3109,7 +3106,7 @@ self.stopDisksStatsCollection() try: - if should_freeze: + if not frozen: freezed = self.freeze() try: self._dom.snapshotCreateXML(snapxml, snapFlags) @@ -3120,7 +3117,7 @@ # Must always thaw, even if freeze failed; in case the guest # did freeze the filesystems, but failed to reply in time. # Libvirt is using same logic (see src/qemu/qemu_driver.c). - if should_freeze: + if not frozen: self.thaw() # We are padding the memory volume with block size of zeroes @@ -3148,7 +3145,7 @@ # Returning quiesce to notify the manager whether the guest agent # froze and flushed the filesystems or not. return {'status': doneCode, - 'quiesce': should_freeze and freezed["status"]["code"] == 0} + 'quiesce': not frozen and freezed["status"]["code"] == 0} def diskReplicateStart(self, srcDisk, dstDisk): try: -- To view, visit https://gerrit.ovirt.org/43299 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8aa7ac0dea8690ca33df8067f84734d788da8bf8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 14

Change in vdsm[master]: log: Use INFO log level as default
by Nir Soffer 27 Jul '16

27 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: log: Use INFO log level as default ...................................................................... log: Use INFO log level as default The current logs are much too verbose which cause trouble for users, and make us look unprofessional. Mature project should not use debug log by default. To debug issues that are not clear enough using INFO logs, the relevant logger level can be modified on a user machine as needed. Change-Id: I767dcd9bad7b9fbeebb438e9ef13cb0ec3f042ee Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/logger.conf.in 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/04/32504/1 diff --git a/vdsm/logger.conf.in b/vdsm/logger.conf.in index 64b154f..8e963dd 100644 --- a/vdsm/logger.conf.in +++ b/vdsm/logger.conf.in @@ -8,18 +8,18 @@ keys=long,simple,none,sysform [logger_root] -level=DEBUG +level=INFO handlers=syslog,logfile propagate=0 [logger_vds] -level=DEBUG +level=INFO handlers=syslog,logfile qualname=vds propagate=0 [logger_Storage] -level=DEBUG +level=INFO handlers=logfile qualname=Storage propagate=0 @@ -31,7 +31,7 @@ propagate=1 [logger_connectivity] -level=DEBUG +level=INFO handlers=connlogfile qualname=connectivity propagate=0 -- To view, visit http://gerrit.ovirt.org/32504 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I767dcd9bad7b9fbeebb438e9ef13cb0ec3f042ee Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 23

Change in vdsm[master]: tests: JsonRpcClient tests suite
by Piotr Kliczewski 25 Jul '16

25 Jul '16

Piotr Kliczewski has uploaded a new change for review. Change subject: tests: JsonRpcClient tests suite ...................................................................... tests: JsonRpcClient tests suite Signed-off-by: pkliczewski <piotr.kliczewski(a)gmail.com> Change-Id: I48c209915f4047958fd9b7e5dd6a202ca14a50cb --- M tests/Makefile.am A tests/jsonrpcClientTests.py M tests/stompAdapterTests.py 3 files changed, 159 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/82/43582/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 9a7f81f..a010bb3 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -57,6 +57,7 @@ iproute2Tests.py \ ipwrapperTests.py \ iscsiTests.py \ + jsonrpcClientTests.py \ jsonrpcServerTests.py \ libvirtconnectionTests.py \ lvmTests.py \ diff --git a/tests/jsonrpcClientTests.py b/tests/jsonrpcClientTests.py new file mode 100644 index 0000000..5d4304f --- /dev/null +++ b/tests/jsonrpcClientTests.py @@ -0,0 +1,148 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +from Queue import Queue +from threading import Event + +from testlib import VdsmTestCase as TestCaseBase +from yajsonrpc.stompreactor import ClientRpcTransportAdapter +from yajsonrpc import JsonRpcClient, JsonRpcRequest, JsonRpcResponse +from yajsonrpc.stomp import Frame, Headers +from vdsm.compat import json + +from stompAdapterTests import TestSubscription + + +ID = 'e8a936a6-d886-4cfa-97b9-2d54209053ff' + + +class TestClient(object): + + def __init__(self): + self._queue = Queue() + self.closed = False + + def send(self, data, destination, headers): + self._queue.put_nowait((data, destination, headers)) + + def pop_message(self): + return self._queue.get(True, 3) + + @property + def has_outgoing_messages(self): + return (self._queue.qsize() > 0) + + def close(self): + self.closed = True + + +class JsonrpcClientTest(TestCaseBase): + + def test_call(self): + subscription = TestSubscription('jms.topic.vdsm_responses', + 'ad052acb-a934-4e10-8ec3-00c7417ef8d1') + client = TestClient() + subscription.set_client(client) + + json_client = JsonRpcClient( + ClientRpcTransportAdapter(subscription, + 'jms.topic.vdsm_requests', + client) + ) + + request = JsonRpcRequest('my_method', (), ID) + json_client.call(request, timeout=0) + + self.assertTrue(client.has_outgoing_messages) + data, destination, headers = client.pop_message() + + req = JsonRpcRequest.fromRawObject(json.loads(data)[0]) + self.assertEquals(req.method, request.method) + self.assertEquals(req.id, request.id) + + self.assertEquals('jms.topic.vdsm_requests', destination) + + self.assertEquals(headers[Headers.CONTENT_TYPE], 'application/json') + self.assertEquals(headers[Headers.REPLY_TO], + 'jms.topic.vdsm_responses') + + def test_async_call_with_response(self): + subscription = TestSubscription('jms.topic.vdsm_responses', + 'ad052acb-a934-4e10-8ec3-00c7417ef8d1') + client = TestClient() + subscription.set_client(client) + + json_client = JsonRpcClient( + ClientRpcTransportAdapter(subscription, + 'jms.topic.vdsm_requests', + client) + ) + + request = JsonRpcRequest('my_method', (), ID) + call = json_client.call_async(request) + + self.assertTrue(client.has_outgoing_messages) + + response = JsonRpcResponse(result='my_response', reqId=ID) + subscription.process(Frame(body=response.encode())) + + self.assertTrue(call.isSet()) + self.assertEquals('my_response', call.responses[0].result) + + def test_receive_event(self): + subscription = TestSubscription('jms.topic.vdsm_responses', + 'ad052acb-a934-4e10-8ec3-00c7417ef8d1') + client = TestClient() + subscription.set_client(client) + ev = Event() + + json_client = JsonRpcClient( + ClientRpcTransportAdapter(subscription, + 'jms.topic.vdsm_requests', + client) + ) + + def callback(client, event, params): + self.assertEquals(event, 'localhost|component|operation|id') + self.assertEquals(params['my_property'], 'my_value') + ev.set() + + json_client.registerEventCallback(callback) + + event = JsonRpcRequest('localhost|component|operation|id', + {'my_property': 'my_value'}) + + subscription.process(Frame(body=event.encode())) + ev.wait(timeout=3) + self.assertTrue(ev.is_set()) + + def test_close(self): + subscription = TestSubscription('jms.topic.vdsm_responses', + 'ad052acb-a934-4e10-8ec3-00c7417ef8d1') + client = TestClient() + + json_client = JsonRpcClient( + ClientRpcTransportAdapter(subscription, + 'jms.topic.vdsm_requests', + client) + ) + + json_client.close() + self.assertTrue(client.closed) + self.assertTrue(subscription.unsubscribed) diff --git a/tests/stompAdapterTests.py b/tests/stompAdapterTests.py index a0be47f..68fae77 100644 --- a/tests/stompAdapterTests.py +++ b/tests/stompAdapterTests.py @@ -60,6 +60,7 @@ def __init__(self, destination, id): self._destination = destination self._id = id + self.unsubscribed = False def set_client(self, client): self._client = TestConnection(client) @@ -76,6 +77,15 @@ def client(self): return self._client + def set_message_handler(self, message_handler): + self.handler = message_handler + + def process(self, data): + self.handler(None, data) + + def unsubscribe(self): + self.unsubscribed = True + class ConnectFrameTest(TestCaseBase): -- To view, visit https://gerrit.ovirt.org/43582 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I48c209915f4047958fd9b7e5dd6a202ca14a50cb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>

2 7

Change in vdsm[master]: tests: JsonRpcServer tests suite
by Piotr Kliczewski 25 Jul '16

25 Jul '16

Piotr Kliczewski has uploaded a new change for review. Change subject: tests: JsonRpcServer tests suite ...................................................................... tests: JsonRpcServer tests suite Change-Id: I1090764c7289544abe331a13ec765ceed2a53afe Signed-off-by: pkliczewski <piotr.kliczewski(a)gmail.com> --- M lib/yajsonrpc/__init__.py M tests/Makefile.am A tests/jsonrpcServerTests.py 3 files changed, 133 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/81/43581/1 diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py index 68af242..a307c7d 100644 --- a/lib/yajsonrpc/__init__.py +++ b/lib/yajsonrpc/__init__.py @@ -293,7 +293,12 @@ self._responses.append(response) def requestDone(self, response): - del self._requests[response.id] + try: + del self._requests[response.id] + except KeyError: + # ignore when request had no id + pass + self.addResponse(response) self.sendReply() diff --git a/tests/Makefile.am b/tests/Makefile.am index 15dbd7b..9a7f81f 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -57,6 +57,7 @@ iproute2Tests.py \ ipwrapperTests.py \ iscsiTests.py \ + jsonrpcServerTests.py \ libvirtconnectionTests.py \ lvmTests.py \ main.py \ diff --git a/tests/jsonrpcServerTests.py b/tests/jsonrpcServerTests.py new file mode 100644 index 0000000..6c508c2 --- /dev/null +++ b/tests/jsonrpcServerTests.py @@ -0,0 +1,126 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +import threading +from contextlib import contextmanager +from Queue import Queue + +from testlib import VdsmTestCase as TestCaseBase +from yajsonrpc import JsonRpcServer, JsonRpcResponse, JsonRpcRequest + + +ID = 'e8a936a6-d886-4cfa-97b9-2d54209053ff' + + +class FakeBridge(object): + + def get_text(self): + return 'my text' + + def register_server_address(self, server_address): + self.server_address = server_address + + def unregister_server_address(self): + self.server_address = None + + +class TestClient(object): + + def __init__(self): + self._queue = Queue() + + def send(self, data): + self._queue.put_nowait(data) + + def pop_message(self): + return self._queue.get(True, 3) + + @property + def has_outgoing_messages(self): + return (self._queue.qsize() > 0) + + +class TestConnection(object): + + def get_local_address(self): + return 'localhost' + + +@contextmanager +def create_server(bridge): + server = JsonRpcServer(bridge, 3600) + t = threading.Thread(target=server.serve_requests, + name='JsonRpcServer') + t.setDaemon(True) + t.start() + + try: + yield server + finally: + server.stop() + + +class JsonrpcServerTest(TestCaseBase): + + def test_method_call(self): + request = JsonRpcRequest(method='get_text', params=(), reqId=ID) + client = TestClient() + + with create_server(FakeBridge()) as server: + server.queueRequest((client, TestConnection(), request.encode())) + + response = client.pop_message() + self.assertIsNot(response, None) + res = JsonRpcResponse.decode(response) + self.assertEquals(res.result, 'my text') + + def test_not_existing_method(self): + request = JsonRpcRequest(method='abcd', params=(), reqId=ID) + client = TestClient() + + with create_server(FakeBridge()) as server: + server.queueRequest((client, TestConnection(), request.encode())) + + response = client.pop_message() + self.assertIsNot(response, None) + res = JsonRpcResponse.decode(response) + self.assertEquals(res.error['code'], -32601) + + def test_wrong_param(self): + request = JsonRpcRequest(method='get_text', params=('param',), + reqId=ID) + client = TestClient() + + with create_server(FakeBridge()) as server: + server.queueRequest((client, TestConnection(), request.encode())) + + response = client.pop_message() + self.assertIsNot(response, None) + res = JsonRpcResponse.decode(response) + self.assertEquals(res.error['code'], -32603) + + def test_no_request_id(self): + request = JsonRpcRequest(method='get_text', params=()) + client = TestClient() + + with create_server(FakeBridge()) as server: + server.queueRequest((client, TestConnection(), request.encode())) + + response = client.pop_message() + self.assertIsNot(response, None) -- To view, visit https://gerrit.ovirt.org/43581 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1090764c7289544abe331a13ec765ceed2a53afe Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>

2 8

Change in vdsm[master]: vm: Cleanup waiting for xml update
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: vm: Cleanup waiting for xml update ...................................................................... vm: Cleanup waiting for xml update This patch cleans up a bit the code for waiting until libvirt xml is updated after pivot was completed. - Clarify confusing log message claiming that pivot failed after it completed successfully - Cleanup creation of volumes lists using generator expression - More clear logic for checking current volumes list - Replace detailed log message and unhelpful exception with detailed exception - Move comment out of the loop to make the loop more clear - Remove unneeded keys() calls when looking up alias in chains This code was added as temporary solution until libvirt is fixed, but I think we would like keep a simplified version of it even after libvirt is fixed, verifying that the operation was successful. Change-Id: I9fec5416a62736bad461ddd0b54093d23960b7a6 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 27 insertions(+), 24 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/38/39938/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index efadbdb..8ece47b 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -5100,40 +5100,43 @@ # synchronized and we may start the vm with a stale volume in the # future. See https://bugzilla.redhat.com/show_bug.cgi?id=1202719 for # more details. - # TODO: Remove once we depend on a libvirt with this bug fixed. # We expect libvirt to show that the original leaf has been removed # from the active volume chain. origVols = sorted([x['volumeID'] for x in self.drive.volumeChain]) - expectedVols = origVols[:] - expectedVols.remove(self.drive.volumeID) + expectedVols = [v for v in origVols if v != self.driveVolumeID] alias = self.drive['alias'] self.vm.log.info("Waiting for libvirt to update the XML after pivot " "of drive %s completed", alias) - while True: - # This operation should complete in either one or two iterations of - # this loop. Until libvirt updates the XML there is nothing to do - # but wait. While we wait we continue to tell engine that the job - # is ongoing. If we are still in this loop when the VM is powered - # off, the merge will be resolved manually by engine using the - # reconcileVolumeChain verb. - chains = self.vm._driveGetActualVolumeChain([self.drive]) - if alias not in chains.keys(): - raise RuntimeError("Failed to retrieve volume chain for " - "drive %s. Pivot failed.", alias) - curVols = sorted([entry.uuid for entry in chains[alias]]) - if curVols == origVols: - time.sleep(1) - elif curVols == expectedVols: + # This operation should complete in either one or two iterations of + # this loop. Until libvirt updates the XML there is nothing to do + # but wait. While we wait we continue to tell engine that the job + # is ongoing. If we are still in this loop when the VM is powered + # off, the merge will be resolved manually by engine using the + # reconcileVolumeChain verb. + # TODO: Check once when we depend on a libvirt with this bug fixed. + + while True: + chains = self.vm._driveGetActualVolumeChain([self.drive]) + if alias not in chains: + raise RuntimeError("Failed to retrieve volume chain for " + "drive %s after pivot completed", alias) + + curVols = sorted(entry.uuid for entry in chains[alias]) + + if curVols == expectedVols: self.vm.log.info("The XML update has been completed") - break - else: - self.log.error("Bad volume chain found for drive %s. Previous " - "chain: %s, Expected chain: %s, Actual chain: " - "%s", alias, origVols, expectedVols, curVols) - raise RuntimeError("Bad volume chain found") + return + + if curVols != origVols: + raise RuntimeError( + "Bad volume chain after pivot for drive %s. Previous " + "chain: %s, Expected chain: %s, Actual chain: %s" % + (alias, origVols, expectedVols, curVols)) + + time.sleep(1) def _devicesWithAlias(domXML): -- To view, visit https://gerrit.ovirt.org/39938 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9fec5416a62736bad461ddd0b54093d23960b7a6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 7

Change in vdsm[master]: fc-scan: Use utilities from vdsm library.
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: fc-scan: Use utilities from vdsm library. ...................................................................... fc-scan: Use utilities from vdsm library. Replace low level threading code with simpler concurrent.tmap() call and duplicate monotonic_time() with utils.monotonic_time(). Change-Id: Ic48748d6a43d41e034e16cb4f636ebe627881590 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/fc-scan 1 file changed, 24 insertions(+), 46 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/66/38466/1 diff --git a/vdsm/storage/fc-scan b/vdsm/storage/fc-scan index 344345d..c746ea4 100755 --- a/vdsm/storage/fc-scan +++ b/vdsm/storage/fc-scan @@ -38,43 +38,11 @@ import logging import os import sys -import threading + +from vdsm import concurrent +from vdsm import utils log = logging.getLogger("fc-scan") - - -class Scan(object): - - def __init__(self, host): - self.host = host - self.succeeded = False - self.thread = None - - def start(self): - self.thread = threading.Thread(target=self.run) - self.thread.daemon = True - self.thread.start() - - def wait(self): - self.thread.join() - - def run(self): - try: - path = "/sys/class/scsi_host/%s/scan" % self.host - log.debug("Scanning %s", path) - start = monotonic_time() - fd = os.open(path, os.O_WRONLY) - try: - os.write(fd, "- - -") - finally: - os.close(fd) - self.succeeded = True - elapsed = monotonic_time() - start - log.debug("Scanned %s in %.2f seconds", path, elapsed) - except OSError as e: - log.error("Scanning %s failed: %s", path, e) - except Exception: - log.exception("Scanning %s failed", path) def main(args): @@ -93,22 +61,32 @@ log.debug("No fc_host found") return 0 - scans = [] - - for host in hosts: - s = Scan(host) - s.start() - scans.append(s) - - for s in scans: - s.wait() + scans = concurrent.tmap(scan_host, hosts) if not all(s.succeeded for s in scans): return 1 + return 0 -def monotonic_time(): - return os.times()[4] + +def scan_host(name): + try: + path = "/sys/class/scsi_host/%s/scan" % name + log.debug("Scanning %s", path) + start = utils.monotonic_time() + fd = os.open(path, os.O_WRONLY) + try: + os.write(fd, "- - -") + finally: + os.close(fd) + elapsed = utils.monotonic_time() - start + log.debug("Scanned %s in %.2f seconds", path, elapsed) + except OSError as e: + log.error("Scanning %s failed: %s", path, e) + raise + except Exception: + log.exception("Scanning %s failed", path) + raise if __name__ == '__main__': -- To view, visit https://gerrit.ovirt.org/38466 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic48748d6a43d41e034e16cb4f636ebe627881590 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

8 24

Change in vdsm[master]: lib: Revert and refine error handling in tmap()
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: lib: Revert and refine error handling in tmap() ...................................................................... lib: Revert and refine error handling in tmap() In commit 2b7155b696 (lib: Simplify and generalize concurrent.tmap()), we simplified error handling by returning a named tuple with function results. This turned out less useful then the original error handling. This patch returns the previous error handling: - Functions passed to tmap() should not raise - if they raise, this is considered a bug in the function. - The last error is raised by tmap() instead of returning the result. This make it easier to fail loudly for unexpected errors. - The original exception is re-raised now with the original traceback. - Error handling is documented properly now Previously you had to make sure function raises to signal failures: def func(): try: code that should not fail... code that may fail... code that should not fail... except ExpectedError: log.error(...) raise except Exception: log.exception(...) raise results = concurrent.tmap(func, values) if not all(r.succeeded for r in results): ... Returning the result as is lets us have nicer code: def func(): code that should not fail... try: code that may fail... except ExpectedError: log.error(...) return False code that should not fail... return True succeeded = concurrent.tmap(func, values) if not all(succeeded): ... We can ignore unexpected errors, since tmap() will log them and fail loudly. We can also minimize try except block for expected errors. Change-Id: I0154b28ff7822c63e77181bbbf444c712bd0c31e Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/concurrent.py M tests/concurrentTests.py 2 files changed, 45 insertions(+), 19 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/39211/1 diff --git a/lib/vdsm/concurrent.py b/lib/vdsm/concurrent.py index 64e072d..5498052 100644 --- a/lib/vdsm/concurrent.py +++ b/lib/vdsm/concurrent.py @@ -18,22 +18,42 @@ # Refer to the README and COPYING files for full details of the license # +import logging import threading -from collections import namedtuple - - -Result = namedtuple("Result", ["succeeded", "value"]) +import sys def tmap(func, iterable): + """ + Run func with arguments from iterable in multiple threads, a returning the + output in order of arguments. + + func should not raise exceptions - we consider this a bug in func, and will + fail the call and re-raise the exception in the caller thread. + + Expected exceptions should be handled in func. If the caller likes to + handle the error later, func should return it: + + def func(value): + try: + return something(value) + except ExpectedError as e: + return e + + Unexpected exceptions should not be handled, as they are logged in the + worker threads and re-raised in the caller thread. If multiple excpetions + raised, only the last one will be re-raised in the caller thread. + """ args = list(iterable) results = [None] * len(args) + error = [None] def worker(i, f, arg): try: - results[i] = Result(True, f(arg)) - except Exception as e: - results[i] = Result(False, e) + results[i] = f(arg) + except Exception: + error[0] = sys.exc_info() + logging.exception("Unhandled exception in tmap worker thread") threads = [] for i, arg in enumerate(args): @@ -45,4 +65,8 @@ for t in threads: t.join() + if error[0] is not None: + t, v, tb = error[0] + raise t, v, tb + return results diff --git a/tests/concurrentTests.py b/tests/concurrentTests.py index 307e397..5c0646b 100644 --- a/tests/concurrentTests.py +++ b/tests/concurrentTests.py @@ -26,13 +26,16 @@ from vdsm import concurrent +class Error(Exception): + pass + + class TMapTests(VdsmTestCase): def test_results(self): values = tuple(range(10)) results = concurrent.tmap(lambda x: x, values) - expected = [concurrent.Result(True, x) for x in values] - self.assertEqual(results, expected) + self.assertEqual(results, list(values)) def test_results_order(self): def func(x): @@ -40,8 +43,7 @@ return x values = tuple(random.random() * 0.1 for x in range(10)) results = concurrent.tmap(func, values) - expected = [concurrent.Result(True, x) for x in values] - self.assertEqual(results, expected) + self.assertEqual(results, list(values)) def test_concurrency(self): start = time.time() @@ -49,12 +51,12 @@ elapsed = time.time() - start self.assertTrue(0.1 < elapsed < 0.2) - def test_error(self): - error = RuntimeError("No result for you!") - + def test_raise_last_error(self): def func(x): - raise error - - results = concurrent.tmap(func, range(10)) - expected = [concurrent.Result(False, error)] * 10 - self.assertEqual(results, expected) + raise Error(x) + try: + concurrent.tmap(func, (1, 2, 3)) + except Error as e: + self.assertEqual(e.args, (3,)) + else: + self.fail("Exception was not raised") -- To view, visit https://gerrit.ovirt.org/39211 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0154b28ff7822c63e77181bbbf444c712bd0c31e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 10

Change in vdsm[master]: virt: Use Drive.diskType instead of networkDev and blockDev
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: virt: Use Drive.diskType instead of networkDev and blockDev ...................................................................... virt: Use Drive.diskType instead of networkDev and blockDev Now that we have explicit diskType we don't need to use the networkDev and blockDev properties. This is very useful when we set libvirt disk type property, or want to check for certain disk type. Change-Id: Id68bc74b3d788dc82fc61bf8c3de5a52164d0989 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/vmStorageTests.py M vdsm/virt/vm.py 2 files changed, 21 insertions(+), 28 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/40472/1 diff --git a/tests/vmStorageTests.py b/tests/vmStorageTests.py index 8e8f7bc..dfe0991 100644 --- a/tests/vmStorageTests.py +++ b/tests/vmStorageTests.py @@ -268,74 +268,69 @@ def test_cdrom(self): conf = drive_config(device='cdrom') drive = Drive({}, self.log, **conf) - self.assertFalse(drive.networkDev) - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.FILE) def test_floppy(self): conf = drive_config(device='floppy') drive = Drive({}, self.log, **conf) - self.assertFalse(drive.networkDev) - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.FILE) def test_network_disk(self): conf = drive_config(diskType=DISK_TYPE.NETWORK) drive = Drive({}, self.log, **conf) - self.assertTrue(drive.networkDev) - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.NETWORK) @MonkeyPatch(utils, 'isBlockDevice', lambda path: True) def test_block_disk(self): conf = drive_config(device='disk') drive = Drive({}, self.log, **conf) - self.assertFalse(drive.networkDev) - self.assertTrue(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.BLOCK) @MonkeyPatch(utils, 'isBlockDevice', lambda path: False) def test_file_disk(self): conf = drive_config(device='disk') drive = Drive({}, self.log, **conf) - self.assertFalse(drive.networkDev) - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.FILE) @MonkeyPatch(utils, 'isBlockDevice', lambda path: False) def test_migrate_from_file_to_block(self): conf = drive_config(path='/filedomain/volume') drive = Drive({}, self.log, **conf) - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.FILE) # Migrate drive to block domain... utils.isBlockDevice = lambda path: True drive.path = "/blockdomain/volume" - self.assertTrue(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.BLOCK) @MonkeyPatch(utils, 'isBlockDevice', lambda path: True) def test_migrate_from_block_to_file(self): conf = drive_config(path='/blockdomain/volume') drive = Drive({}, self.log, **conf) - self.assertTrue(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.BLOCK) # Migrate drive to file domain... utils.isBlockDevice = lambda path: False drive.path = "/filedomain/volume" - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.FILE) @MonkeyPatch(utils, 'isBlockDevice', lambda path: True) def test_migrate_from_block_to_network(self): conf = drive_config(path='/blockdomain/volume') drive = Drive({}, self.log, **conf) - self.assertTrue(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.BLOCK) # Migrate drive to network disk... drive.path = "pool/volume" drive.diskType = DISK_TYPE.NETWORK - self.assertFalse(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.NETWORK) @MonkeyPatch(utils, 'isBlockDevice', lambda path: True) def test_migrate_network_to_block(self): conf = drive_config(diskType=DISK_TYPE.NETWORK, path='pool/volume') drive = Drive({}, self.log, **conf) - self.assertTrue(drive.networkDev) + self.assertEqual(drive.diskType, DISK_TYPE.NETWORK) # Migrate drive to block domain... drive.path = '/blockdomain/volume' drive.diskType = None - self.assertTrue(drive.blockDev) + self.assertEqual(drive.diskType, DISK_TYPE.BLOCK) @expandPermutations diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 78b55c9..a6e742e 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1862,7 +1862,7 @@ def _changeDisk(self, diskDeviceXmlElement): diskType = diskDeviceXmlElement.getAttribute('type') - if diskType not in ['file', 'block']: + if diskType not in (DISK_TYPE.BLOCK, DISK_TYPE.FILE): return diskSerial = diskDeviceXmlElement. \ @@ -1871,13 +1871,12 @@ for vmDrive in self._devices[hwclass.DISK]: if vmDrive.serial == diskSerial: # update the type - diskDeviceXmlElement.setAttribute( - 'type', 'block' if vmDrive.blockDev else 'file') + diskDeviceXmlElement.setAttribute('type', vmDrive.diskType) # update the path + attr = 'dev' if vmDrive.diskType == DISK_TYPE.BLOCK else 'file' diskDeviceXmlElement.getElementsByTagName('source')[0]. \ - setAttribute('dev' if vmDrive.blockDev else 'file', - vmDrive.path) + setAttribute(attr, vmDrive.path) # update the format (the disk might have been collapsed) diskDeviceXmlElement.getElementsByTagName('driver')[0]. \ @@ -2773,7 +2772,7 @@ # we specify type='block' and dev=path for block volumes but we # always speficy the file=path for backwards compatibility. args = {'type': sourceType, 'file': newPath} - if sourceType == 'block': + if sourceType == DISK_TYPE.BLOCK: args['dev'] = newPath disk.appendChildWithArgs('source', **args) return disk @@ -2881,7 +2880,7 @@ newDrives[vmDevName]["format"] = "cow" # We need to keep track of the drive object because we cannot - # safely access the blockDev property until after prepareVolumePath + # safely access the diskType property until after prepareVolumePath vmDrives[vmDevName] = vmDrive # If all the drives are the current ones, return success @@ -2905,9 +2904,8 @@ _rollbackDrives(preparedDrives) return errCode['snapshotErr'] - snapType = 'block' if vmDrives[vmDevName].blockDev else 'file' snapelem = _diskSnapshot(vmDevName, newDrives[vmDevName]["path"], - snapType) + vmDrives[vmDevName].diskType) disks.appendChild(snapelem) snap.appendChild(disks) @@ -4553,7 +4551,7 @@ sourceXML = find_element_by_name(diskXML, 'source') if not sourceXML: break - sourceAttr = ('file', 'dev')[drive.blockDev] + sourceAttr = 'dev' if drive.diskType == DISK_TYPE.BLOCK else 'file' path = sourceXML.getAttribute(sourceAttr) # TODO: Allocation information is not available in the XML. Switch -- To view, visit https://gerrit.ovirt.org/40472 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id68bc74b3d788dc82fc61bf8c3de5a52164d0989 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 11

Change in vdsm[master]: misc: Safer and simpler itmap
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: misc: Safer and simpler itmap ...................................................................... misc: Safer and simpler itmap The previous code had few issues: - It used unlimited number of threads by default. This may lead to creation of 100's of threads if you do not specify a value. - It used non-daemon threads, which could lead to unwanted delay during vdsm shutdown. - It tried to yield results before all arguments were handled. This could lead to unwanted delay in argument processing, if the caller would block processing the results. - It started one thread per value, even if maxthreads was smaller than number of values. - It was too complicated. Changes: - The caller must specify the maximum number of threads. - Use daemon threads - Queue all values before yielding results - Start up to maxthreads worker threads, each processing multiple values - Simplify the code - Add test for error handling Change-Id: Iba6116ac4003702c8e921cebaf494491a6f9afaf Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/miscTests.py M vdsm/storage/misc.py 2 files changed, 42 insertions(+), 42 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/39119/1 diff --git a/tests/miscTests.py b/tests/miscTests.py index 31f64fa..4b3e3c3 100644 --- a/tests/miscTests.py +++ b/tests/miscTests.py @@ -196,7 +196,7 @@ # outOfProcess operation + 1. it let us know that oop and itmap operate # properly with their limitations data = frozenset(range(oop.HELPERS_PER_DOMAIN + 1)) - ret = frozenset(misc.itmap(dummy, data, misc.UNLIMITED_THREADS)) + ret = frozenset(misc.itmap(dummy, data, len(data))) self.assertEquals(ret, data) def testMoreThreadsThanArgs(self): @@ -207,6 +207,13 @@ data = 1 self.assertRaises(ValueError, misc.itmap(int, data, 0).next) + def testErrors(self): + err = Exception() + def dummy(arg): + raise err + data = [1, 2, 3] + self.assertEqual(list(misc.itmap(dummy, data, 4)), [err] * len(data)) + class RotateFiles(TestCaseBase): diff --git a/vdsm/storage/misc.py b/vdsm/storage/misc.py index eb484c7..463fd04 100644 --- a/vdsm/storage/misc.py +++ b/vdsm/storage/misc.py @@ -58,7 +58,6 @@ STR_UUID_SIZE = 36 UUID_HYPHENS = [8, 13, 18, 23] MEGA = 1 << 20 -UNLIMITED_THREADS = -1 log = logging.getLogger('Storage.Misc') @@ -882,53 +881,47 @@ raise exception -def itmap(func, iterable, maxthreads=UNLIMITED_THREADS): +def itmap(func, iterable, maxthreads): """ - Make an iterator that computes the function using - arguments from the iterable. It works similar to tmap - by running each operation in a different thread, this - causes the results not to return in any particular - order so it's good if you don't care about the order - of the results. - maxthreads stands for maximum threads that we can initiate simultaneosly. - If we reached to max threads the function waits for thread to - finish before initiate the next one. + Return an iterator calling func with arguments from iterable in multiple threads. + + Unlike tmap, the results are not returned in the original order of the + arguments, and number of threads is limited to maxthreads. """ - if maxthreads < 1 and maxthreads != UNLIMITED_THREADS: - raise ValueError("Wrong input to function itmap: %s", maxthreads) + if maxthreads < 1: + raise ValueError("Invalid maxthreads value: %s" % maxthreads) - respQueue = Queue.Queue() + DONE = object() + values = Queue.Queue() + results = Queue.Queue() - def wrapper(value): - try: - respQueue.put(func(value)) - except Exception as e: - respQueue.put(e) + def worker(): + while True: + value = values.get() + if value is DONE: + return + try: + results.put(func(value)) + except Exception as e: + results.put(e) - threadsCount = 0 - for arg in iterable: - if maxthreads != UNLIMITED_THREADS: - if maxthreads == 0: - # This not supposed to happened. If it does, it's a bug. - # maxthreads should get to 0 only after threadsCount is - # greater than 1 - if threadsCount < 1: - raise RuntimeError("No thread initiated") - else: - yield respQueue.get() - # if yield returns one thread stopped, so we can run - # another thread in queue - maxthreads += 1 - threadsCount -= 1 + count = 0 + threads = 0 - t = threading.Thread(target=wrapper, args=(arg,)) - t.start() - threadsCount += 1 - maxthreads -= 1 + for value in iterable: + values.put(value) + count += 1 + if threads < maxthreads: + t = threading.Thread(target=worker) + t.daemon = True + t.start() + threads += 1 - # waiting for rest threads to end - for i in xrange(threadsCount): - yield respQueue.get() + for _ in range(threads): + values.put(DONE) + + for _ in xrange(count): + yield results.get() def isAscii(s): -- To view, visit https://gerrit.ovirt.org/39119 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iba6116ac4003702c8e921cebaf494491a6f9afaf Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 38

Change in vdsm[master]: udevadm: More precise error handling
by Nir Soffer 24 Jul '16

24 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: udevadm: More precise error handling ...................................................................... udevadm: More precise error handling udevadm provides a --timeout option, but there is no robust way to detect a timeout in EL6, EL7, and Fedora 20. In Fedora 21 and upstream, udevadm ignores the timeout option. This patch improves error handling by using our own timeout. udevadm.settle() raises now udevadm.Failure or udevadm.Timeout, and the caller is responsible to handle the error. In both multipath.rescan() and IscsiConnection.connect(), we warn about timeout but do not handle other errors, so real errors in udevadm will fail loudly. Change-Id: Ia0a7380b1b181ec93399ea741122cfa2e98086fb Relates-To: https://bugzilla.redhat.com/1209474 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A tests/udevadmTests.py M vdsm/storage/multipath.py M vdsm/storage/storageServer.py M vdsm/storage/udevadm.py 4 files changed, 106 insertions(+), 21 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/40/39740/1 diff --git a/tests/udevadmTests.py b/tests/udevadmTests.py new file mode 100644 index 0000000..90841b2 --- /dev/null +++ b/tests/udevadmTests.py @@ -0,0 +1,52 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + + +from monkeypatch import MonkeyPatch +from testlib import VdsmTestCase + +from vdsm import utils +from storage import udevadm + +TRUE = utils.CommandPath("true", "/bin/true", "/usr/bin/true") +FALSE = utils.CommandPath("false", "/bin/false", "/usr/bin/false") +READ = utils.CommandPath("read", "/bin/read", "/usr/bin/read") + + +class UdevadmSettleTests(VdsmTestCase): + + @MonkeyPatch(udevadm, "_UDEVADM", TRUE) + def test_success(self): + udevadm.settle(5) + + @MonkeyPatch(udevadm, "_UDEVADM", FALSE) + def test_error(self): + try: + udevadm.settle(5) + except udevadm.Failure as e: + self.assertEqual(e.rc, 1) + self.assertEqual(e.out, "") + self.assertEqual(e.err, "") + else: + self.fail("Failure not raised") + + @MonkeyPatch(udevadm, "_UDEVADM", READ) + def test_timeout(self): + self.assertRaises(udevadm.Timeout, udevadm.settle, 1) diff --git a/vdsm/storage/multipath.py b/vdsm/storage/multipath.py index a1c42b3..925c411 100644 --- a/vdsm/storage/multipath.py +++ b/vdsm/storage/multipath.py @@ -73,7 +73,10 @@ # events are processed, ensuring detection of new devices and creation or # update of multipath devices. timeout = config.getint('irs', 'scsi_settle_timeout') - udevadm.settle(timeout) + try: + udevadm.settle(timeout) + except udevadm.Timeout as e: + log.warning("Timeout waiting for udev events: %s", e) def deduceType(a, b): diff --git a/vdsm/storage/storageServer.py b/vdsm/storage/storageServer.py index 22a90d1..c19fb8d 100644 --- a/vdsm/storage/storageServer.py +++ b/vdsm/storage/storageServer.py @@ -382,7 +382,10 @@ def connect(self): iscsi.addIscsiNode(self._iface, self._target, self._cred) timeout = config.getint("irs", "scsi_settle_timeout") - udevadm.settle(timeout) + try: + udevadm.settle(timeout) + except udevadm.Timeout as e: + self.log.warning("Timeout waiting for udev events: %s", e) def _match(self, session): target = session.target diff --git a/vdsm/storage/udevadm.py b/vdsm/storage/udevadm.py index 4b4b54a..a2afd04 100644 --- a/vdsm/storage/udevadm.py +++ b/vdsm/storage/udevadm.py @@ -18,22 +18,39 @@ # Refer to the README and COPYING files for full details of the license # -import logging +import errno +import signal + from vdsm import utils +from vdsm.infra import zombiereaper _UDEVADM = utils.CommandPath("udevadm", "/sbin/udevadm", "/usr/sbin/udevadm") class Error(Exception): + message = None - def __init__(self, rc, out, err): + def __str__(self): + return self.message.format(self=self) + + +class Failure(Error): + message = ("udevadm failed cmd={self.cmd} rc={self.rc} out={self.out!r} " + "err={self.err!r}") + + def __init__(self, cmd, rc, out, err): + self.cmd = cmd self.rc = rc self.out = out self.err = err - def __str__(self): - return "Process failed with rc=%d out=%r err=%r" % ( - self.rc, self.out, self.err) + +class Timeout(Error): + message = ("udevadm timed out cmd={self.cmd} timeout={self.timeout}") + + def __init__(self, cmd, timeout): + self.cmd = cmd + self.timeout = timeout def settle(timeout, exit_if_exists=None): @@ -44,25 +61,35 @@ Arguments: timeout Maximum number of seconds to wait for the event queue to - become empty. A value of 0 will check if the queue is empty - and always return immediately. + become empty. exit_if_exists Stop waiting if file exists. + + Raises Failure if udevadm failed, or Timeout if udevadm did not terminate + within the requested timeout. """ - args = ["settle", "--timeout=%s" % timeout] + cmd = [_UDEVADM.cmd, "settle"] if exit_if_exists: - args.append("--exit-if-exists=%s" % exit_if_exists) + cmd.append("--exit-if-exists=%s" % exit_if_exists) - try: - _run_command(args) - except Error as e: - logging.error("%s", e) + _run_command(cmd, timeout) -def _run_command(args): - cmd = [_UDEVADM.cmd] - cmd.extend(args) - rc, out, err = utils.execCmd(cmd, raw=True) - if rc != 0: - raise Error(rc, out, err) +def _run_command(cmd, timeout=None): + proc = utils.execCmd(cmd, sync=False, deathSignal=signal.SIGKILL) + + if not proc.wait(timeout): + try: + proc.kill() + except OSError as e: + if e.errno != errno.ESRCH: + raise + finally: + zombiereaper.autoReapPID(proc.pid) + raise Timeout(cmd, timeout) + + if proc.returncode != 0: + out = "".join(proc.stdout) + err = "".join(proc.stderr) + raise Failure(cmd, proc.returncode, out, err) -- To view, visit https://gerrit.ovirt.org/39740 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia0a7380b1b181ec93399ea741122cfa2e98086fb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 10

Change in vdsm[master]: monitor: Add udev monitor
by rmohr＠redhat.com 24 Jul '16

24 Jul '16

Roman Mohr has uploaded a new change for review. Change subject: monitor: Add udev monitor ...................................................................... monitor: Add udev monitor Allow easy monitoring of device hotplug and device state changes which are covered by udev. Change-Id: I4b91753424d83896fa538eb6b57f8653b6332fbb Signed-off-by: Roman Mohr <rmohr(a)redhat.com> --- A lib/vdsm/udev/monitor.py M tests/Makefile.am A tests/udevMonitorTests.py M vdsm.spec.in 4 files changed, 272 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/29/47729/1 diff --git a/lib/vdsm/udev/monitor.py b/lib/vdsm/udev/monitor.py new file mode 100644 index 0000000..8698bf3 --- /dev/null +++ b/lib/vdsm/udev/monitor.py @@ -0,0 +1,133 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +from functools import partial +import logging +import threading +from time import sleep + +from pyudev import Context, Monitor, MonitorObserver + + +class UdevMonitor(object): + + """ + ``udev`` event monitor. Usage: + + The monitor is a thread-save thin wrapper arount pyudev.MonitorObserver. + This allows multiple callbacks for the same netlink socket. To avoid + listening for udev events the application is not interested in, the + monitoring thread only starts listening on the socket when the monitor is + started and at least one subscriber is added. + + The simplest way to use the monitor is to subscribe a callback to a + specific subsystem event and let the callback do the work: + + dev listen_for_disabled_cpu_events(device): + if device.action == 'offline': + print('CPU {0.name} is now offline'.format(device)) + + monitor = UdevMonitor() + monitor.start() + monitor.subscribe(listen_for_disabled_cpu_events, subsystem='cpu') + + Another approach would be to just enqueue the udev event in a queue and do + the actual work in another thread or a thread pool. This is one approach to + do this: + + queue = Queue.Queue() + def new_device_listener(device): + if device.action == 'add': + queue.put(device) + + monitor = UdevMonitor() + monitor.subscribe(new_device_listener, subsystem='usb', + device_type='usb_device') + monitor.start() + """ + + def __init__(self): + self._subsystems = {} + self._context = Context() + self._monitor = Monitor.from_netlink(self._context) + self._observer = MonitorObserver(self._monitor, callback=partial( + UdevMonitor._event_loop, self), name='udev-monitor') + self._filter_lock = threading.Lock() + self._is_started = False + self._can_start = False + + def _event_loop(self, device): + subsystem = self._subsystems[device.subsystem] + for callback in subsystem.get(device.device_type, []): + _execute_callback(callback, device) + if device.device_type: + for callback in subsystem.get(None, []): + _execute_callback(callback, device) + + def subscribe(self, callback, subsystem, device_type=None): + """ + Raise :exc:`~exceptions.ValueError` if the callback is None + + :param callback: function to invoke + :param subsystem: byte or unicode string representing the subsystem to + listen on (e.g. ``'cpu'``, ``'usb'``) + :param device_type: byte or unicode string representing the device type + to listen for changes (e.g. ``'usb_device'``, + ``'block'``) + :return: None + """ + if callback is None: + raise ValueError('callback missing') + with self._filter_lock: + self._monitor.filter_by(subsystem, device_type) + device_types = self._subsystems.get(subsystem, {}) + callbacks = device_types.get(device_type, []) + callbacks.append(callback) + device_types[device_type] = callbacks + self._subsystems[subsystem] = device_types + self._start_if_necessary() + + def start(self): + self._can_start = True + with self._filter_lock: + if self._subsystems: + self._start_if_necessary() + + def _start_if_necessary(self): + if self._can_start and not self._is_started: + self._observer.start() + self._is_started = True + + def stop(self): + """ + Stops the monitoring thread. It is guaranteed that callbacks are no + longer invoked after calling this method. The method can be called + multiple times. + + Note that this only stops the monitoring thread and not the monitor + itself. The monitor stops listening when it is dereferenced. + """ + self._observer.stop() + + +def _execute_callback(callback, device): + try: + callback(device) + except Exception as callbackException: + logging.error( + 'Callback execution threw an exception: %s', callbackException) diff --git a/tests/Makefile.am b/tests/Makefile.am index 1168862..7483d0c 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -110,6 +110,7 @@ testlibTests.py \ toolTests.py \ transportWrapperTests.py \ + udevMonitorTests.py \ utilsTests.py \ vdscliTests.py \ vdsClientTests.py \ diff --git a/tests/udevMonitorTests.py b/tests/udevMonitorTests.py new file mode 100644 index 0000000..2526459 --- /dev/null +++ b/tests/udevMonitorTests.py @@ -0,0 +1,137 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from functools import partial + +import Queue + +from nettestlib import Bridge +from testValidation import ValidateRunningAsRoot +from testlib import VdsmTestCase +from vdsm.udev.monitor import UdevMonitor + + +class UdevMonitorTest(VdsmTestCase): + + def setUp(self): + self._queue = Queue.Queue() + self._monitor = UdevMonitor() + self._monitor.start() + + def tearDown(self): + self._monitor.stop() + + @ValidateRunningAsRoot + def testAddDevice(self): + self._monitor.subscribe( + partial(UdevMonitorTest._device_listener, self), + subsystem='net', + device_type='bridge') + bridge = Bridge() + try: + bridge.addDevice() + device = None + try: + device = self._queue.get(timeout=1) + except Queue.Empty: + pass + self.assertIsNotNone(device, + msg='Should have detected an event') + self.assertEqual(device.action, 'add') + finally: + bridge.delDevice() + + @ValidateRunningAsRoot + def testAddDeviceForSubsystem(self): + self._monitor.subscribe( + partial(UdevMonitorTest._device_listener, self), + subsystem='net', + device_type=None) + bridge = Bridge() + try: + bridge.addDevice() + device = None + try: + device = self._queue.get(timeout=1) + except Queue.Empty: + pass + self.assertIsNotNone(device, + msg='Should have detected an event') + self.assertEqual(device.action, 'add') + finally: + bridge.delDevice() + + @ValidateRunningAsRoot + def testRemoveDevice(self): + + bridge = Bridge() + try: + bridge.addDevice() + self._monitor.subscribe( + partial(UdevMonitorTest._device_listener, self), + subsystem='net', + device_type='bridge') + bridge.delDevice() + device = None + try: + device = self._queue.get(timeout=1) + except Queue.Empty: + pass + self.assertIsNotNone(device, + msg='Should have detected an event') + self.assertEqual(device.action, 'remove') + except: + bridge.delDevice() + + @ValidateRunningAsRoot + def testEventSeries(self): + + bridge1 = Bridge() + bridge2 = Bridge() + try: + self._monitor.subscribe( + partial(UdevMonitorTest._device_listener, self), + subsystem='net', + device_type='bridge') + bridge1.addDevice() + bridge1.delDevice() + bridge2.addDevice() + bridge2.delDevice() + events = [] + try: + events.append(self._queue.get(timeout=1)) + events.append(self._queue.get(timeout=1)) + events.append(self._queue.get(timeout=1)) + events.append(self._queue.get(timeout=1)) + except Queue.Empty: + pass + self.assertEqual(len(events), 4) + except: + _delBridge(bridge1) + _delBridge(bridge2) + + def _device_listener(self, device): + self._queue.put(device) + + +def _delBridge(bridge): + try: + bridge.delDevice() + except: + pass diff --git a/vdsm.spec.in b/vdsm.spec.in index 8f758ca..ae5e483 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -126,6 +126,7 @@ Requires: python-pthreading >= 0.1.3-3 Requires: python-six Requires: python-requests +Requires: python-udev >= 0.15 Requires: %{name}-infra = %{version}-%{release} Requires: rpm-python Requires: nfs-utils -- To view, visit https://gerrit.ovirt.org/47729 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4b91753424d83896fa538eb6b57f8653b6332fbb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Roman Mohr <rmohr(a)redhat.com>

4 13

Change in vdsm[master]: FileVolumeMetadata: split getMetadata
by alitke＠redhat.com 20 Jul '16

20 Jul '16

Adam Litke has uploaded a new change for review. Change subject: FileVolumeMetadata: split getMetadata ...................................................................... FileVolumeMetadata: split getMetadata Change-Id: I1d3fb61831de5b50a3e562b80bf38ef15ede254f Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/fileVolume.py 1 file changed, 16 insertions(+), 13 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/69/44569/1 diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 08e3f50..74aa911 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -123,18 +123,10 @@ """ return (self.getVolumePath(),) - def getMetadata(self, metaId=None): - """ - Get Meta data array of key,values lines - """ - if not metaId: - metaId = self.getMetadataId() - - volPath, = metaId - metaPath = self._getMetaVolumePath(volPath) - + @classmethod + def read_metadata(cls, oop, meta_path): try: - f = self.oop.directReadLines(metaPath) + f = oop.directReadLines(meta_path) out = {} for l in f: if l.startswith("EOF"): @@ -145,11 +137,22 @@ out[key.strip()] = value.strip() except Exception as e: - self.log.error(e, exc_info=True) - raise se.VolumeMetadataReadError("%s: %s" % (metaId, e)) + cls.log.error(e, exc_info=True) + raise se.VolumeMetadataReadError("%s: %s" % (meta_path, e)) return out + def getMetadata(self, metaId=None): + """ + Get Meta data array of key,values lines + """ + if not metaId: + metaId = self.getMetadataId() + + volPath, = metaId + metaPath = self._getMetaVolumePath(volPath) + return self.read_metadata(self.oop, metaPath) + def getParentId(self): """ Return parent volume UUID -- To view, visit https://gerrit.ovirt.org/44569 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1d3fb61831de5b50a3e562b80bf38ef15ede254f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 6

Change in vdsm[master]: VolumeMetadata: move prepare and teardown
by alitke＠redhat.com 20 Jul '16

20 Jul '16

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move prepare and teardown ...................................................................... VolumeMetadata: move prepare and teardown Change-Id: Iba0954ace3b1da5ea9afc41aeb6ea69a729fe29c Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 117 insertions(+), 109 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/50/44050/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 0168d64..597e7ee 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -74,6 +74,8 @@ volume.VolumeMetadata.__init__(self, repoPath, sdUUID, imgUUID, volUUID) self.metaoff = None + self.lvmActivationNamespace = sd.getNamespace(self.sdUUID, + LVM_ACTIVATION_NAMESPACE) def getMetadataId(self): """ @@ -332,6 +334,49 @@ lvs = lvm.lvsByTag(sdUUID, "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)) return [lv.name for lv in lvs] + @logskip("ResourceManager") + def llPrepare(self, rw=False, setrw=False): + """ + Perform low level volume use preparation + + For the Block Volumes the actual LV activation is wrapped + into lvmActivation resource. It is being initialized by the + storage domain sitting on top of the encapsulating VG. + We just use it here. + """ + if setrw: + self.setrw(rw=rw) + access = rm.LockType.exclusive if rw else rm.LockType.shared + activation = rmanager.acquireResource(self.lvmActivationNamespace, + self.volUUID, access) + activation.autoRelease = False + + @classmethod + def teardown(cls, sdUUID, volUUID, justme=False): + """ + Deactivate volume and release resources. + Volume deactivation occurs as part of resource releasing. + If justme is false, the entire COW chain should be torn down. + """ + cls.log.info("Tearing down volume %s/%s justme %s" + % (sdUUID, volUUID, justme)) + lvmActivationNamespace = sd.getNamespace(sdUUID, + LVM_ACTIVATION_NAMESPACE) + rmanager.releaseResource(lvmActivationNamespace, volUUID) + if not justme: + try: + pvolUUID = _getVolumeTag(sdUUID, volUUID, TAG_PREFIX_PARENT) + except Exception as e: + # If storage not accessible or lvm error occurred + # we will failure to get the parent volume. + # We can live with it and still succeed in volume's teardown. + pvolUUID = volume.BLANK_UUID + cls.log.warn("Failure to get parent of volume %s/%s (%s)" + % (sdUUID, volUUID, e)) + + if pvolUUID != volume.BLANK_UUID: + cls.teardown(sdUUID=sdUUID, volUUID=pvolUUID, justme=False) + class BlockVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -341,8 +386,6 @@ def __init__(self, repoPath, sdUUID, imgUUID, volUUID): md = self.MetadataClass(repoPath, sdUUID, imgUUID, volUUID) volume.Volume.__init__(self, md) - self.lvmActivationNamespace = sd.getNamespace(self.sdUUID, - LVM_ACTIVATION_NAMESPACE) @property def metaoff(self): @@ -609,49 +652,6 @@ def shareVolumeRollback(cls, taskObj, volPath): cls.log.info("Volume rollback for volPath=%s", volPath) utils.rmFile(volPath) - - @logskip("ResourceManager") - def llPrepare(self, rw=False, setrw=False): - """ - Perform low level volume use preparation - - For the Block Volumes the actual LV activation is wrapped - into lvmActivation resource. It is being initialized by the - storage domain sitting on top of the encapsulating VG. - We just use it here. - """ - if setrw: - self.setrw(rw=rw) - access = rm.LockType.exclusive if rw else rm.LockType.shared - activation = rmanager.acquireResource(self.lvmActivationNamespace, - self.volUUID, access) - activation.autoRelease = False - - @classmethod - def teardown(cls, sdUUID, volUUID, justme=False): - """ - Deactivate volume and release resources. - Volume deactivation occurs as part of resource releasing. - If justme is false, the entire COW chain should be torn down. - """ - cls.log.info("Tearing down volume %s/%s justme %s" - % (sdUUID, volUUID, justme)) - lvmActivationNamespace = sd.getNamespace(sdUUID, - LVM_ACTIVATION_NAMESPACE) - rmanager.releaseResource(lvmActivationNamespace, volUUID) - if not justme: - try: - pvolUUID = _getVolumeTag(sdUUID, volUUID, TAG_PREFIX_PARENT) - except Exception as e: - # If storage not accessible or lvm error occurred - # we will failure to get the parent volume. - # We can live with it and still succeed in volume's teardown. - pvolUUID = volume.BLANK_UUID - cls.log.warn("Failure to get parent of volume %s/%s (%s)" - % (sdUUID, volUUID, e)) - - if pvolUUID != volume.BLANK_UUID: - cls.teardown(sdUUID=sdUUID, volUUID=pvolUUID, justme=False) def getVolumeTag(self, tagPrefix): return self.md.getVolumeTag(tagPrefix) diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 2dad91f..08e3f50 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -337,6 +337,26 @@ volList.append(volid) return volList + def llPrepare(self, rw=False, setrw=False): + """ + Make volume accessible as readonly (internal) or readwrite (leaf) + """ + volPath = self.getVolumePath() + + # Volumes leaves created in 2.2 did not have group writeable bit + # set. We have to set it here if we want qemu-kvm to write to old + # NFS volumes. + self.oop.fileUtils.copyUserModeToGroup(volPath) + + if setrw: + self.setrw(rw=rw) + if rw: + if not self.oop.os.access(volPath, os.R_OK | os.W_OK): + raise se.VolumeAccessError(volPath) + else: + if not self.oop.os.access(volPath, os.R_OK): + raise se.VolumeAccessError(volPath) + class FileVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -488,26 +508,6 @@ procPool.utils.rmFile(volPath) procPool.utils.rmFile(cls.__metaVolumePath(volPath)) procPool.utils.rmFile(cls.MetadataClass._leaseVolumePath(volPath)) - - def llPrepare(self, rw=False, setrw=False): - """ - Make volume accessible as readonly (internal) or readwrite (leaf) - """ - volPath = self.getVolumePath() - - # Volumes leaves created in 2.2 did not have group writeable bit - # set. We have to set it here if we want qemu-kvm to write to old - # NFS volumes. - self.oop.fileUtils.copyUserModeToGroup(volPath) - - if setrw: - self.setrw(rw=rw) - if rw: - if not self.oop.os.access(volPath, os.R_OK | os.W_OK): - raise se.VolumeAccessError(volPath) - else: - if not self.oop.os.access(volPath, os.R_OK): - raise se.VolumeAccessError(volPath) @classmethod def __putMetadata(cls, metaId, meta): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index a098eb7..22f7ead 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -525,6 +525,56 @@ return apparentSize return req_size + def prepare(self, rw=True, justme=False, + chainrw=False, setrw=False, force=False): + """ + Prepare volume for use by consumer. + If justme is false, the entire COW chain is prepared. + Note: setrw arg may be used only by SPM flows. + """ + self.log.info("Volume: preparing volume %s/%s", + self.sdUUID, self.volUUID) + + if not force: + # Cannot prepare ILLEGAL volume + if not self.isLegal(): + raise se.prepareIllegalVolumeError(self.volUUID) + + if rw and self.isShared(): + if chainrw: + rw = False # Shared cannot be set RW + else: + raise se.SharedVolumeNonWritable(self) + + if (not chainrw and rw and self.isInternal() and setrw and + not self.recheckIfLeaf()): + raise se.InternalVolumeNonWritable(self) + + self.llPrepare(rw=rw, setrw=setrw) + self.updateInvalidatedSize() + + try: + if justme: + return True + pvol = self.produceParent() + if pvol: + pvol.prepare(rw=chainrw, justme=False, + chainrw=chainrw, setrw=setrw) + except Exception: + self.log.error("Unexpected error", exc_info=True) + self.teardown(self.sdUUID, self.volUUID) + raise + + return True + + @classmethod + def teardown(cls, sdUUID, volUUID, justme=False): + """ + Teardown volume. + If justme is false, the entire COW chain is teared down. + """ + pass + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -1091,53 +1141,11 @@ def prepare(self, rw=True, justme=False, chainrw=False, setrw=False, force=False): - """ - Prepare volume for use by consumer. - If justme is false, the entire COW chain is prepared. - Note: setrw arg may be used only by SPM flows. - """ - self.log.info("Volume: preparing volume %s/%s", - self.sdUUID, self.volUUID) - - if not force: - # Cannot prepare ILLEGAL volume - if not self.isLegal(): - raise se.prepareIllegalVolumeError(self.volUUID) - - if rw and self.isShared(): - if chainrw: - rw = False # Shared cannot be set RW - else: - raise se.SharedVolumeNonWritable(self) - - if (not chainrw and rw and self.isInternal() and setrw and - not self.recheckIfLeaf()): - raise se.InternalVolumeNonWritable(self) - - self.llPrepare(rw=rw, setrw=setrw) - self.updateInvalidatedSize() - - try: - if justme: - return True - pvol = self.produceParent() - if pvol: - pvol.prepare(rw=chainrw, justme=False, - chainrw=chainrw, setrw=setrw) - except Exception: - self.log.error("Unexpected error", exc_info=True) - self.teardown(self.sdUUID, self.volUUID) - raise - - return True + return self.md.prepare(rw, justme, chainrw, setrw, force) @classmethod def teardown(cls, sdUUID, volUUID, justme=False): - """ - Teardown volume. - If justme is false, the entire COW chain is teared down. - """ - pass + cls.MetadataClass.teardown(sdUUID, volUUID, justme) @classmethod def newMetadata(cls, metaId, sdUUID, imgUUID, puuid, size, format, type, -- To view, visit https://gerrit.ovirt.org/44050 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iba0954ace3b1da5ea9afc41aeb6ea69a729fe29c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 7

Change in vdsm[master]: sdc: Allow StorageDomainCache to cache StorageDomainManifest...
by alitke＠redhat.com 20 Jul '16

20 Jul '16

Adam Litke has uploaded a new change for review. Change subject: sdc: Allow StorageDomainCache to cache StorageDomainManifest objects ...................................................................... sdc: Allow StorageDomainCache to cache StorageDomainManifest objects When HSM is operating without SPM the system should always be using StorageDomainManifest objects instead of StorageDomains. With this patch we can instruct the cache to serve the correct type of object without changing all consumers of sdCache. Change-Id: I9a3dc7d9bf24f7d8b60ddff6f5364b65a9354e45 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/sdc.py 1 file changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/41/44041/1 diff --git a/vdsm/storage/sdc.py b/vdsm/storage/sdc.py index ecb9708..cbdaed9 100644 --- a/vdsm/storage/sdc.py +++ b/vdsm/storage/sdc.py @@ -64,12 +64,13 @@ STORAGE_STALE = 1 STORAGE_REFRESHING = 2 - def __init__(self, storage_repo): + def __init__(self, storage_repo, use_manifests=False): self._syncroot = threading.Condition() self.__domainCache = {} self.__inProgress = set() self.__staleStatus = self.STORAGE_STALE self.storage_repo = storage_repo + self.use_manifests = use_manifests self.knownSDs = {} # {sdUUID: mod.findDomain} def invalidateStorage(self): @@ -162,12 +163,18 @@ # this changes, please update the order. for mod in (blockSD, glusterSD, localFsSD, nfsSD): try: - return mod.findDomain(sdUUID) + if self.use_manifests: + ret = mod.findDomainManifest(sdUUID) + else: + ret = mod.findDomain(sdUUID) except se.StorageDomainDoesNotExist: pass except Exception: self.log.error("Error while looking for domain `%s`", sdUUID, exc_info=True) + else: + self.log.debug("Found domain %s", ret) + return ret raise se.StorageDomainDoesNotExist(sdUUID) @@ -181,10 +188,12 @@ return uuids - def refresh(self): + def refresh(self, use_manifests=None): with self._syncroot: lvm.invalidateCache() self.__domainCache.clear() + if use_manifests is not None: + self.use_manifests = use_manifests def manuallyAddDomain(self, domain): with self._syncroot: -- To view, visit https://gerrit.ovirt.org/44041 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9a3dc7d9bf24f7d8b60ddff6f5364b65a9354e45 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 7

Change in vdsm[master]: VolumeMetadata: Move getParentVolume
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: Move getParentVolume ...................................................................... VolumeMetadata: Move getParentVolume Move getParentVolume into VolumeMetadata. If HSM configured sdCache to return StorageDomainManifest objects then this will return VolumeMetadata objects. Otherwise it will continue to return Volume objects as it always has. Change-Id: I74def8861e0b7b97ca4d218a969440ccba7f07dd Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/volume.py 1 file changed, 11 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/42/44042/1 diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index c40ba77..f7e5bc0 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -485,6 +485,16 @@ """ pass # Do not remove this method or the V3 upgrade will fail. + def getParentVolume(self): + """ + Return parent volume object + """ + puuid = self.getParent() + if puuid and puuid != BLANK_UUID: + return sdCache.produce(self.sdUUID).produceVolume(self.imgUUID, + puuid) + return None + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -1128,11 +1138,7 @@ """ Return parent volume object """ - puuid = self.getParent() - if puuid and puuid != BLANK_UUID: - return sdCache.produce(self.sdUUID).produceVolume(self.imgUUID, - puuid) - return None + return self.md.getParentVolume() def setParent(self, puuid): """ -- To view, visit https://gerrit.ovirt.org/44042 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I74def8861e0b7b97ca4d218a969440ccba7f07dd Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 7

Change in vdsm[master]: storage: Make Image.__chainSizeCalc public
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: storage: Make Image.__chainSizeCalc public ...................................................................... storage: Make Image.__chainSizeCalc public The new SDM copyVolumeData wants to make use of the same logic being used by the classic copy flows to extend the size of the target volume to the appropriate size. Make Image.__chainSizeCalc public so it can be accessed from the SDM code. Change-Id: Id079eb5067c16f934370e42b5f4e09bbcef1512b Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/image.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/95/38995/1 diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 791b48c..8d5e8c2 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -140,7 +140,7 @@ randomStr = misc.randomStr(RENAME_RANDOM_STRING_LEN) return "%s%s_%s" % (sd.REMOVED_IMAGE_PREFIX, randomStr, uuid) - def __chainSizeCalc(self, sdUUID, imgUUID, volUUID, size): + def chainSizeCalc(self, sdUUID, imgUUID, volUUID, size): """ Compute an estimate of the whole chain size using the sum of the actual size of the chain's volumes @@ -763,7 +763,7 @@ if volParams['volFormat'] != volume.COW_FORMAT or \ volParams['prealloc'] != volume.SPARSE_VOL: raise se.IncorrectFormat(self) - volParams['apparentsize'] = self.__chainSizeCalc( + volParams['apparentsize'] = self.chainSizeCalc( sdUUID, srcImgUUID, srcVolUUID, volParams['size']) # Find out dest volume parameters -- To view, visit https://gerrit.ovirt.org/38995 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id079eb5067c16f934370e42b5f4e09bbcef1512b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

6 34

Change in vdsm[master]: SDM: Add extendVolumeContainer API
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: SDM: Add extendVolumeContainer API ...................................................................... SDM: Add extendVolumeContainer API The extendVolumeContainer API is used to extend LVM logical volumes which store thinly-provisioned vdsm volumes. Change-Id: I6a128ba3eab4116ff4e794e94a171e51d9e432de Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py M vdsm/rpc/vdsmapi-schema.json M vdsm/storage/hsm.py M vdsm/storage/sdm/__init__.py M vdsm/storage/sdm/blockstore.py M vdsm/storage/sdm/filestore.py 8 files changed, 123 insertions(+), 9 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/96/39696/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 23bfc1a..f6c0c83 100755 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1953,6 +1953,17 @@ else: return status['status']['code'], status['status']['message'] + def extendVolumeContainer(self, args): + if len(args) != 4: + raise ValueError("Wrong number of arguments") + + sdUUID, imgUUID, volUUID, size = args + status = self.s.extendVolumeContainer(sdUUID, imgUUID, volUUID, size) + if status['status']['code'] == 0: + return 0, '' + else: + return status['status']['code'], status['status']['message'] + if __name__ == '__main__': if _glusterEnabled: @@ -2844,6 +2855,11 @@ '<srcImage> <dstImage> <collapse>', 'Copy the date from one volume into another.' )), + 'extendVolumeContainer': ( + serv.extendVolumeContainer, ( + '<sdUUID> <imgUUID> <volUUID>', + 'Extend a thinly-provisioned block volume.' + )), } if _glusterEnabled: commands.update(ge.getGlusterCmdDict(serv)) diff --git a/vdsm/API.py b/vdsm/API.py index b4b7308..5661b82 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1675,6 +1675,10 @@ def copyData(self, srcImage, dstImage, collapse): return self._cif.irs.copyData(srcImage, dstImage, collapse) + def extendVolumeContainer(self, sdUUID, imgUUID, volUUID, size): + return self._cif.irs.extendVolumeContainer(sdUUID, imgUUID, volUUID, + size) + # take a rough estimate on how much free mem is available for new vm # memTotal = memFree + memCached + mem_used_by_non_qemu + resident . # simply returning (memFree + memCached) is not good enough, as the diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index 3e70e28..ae43614 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -993,6 +993,10 @@ api = API.Global() return api.copyData(srcImage, dstImage, collapse) + def extendVolumeContainer(self, sdUUID, imgUUID, volUUID, size): + api = API.Global() + return api.extendVolumeContainer(sdUUID, imgUUID, volUUID, size) + def getGlobalMethods(self): return ((self.vmDestroy, 'destroy'), (self.vmCreate, 'create'), @@ -1143,7 +1147,8 @@ 'storageServer_ConnectionRefs_statuses'), (self.volumeCreateContainer, 'createVolumeContainer'), (self.volumeRemove, 'removeVolume'), - (self.copyData, 'copyData')) + (self.copyData, 'copyData'), + (self.extendVolumeContainer, 'extendVolumeContainer')) def wrapApiMethod(f): diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 8720703..f507324 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -4182,6 +4182,24 @@ 'data': {'srcImage': 'VolumeSpec', 'dstImage': 'VolumeSpec', 'collapse': 'bool'}} +## +# @Host.extendVolumeContainer: +# +# Extend a thinly-provisioned volume. +# +# @sdUUID: The UUID of the storage domain containing the volume +# +# @imgUUID: The UUID of the image containing the volume +# +# @volUUID: The UUID of the volume +# +# @size: The new desired size (in bytes) +# +# Since: 4.18.0 +## +{'command': {'class': 'Host', 'name': 'extendVolumeContainer'}, + 'data': {'sdUUID': 'UUID', 'imgUUID': 'UUID', 'volUUID': 'UUID', + 'size': 'uint'}} ## Category: @ConnectionRefs ################################################## ## diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 2236457..34730dd 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -854,13 +854,22 @@ """ newSize = misc.validateN(newSize, "newSize") / 2 ** 20 - try: - pool = self.getPool(spUUID) - except se.StoragePoolUnknown: - pass + enableSDM = True # Replace this with a check on the SD version + if enableSDM: + domain = sdCache.produce(volDict['domainID']) + self._sdmSchedule('extendVolumeContainer', + sdm.extendVolumeContainer, domain, + volDict['imageID'], volDict['volumeID'], newSize, + callbackFunc, volDict) else: - if pool.hsmMailer: - pool.hsmMailer.sendExtendMsg(volDict, newSize, callbackFunc) + try: + pool = self.getPool(spUUID) + except se.StoragePoolUnknown: + pass + else: + if pool.hsmMailer: + pool.hsmMailer.sendExtendMsg(volDict, newSize, + callbackFunc) def _spmSchedule(self, spUUID, name, func, *args): self.validateSPM(spUUID) @@ -3744,3 +3753,18 @@ se.VolumeCopyError("Unsupported combination of image types: " "src:%s, dst:%s" % (src.get('type'), dst.get('type'))) + + @public + def extendVolumeContainer(self, sdUUID, imgUUID, volUUID, size): + vars.task.setDefaultException( + se.VolumeExtendingError("sdUUID=%s, volumeUUID=%s, size=%s" % ( + sdUUID, volUUID, size))) + size = misc.validateN(size, "size") + # ExtendVolume expects size in MB + size = math.ceil(size / 2 ** 20) + + dom = sdCache.produce(sdUUID=sdUUID) + misc.validateUUID(imgUUID, 'imgUUID') + misc.validateUUID(volUUID, 'volUUID') + vars.task.getSharedLock(STORAGE, sdUUID) + return sdm.extendVolumeContainer(dom, imgUUID, volUUID, size) diff --git a/vdsm/storage/sdm/__init__.py b/vdsm/storage/sdm/__init__.py index 9d74bd5..7f80ca2 100644 --- a/vdsm/storage/sdm/__init__.py +++ b/vdsm/storage/sdm/__init__.py @@ -195,3 +195,16 @@ finally: dstDom.releaseVolumeLease(dstImage['imgUUID'], dstImage['volUUID']) srcDom.releaseVolumeLease(srcImage['imgUUID'], srcImage['volUUID']) + + +def extendVolumeContainer(domain, imgUUID, volUUID, size, + cbFn=None, cbData=None): + cls = __getStoreClass(domain) + hostId = getDomainHostId(domain.sdUUID) + domain.acquireClusterLock(hostId) + try: + cls.extendVolume(domain, imgUUID, volUUID, size) + finally: + domain.releaseClusterLock() + if cbFn: + cbFn(cbData) diff --git a/vdsm/storage/sdm/blockstore.py b/vdsm/storage/sdm/blockstore.py index b80f869..73a12a8 100644 --- a/vdsm/storage/sdm/blockstore.py +++ b/vdsm/storage/sdm/blockstore.py @@ -20,6 +20,7 @@ import os import logging +import math import vdsm.utils as utils from vdsm.config import config @@ -27,9 +28,13 @@ import volumestore from .. import blockVolume from .. import lvm +from .. import resourceManager as rm +from .. import sd from .. import storage_exception as se from .. import volume +from ..resourceFactories import IMAGE_NAMESPACE +rmanager = rm.ResourceManager.getInstance() log = logging.getLogger('Storage.sdm.blockstore') SECTORS_TO_MB = (1 << 20) / volume.BLOCK_SIZE @@ -37,6 +42,9 @@ class BlockStore(volumestore.VolumeStore): volClass = blockVolume.BlockVolume + + # Estimate of the additional space needed for qcow format internal data. + VOLWM_COW_OVERHEAD = 1.1 @classmethod def volFormatToPreallocate(cls, volFormat): @@ -109,7 +117,7 @@ parent = tag[len(blockVolume.TAG_PREFIX_PARENT):] if parent and image: break - vols.append(volumestore.GCVol(lv.name, volUUID, image, parent)) + vols.append(volumestore.GCVol(lv.name, volUUID, image, parent) return vols @classmethod @@ -119,3 +127,24 @@ except se.VolumeMetadataReadError: pass lvm.removeLVs(dom.sdUUID, volName) + + @classmethod + def extendVolume(cls, dom, imgUUID, volUUID, size): + imageResourcesNamespace = sd.getNamespace(dom.sdUUID, IMAGE_NAMESPACE) + with rmanager.acquireResource(imageResourcesNamespace, imgUUID, + rm.LockType.shared): + # Verify that the requested size is valid + vol = dom.produceVolume(imgUUID, volUUID) + volInfo = vol.getInfo() + maxSize = int(volInfo['capacity']) + if volInfo['format'] == volume.type2name(volume.COW_FORMAT): + maxSize = maxSize * cls.VOLWM_COW_OVERHEAD + maxSize = math.ceil(maxSize / 2 ** 20) + if size > maxSize: + raise se.VolumeExtendingError( + "Size %i exceeds the maximum extend size of %i for volume " + "%s" % (size, maxSize, volUUID)) + + dom.extendVolume(volUUID, size) + + diff --git a/vdsm/storage/sdm/filestore.py b/vdsm/storage/sdm/filestore.py index dd6c58f..cc3e2bb 100644 --- a/vdsm/storage/sdm/filestore.py +++ b/vdsm/storage/sdm/filestore.py @@ -157,4 +157,9 @@ except se.ImageDeleteError: dom.imageGarbageCollector() else: - dom.oop.os.unlink(volPath) \ No newline at end of file + dom.oop.os.unlink(volPath) + + @classmethod + def extendVolume(cls, dom, imgUUID, volUUID, size): + # There is nothing to do for file domains. The filesystem handles it, + pass -- To view, visit https://gerrit.ovirt.org/39696 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6a128ba3eab4116ff4e794e94a171e51d9e432de Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 16

Change in vdsm[master]: SDM: isolateVolumes API
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: SDM: isolateVolumes API ...................................................................... SDM: isolateVolumes API Change-Id: I9b67e2df82afba9956e8246c1a4f9093aed729f2 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py M vdsm/rpc/vdsmapi-schema.json M vdsm/storage/hsm.py M vdsm/storage/sdm/__init__.py M vdsm/storage/sdm/blockstore.py M vdsm/storage/sdm/filestore.py M vdsm/storage/sdm/volumestore.py M vdsm/storage/storage_exception.py 10 files changed, 134 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/40379/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 633d1b7..d3c482e 100755 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1963,6 +1963,17 @@ else: return status['status']['code'], status['status']['message'] + def isolateVolumes(self, args): + if len(args) != 4: + raise ValueError('Wrong number of arguments') + sdUUID, srcImgUUID, dstImgUUID, volStr = args + volList = volStr.split(',') + status = self.s.isolateVolumes(sdUUID, srcImgUUID, dstImgUUID, volList) + if status['status']['code'] == 0: + return 0, '' + else: + return status['status']['code'], status['status']['message'] + if __name__ == '__main__': if _glusterEnabled: @@ -2855,6 +2866,12 @@ '<sdUUID> <imgUUID> <volUUID>', 'Extend a thinly-provisioned block volume.' )), + 'isolateVolumes': ( + serv.isolateVolumes, ( + '<sdUUID> <srcImgUUID> <dstImgUUID> <volUUID>[...,<volUUID>]', + 'Isolate volumes from one image into a new image for ' + 'post-processing.' + )), } if _glusterEnabled: commands.update(ge.getGlusterCmdDict(serv)) diff --git a/vdsm/API.py b/vdsm/API.py index a50025f..44dddb4 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1049,6 +1049,10 @@ def validate(self): return self._irs.validateStorageDomain(self._UUID) + def isolateVolumes(self, srcImageID, dstImageID, volumeList): + return self._irs.isolateVolumes(self._UUID, srcImageID, dstImageID, + volumeList) + class StoragePool(APIBase): ctorArgs = ['storagepoolID'] diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index 7834e83..0e1e5e4 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -1000,6 +1000,10 @@ api = API.Global() return api.extendVolumeContainer(sdUUID, imgUUID, volUUID, size) + def isolateVolumes(self, sdUUID, srcImgUUID, dstImgUUID, volumeList): + api = API.StorageDomain(sdUUID) + return api.isolateVolumes(srcImgUUID, dstImgUUID, volumeList) + def getGlobalMethods(self): return ((self.vmDestroy, 'destroy'), (self.vmCreate, 'create'), @@ -1151,7 +1155,8 @@ 'storageServer_ConnectionRefs_statuses'), (self.volumeCreateContainer, 'createVolumeContainer'), (self.copyData, 'copyData'), - (self.extendVolumeContainer, 'extendVolumeContainer')) + (self.extendVolumeContainer, 'extendVolumeContainer'), + (self.isolateVolumes, 'isolateVolumes')) def wrapApiMethod(f): diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index c0d8caf..a873d22 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -5459,6 +5459,25 @@ {'command': {'class': 'StorageDomain', 'name': 'validate'}, 'data': {'storagedomainID': 'UUID'}} +## +# @StorageDomain.isolateVolumes: +# +# Isolate volumes from one image into a new image. +# +# @storagedomainID: The UUID of the Storage Domain +# +# @srcImageID: The UUID of the Image containing the volumes +# +# @dstImageID: The UUID of the destination Image +# +# @volumeList: Identifies a set of volumes to move +# +# Since: 4.18.0 +## +{'command': {'class': 'StorageDomain', 'name': 'isolateVolumes'}, + 'data': {'storagedomainID': 'UUID', 'srcImageID': 'UUID', + 'dstImageID': 'UUID', 'volumeList': ['UUID']}} + ## Category: @StoragePool ##################################################### ## # @StoragePool: diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 63c9b3b..746423d 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3760,3 +3760,13 @@ misc.validateUUID(volUUID, 'volUUID') vars.task.getSharedLock(STORAGE, sdUUID) return sdm.extendVolumeContainer(dom, imgUUID, volUUID, size) + + @public + def isolateVolumes(self, sdUUID, srcImgUUID, dstImgUUID, volumeList): + vars.task.setDefaultException( + se.IsolateVolumesError(sdUUID, srcImgUUID, dstImgUUID, volumeList)) + dom = sdCache.produce(sdUUID=sdUUID) + misc.validateUUID(srcImgUUID, 'srcImgUUID') + misc.validateUUID(dstImgUUID, 'dstImgUUID') + vars.task.getSharedLock(STORAGE, sdUUID) + return sdm.isolateVolumes(dom, srcImgUUID, dstImgUUID, volumeList) diff --git a/vdsm/storage/sdm/__init__.py b/vdsm/storage/sdm/__init__.py index da1e858..1636e63 100644 --- a/vdsm/storage/sdm/__init__.py +++ b/vdsm/storage/sdm/__init__.py @@ -208,3 +208,21 @@ domain.releaseClusterLock() if cbFn: cbFn(cbData) + + +def isolateVolumes(domain, srcImgUUID, dstImgUUID, volumeList): + cls = __getStoreClass(domain) + imageResourcesNamespace = sd.getNamespace(domain.sdUUID, IMAGE_NAMESPACE) + + hostId = getDomainHostId(domain.sdUUID) + domain.acquireClusterLock(hostId) + try: + with nested(rmanager.acquireResource(imageResourcesNamespace, + srcImgUUID, + rm.LockType.exclusive), + rmanager.acquireResource(imageResourcesNamespace, + dstImgUUID, + rm.LockType.exclusive)): + cls.isolateVolumes(domain, srcImgUUID, dstImgUUID, volumeList) + finally: + domain.releaseClusterLock() diff --git a/vdsm/storage/sdm/blockstore.py b/vdsm/storage/sdm/blockstore.py index 06a77e4..bd9d3f1 100644 --- a/vdsm/storage/sdm/blockstore.py +++ b/vdsm/storage/sdm/blockstore.py @@ -101,6 +101,18 @@ return newName @classmethod + def _isolateVolume(cls, dom, srcImgUUID, dstImgUUID, vol): + pVolUUID = vol.getParent() + toAdd = [blockVolume.TAG_PREFIX_PARENT + volume.BLANK_UUID, + blockVolume.TAG_PREFIX_IMAGE + dstImgUUID] + toDel = [blockVolume.TAG_PREFIX_PARENT + pVolUUID, + blockVolume.TAG_PREFIX_IMAGE + srcImgUUID] + lvm.changeLVTags(dom.sdUUID, vol.volUUID, addTags=toAdd, delTags=toDel) + if pVolUUID and pVolUUID != volume.BLANK_UUID: + pVol = dom.produceVolume(srcImgUUID, pVolUUID) + cls.recheckIfLeaf(pVol) + + @classmethod def _getGCVolumes(cls, dom, onlyImg, onlyVol): lvs = lvm.getLV(dom.sdUUID) vols = [] diff --git a/vdsm/storage/sdm/filestore.py b/vdsm/storage/sdm/filestore.py index 3e99ee9..9385d87 100644 --- a/vdsm/storage/sdm/filestore.py +++ b/vdsm/storage/sdm/filestore.py @@ -98,6 +98,21 @@ return newName @classmethod + def _isolateVolume(cls, dom, srcImgUUID, dstImgUUID, vol): + srcImgPath = os.path.join(dom.getRepoPath(), dom.sdUUID, + sd.DOMAIN_IMAGES, srcImgUUID) + dstImgPath = os.path.join(dom.getRepoPath(), dom.sdUUID, + sd.DOMAIN_IMAGES, dstImgUUID) + pUUID = vol.getParent() + vol._share(dstImgPath) + dstVol = dom.produceVolume(dstImgUUID, vol.volUUID) + dstVol.setParent(volume.BLANK_UUID) + dstVol.setImage(dstImgUUID) + newName = cls._beginRemoveVolume(dom, srcImgPath, vol.volUUID) + volInfo = volumestore.GCVol(newName, vol.volUUID, srcImgUUID, pUUID) + cls._garbageCollectVolume(dom, volInfo) + + @classmethod def _getGCVolumes(cls, dom, onlyImg, onlyVol): vols = [] volPaths = [] diff --git a/vdsm/storage/sdm/volumestore.py b/vdsm/storage/sdm/volumestore.py index d518f20..6568d75 100644 --- a/vdsm/storage/sdm/volumestore.py +++ b/vdsm/storage/sdm/volumestore.py @@ -355,3 +355,27 @@ newName = cls._beginRemoveVolume(dom, imageDir, volUUID) volInfo = GCVol(newName, volUUID, imgUUID, pUUID) cls._garbageCollectVolume(dom, volInfo) + + @classmethod + def isolateVolumes(cls, dom, srcImgUUID, dstImgUUID, volumeList): + repoPath = dom.getRepoPath() + # Create dest image + cls.createImage(repoPath, dom.sdUUID, dstImgUUID) + # Verify dest image contains only volumes in volumeList + uuidList = cls.volClass.getImageVolumes(repoPath, dom.sdUUID, + dstImgUUID) + extraVols = set(uuidList) - set(volumeList) + if extraVols: + log.error("Destination image contains unexpected volumes: %s", + extraVols) + raise se.IsolateVolumesError(dom.sdUUID, srcImgUUID, + dstImgUUID, volumeList) + # Iterate over volumes in volumeList + for volUUID in volumeList: + try: + vol = cls.volClass(repoPath, dom.sdUUID, srcImgUUID, volUUID) + except se.VolumeDoesNotExist: + log.debug("Skipping non-existent source volume %s", volUUID) + continue + vol.validateDelete() + cls._isolateVolume(dom, srcImgUUID, dstImgUUID, vol) diff --git a/vdsm/storage/storage_exception.py b/vdsm/storage/storage_exception.py index 1cfc8e4..a695cf4 100644 --- a/vdsm/storage/storage_exception.py +++ b/vdsm/storage/storage_exception.py @@ -453,6 +453,15 @@ message = "Image does not exist in domain" +class IsolateVolumesError(StorageException): + def __init__(self, sdUUID, srcImgUUID, dstImgUUID, volumeList): + self.value = ("domain=%s srcImg=%s dstImg=%s " + "volumes=%s" % (sdUUID, srcImgUUID, dstImgUUID, + volumeList)) + code = 269 + message = "Unable to isolate volumes" + + ################################################# # Pool Exceptions ################################################# -- To view, visit https://gerrit.ovirt.org/40379 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9b67e2df82afba9956e8246c1a4f9093aed729f2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 8

Change in vdsm[master]: HACK: run GC in domain monitor
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: HACK: run GC in domain monitor ...................................................................... HACK: run GC in domain monitor Change-Id: I3c560e6fbccdf50b135cc9c90b23824ae04b0376 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/monitor.py M vdsm/storage/sdm/__init__.py 2 files changed, 22 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/80/40380/1 diff --git a/vdsm/storage/monitor.py b/vdsm/storage/monitor.py index ef032a5..a2a1bd7 100644 --- a/vdsm/storage/monitor.py +++ b/vdsm/storage/monitor.py @@ -28,6 +28,7 @@ from . import clusterlock from . import misc +from . import sdm from .sdc import sdCache @@ -249,6 +250,7 @@ self._performDomainSelftest() self._checkReadDelay() self._collectStatistics() + self._garbageCollect() except Exception as e: self.log.exception("Error monitoring domain %s", self.sdUUID) self.nextStatus.error = e @@ -340,6 +342,14 @@ self.nextStatus.isoPrefix = self.isoPrefix self.nextStatus.version = self.domain.getVersion() + def _garbageCollect(self): + if True: # XXX: limit this to domain ver 4 or later + try: + sdm.garbageCollectStorageDomain(self.domain) + except: + self.log.exception("Garbage collection failed for domain %s", + self.domain.sdUUID) + # Managing host id def _shouldAcquireHostId(self): diff --git a/vdsm/storage/sdm/__init__.py b/vdsm/storage/sdm/__init__.py index 1636e63..4a31332 100644 --- a/vdsm/storage/sdm/__init__.py +++ b/vdsm/storage/sdm/__init__.py @@ -226,3 +226,15 @@ cls.isolateVolumes(domain, srcImgUUID, dstImgUUID, volumeList) finally: domain.releaseClusterLock() + + +def garbageCollectStorageDomain(domain): + if domain.isISO(): + return + cls = __getStoreClass(domain) + hostId = getDomainHostId(domain.sdUUID) + domain.acquireClusterLock(hostId) + try: + cls.garbageCollectStorageDomain(domain) + finally: + domain.releaseClusterLock() -- To view, visit https://gerrit.ovirt.org/40380 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3c560e6fbccdf50b135cc9c90b23824ae04b0376 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 9

Change in vdsm[master]: Add wipeVolume API
by alitke＠redhat.com 10 Jul '16

10 Jul '16

Adam Litke has uploaded a new change for review. Change subject: Add wipeVolume API ...................................................................... Add wipeVolume API Change-Id: I572099e1fee4ae847de95e27973405ba90b56a9b Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py M vdsm/rpc/vdsmapi-schema.json M vdsm/storage/hsm.py M vdsm/storage/sdm/__init__.py 6 files changed, 70 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/28/40628/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index bac146e..c267b75 100755 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1973,6 +1973,16 @@ else: return status['status']['code'], status['status']['message'] + def wipeVolume(self, args): + if len(args) != 3: + raise ValueError('Wrong number of arguments') + sdUUID, imgUUID, volUUID = args + status = self.s.wipeVolume(sdUUID, imgUUID, volUUID) + if status['status']['code'] == 0: + return 0, '' + else: + return status['status']['code'], status['status']['message'] + if __name__ == '__main__': if _glusterEnabled: diff --git a/vdsm/API.py b/vdsm/API.py index 6adefa8..a49448b 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -840,6 +840,9 @@ return self._irs.setVolumeLegality(self._sdUUID, self._spUUID, self._imgUUID, self._UUID, legality) + def wipe(self): + return self._irs.wipeVolume(self._sdUUID, self._imgUUID, self._UUID) + class Image(APIBase): ctorArgs = ['imageID', 'storagepoolID', 'storagedomainID'] diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index f110468..270f7a1 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -1004,6 +1004,10 @@ api = API.StorageDomain(sdUUID) return api.isolateVolume(srcImgUUID, dstImgUUID, volumeID) + def wipeVolume(self, sdUUID, imgUUID, volUUID): + volume = API.Volume(volUUID, None, sdUUID, imgUUID) + return volume.wipe() + def getGlobalMethods(self): return ((self.vmDestroy, 'destroy'), (self.vmCreate, 'create'), @@ -1156,7 +1160,8 @@ (self.volumeCreateContainer, 'createVolumeContainer'), (self.copyData, 'copyData'), (self.extendVolumeContainer, 'extendVolumeContainer'), - (self.isolateVolume, 'isolateVolume')) + (self.isolateVolume, 'isolateVolume'), + (self.wipeVolume, 'wipeVolume')) def wrapApiMethod(f): diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index a76db8d..80b1968 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -8312,6 +8312,22 @@ 'legality': 'VolumeLegality'}} ## +# @Volume.wipe: +# +# Wipe a volume to ensure previous data is erased. +# +# @volumeID: The UUID of the Volume +# +# @storagedomainID: The Storage Domain associated with the Volume +# +# @imageID: The Image associated with the Volume +# +# Since: 4.18.0 +## +{'command': {'class': 'Volume', 'name': 'wipe'}, + 'data': {'volumeID': 'UUID', 'storagedomainID': 'UUID', 'imageID': 'UUID'}} + +## # @VM.setNumberOfCpus: # # Set the number CPUs for a VM diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 432c8f9..cb51cd1 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3776,3 +3776,16 @@ misc.validateUUID(dstImgUUID, 'dstImgUUID') vars.task.getSharedLock(STORAGE, sdUUID) return sdm.isolateVolume(dom, srcImgUUID, dstImgUUID, volumeID) + + @public + def wipeVolume(self, sdUUID, imgUUID, volUUID): + argsStr = ("sdUUID=%s, imgUUID=%s, volUUID=%s" % + (sdUUID, imgUUID, volUUID)) + vars.task.setDefaultException(se.VolumesZeroingError(argsStr)) + misc.validateUUID(imgUUID, 'imgUUID') + misc.validateUUID(volUUID, 'volUUID') + vars.task.getSharedLock(STORAGE, sdUUID) + imageSpec = {'sdUUID': sdUUID, 'imgUUID': imgUUID, 'volUUID': volUUID} + with sdm.prepareCopy(imageSpec, False, 0, True): + self._sdmSchedule('wipeVolume', sdm.wipeVolume, sdUUID, imgUUID, + volUUID) diff --git a/vdsm/storage/sdm/__init__.py b/vdsm/storage/sdm/__init__.py index 4738813..2113d26 100644 --- a/vdsm/storage/sdm/__init__.py +++ b/vdsm/storage/sdm/__init__.py @@ -22,6 +22,7 @@ from contextlib import contextmanager, nested from .. import image +from .. import misc from .. import resourceManager as rm from .. import sd from .. import storage_exception as se @@ -234,6 +235,27 @@ domain.releaseClusterLock() +def wipeVolume(sdUUID, imgUUID, volUUID): + dom = sdCache.produce(sdUUID) + imageResourcesNamespace = sd.getNamespace(sdUUID, IMAGE_NAMESPACE) + try: + with rmanager.acquireResource(imageResourcesNamespace, + imgUUID, rm.LockType.exclusive): + vol = dom.produceVolume(imgUUID, volUUID) + size = vol.getVolumeSize(bs=1) + vol_path = vol.getVolumePath() + vol.prepare(rw=True, justme=True) + try: + vol.setLegality(volume.ILLEGAL_VOL) + misc.ddWatchCopy( + "/dev/zero", vol_path, vars.task.aborting, int(size)) + finally: + vol.tearDown(imgUUID, volUUID) + vol.setLegality(volume.LEGAL_VOL) + finally: + dom.releaseVolumeLease(imgUUID, volUUID) + + def garbageCollectStorageDomain(domain): if domain.isISO(): return -- To view, visit https://gerrit.ovirt.org/40628 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I572099e1fee4ae847de95e27973405ba90b56a9b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 4

Change in vdsm[master]: api: support multiple 'ipv6addrs' per network and deprecate ...
by osvoboda＠redhat.com 10 Jul '16

10 Jul '16

Ondřej Svoboda has uploaded a new change for review. Change subject: api: support multiple 'ipv6addrs' per network and deprecate 'ipv6addr' ...................................................................... api: support multiple 'ipv6addrs' per network and deprecate 'ipv6addr' Next, the 'ipv6' hook will be extended to understand a whitespace-separated list of IPv6 addresses and also testStaticNetworkConfig will learn to pass multiple IPv6 addresses to VDSM. IPv4 support is planned but needs discussion (should we accept netmasks or just CIDR notation?). Change-Id: I8d16083781919fe5e5d9c75e9dd4ab744afe45be Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm/network/api.py M vdsm/network/configurators/ifcfg.py M vdsm/network/configurators/iproute2.py M vdsm/network/configurators/pyroute_two.py M vdsm/network/models.py 5 files changed, 45 insertions(+), 24 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/07/39307/1 diff --git a/vdsm/network/api.py b/vdsm/network/api.py index de92fbf..096f89f 100755 --- a/vdsm/network/api.py +++ b/vdsm/network/api.py @@ -82,7 +82,7 @@ def _objectivizeNetwork(bridge=None, vlan=None, bonding=None, bondingOptions=None, nics=None, mtu=None, ipaddr=None, netmask=None, gateway=None, bootproto=None, - ipv6addr=None, ipv6gateway=None, ipv6autoconf=None, + ipv6addrs=None, ipv6gateway=None, ipv6autoconf=None, dhcpv6=None, defaultRoute=None, _netinfo=None, configurator=None, blockingdhcp=None, implicitBonding=None, opts=None): @@ -100,7 +100,7 @@ :param netmask: IPv4 mask in dotted decimal format. :param gateway: IPv4 address in dotted decimal format. :param bootproto: protocol for getting IP config for the net, e.g., 'dhcp' - :param ipv6addr: IPv6 address in format address[/prefixlen]. + :param ipv6addrs: list of IPv6 addresses in the format address[/prefixlen] :param ipv6gateway: IPv6 address in format address[/prefixlen]. :param ipv6autoconf: whether to use IPv6's stateless autoconfiguration. :param dhcpv6: whether to use DHCPv6. @@ -162,7 +162,7 @@ raise ConfigNetworkError(ne.ERR_BAD_PARAMS, 'Network defined without ' 'devices.') ipv4 = IPv4(ipaddr, netmask, gateway, defaultRoute) - ipv6 = IPv6(ipv6addr, ipv6gateway, defaultRoute) + ipv6 = IPv6(ipv6addrs, ipv6gateway, defaultRoute) blockingdhcp = configurator._inRollback or utils.tobool(blockingdhcp) topNetDev.ipconfig = IpConfig( ipv4=ipv4, ipv6=ipv6, bootproto=bootproto, blockingdhcp=blockingdhcp, @@ -249,6 +249,7 @@ @_alterRunningConfig def _addNetwork(network, vlan=None, bonding=None, nics=None, ipaddr=None, netmask=None, prefix=None, mtu=None, gateway=None, dhcpv6=None, + ipv6addrs=None, ipv6addr=None, ipv6gateway=None, ipv6autoconf=None, force=False, configurator=None, bondingOptions=None, bridged=True, _netinfo=None, hostQos=None, @@ -279,6 +280,12 @@ except ValueError as ve: raise ConfigNetworkError(ne.ERR_BAD_ADDR, "Bad prefix: %s" % ve) + if ipv6addr: + if ipv6addrs: + raise ConfigNetworkError(ne.ERR_BAD_PARAMS, + 'Both ipv6addrs and ipv6addr supplied') + else: + ipv6addrs = [ipv6addr] if not utils.tobool(force): logging.debug('validating network...') @@ -312,7 +319,8 @@ bridge=network if bridged else None, vlan=vlan, bonding=bonding, bondingOptions=bondingOptions, nics=nics, mtu=mtu, ipaddr=ipaddr, netmask=netmask, gateway=gateway, bootproto=bootproto, dhcpv6=dhcpv6, - blockingdhcp=blockingdhcp, ipv6addr=ipv6addr, ipv6gateway=ipv6gateway, + blockingdhcp=blockingdhcp, ipv6addrs=ipv6addrs, + ipv6gateway=ipv6gateway, ipv6autoconf=ipv6autoconf, defaultRoute=defaultRoute, _netinfo=_netinfo, configurator=configurator, opts=options) @@ -783,7 +791,8 @@ netmask="<ipv4>" gateway="<ipv4>" bootproto="..." - ipv6addr="<ipv6>[/<prefixlen>]" + ipv6addrs=["<ipv6>[/<prefixlen>]", ...] + ipv6addr="<ipv6>[/<prefixlen>]" (deprecated) ipv6gateway="<ipv6>" ipv6autoconf="0|1" dhcpv6="0|1" diff --git a/vdsm/network/configurators/ifcfg.py b/vdsm/network/configurators/ifcfg.py index 866ade8..f228306 100644 --- a/vdsm/network/configurators/ifcfg.py +++ b/vdsm/network/configurators/ifcfg.py @@ -584,10 +584,13 @@ if ipv4.defaultRoute is not None: cfg += 'DEFROUTE=%s\n' % _to_ifcfg_bool(ipv4.defaultRoute) cfg += 'NM_CONTROLLED=no\n' - if ipv6.address or ipconfig.ipv6autoconf or ipconfig.dhcpv6: + if ipv6.addresses or ipconfig.ipv6autoconf or ipconfig.dhcpv6: cfg += 'IPV6INIT=yes\n' - if ipv6.address is not None: - cfg += 'IPV6ADDR=%s\n' % pipes.quote(ipv6.address) + if ipv6.addresses: + cfg += 'IPV6ADDR=%s\n' % pipes.quote(ipv6.addresses[0]) + if len(ipv6.addresses) > 1: + cfg += 'IPV6ADDR_SECONDARIES=%s\n' % pipes.quote( + ' '.join(ipv6.addresses[1:])) if ipv6.gateway is not None: cfg += 'IPV6_DEFAULTGW=%s\n' % pipes.quote(ipv6.gateway) elif ipconfig.dhcpv6: @@ -701,7 +704,10 @@ if ipv4.defaultRoute is None and confParams.get('DEFROUTE'): ipv4.defaultRoute = _from_ifcfg_bool(confParams['DEFROUTE']) if confParams.get('IPV6INIT') == 'yes': - ipv6.address = confParams.get('IPV6ADDR') + ipv6addr = confParams.get('IPV6ADDR') + ipv6addrs = confParams.get('IPV6ADDR_SECONDARIES') + ipv6.addresses = ([ipv6addr] if ipv6addr else []) + ( + ipv6addrs.split() if ipv6addrs else []) ipv6.gateway = confParams.get('IPV6_DEFAULTGW') ipconfig.ipv6autoconf = ( confParams.get('IPV6_AUTOCONF') == 'yes') diff --git a/vdsm/network/configurators/iproute2.py b/vdsm/network/configurators/iproute2.py index 1cde1cc..045ed66 100644 --- a/vdsm/network/configurators/iproute2.py +++ b/vdsm/network/configurators/iproute2.py @@ -225,16 +225,17 @@ ipconfig = iface.ipconfig ipv4 = ipconfig.ipv4 ipv6 = ipconfig.ipv6 - if ipv4.address or ipv6.address: + if ipv4.address or ipv6.addresses: self.removeIpConfig(iface) if ipv4.address: ipwrapper.addrAdd(iface.name, ipv4.address, ipv4.netmask) if ipv4.gateway and ipv4.defaultRoute: ipwrapper.routeAdd(['default', 'via', ipv4.gateway]) - if ipv6.address: - ipv6addr, ipv6netmask = ipv6.address.split('/') - ipwrapper.addrAdd(iface.name, ipv6addr, ipv6netmask, family=6) + if ipv6.addresses: + for address in ipv6.addresses: + ipv6addr, ipv6netmask = address.split('/') + ipwrapper.addrAdd(iface.name, ipv6addr, ipv6netmask, family=6) if ipv6.gateway: ipwrapper.routeAdd(['default', 'via', ipv6.gateway], dev=iface.name, family=6) diff --git a/vdsm/network/configurators/pyroute_two.py b/vdsm/network/configurators/pyroute_two.py index 221e142..a6833d5 100644 --- a/vdsm/network/configurators/pyroute_two.py +++ b/vdsm/network/configurators/pyroute_two.py @@ -59,7 +59,7 @@ ipconfig = iface.ipconfig ipv4 = ipconfig.ipv4 ipv6 = ipconfig.ipv6 - if ipv4.address or ipv6.address: + if ipv4.address or ipv6.addresses: self.removeIpConfig(iface) if ipv4.address: with self.ip.interfaces[iface.name] as i: @@ -67,9 +67,10 @@ if ipv4.gateway and ipv4.defaultRoute: self.ip.routes.add({'dst': 'default', 'gateway': ipv4.gateway}).commit() - if ipv6.address: + if ipv6.addresses: with self.ip.interfaces[iface.name] as i: - i.add_ip(ipv6.address) + for address in ipv6.addresses: + i.add_ip(address) if ipv6.gateway: self.ip.routes.add({'dst': 'default', 'gateway': ipv6.gateway}).commit() diff --git a/vdsm/network/models.py b/vdsm/network/models.py index 1d34c88..2c577de 100644 --- a/vdsm/network/models.py +++ b/vdsm/network/models.py @@ -410,20 +410,24 @@ class IPv6(object): - def __init__(self, address=None, gateway=None, defaultRoute=None): - if address: - self.validateAddress(address) + def __init__(self, addresses=None, gateway=None, defaultRoute=None): + if addresses: + # TODO: Do we want this? + # if isinstance(addresses, str): + # addresses = addresses.split() + for address in addresses: + self.validateAddress(address) if gateway: self.validateGateway(gateway) elif gateway: raise ConfigNetworkError(ne.ERR_BAD_ADDR, 'Specified gateway but ' 'not ip address.') - self.address = address + self.addresses = addresses self.gateway = gateway self.defaultRoute = defaultRoute def __repr__(self): - return 'IPv6(%s, %s, %s)' % (self.address, self.gateway, + return 'IPv6(%s, %s, %s)' % (self.addresses, self.gateway, self.defaultRoute) @classmethod @@ -465,7 +469,7 @@ if ipv6 is None: ipv6 = IPv6() if (ipv4.address and bootproto == 'dhcp') or \ - (ipv6.address and (ipv6autoconf or dhcpv6)): + (ipv6.addresses and (ipv6autoconf or dhcpv6)): raise ConfigNetworkError(ne.ERR_BAD_ADDR, 'Static and dynamic IP ' 'configurations are mutually exclusive.') self.ipv4 = ipv4 @@ -477,8 +481,8 @@ def __nonzero__(self): # iproute2 and pyroute_two check that IP configuration is not empty - return bool(self.ipv4.address or self.ipv6.address or self.bootproto or - self.ipv6autoconf or self.dhcpv6) + return bool(self.ipv4.address or self.ipv6.addresses or self.bootproto + or self.ipv6autoconf or self.dhcpv6) def __repr__(self): return 'IpConfig(%r, %r, %s, %s, %s)' % (self.ipv4, self.ipv6, -- To view, visit https://gerrit.ovirt.org/39307 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8d16083781919fe5e5d9c75e9dd4ab744afe45be Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

5 32

Change in vdsm[master]: contrib: Add tool for getting hsm stats
by Nir Soffer 03 Jul '16

03 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: contrib: Add tool for getting hsm stats ...................................................................... contrib: Add tool for getting hsm stats This is a very simple tool showing historgam of hsm calls in the given log files. Example: $ hsmstat vdsm.log.1 vdsm.log.2 8515 getVolumeSize 2258 repoStats 51 getVolumeInfo 30 getVolumesList 19 teardownImage ... We can get same output or better using logdb by writing some smart sql. Change-Id: Ia6013da71ca96535b98edace8577a4d8bb529465 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A contrib/hsmstat 1 file changed, 39 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/48/38748/1 diff --git a/contrib/hsmstat b/contrib/hsmstat new file mode 100755 index 0000000..d3400d2 --- /dev/null +++ b/contrib/hsmstat @@ -0,0 +1,39 @@ +#!/usr/bin/env python +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from collections import defaultdict +import fileinput +import re + +verb_re = re.compile(r" Run and protect: (?P<name>.+)\(") + +verbs = defaultdict(int) + +for line in fileinput.input(): + match = verb_re.search(line) + if match: + name = match.group('name') + verbs[name] += 1 + +histogram = sorted(((count, verb) for verb, count in verbs.iteritems()), + reverse=True) +for count, name in histogram: + print " %6d %s" % (count, name) -- To view, visit https://gerrit.ovirt.org/38748 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6013da71ca96535b98edace8577a4d8bb529465 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 8

Change in vdsm[master]: contrib: Add tool for getting repoStats statistics
by Nir Soffer 03 Jul '16

03 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: contrib: Add tool for getting repoStats statistics ...................................................................... contrib: Add tool for getting repoStats statistics This tool is useful when debugging storage domain monitoring issues. Examples: $ repostat vdsm.log.* domain: ebcd25b1-1ab0-47e3-aa8b-83ee5f91b5bd delay avg: 0.093941 min: 0.000430 max: 1.499200 last check avg: 5.117508 min: 0.000000 max: 12.300000 domain: 6d7f7a48-4511-43b8-8abf-d323f7118b19 delay avg: 0.001090 min: 0.000300 max: 0.030056 last check avg: 4.975313 min: 0.000000 max: 10.200000 Change-Id: Id5f7828216058bb401f8a472fa5601e79542def1 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M Makefile.am A contrib/repostat 2 files changed, 55 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/49/38749/1 diff --git a/Makefile.am b/Makefile.am index 34885a1..7407b22 100644 --- a/Makefile.am +++ b/Makefile.am @@ -49,6 +49,7 @@ autogen.sh \ build-aux/pkg-version \ contrib/profile-stats \ + contrib/repostat \ pylintrc \ vdsm.spec \ vdsm.spec.in \ @@ -80,6 +81,7 @@ WHITELIST = \ contrib/profile-stats \ + contrib/repostat \ init/daemonAdapter \ vdsm/get-conf-item \ vdsm/set-conf-item \ diff --git a/contrib/repostat b/contrib/repostat new file mode 100755 index 0000000..c140b4c --- /dev/null +++ b/contrib/repostat @@ -0,0 +1,53 @@ +#!/usr/bin/env python +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +""" +Parse repoStats log lines and calculate statistics. + +Usage: repostat vdsm.log [...] +""" + +import fileinput +import re + +repostat_re = re.compile(r' Run and protect: repoStats, Return response: ') + +stats = {} + + +def liststat(a): + b = sorted(a) + return sum(b) / len(b), b[0], b[-1] + + +for line in fileinput.input(): + match = repostat_re.search(line) + if not match: + continue + response = eval(line[match.end():]) + for uuid, info in response.items(): + stats.setdefault(uuid, {'delays': [], 'checks': []}) + stats[uuid]['delays'].append(float(info['delay'])) + stats[uuid]['checks'].append(float(info['lastCheck'])) + +for uuid, info in stats.iteritems(): + print 'domain:', uuid + print ' delay avg: %f min: %f max: %f' % liststat(info['delays']) + print ' last check avg: %f min: %f max: %f' % liststat(info['checks']) -- To view, visit https://gerrit.ovirt.org/38749 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id5f7828216058bb401f8a472fa5601e79542def1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 6

Change in vdsm[master]: gluster: rename glusterCreateBrick to glusterBrickCreate
by barumuga＠redhat.com 03 Jul '16

03 Jul '16

Hello Piotr Kliczewski, Timothy Asir, Dan Kenigsberg, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/42308 to review the following change. Change subject: gluster: rename glusterCreateBrick to glusterBrickCreate ...................................................................... gluster: rename glusterCreateBrick to glusterBrickCreate To make consistency in function names in supervdsm and verb names in vdsm, this patch renames verb glusterCreateBrick to glusterBrickCreate Change-Id: I367a864bc9bf4aa9b6a0d36b64f541fa89c61c0d Signed-off-by: Bala.FA <barumuga(a)redhat.com> --- M client/vdsClientGluster.py M vdsm/gluster/api.py M vdsm/gluster/apiwrapper.py M vdsm/gluster/storagedev.py M vdsm/rpc/vdsmapi-gluster-schema.json 5 files changed, 11 insertions(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/08/42308/1 diff --git a/client/vdsClientGluster.py b/client/vdsClientGluster.py index a4bce63..76b91e1 100644 --- a/client/vdsClientGluster.py +++ b/client/vdsClientGluster.py @@ -633,7 +633,7 @@ pp.pprint(status) return status['status']['code'], status['status']['message'] - def do_glusterCreateBrick(self, args): + def do_glusterBrickCreate(self, args): params = self._eqSplit(args) devList = params.get('devices', '').split(',') brickName = params.get('brickName', '') @@ -646,7 +646,7 @@ raidParams['stripeSize'] = int(params.get('stripeSize', 0)) raidParams['pdCount'] = int(params.get('pdCount', 0)) - status = self.s.glusterCreateBrick(brickName, mountPoint, + status = self.s.glusterBrickCreate(brickName, mountPoint, devList, fsType, raidParams) pp.pprint(status) return status['status']['code'], status['status']['message'] @@ -1172,8 +1172,8 @@ ('[volumeName=<volume_name>]', 'snapshot list for given volume' )), - 'glusterCreateBrick': ( - serv.do_glusterCreateBrick, + 'glusterBrickCreate': ( + serv.do_glusterBrickCreate, ('brickName=<brick_name> mountPoint=<mountPoint> ' 'devices=<device[,device, ...]> ' '[raidType=<raid_type>] [stripeSize=<stripe_size>] ' diff --git a/vdsm/gluster/api.py b/vdsm/gluster/api.py index 24b8230..918c3cc 100644 --- a/vdsm/gluster/api.py +++ b/vdsm/gluster/api.py @@ -672,9 +672,9 @@ return {'snapshotList': status} @exportAsVerb - def createBrick(self, name, mountPoint, devList, fsType=None, + def brickCreate(self, name, mountPoint, devList, fsType=None, raidParams={}, options=None): - status = self.svdsmProxy.glusterCreateBrick(name, + status = self.svdsmProxy.glusterBrickCreate(name, mountPoint, devList, fsType, diff --git a/vdsm/gluster/apiwrapper.py b/vdsm/gluster/apiwrapper.py index 4768a86..f607dd4 100644 --- a/vdsm/gluster/apiwrapper.py +++ b/vdsm/gluster/apiwrapper.py @@ -81,9 +81,9 @@ def storageDevicesList(self, options=None): return self._gluster.storageDevicesList() - def createBrick(self, name, mountPoint, devList, fsType=None, + def brickCreate(self, name, mountPoint, devList, fsType=None, raidParams={}): - return self._gluster.createBrick(name, mountPoint, + return self._gluster.brickCreate(name, mountPoint, devList, fsType, raidParams) diff --git a/vdsm/gluster/storagedev.py b/vdsm/gluster/storagedev.py index 2aeabad..f4178e9 100644 --- a/vdsm/gluster/storagedev.py +++ b/vdsm/gluster/storagedev.py @@ -123,7 +123,7 @@ @makePublic -def createBrick(brickName, mountPoint, devNameList, fsType=DEFAULT_FS_TYPE, +def brickCreate(brickName, mountPoint, devNameList, fsType=DEFAULT_FS_TYPE, raidParams={}): def _getDeviceList(devNameList): return [blivetEnv.devicetree.getDeviceByName(devName.split("/")[-1]) diff --git a/vdsm/rpc/vdsmapi-gluster-schema.json b/vdsm/rpc/vdsmapi-gluster-schema.json index c86f43c..209bfbd 100644 --- a/vdsm/rpc/vdsmapi-gluster-schema.json +++ b/vdsm/rpc/vdsmapi-gluster-schema.json @@ -1274,7 +1274,7 @@ 'data': {'type': 'str', 'stripeSize': 'int', 'diskCnt': 'int'}} ## -# @GlusterHost.createBrick: +# @GlusterHost.brickCreate: # # Create a brick for the gluster volume # @@ -1293,7 +1293,7 @@ # # Since: 4.17.0 ## -{'command': {'class': 'GlusterHost', 'name': 'createBrick'}, +{'command': {'class': 'GlusterHost', 'name': 'brickCreate'}, 'data': {'name': 'str', 'mountPoint': 'str', 'devList': ['str'], '*fsType': 'str', '*raidParams': 'RaidDevice'}, 'returns': 'bool'} -- To view, visit https://gerrit.ovirt.org/42308 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I367a864bc9bf4aa9b6a0d36b64f541fa89c61c0d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Bala.FA <barumuga(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com> Gerrit-Reviewer: Timothy Asir <tjeyasin(a)redhat.com>

4 4

Change in vdsm[master]: tests: Add time per test if nose-timer is available
by asegurap＠redhat.com 03 Jul '16

03 Jul '16

Antoni Segura Puimedon has uploaded a new change for review. Change subject: tests: Add time per test if nose-timer is available ...................................................................... tests: Add time per test if nose-timer is available This patch adds an output like: resourceManagerTests.ResourceManagerTests.testStressTest: 12.8313s resourceManagerTests.ResourceManagerTests.testResourceAcquireTimeout: 1.0069s resourceManagerTests.ResourceManagerTests.testResourceLockSwitch: 0.0108s resourceManagerTests.ResourceManagerTests.testResourceAutorelease: 0.0101s resourceManagerTests.ResourceManagerTests.testResourceInvalidation: 0.0093s resourceManagerTests.ResourceManagerTests.testCrashOnSwitch: 0.0067s resourceManagerTests.ResourceManagerTests.testResourceSwitchLockTypeFail: 0.0062s resourceManagerTests.ResourceManagerTests.testCancelExclusiveBetweenShared: 0.0049s resourceManagerTests.ResourceManagerTests.testAcquireResourceExclusive: 0.0042s resourceManagerTests.ResourceManagerTests.testCancelRequest: 0.0031s resourceManagerTests.ResourceManagerTests.testRequestRefCmp: 0.0030s resourceManagerTests.ResourceManagerTests.testRequestRecancel: 0.0030s resourceManagerTests.ResourceManagerTests.testResourceStatuses: 0.0029s resourceManagerTests.ResourceManagerTests.testAccessAttributeNotExposedByRequestRef: 0.0029s resourceManagerTests.ResourceManagerTests.testFailCreateAfterSwitch: 0.0025s resourceManagerTests.ResourceManagerTests.testRequestWithBadCallbackOnCancel: 0.0025s resourceManagerTests.ResourceManagerTests.testAcquireResourceShared: 0.0023s resourceManagerTests.ResourceManagerTests.testRereleaseResource: 0.0022s resourceManagerTests.ResourceManagerTests.testResourceWrapper: 0.0020s resourceManagerTests.ResourceManagerTests.testRequestWithBadCallbackOnGrant: 0.0020s resourceManagerTests.ResourceManagerTests.testRequestRefStr: 0.0020s resourceManagerTests.ResourceManagerTests.testAccessAttributeNotExposedByWrapper: 0.0019s resourceManagerTests.ResourceManagerTests.testErrorInFactory: 0.0017s resourceManagerTests.ResourceManagerTests.testAcquireNonExistingResource: 0.0016s resourceManagerTests.ResourceManagerTests.testRequestInvalidResource: 0.0015s resourceManagerTests.ResourceManagerTests.testRequestRegrant: 0.0015s resourceManagerTests.ResourceManagerTests.testResourceAcquireInvalidTimeout: 0.0014s resourceManagerTests.ResourceManagerTests.testReleaseInvalidResource: 0.0014s resourceManagerTests.ResourceManagerTests.testSingleton: 0.0014s resourceManagerTests.ResourceManagerTests.testForceRegisterNamespace: 0.0013s resourceManagerTests.ResourceManagerTests.testListNamespaces: 0.0013s resourceManagerTests.ResourceManagerTests.testReregisterNamespace: 0.0012s resourceManagerTests.ResourceManagerTests.testRegisterInvalidNamespace: 0.0012s At the end of the usual tests if nose-timer is detected in the system. It shows the tests over 500ms in yellow and those over 5s in red. If you want to get this output, do: pip install nose-timer Change-Id: I4960b30532a84abd259fb268c7e40a1751847a96 Signed-off-by: Antoni S. Puimedon <asegurap(a)redhat.com> --- M tests/run_tests_local.sh.in M tests/testrunner.py 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/30171/1 diff --git a/tests/run_tests_local.sh.in b/tests/run_tests_local.sh.in index d19f22c..f51473d 100644 --- a/tests/run_tests_local.sh.in +++ b/tests/run_tests_local.sh.in @@ -7,4 +7,9 @@ @top_srcdir@/tests/makecert.sh fi -PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@top_srcdir@/lib:@top_srcdir@/vdsm:@top_srcdir@/client:@top_srcdir@/vdsm_api:$PYTHONPATH" "$PYTHON_EXE" @top_srcdir@/tests/testrunner.py --local-modules $@ +python -c 'import nosetimer' > /dev/null 2>&1 +if [ $? -eq 0 ]; then + EXTRA_ARGS='--with-timer' +fi + +PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@top_srcdir@/lib:@top_srcdir@/vdsm:@top_srcdir@/client:@top_srcdir@/vdsm_api:$PYTHONPATH" "$PYTHON_EXE" @top_srcdir@/tests/testrunner.py --local-modules $@ "$EXTRA_ARGS" diff --git a/tests/testrunner.py b/tests/testrunner.py index eed743f..ec24c2b 100644 --- a/tests/testrunner.py +++ b/tests/testrunner.py @@ -40,6 +40,16 @@ from nose import config from nose import core from nose import result +try: + from nosetimer.plugin import TimerPlugin +except ImportError: + timer_plugin = None +else: + timer_plugin = TimerPlugin() + timer_plugin.enabled = True + timer_plugin.timer_ok = 500 # Okay <= 500ms + timer_plugin.timer_warning = 5000 # Warn > 5s + timer_plugin.timer_no_color = False # Enable color output from testValidation import SlowTestsPlugin, StressTestsPlugin @@ -378,6 +388,8 @@ plugins=core.DefaultPluginManager()) conf.plugins.addPlugin(SlowTestsPlugin()) conf.plugins.addPlugin(StressTestsPlugin()) + if timer_plugin is not None: + conf.plugins.addPlugin(timer_plugin) runner = VdsmTestRunner(stream=conf.stream, verbosity=conf.verbosity, -- To view, visit http://gerrit.ovirt.org/30171 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4960b30532a84abd259fb268c7e40a1751847a96 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Antoni Segura Puimedon <asegurap(a)redhat.com>

6 13

Change in vdsm[master]: ImageManifest: create_image_dir
by alitke＠redhat.com 03 Jul '16

03 Jul '16

Adam Litke has uploaded a new change for review. Change subject: ImageManifest: create_image_dir ...................................................................... ImageManifest: create_image_dir Change-Id: I94a8cfdd9fb1f3c3fd4e4e2ab409bfd287829351 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/image_manifest_tests.py M vdsm/storage/image.py 2 files changed, 20 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/43557/1 diff --git a/tests/image_manifest_tests.py b/tests/image_manifest_tests.py index a2ac1c0..a27eabd 100644 --- a/tests/image_manifest_tests.py +++ b/tests/image_manifest_tests.py @@ -23,6 +23,7 @@ from testlib import VdsmTestCase, namedTemporaryDir from storage import sd, image +from storage.sdm import sdm_verb class ImageManifestTests(VdsmTestCase): @@ -33,3 +34,14 @@ manifest = image.ImageManifest(tmpdir) expected = os.path.join(tmpdir, sduuid, sd.DOMAIN_IMAGES, imguuid) self.assertEquals(expected, manifest.getImageDir(sduuid, imguuid)) + + @sdm_verb + def test_create_image_dir(self): + with namedTemporaryDir() as tmpdir: + sduuid = str(uuid.uuid4()) + imguuid = str(uuid.uuid4()) + manifest = image.ImageManifest(tmpdir) + os.makedirs(os.path.join(tmpdir, sduuid, sd.DOMAIN_IMAGES)) + manifest.create_image_dir(sduuid, imguuid) + self.assertTrue(os.path.exists(manifest.getImageDir(sduuid, + imguuid))) \ No newline at end of file diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 81b2043..5f87921 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -38,6 +38,7 @@ from threadLocal import vars import resourceFactories import resourceManager as rm +from sdm import require_sdm log = logging.getLogger('Storage.Image') rmanager = rm.ResourceManager.getInstance() @@ -97,6 +98,13 @@ """ return os.path.join(self.repoPath, sdUUID, sd.DOMAIN_IMAGES, imgUUID) + @require_sdm + def create_image_dir(self, sd_id, img_id): + img_dir = self.getImageDir(sd_id, img_id) + if not os.path.isdir(img_dir): + log.info("Create placeholder %s for image's volumes", img_dir) + os.mkdir(img_dir) + return img_dir class Image: """ Actually represents a whole virtual disk. -- To view, visit https://gerrit.ovirt.org/43557 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I94a8cfdd9fb1f3c3fd4e4e2ab409bfd287829351 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 10

Change in vdsm[master]: tests: Add a live merge functional test
by alitke＠redhat.com 03 Jul '16

03 Jul '16

Adam Litke has uploaded a new change for review. Change subject: tests: Add a live merge functional test ...................................................................... tests: Add a live merge functional test Test whether we can successfully merge the active layer. Uses lots of the functional test infrastructure! Only runs if vdsm says it can support live merge. Change-Id: Idd5a2f7eedaef9e90981256de66fc3ed21658e89 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/functional/utils.py M tests/functional/virtTests.py 2 files changed, 185 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/29824/1 diff --git a/tests/functional/utils.py b/tests/functional/utils.py index 494be98..e3cdba6 100644 --- a/tests/functional/utils.py +++ b/tests/functional/utils.py @@ -228,3 +228,41 @@ def updateVmPolicy(self, vmId, vcpuLimit): result = self.vdscli.updateVmPolicy([vmId, vcpuLimit]) return result['status']['code'], result['status']['message'] + + def getTaskStatus(self, taskId): + result = self.vdscli.getTaskStatus(taskId) + return result['status']['code'], result['status']['message'],\ + result['taskStatus'] + + def getVolumeInfo(self, sdId, spId, imgId, volId): + result = self.vdscli.getVolumeInfo(sdId, spId, imgId, volId) + return result['status']['code'], result['status']['message'],\ + result['info'] + + def createVolume(self, sdId, spId, imgId, size, volFormat, preallocate, + diskType, volId, desc, baseImgId, baseVolId): + result = self.vdscli.createVolume(sdId, spId, imgId, size, volFormat, + preallocate, diskType, volId, desc, + baseImgId, baseVolId) + return result['status']['code'], result['status']['message'],\ + result['uuid'] + + def deleteVolume(self, sdId, spId, imgId, volIds, postZero=False, + force=False): + result = self.vdscli.deleteVolume(sdId, spId, imgId, volIds, postZero, + force) + return result['status']['code'], result['status']['message'],\ + result['uuid'] + + def snapshot(self, vmId, snapDrives, snapMemVolHandle=''): + result = self.vdscli.snapshot(vmId, snapDrives, snapMemVolHandle) + return result['status']['code'], result['status']['message'] + + def merge(self, vmId, drive, base, top, bandwidth, jobId): + result = self.vdscli.merge(vmId, drive, base, top, bandwidth, jobId) + return result['status']['code'], result['status']['message'] + + def list(self, fullStatus=False, vmList=()): + result = self.vdscli.list(fullStatus, vmList) + return result['status']['code'], result['status']['message'], \ + result['vmList'] diff --git a/tests/functional/virtTests.py b/tests/functional/virtTests.py index 94ce240..b811b92 100644 --- a/tests/functional/virtTests.py +++ b/tests/functional/virtTests.py @@ -22,6 +22,7 @@ import math import tempfile import logging +import uuid from stat import S_IROTH from functools import partial, wraps @@ -32,7 +33,8 @@ from testrunner import temporaryPath from vdsm.utils import CommandPath, RollbackContext -import storageTests as storage +import storageTests +import storage from storage.misc import execCmd from utils import VdsProxy, SUCCESS @@ -109,6 +111,18 @@ return method(self, *args, **kwargs) else: raise SkipTest('KVM is not enabled') + return wrapped + + +def requireLiveMerge(method): + @wraps(method) + def wrapped(self, *args, **kwargs): + status, msg, result = self.vdsm.getVdsCapabilities() + self.assertEqual(status, SUCCESS, msg) + if result.get('liveMerge') == 'true': + return method(self, *args, **kwargs) + else: + raise SkipTest('Live Merge is not available') return wrapped @@ -227,9 +241,9 @@ @requireKVM @permutations([['localfs'], ['iscsi'], ['nfs']]) def testVmWithStorage(self, backendType): - disk = storage.StorageTest() + disk = storageTests.StorageTest() disk.setUp() - conf = storage.storageLayouts[backendType] + conf = storageTests.storageLayouts[backendType] drives = disk.generateDriveConf(conf) customization = {'vmId': '88888888-eeee-ffff-aaaa-111111111111', 'vmName': 'testVmWithStorage' + backendType, @@ -247,8 +261,8 @@ def testVmWithDevice(self, *devices): customization = {'vmId': '77777777-ffff-3333-bbbb-222222222222', 'vmName': 'testVm', 'devices': [], 'display': 'vnc'} - storageLayout = storage.storageLayouts['localfs'] - diskSpecs = storage.StorageTest.generateDriveConf(storageLayout) + storageLayout = storageTests.storageLayouts['localfs'] + diskSpecs = storageTests.StorageTest.generateDriveConf(storageLayout) pciSpecs = {'bus': '0x00', 'domain': '0x0000', 'function': '0x0', 'type': 'pci'} ccidSpecs = {'slot': '0', 'controller': '0', 'type': 'ccid'} @@ -412,3 +426,131 @@ self.vdsm.updateVmPolicy(customization['vmId'], '50') self.assertEqual(status, SUCCESS, msg) + + +@expandPermutations +class LiveMergeTest(VirtTestBase): + def _waitTask(self, taskId): + def assertTaskOK(): + status, msg, result = self.vdsm.getTaskStatus(taskId) + self.assertEqual(status, SUCCESS, msg) + self.assertEquals(result['taskState'], 'finished') + + self.retryAssert(assertTaskOK, timeout=60) + + def _waitBlockJobs(self, vmId, jobIds): + def assertJobsGone(): + status, msg, result = self.vdsm.getVmStats(vmId) + self.assertEqual(status, SUCCESS, msg) + self.assertTrue('vmJobs' in result) + self.assertTrue(all([x not in result['vmJobs'].keys() + for x in jobIds])) + + self.retryAssert(assertJobsGone, timeout=60) + + def _snapshotVM(self, vmId, drives, rollback): + snapDrives = [] + for drive in drives: + sd = drive['domainID'] + sp = drive['poolID'] + img = drive['imageID'] + vol = drive['volumeID'] + newVol = str(uuid.uuid4()) + volFormat = storage.volume.COW_FORMAT + preallocate = storage.volume.SPARSE_VOL + desc = 'snapshot for %s' % vol + + # Create volume and wait + status, msg, result = self.vdsm.getVolumeInfo(sd, sp, img, vol) + self.assertEqual(status, SUCCESS, msg) + size = result['capacity'] + diskType = result['disktype'] + + status, msg, taskId = self.vdsm.createVolume(sd, sp, img, size, + volFormat, + preallocate, diskType, + newVol, desc, img, + vol) + self.assertEqual(status, SUCCESS, msg) + self._waitTask(taskId) + undo = lambda sd=sd, sp=sp, img=img, vol=newVol: \ + self._waitTask(self.vdsm.deleteVolume(sd, sp, img, vol)[2]) + rollback.prependDefer(undo) + + snapDrives.append({'domainID': sd, + 'imageID': img, + 'volumeID': newVol, + 'baseVolumeID': vol}) + + # Create snapshot + status, msg = self.vdsm.snapshot(vmId, snapDrives) + self.assertEqual(status, SUCCESS, msg) + return snapDrives + + def _orderChain(self, vmId, dev, chain): + parentMap = {} + for vol in chain: + status, msg, info = self.vdsm.getVolumeInfo(dev['domainID'], + dev['poolID'], + dev['imageID'], vol) + self.assertEqual(status, SUCCESS, msg) + parent = info['parent'] + parentMap[vol] = parent + + vol = dev['volumeID'] + chain = list() + while True: + chain.insert(0, vol) + vol = parentMap.get(vol, '00000000-0000-0000-0000-000000000000') + if vol == '00000000-0000-0000-0000-000000000000': + break + return chain + + def _getVolumeChains(self, vmId): + chains = {} + status, msg, result = self.vdsm.list(True, (vmId,)) + self.assertEqual(status, SUCCESS, msg) + vmDef = result[0] + for dev in vmDef['devices']: + if dev['device'] != 'disk': + continue + chains[dev['imageID']] = self._orderChain(vmId, dev, + [x['volumeID'] for x in + dev['volumeChain']]) + return chains + + @requireKVM + @requireLiveMerge + def testCapable(self): + pass + + @permutations([['localfs']]) + def testMergeActiveLayer(self, backendType): + disk = storageTests.StorageTest() + disk.setUp() + conf = storageTests.storageLayouts[backendType] + drives = disk.generateDriveConf(conf) + vmId = '12121212-abab-baba-abab-222222222222' + customization = {'vmId': vmId, + 'vmName': 'testMergeActive' + backendType, + 'drives': drives, + 'display': 'vnc'} + + with RollbackContext() as rollback: + disk.createVdsmStorageLayout(conf, 3, rollback) + with RunningVm(self.vdsm, customization) as vm: + self._waitForStartup(vm, VM_MINIMAL_UPTIME) + snapDrives = self._snapshotVM(vmId, drives, rollback) + chains = {} + jobIds = [] + for drive in snapDrives: + base = drive['baseVolumeID'] + top = drive['volumeID'] + jobId = str(uuid.uuid4()) + chains[drive['imageID']] = [base, top] + status, msg = self.vdsm.merge(vmId, drive, base, top, 0, + jobId) + jobIds.append(jobId) + self._waitBlockJobs(vmId, jobIds) + actual = self._getVolumeChains(vmId) + self.assertEquals(chains, actual) -- To view, visit http://gerrit.ovirt.org/29824 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idd5a2f7eedaef9e90981256de66fc3ed21658e89 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 13

Change in vdsm[master]: vm: Support for non-ascii vm name
by Nir Soffer 03 Jul '16

03 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: vm: Support for non-ascii vm name ...................................................................... vm: Support for non-ascii vm name Libvirt does not support non-ascii vm name, failing creation of such vm. Use generated vm name based on vm id, used when vm name is not provided. For easier debugging, we log he non-ascii name when starting a vm: Non-ASCII vm name 'מכונה-ויראטואלית' And also the generaed name (same when vm name is not provided): Using generated vm name 'nb938182d-7f19-435b-9499-4cb8f1578e1b' Note: this change may break engine side, if it expects the original vm name in vdsm response. Fixing this on the engine side will avoid this issue and may be better idea. Tested with both jsonrpc and xmlrpc: - Creating vm with non-ascii name - Staring vm - Opening spice console - Stopping vm - Restarting vdsm while vm is running - Migrating to another host (broken with jsonrpc see bz1282054) Change-Id: I2386c0f98aeb8161feaf19c2862be229f73eb0df Bug-Url: https://bugzilla.redhat.com/1260131 Relates-To: https://bugzilla.redhat.com/1062943 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 17 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/70/48570/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index f9bc25c..a6d2bc8 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -299,8 +299,7 @@ self._devices = self._emptyDevMap() self._connection = libvirtconnection.get(cif) - if 'vmName' not in self.conf: - self.conf['vmName'] = 'n%s' % self.id + self.conf['vmName'] = self._normalizedVmName(params) self._guestSocketFile = self._makeChannelPath(_VMCHANNEL_DEVICE_NAME) self._qemuguestSocketFile = self._makeChannelPath(_QEMU_GA_DEVICE_NAME) self.guestAgent = guestagent.GuestAgent( @@ -380,6 +379,22 @@ break return str(idx) + def _normalizedVmName(self, params): + name = params.get('vmName') + if isinstance(name, unicode): + # Libvirt does not support non-ascii vm name + # See https://bugzilla.redhat.com/1260131 + try: + name = name.encode('ascii') + except UnicodeEncodeError: + self.log.debug("Non-ASCII vm name '%s'", name.encode('utf8')) + name = None + if name is None: + # Missing or non-ascii name + name = 'n' + params['vmId'] + self.log.debug("Using generated vm name %r", name) + return name + def _normalizeVdsmImg(self, drv): drv['reqsize'] = drv.get('reqsize', '0') # Backward compatible if 'device' not in drv: -- To view, visit https://gerrit.ovirt.org/48570 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2386c0f98aeb8161feaf19c2862be229f73eb0df Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 7

Change in vdsm[master]: automation: Enable xunit report
by Nir Soffer 03 Jul '16

03 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: automation: Enable xunit report ...................................................................... automation: Enable xunit report This used to segfault python in the past, but I think I fixed the root cause long time ago. Trying to use this again since it seems to give interesting output like the time it took to run each test. Change-Id: Ie8c09cf21e7cf4a97b7b006f2b3ebe4943ed72b9 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M automation/check-patch.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/40/46640/1 diff --git a/automation/check-patch.sh b/automation/check-patch.sh index 15707e8..1dba426 100755 --- a/automation/check-patch.sh +++ b/automation/check-patch.sh @@ -14,7 +14,7 @@ " ./autogen.sh --system --enable-hooks -make check +make check NOSE_WITH_XUNIT=1 ./automation/build-artifacts.sh -- To view, visit https://gerrit.ovirt.org/46640 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie8c09cf21e7cf4a97b7b006f2b3ebe4943ed72b9 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 18

Change in vdsm[master]: fileSD: Keep deleted images name unique
by Nir Soffer 03 Jul '16

03 Jul '16

Nir Soffer has uploaded a new change for review. Change subject: fileSD: Keep deleted images name unique ...................................................................... fileSD: Keep deleted images name unique If an image delete fails, and same image is created again, the second image cannot be deleted, since the first image occupy the deleted image name. We use now _remove_me_<imagid>_<taskid>. This make it easy to locate the failed operation in vdsm or engine logs. Change-Id: Ia0582896d7ede028e9096da1a688e3e449ab0258 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/fileSD.py 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/32/48032/1 diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index dc4a68e..7d8074a 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -34,6 +34,7 @@ import outOfProcess as oop from remoteFileHandler import Timeout from persistentDict import PersistentDict, DictValidator +from threadLocal import vars from vdsm import constants from vdsm.utils import stripNewLines from vdsm import supervdsm @@ -204,7 +205,8 @@ def deleteImage(self, sdUUID, imgUUID, volsImgs): currImgDir = self.getImagePath(imgUUID) dirName, baseName = os.path.split(currImgDir) - toDelDir = os.path.join(dirName, sd.REMOVED_IMAGE_PREFIX + baseName) + newName = "%s%s_%s" % (sd.REMOVED_IMAGE_PREFIX, baseName, vars.task.id) + toDelDir = os.path.join(dirName, newName) self.log.debug("Renaming dir %s to %s", currImgDir, toDelDir) try: self.oop.os.rename(currImgDir, toDelDir) -- To view, visit https://gerrit.ovirt.org/48032 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia0582896d7ede028e9096da1a688e3e449ab0258 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 4

Change in vdsm[master]: ipv6 hook: support ipv6addrs
by osvoboda＠redhat.com 03 Jul '16

03 Jul '16

Ondřej Svoboda has uploaded a new change for review. Change subject: ipv6 hook: support ipv6addrs ...................................................................... ipv6 hook: support ipv6addrs Change-Id: I722d0007840060cdccdb12e4fba2a2066f33c62f Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm_hooks/ipv6/README M vdsm_hooks/ipv6/ipv6.py 2 files changed, 12 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/39359/1 diff --git a/vdsm_hooks/ipv6/README b/vdsm_hooks/ipv6/README index 05bd990..a96e2c2 100644 --- a/vdsm_hooks/ipv6/README +++ b/vdsm_hooks/ipv6/README @@ -5,13 +5,18 @@ Requirements: * oVirt-3.5 (started supporting custom properties) +New in version 4.18: (?) +-------------------- +Multiple IPv6 addresses are now accepted, see Usage. Please use 'ipv6addrs' +property instead of 'ipv6addr' which is deprecated. + Preparation on a host: ---------------------- yum install vdsm-hook-ipv6 Preparation on the engine side: ------------------------------- -PROPERTIES='ipv6addr=.*;ipv6gateway=.*;ipv6autoconf=.*;dhcpv6=.*' +PROPERTIES='ipv6addrs=.*;ipv6gateway=.*;ipv6autoconf=.*;dhcpv6=.*' engine-config -s "UserDefinedNetworkCustomProperties=$PROPERTIES" --cver='3.5' Don't forget to include the names of other custom network properties you may @@ -20,9 +25,10 @@ Usage: ------ In the oVirt UI open the 'Setup Host Networks' dialog. Proceed to editing -a desired logical network's properties. Among them you will find 'ipv6addr' -and 'ipv6gateway', which accept custom IPv6 addresses, plus 'ipv6autoconf' -and 'dhcpv6', which accept '0', '1' or 'false', 'true'. +a desired logical network's properties. Among them you will find 'ipv6addrs' +(which accepts a space-separated list of IPv6 addresses), 'ipv6gateway' +(a single address), plus 'ipv6autoconf' and 'dhcpv6', which accept '0', '1' or +'false', 'true'. You may be warned that the network is in use when confirming the 'Setup Host Networks' dialog. Make sure to stop any VMs that are using the network. diff --git a/vdsm_hooks/ipv6/ipv6.py b/vdsm_hooks/ipv6/ipv6.py index 6c00475..9385fab 100644 --- a/vdsm_hooks/ipv6/ipv6.py +++ b/vdsm_hooks/ipv6/ipv6.py @@ -34,7 +34,8 @@ def _process_network(attrs): - for property_name in ('ipv6addr', 'ipv6gateway', 'ipv6autoconf', 'dhcpv6'): + for property_name in ('ipv6addrs', 'ipv6addr', 'ipv6gateway', + 'ipv6autoconf', 'dhcpv6'): value = attrs['custom'].get(property_name) if value is not None: attrs[property_name] = value -- To view, visit https://gerrit.ovirt.org/39359 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I722d0007840060cdccdb12e4fba2a2066f33c62f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

5 20

Change in vdsm[master]: sourceroute: split gathering of routes and rules off configu...
by osvoboda＠redhat.com 03 Jul '16

03 Jul '16

Ondřej Svoboda has uploaded a new change for review. Change subject: sourceroute: split gathering of routes and rules off configure() and remove() ...................................................................... sourceroute: split gathering of routes and rules off configure() and remove() The code is too complex (for me) to understand. I want to remove the use of network configurators from the module because they actually call sourceroute classes' methods only to be called back: Ifcfg.configureNic, Configurator._addSourceRoute, StaticSourceRoute.configure, Ifcfg.configureSourceRoute Ifcfg.removeNic, Ifcfg._ifaceDownAndCleanup, Configurator._removeSourceRoute, StaticSourceRoute.remove (or DynamicSourceRoute), Ifcfg.removeSourceRoute Change-Id: I69d8949afbd5b8ba2cf14930533cda99a4580eb0 Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm/network/sourceroute.py 1 file changed, 30 insertions(+), 22 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/93/41593/1 diff --git a/vdsm/network/sourceroute.py b/vdsm/network/sourceroute.py index 5f06de0..d7916ed 100644 --- a/vdsm/network/sourceroute.py +++ b/vdsm/network/sourceroute.py @@ -25,11 +25,7 @@ from vdsm import netinfo from vdsm.constants import P_VDSM_RUN -from vdsm.ipwrapper import IPRoute2Error -from vdsm.ipwrapper import Route -from vdsm.ipwrapper import routeShowTable -from vdsm.ipwrapper import Rule -from vdsm.ipwrapper import ruleList +from vdsm.ipwrapper import IPRoute2Error, Route, routeShowTable, Rule, ruleList from vdsm.utils import rmFile @@ -67,23 +63,32 @@ Rule(destination=self._network, table=self._table, srcDevice=self.device)] - def configure(self): + def _prepare_configuration(self): logging.info(("Configuring gateway - ip: %s, network: %s, " + "subnet: %s, gateway: %s, table: %s, device: %s") % (self._ipaddr, self._network, self._mask, self._gateway, self._table, self.device)) - routes = self._buildRoutes() - rules = self._buildRules() + self.routes = self._buildRoutes() + self.rules = self._buildRules() + def configure(self): + self._prepare_configuration() try: - self._configurator.configureSourceRoute(routes, rules, self.device) + self._configurator.configureSourceRoute(self.routes, self.rules, + self.device) except IPRoute2Error as e: logging.error('ip binary failed during source route configuration' ': %s', e.message) + def _prepare_removal(self): + self.routes = None + self.rules = None + def remove(self): - self._configurator.removeSourceRoute(None, None, self.device) + self._prepare_removal() + self._configurator.removeSourceRoute(self.routes, self.rules, + self.device) class DynamicSourceRoute(StaticSourceRoute): @@ -157,20 +162,23 @@ return rules + def _prepare_removal(self): + self.rules = self._getRules(self.device) + if self.rules: + self.table = self._getTable(self.rules) + if self.table: + self.routes = self._getRoutes(self.table) + def remove(self): logging.info("Removing gateway - device: %s", self.device) - - rules = self._getRules(self.device) - if rules: - table = self._getTable(rules) - if table: - try: - self._configurator.removeSourceRoute( - self._getRoutes(table), rules, - self.device) - except IPRoute2Error as e: - logging.error('ip binary failed during source route ' - 'removal: %s' % e.message) + self._prepare_removal() + if self.rules and self.table: + try: + self._configurator.removeSourceRoute( + self.routes, self.rules, self.device) + except IPRoute2Error as e: + logging.error('ip binary failed during source route ' + 'removal: %s' % e.message) @staticmethod def _isLibvirtInterfaceFallback(device): -- To view, visit https://gerrit.ovirt.org/41593 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I69d8949afbd5b8ba2cf14930533cda99a4580eb0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

3 6

Change in vdsm[master]: sourceroute: move exception handling to the respective confi...
by osvoboda＠redhat.com 03 Jul '16

03 Jul '16

Ondřej Svoboda has uploaded a new change for review. Change subject: sourceroute: move exception handling to the respective configurator ...................................................................... sourceroute: move exception handling to the respective configurator Change-Id: I579b1789522eaed025d1c059bcf7a900e499ef32 Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm/network/configurators/iproute2.py M vdsm/network/sourceroute.py 2 files changed, 16 insertions(+), 19 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/94/41594/1 diff --git a/vdsm/network/configurators/iproute2.py b/vdsm/network/configurators/iproute2.py index c260bfe..bec32f7 100644 --- a/vdsm/network/configurators/iproute2.py +++ b/vdsm/network/configurators/iproute2.py @@ -201,17 +201,22 @@ @staticmethod def configureSourceRoute(routes, rules, device): - for route in routes: - routeAdd(route) - - for rule in rules: - ruleAdd(rule) + try: + for route in routes: + routeAdd(route) + for rule in rules: + ruleAdd(rule) + except IPRoute2Error as e: + logging.error('Source route configuration failed: %s', e.message) @staticmethod def removeSourceRoute(routes, rules, device): # Routes are removed automatically when the link goes down. - for rule in rules: - ruleDel(rule) + try: + for rule in rules: + ruleDel(rule) + except IPRoute2Error as e: + logging.error('Source route removal failed: %s' % e.message) class ConfigApplier(object): diff --git a/vdsm/network/sourceroute.py b/vdsm/network/sourceroute.py index d7916ed..1fb9626 100644 --- a/vdsm/network/sourceroute.py +++ b/vdsm/network/sourceroute.py @@ -74,12 +74,8 @@ def configure(self): self._prepare_configuration() - try: - self._configurator.configureSourceRoute(self.routes, self.rules, - self.device) - except IPRoute2Error as e: - logging.error('ip binary failed during source route configuration' - ': %s', e.message) + self._configurator.configureSourceRoute(self.routes, self.rules, + self.device) def _prepare_removal(self): self.routes = None @@ -173,12 +169,8 @@ logging.info("Removing gateway - device: %s", self.device) self._prepare_removal() if self.rules and self.table: - try: - self._configurator.removeSourceRoute( - self.routes, self.rules, self.device) - except IPRoute2Error as e: - logging.error('ip binary failed during source route ' - 'removal: %s' % e.message) + self._configurator.removeSourceRoute(self.routes, self.rules, + self.device) @staticmethod def _isLibvirtInterfaceFallback(device): -- To view, visit https://gerrit.ovirt.org/41594 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I579b1789522eaed025d1c059bcf7a900e499ef32 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

3 7

Change in vdsm[master]: register: handle non uuid machines
by Douglas Schilling Landgraf 03 Jul '16

03 Jul '16

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: register: handle non uuid machines ...................................................................... register: handle non uuid machines If user machines doesn't provide UUID data this patch collects a random UUID from system. Change-Id: I9429e46895a73178cf459d5ced5d39499e993922 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1268854 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/tool/register.py 1 file changed, 11 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/85/47385/1 diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py index 47c8412..d088bfc 100644 --- a/lib/vdsm/tool/register.py +++ b/lib/vdsm/tool/register.py @@ -256,7 +256,17 @@ if self.vdsm_uuid: self.uuid = self.vdsm_uuid else: - self.uuid = getHostUUID(legacy=False) + uuid_node = getHostUUID(legacy=False) + if uuid_node is None: + self.logger.debug("Collecting random UUID from system") + random_uuid_path = '/proc/sys/kernel/random/uuid' + if os.path.exists(random_uuid_path): + with open(random_uuid_path, 'r') as f: + self.uuid = f.readline().strip() + else: + raise RuntimeError("Unable to determine host UUID!") + else: + self.uuid = uuid_node self.url_reg += "&uniqueId={u}".format(u=self.uuid) -- To view, visit https://gerrit.ovirt.org/47385 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9429e46895a73178cf459d5ced5d39499e993922 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

4 6

Change in vdsm[master]: vm: do not call before_vm_create in dehibernation
by fromani＠redhat.com 28 Jun '16

28 Jun '16

Francesco Romani has uploaded a new change for review. Change subject: vm: do not call before_vm_create in dehibernation ...................................................................... vm: do not call before_vm_create in dehibernation For reasons unknown, VDSM calls the before_vm_create hook also in the dehibernation path. This is wasteful and useless at best, and most likely misleading and wrong, because VDSM logs the XML it produced, which is *not* what is gonna be used in the dehibernation path. The hook is fed with phony data, and the output of the hook is discarded. This patch dispel the lie and avoid to call the before_vm_create hook in the dehibernation path. Change-Id: I4c1377346b1ae63a1feb15e1458b5141c255c849 X-Backport-To: 3.6 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 0 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/74/45074/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index c54853a..b77b9df 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1887,8 +1887,6 @@ hooks.before_vm_dehibernate(srcDomXML, self.conf, {'FROM_SNAPSHOT': str(fromSnapshot)}) - # TODO: this silliness is only to preserve the old behaviour - hooks.before_vm_start(self._buildDomainXML(), self.conf) # TODO: this is debug information. For 3.6.x we still need to # see the XML even with 'info' as default level. self.log.info(srcDomXML) -- To view, visit https://gerrit.ovirt.org/45074 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4c1377346b1ae63a1feb15e1458b5141c255c849 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 8

Change in vdsm[master]: api: allow setLogLevel to tune a specific logger
by fromani＠redhat.com 28 Jun '16

28 Jun '16

Francesco Romani has uploaded a new change for review. Change subject: api: allow setLogLevel to tune a specific logger ...................................................................... api: allow setLogLevel to tune a specific logger the not widely known setLogLevel VDSM verb allows to dinamically tune the log level of the root logger, until the next restart of VDSM. This patch extends the API to let the client tune any specific logger. Change-Id: I8f40488fac04031552f36b9de026a0062ab81db0 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py M vdsm/rpc/vdsmapi-schema.json 3 files changed, 12 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/26/38426/1 diff --git a/vdsm/API.py b/vdsm/API.py index 85478b1..593f336 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1360,7 +1360,7 @@ return {'status': doneCode, 'info': stats} - def setLogLevel(self, level): + def setLogLevel(self, level, name=None): """ Set verbosity level of vdsm's log. @@ -1382,8 +1382,10 @@ except KeyError: return errCode['unavail'] else: - logging.info('Setting loglevel to %s (%d)', level, log_level) - _set_log_level(logging.getLogger(), log_level) + logger = logging.getLogger(name) + logging.info('Setting loglevel on %s to %s (%d)', + logger.name, level, log_level) + _set_log_level(logger, log_level) return {'status': doneCode} # VM-related functions diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index b01b661..ea0e0c3 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -583,9 +583,9 @@ return api.fenceNode(addr, port, agent, username, password, action, secure, options, policy) - def setLogLevel(self, level): + def setLogLevel(self, level, name=None): api = API.Global() - return api.setLogLevel(level) + return api.setLogLevel(level, name) def setMOMPolicy(self, policy): api = API.Global() diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 1d919e8..854e7aa 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -4030,10 +4030,14 @@ # # @level: The new verosity level desired # +# @name: #optional If present, change settings only for the logger +# with this name. Otherwise, change settings for the top-level +# logger (new in version 4.17.0). +# # Since: 4.10.0 ## {'command': {'class': 'Host', 'name': 'setLogLevel'}, - 'data': {'level': 'LoggingLevel'}} + 'data': {'level': 'LoggingLevel', '*name': 'str'}} ## # @Host.setSafeNetworkConfig: -- To view, visit https://gerrit.ovirt.org/38426 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8f40488fac04031552f36b9de026a0062ab81db0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

6 36

Change in vdsm[master]: API: streamline and make setLogLevel correct
by fromani＠redhat.com 28 Jun '16

28 Jun '16

Francesco Romani has uploaded a new change for review. Change subject: API: streamline and make setLogLevel correct ...................................................................... API: streamline and make setLogLevel correct According to the schema, setLogLevel should accept a log level string ('DEBUG', 'INFO',...). But the code actually blindly casted the given value to int(), and this suggests the code actually expected an integer value. To make things a bit user friendlier and comformant to schema, change the argument to actually be a log level in string format. Along the way, this patch streamlines the implementation of setLogLevel and makes it simpler. Change-Id: Iaddbb12d13bdbaa7a02255ab209da11e42d2ea97 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/API.py 1 file changed, 21 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/25/38425/1 diff --git a/vdsm/API.py b/vdsm/API.py index 5cbda50..85478b1 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1369,13 +1369,22 @@ Doesn't survive a restart """ - logging.info('Setting loglevel to %s', level) - handlers = logging.getLogger().handlers - [fileHandler] = [h for h in handlers if - isinstance(h, logging.FileHandler)] - fileHandler.setLevel(int(level)) + LEVELS = { + 'DEBUG': logging.DEBUG, + 'INFO': logging.INFO, + 'WARNING': logging.WARNING, + 'ERROR': logging.ERROR, + 'CRITICAL': logging.CRITICAL + } - return {'status': doneCode} + try: + log_level = LEVELS[level] + except KeyError: + return errCode['unavail'] + else: + logging.info('Setting loglevel to %s (%d)', level, log_level) + _set_log_level(logging.getLogger(), log_level) + return {'status': doneCode} # VM-related functions def getVMList(self, fullStatus=False, vmList=()): @@ -1772,3 +1781,9 @@ logging.warn("options %s is deprecated. Use %s instead" % (k, _translationMap[k])) options[_translationMap[k]] = options.pop(k) + + +def _set_log_level(logger, log_level): + for handler in logger.handlers: + if isinstance(handler, logging.FileHandler): + handler.setLevel(log_level) -- To view, visit https://gerrit.ovirt.org/38425 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaddbb12d13bdbaa7a02255ab209da11e42d2ea97 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

6 38

Change in vdsm[master]: virt: Correct VM state before vm.cont() in _recover()
by smelamud＠redhat.com 16 Jun '16

16 Jun '16

Shmuel Leib Melamud has uploaded a new change for review. Change subject: virt: Correct VM state before vm.cont() in _recover() ...................................................................... virt: Correct VM state before vm.cont() in _recover() In SourceThread._recover(), if error occured while hibernating a VM, set VM status to PAUSED before calling vm.cont(). Otherwise, vm.cont() will fail to run in SAVING_STATE state and the VM will be left paused. Change-Id: I5b1c7b4eecacf87ece48dc563fd2da294af0510b Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1238536 Signed-off-by: Shmuel Melamud <smelamud(a)redhat.com> --- M vdsm/virt/migration.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/27/47527/1 diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py index 32caccb..f950a6f 100644 --- a/vdsm/virt/migration.py +++ b/vdsm/virt/migration.py @@ -213,6 +213,7 @@ self.log.exception("Failed to destroy remote VM") # if the guest was stopped before migration, we need to cont it if self.hibernating: + self._vm.lastStatus = vmstatus.PAUSED self._vm.cont() # either way, migration has finished self._vm.lastStatus = vmstatus.UP -- To view, visit https://gerrit.ovirt.org/47527 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5b1c7b4eecacf87ece48dc563fd2da294af0510b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shmuel Leib Melamud <smelamud(a)redhat.com>

4 9

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches April 2016