autoqa-devel July 2010

autoqa-devel@lists.fedorahosted.org

7 participants
85 discussions

Upgrade Path
by Josef Skladanka 21 Jul '10

21 Jul '10

Hey, gang! Yesterday, Kamil, Vojta and me were discussing some Upgrade Path issues, since we're not really sure, what are the problematic situations we need to catch. But from the discussion below the #123 we have come to this problem we're not really sure about: There is a package Foo in Fedora-Updates repository. Lets say: [updates] F13: foo-1.1 F12: foo-1.0.3 F11: foo-1.0 And the maintainer/developer/whoever decides to deliver new version foo-2.0 in all of these, so, he adds foo-1.2 into the all respective Updates-Testing repos. [updates-testing] F13: foo-1.2 F12: foo-1.2 F11: foo-1.2 // Note, that updates-testing is not enabled by default. Now, let's imagine a user on F11, who has Updates-testing repository enabled, and installs foo-1.2. Then he disables the testing repo, and decides to upgrade to F12 (which has only access to foo-1.0.3). Is this still breaking the upgrade path, or not? From our point of view, it should not. The thing is, that foo-1.2 needs to stay in bodhi for certain amount of time (lets say a week), before it gets pushed from updates-testing to updates. If the before mentioned scheme broke the upgrade path, then the workflow would need to look like this: (day 0): add foo-1.2 to F13 updates-testing (day 1-6): wait in koji (day 7): push from F13 updates-testing to F13 updates (day 7): add foo-1.2 to F12 updates-testing ... You can see the terrible delay... And yes, i know that karma can make it push faster, but still, you need to work sequentialy for all the releases, which does not seem to be OK. Joza

3 4

[AutoQA] #115: Update post-tree-compose to recognize new stage2 URL
by fedora-badges 21 Jul '10

21 Jul '10

#115: Update post-tree-compose to recognize new stage2 URL -------------------------+-------------------------------------------------- Reporter: jlaska | Owner: Type: enhancement | Status: new Priority: major | Milestone: Automate installation test plan Component: watchers | Version: 1.0 Keywords: | -------------------------+-------------------------------------------------- During Fedora 13, release engineering will provide schedule drops of rawhide for automated testing against rats_install. The provided data will include a compose tree (without packages). For proper automated testing of these images, post-tree-compose will need to monitor a new URL for testing (likely candidate is http://alt.fedoraproject.org/pub/alt/stage/rawhide-testing) In addition, rats_install will need to support installing with stage2 and the package repo in different locations. That will be tracked in another ticket. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/115> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #176: Initscript review: rsyslog
by fedora-badges 21 Jul '10

21 Jul '10

#176: Initscript review: rsyslog ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: minor | Milestone: Initscripts review Component: tests | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- Check the test according to the <https://fedorahosted.org/autoqa/wiki/initscripts> -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/176> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 4

Re: [PATCH 4/4] Modified templates for all hooks for new autoqa script
by Will Woods 21 Jul '10

21 Jul '10

----- "Vojtěch Aschenbrenner" <vaschenb(a)redhat.com> wrote: > -# post-bodhi-update tests can expect the following variables from > autoqa: > -# title: title of the update request > -# updateid: Unique ID of the update request (may be empty) > -# targettag: target koji tag for this update > -# envrs: comma-separated list of package envrs in this update > request Removing the argument list from the template is probably not a good idea, since the templates are a primary source of documentation for the test hook. It'd be better to reword the comments to something like: "post-bodhi-update tests can expect the following keys to be present in the autoqa_args dict." -w

2 1

[AutoQA] #198: complete DVD auto install test driver
by fedora-badges 21 Jul '10

21 Jul '10

#198: complete DVD auto install test driver -------------------+-------------------------------------------------------- Reporter: liam | Owner: liam Type: task | Status: new Priority: major | Milestone: Automate installation test plan Component: tests | Version: 1.0 Keywords: | -------------------+-------------------------------------------------------- We are developing dvd auto install test driver: http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=tests/anaconda/dvd_i… to meet our auto install requirement: https://fedoraproject.org/wiki/Is_anaconda_broken_roadmap The tests can run like: PYTHONPATH=../../lib/python/ python dvd_install.py -i /path/to/Fedora-13-x86_64-DVD.iso -k /path/to/ks.cfg Issues we are facing: 1. stage2 to located install.img may can not be achieved in this version, more details see: https://bugzilla.redhat.com/show_bug.cgi?id=609379 2. the minimon can not get logs when provide a local ks file and the network is not enabled in ks file. In this case, the network would not active during install. the minimon should configure the IP address for guest when start to run in guest and did not find IP? 3. Optimize the lib code in anaconda.py, to inherit the class VirtGuest and extend some methods like Will Woods said last time. Some methods may need to rewrite when write cd_insall.py 4. In order to support some install like: updates, graphic/test/vnc, ksdevice, ip/dns/netmask/gateway, we may need to add some options to support them.But this also may confuse testers. So I only provide a "-x" option to support them. If tester wants to support them, can run like: PYTHONPATH=../../lib/python/ python dvd_install.py -i /path/to/Fedora-13-x86_64-DVD.iso -k /path/to/ks.cfg -x "serial console=tty0 console=ttyS ksdevice=eth0 ip=192.168.1.2 netmask=255.255.255.0 updates=http://url/to/updates". Make sense? -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/198> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #129: create auto install test script which maps 1x1 to the install method.
by fedora-badges 21 Jul '10

21 Jul '10

#129: create auto install test script which maps 1x1 to the install method. -------------------+-------------------------------------------------------- Reporter: liam | Owner: Type: task | Status: new Priority: major | Milestone: Automate installation test plan Component: tests | Version: 1.0 Keywords: | -------------------+-------------------------------------------------------- Each install method should have an install test script, For example: * Single ISO install - dvd_install.py * Multi ISO install - cd_install.py * Hard Drive ISO install - hdiso_install.py * NFS install - nfs_install.py * NFS ISO install - nfsiso_install.py * HTTP/FTP install - url_install.py <-- similar to rats_install.py now we need to create these scripts, but each of the script should share code as much as possible. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/129> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

Re: [PATCH] add helloworld test
by Kamil Paral 20 Jul '10

20 Jul '10

----- "Vojtěch Aschenbrenner" <vaschenb(a)redhat.com> wrote: > On Wed, Jul 14, 2010 at 05:41:29PM -0400, Will Woods wrote: > > job.run_test('helloworld', config=autoqa_conf, **locals()) > > > > (doing **locals() passes the contents of locals as if they had all > been > > given to the function as "var_name=var_value" pairs) > > This is problem, because locals() will pass many parameters (not only > ours), output included at the end of mail (sorry for LONGLONG mail). > > > Probably we want *args as well as **kwargs here, just in case. > > OK, *args included now, but here is problem with this mess in > arguments > from locals(). I didn't occur to me that there will be some many local variables. I think we one several choices: == Temporary solutions == 1. Leave helloworld available just for post-koji-build 2. Test the presence of all variables (defined for any hook) in helloworld control file and pass to the job only those that are currently available. This means more work when maintaining this test. == Proper solutions == 3. Pass all autoqa variables into the control file as a dictionary, instead of a pile of variables. We have talked about this some time ago. It will need a modification of all the test cases and some tweaks in autoqa harness. But overall it shouldn't be such a hard change. Vojtěch could do it easily I believe. What do you think?

5 12

aqc-create.py: install autoqa clients easily
by Kamil Paral 19 Jul '10

19 Jul '10

Hello, I have created a simple python script to simplify installation of autoqa clients into virtual machines. As our autoqa efforts progress (setting up staging servers etc) it is quite common task to do, so this script will save you a lot of manual labor. Script is called "aqc-create.py" ("_Auto_QA _client - create") and is available in 'aqc' branch in utils/ directory in our git: http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=utils;hb=aqc Usage is simple: $ aqc-create.py -n Example will create Fedora 13 virtual machine with <your-arch> arch, 512MB RAM and storage /dev/vg_autoqa/Example. $ aqc-create.py -n Example2 -f 12 -a i386 -r 1024 --disk /tmp/Example2.img -v -d will create Fedora 12 machine with i386 arch, 1024MB RAM and storage /tmp/Example2.img. It will be verbose and it will not attach you to the serial console of the new machine, so the installation will run on background. More options are available, see --help. The installation is fully automatized and it uses kickstart generated by simple wsgi application available in utils/aqc-ks/ directory. It is currently deployed at our staging server: * help usage: http://test1185.test.redhat.com/wsgi/aqc-ks/aqc-ks * example usage: http://test1185.test.redhat.com/wsgi/aqc-ks/aqc-ks?release=13&arch=x86_64 (This is just an implementation detail, the script uses it automatically and you don't have to care about it.) aqc-create.py will do these things for you: * call virt-install with correct options (kickstart path, serial console redirection, boot kernel location, etc) * generate a kickstart * add jlaska's autoqa repo * install autotest and autoqa packages and disable their cron tasks * add autotest-server's ssh certificate to /root/.ssh/authorized_keys (currently the staging server's certificate is added by default) The client is fully prepared automatically and all you have to do at the autotest-server is call these commands: $ ssh root@<hostname> 'true' # just to add the fingerprint to known_hosts $ atest host create -t x86_64 -b fc13,virt <hostname> # change labels if needed (If you have ideas how to get rid of the first line, comments are welcome.) So, please comment and report problems, but I have used it many times in the last few days and it seems to be working really well. Maybe we can even use it in our tutorial wiki pages, it may help beginners to start with AutoQA. I'll try to document it later. If you think this script is valuable, I would like to integrate it into our master branch. Do you think the name (aqc-create/aqc-ks) and the location (utils/) is fine, or do you have other suggestions? Or maybe we want to have it stored somewhere else than in autoqa source code? Let me know. Kamil

2 2

Re: [PATCH] add autotest labels support
by Kamil Paral 19 Jul '10

19 Jul '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > So. I think the sensible thing to do might be to have a new file > where > we can store/override test metadata - things like labels and hooks. > Yes, > this means another file for each test (boo! more files!) but if we're > smart about the file format/syntax we won't need to add any others. I > hope. Ok, you bought me in. I have meditated over all the possible stuff we could use the new file for and this is came out of my fantasies: control.autoqa: ============================================= # This file can expect some variables pre-defined: # hook: name of the hook to run this test # archs: list of host archs to run this test on # execute: whether to execute this test at all, by default True # all test arguments (envr, name, kojitag, etc) # 1. Metadata definitions # a) Autotest labels needed for this test to run labels = ['virt'] labels.append(envr.split('.')[-1]) # distro part of envr (like 'fc13') # b) Host architectures override # Some tests may need to override the default set of pre-selected # host architectures. That is logical, because the watcher can't know # all the specifics of a certain test, so the architecture selection # may not be optimal. For example some tests need to be executed only # once on an arbitrary architecture, even though the packages are built # for several architectures. Rpmlint as an example of such a test. # These tests can override the default selection here. archs = ['noarch'] # this is rpmlint, we will test all archs at once, # no need to run it on several hosts # 2. Decision whether to run this test at all # This is a place where a test may decide whether it should be run # under specified conditions, or just skipped. There might be many # reasons why to skip this test: # - it doesn't make sense to execute this test under current hook # - the test doesn't support current combination of specified # arguments # - combination of the above (it doesn't make sense to run this test # under this hook with these arguments) # If you don't want to run this test, just set 'execute' to False # a) Hook decision supported_hooks = ['post-koji-build', 'post-bodhi-build'] if hook not in supported_hooks: execute = False # b) Arguments decision # this is an initscripts test, we support only a few packages where # initscripts should be tested. it doesn't make sense to run this # test for all other packages. therefore if we don't have a testfile # for this package in our tests/ subdirectory, just skip this test if name not in os.listdir('tests'): # not supported execute = False ============================================= (EOF) <implementation> In the autoqa harness we would just evaluate such control.autoqa file for every test available and then look at some output variables (execute, archs, labels) and schedule jobs accordingly. The watchers would no longer pass testlist option to autoqa, that would be resolved inside autoqa script. </implementation> It's good to mention: * That was just an example file. Real files may be much simpler or much more complicated. * I have tried to come up with all the important stuff we might need, but it's certain we will discover further metadata/decision items in the future. And some questions: 1. Are we able to do all this with just a template/config file? Personally the security risk of scripting language seems negligible to me and defining this stuff with config files seems overcomplicated to me (and some parts would have to be left out completely). 2. This new file seems to solve several of our issues at once, I think it's quite neat (self-applause! booo!) :) But is also seems like a major architecture overhaul. Do we want to do it right now? I'll gladly volunteer, but first we must decide - do we want to concentrate purely on depcheck/other PUATP stuff right now and just add a quick dirty labels support (with plans of doing it properly in the future), or is this worth doing properly right now? > > Ah, true. But if you're iterating over the keys of a dict, you can > do: > for key in vars.keys(): > rather than copying the entire dict. Err, I thought it returns an iterator, but it returns a standard list. One is still learning, isn't he? :) Patched, thanks.

3 2

Re: [PATCH] add autotest labels support
by Kamil Paral 16 Jul '10

16 Jul '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Wed, 2010-07-14 at 05:23 -0400, Kamil Paral wrote: > > > Personally the security risk of scripting language seems negligible > > to me and defining this stuff with config files seems > overcomplicated > > to me (and some parts would have to be left out completely). > > I'm inclined to agree. Let's go with actual Python code, but try to > keep > it simple - so test authors don't have to depend on Super-Clever QA > Boffins like us just to get a test working (heh). > > > 2. This new file seems to solve several of our issues at once, I > think > > it's quite neat (self-applause! booo!) :) But is also seems like a > > major architecture overhaul. Do we want to do it right now? > > > > I'll gladly volunteer, but first we must decide - do we want to > > concentrate purely on depcheck/other PUATP stuff right now and just > > > add a quick dirty labels support (with plans of doing it properly > > in the future), or is this worth doing properly right now? > > I like it, and I think it's worth doing this properly right now - I > think we had identified this as one of the prerequisites for the > depcheck/PUATP work anyway. > > (Also, I think we have some time in the schedule to do foundation > work, > since Alpha isn't until September 21 or so.) > > The thing I wonder about is this: if we're going to exec the > 'control.autoqa' file as-is for each test, don't we risk having each > file stomp on the variables set by the previous one? It seems like > we'll > end up needing to maintain different environment/scope/namespace for > each file. So maybe we want to be able to import them as normal > python > modules, rather than manipulating the global/local namespace > ourselves? Oh, that shouldn't be a problem (at least if I understand your concerns properly), exec statement supports execution in a custom (dictionary handled) namespace: http://docs.python.org/reference/simple_stmts.html#the-exec-statement I already use it in the provided patch (autoqa, line 106). So we can have a new dictionary for every test we want to evaluate. > > We already import the 'hook.py' for each hook, so there's some > precedent > there, but I'd be happy with whichever way you think is cleanest. > > And thanks for volunteering - I'm looking forward to seeing how it > works! I will try to provide some sample patch and we can fine-tune the details then. > > -w > > _______________________________________________ > autoqa-devel mailing list > autoqa-devel(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/autoqa-devel

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel July 2010