autoqa-devel July 2010

autoqa-devel@lists.fedorahosted.org

7 participants
85 discussions

(mostly) Out of Office until July 22
by Will Woods 15 Jul '10

15 Jul '10

Hey folks, I'm going to be out of the office (going to NYC for HOPE[1]) until July 22. I know we have a lot of work in-progress: * helloworld test / autoqa-args-in-a-dict / multihook tests * autoqa labels / control.autoqa * AutoQATest base class * depcheck improvements * automation installation work * new test ideas coming in, watcher updates, etc.. So! I'll try to check in as much as I can, and I'll be happy to answer questions anyone sends by email - just know it might take me a day or two to respond[2] - and if I can get time I'll still review patches sent to the list. But if something REALLY needs to get reviewed/merged to autoqa master immediately, well: if Kamil and Josef both think it's a good idea, go for it. Okay. See you next week! -w [1] http://thenexthope.org/ [2] "How's that different from any other day, Will?" Hm. Good point.

1 0

[PATCH] add helloworld test
by Vojtěch Aschenbrenner 15 Jul '10

15 Jul '10

Hello, I created new test for AutoQA named 'helloworld', ticket #195. It's simple test, that will only forward parameters that were passed in control file. Main purpose is to show, how to write new tests for AutoQA. Patch included. --- >From 78a996deb74201984a42d3d3783c1a47e213396f Mon Sep 17 00:00:00 2001 From: vaschenb <vaschenb(a)redhat.com> Date: Tue, 13 Jul 2010 15:37:26 +0200 Subject: [PATCH] 195: Add helloworld test for AutoQA --- tests/helloworld/control | 18 +++++++++++++++++ tests/helloworld/helloworld.py | 41 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 0 deletions(-) create mode 100644 tests/helloworld/control create mode 100644 tests/helloworld/helloworld.py diff --git a/tests/helloworld/control b/tests/helloworld/control new file mode 100644 index 0000000..af19252 --- /dev/null +++ b/tests/helloworld/control @@ -0,0 +1,18 @@ +# vim: set syntax=python +TIME="SHORT" +AUTHOR = "Vojtech Aschenbrenner <vaschenb(a)redhat.com>" +DOC = """ +This test runs helloworld. It will only print params what it will get. +Main purpose is to show, how to write tests. +""" +NAME = 'helloworld' +TEST_TYPE = 'CLIENT' # SERVER can be used for tests that need multiple machines +TEST_CLASS = 'General' +TEST_CATEGORY = 'Functional' + +# post-koji-build tests can expect the following variables from autoqa: +# envr: package NVR (required, epoch can be skipped) +# name: package name +# kojitag: koji tag applied to this package + +job.run_test('helloworld', name=name, envr=envr, kojitag=kojitag, config=autoqa_conf) diff --git a/tests/helloworld/helloworld.py b/tests/helloworld/helloworld.py new file mode 100644 index 0000000..6d1bdf8 --- /dev/null +++ b/tests/helloworld/helloworld.py @@ -0,0 +1,41 @@ +# +# Copyright 2010, Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# Author: Vojtech Aschenbrenner <vaschenb(a)redhat.com> + +from autotest_lib.client.bin import test +from autotest_lib.client.bin.test_config import config_loader +import autoqa.util +import os + +class helloworld(test.test): + version = 1 # increment if setup() changes + + def initialize(self, envr, config): + self.config = config_loader(config, self.tmpdir) + self.autotest_url = autoqa.util.make_autotest_url(self.config) + + def setup(self): + pass + + def run_once(self, **kwargs): + self.arguments = kwargs + print "===Printing passed params===" + for self.name, self.arg in self.arguments.iteritems(): + print "%s = %r" % (self.name, self.arg) + print "%s = %r" % ("autotest_url", self.autotest_url) + print "===End of passed params===" -- 1.7.1.1

2 2

[PATCH] Create a common base class for autoqa tests
by Josef Skladanka 14 Jul '10

14 Jul '10

Hello gang, while I was playing around with ResultsDB, I've come to some issues, which bothered me to the level I wrote a fix :) 1) Common base class for the tests ----------------------------------- [patch 0001 - lib/python/test.py] While rewriting the tests, so they store the results in resultsdb, i found myself doing repetitive work - i needed to write the same code to __all__ the test files, which is a boring job, which i do not want to do ever again :) Fortunately this can be solved by creating a common parent class (lets call it BaseTest) for all the tests. This class will then implement all the repetitive code (e.g. sending the email, loading config...), and only local "specialities" need to be handled in the test. The main thought behind the BaseTest is that if we setup few common variables (result, summary, highlights, outputs) as instance variables, we can then use the postprocess function (which is called after the run_once successfully [and note the 'successfully' for later] ends). [note] This is also a pre-step to switching to ResultsDB as all the changes required to add basic resultsdb storage is about 7 lines in the BaseTest. [/note] [patch 0002] For the email's sake in current 'send email to results list' model, the postprocess fuction implements the email-sending. Summary is the subject and highlights + outputs are body. The self.mail_to list is filled in the tests which have the "inform package owners" fuctionality. 2) ExceptionCatcher decorator ----------------------------- [patch 0001 - lib/python/decorators.py] What the heck is this?!? Well, you remember, that the 'postprocess' function is called only if the run_once ends well (i.e. no exception is thrown from the run_once). This, of course, is absolutely not OK, if one wants to have some common "out" point - e.g. call the postprocess function. This decorator then watches over the decorated function, and if exception is raised, the decorator stores it, calls the 'exception_happened' function (which is given as parameter to the decorator, see the docstring for more detail), and then re-raises the exception. This practically gives us the opportunity to call the postprocess method. In fact, I call "FOOBAR_failed" method, which potentially sets the result & summary (if not already set) and then calls the postprocess. Using the decorator has also a small drawback - **kwargs needs to be set as last parameter of the decorated function. This is because autotest does some nasty magic with introspection to find out which parameters the function (initialize, run_once) has and it fails on the decorated function, so it just blindly uses all the parameters specified in the control file. 3) Tests & templates rewritten ------------------------------ [both patches] I also rewrote the tests & templates so they implement this functionality, I did not have time to test it yet, so it's probable, that there are some typing errors and stuff, but i think it's important to show you, how will the new style of test writing work, once this patch is pushed. Please comment, ask questions, share ideas - I believe that this is an important issue and would like all of you to be happy about it. joza

2 6

Re: [PATCH] add autotest labels support
by Kamil Paral 13 Jul '10

13 Jul '10

Thanks for your remarks, they were really valuable. Comments below. ----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Mon, 2010-07-12 at 09:18 -0400, Kamil Paral wrote: > > I like what this patch is doing overall; a couple comments below. > > > +def get_aq_vars(controlfile, extradata): > > + '''Extract AutoQA specific test variables from control file > and > > + return them in a dictionary. All these variables start with > prefix 'aq_'. > > + Arguments: > > + * controlfile - control file > > + * extradata - dictionary with extra variables received from > hook options > > + Returns: dictionary with AutoQA specific test variables > > + ''' > > + lines = [line for line in open(controlfile).readlines() if > line.startswith('aq_')] > > + vars = extradata.copy() > > + for line in lines: > > + try: > > + exec line in vars > > Interesting - rather than actually interpreting the entire control > file, > you're executing *only* those lines that start with 'aq_'. > > Which means: > 1) that variable needs to be *literal* - this wouldn't work: > release_ver = figure_out_release_ver() > aq_labels = [release_ver] > Nor this: > if rawhide: > aq_labels=['rawhide'] > else: > aq_labels=['f13'] Ah, you got me, I didn't imagine that. I wanted to evaluate the whole control file but that would mean I also run "job.run_test(...)" line, which I don't want to. So I just parsed out aq_* variables. Alright, possible solutions: 1. Make separate file for AutoQA variables, like control.autoqa. We can then easily evaluate the whole file. But it's another file for every test... I don't like many files :) 2. We can find "job.runtest(...)" line in control file, cut it out/ comment it out, and evaluate the rest of the file. That's easy to do, on the other hand maybe some more complex stuff may be present in the control file in the future and some problems may arise. Which approach is better? Personally I would do the second one. We can always redesign it in the future. Or do you have some better idea? > > 2) Any test can execute any python code inside the 'autoqa' script - > which usually runs as root. This makes me a little nervous, although > I'm > not sure what user the control file normally runs as, and we're > making > no guarantees about security anyway. I understand it. Currently only autoqa script and watchers script run on autotest server machine. Now also parts of tests will run there. But there are these two points: 1. I think we will always review any test that would be accepted and deployed to our production servers. It's very easy to see any mischief in the control file. I don't think it is probable some malicious code could be put in there and we haven't seen it. 2. I have the feeling we can't do without it. We need to use autotest labels, right? They must be defined per test, not per hook. Also we need a dynamic definition ("because test XX runs on YY.fc13 package, it needs 'fc13' autotest label" - you can hardly define this statically). Dynamic definition means some scripting language to provide the logic, you can't get by with just a plain text file or smth. Well and that means some Python script in the end. To be correct, in the aforementioned example we could get by with a static mapping of ENVR suffixes -> autotest labels. But I expect some more complex needs will arise in the future. So, it is my impression we can't really do much about it, we need it. And I don't really see it as a big security problem. > > So anyway, I'm not sure how I feel about this. At the very least it > seems confusing to have lines in the control file that get > *partially* > evaluated by autoqa, although not as you might expect. > > How does autotest evaluate the control files as a whole? Yes, I believe autotest executes the whole control file. > > > + except: > > + print "Evaluating of AutoQA variables from control file > failed:" > > + print " Line: %s" % line.strip() > > + print " Control file: %s" % controlfile > > + return {} > > + # let's leave only aq_* keys present > > + for key in vars.copy(): > > Why vars.copy() instead of just 'for key in vars:'? Because you can't delete keys from a collection that you're traversing at the same time. There might be some... side effects :) > > > + if not key.startswith('aq_'): > > + del vars[key] > > + return vars > > + > > The rest seems fine - I pushed a commit to clean up some trailing > whitespace. I should finally define some vim shortcuts to fix this up for me... > > Anyway, if we can figure out how to make the partial evaluation of > the > control files a bit saner (or you can help me understand why this is > actually sane and I'm just confused) I'll be happy to merge this into > master. > > -w > > _______________________________________________ > autoqa-devel mailing list > autoqa-devel(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/autoqa-devel

2 1

[PATCH] watch-koji-builds: improve verbose output
by Kamil Paral 13 Jul '10

13 Jul '10

Trivial patch, already applied to master.

2 2

[PATCH] add autotest labels support
by Kamil Paral 12 Jul '10

12 Jul '10

Hello, This a patch that adds autotest labels support to AutoQA. I have used labels defined at https://fedoraproject.org/wiki/Managing_autotest_labels. The patch is tested and should work ok. The only tests that currently need to use this are initscripts and virt_install, I believe. If I have missed something, let me know. Initscripts are now forced to run in virtual machine and they run on the Fedora release for which the package was built. Rats_install now requires a machine that is virt_capable. In the future package_sanity will also require this patch's functionality. I have also added a two line code that disables machine reboot before and after each test is run (it should be a separate patch, right, but...:)).

2 2

[PATCH] replace repoinfo.conf on upgrade
by Kamil Paral 12 Jul '10

12 Jul '10

repoinfo.conf (as opposed to autoqa.conf) is not really a configuration file where the user could set some useful setting. It's more of a definition file for AutoQA's internal use only. Therefore we move it from %config(noreplace) behaviour to %config behaviour. That will save us some trouble when upgrading autoqa (often you forget about updating repoinfo.conf and then it throws weird error messages, at best). --- Makefile | 2 +- autoqa.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 86c7105..0931a69 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ install: build [ -f $(PREFIX)/etc/cron.d/autoqa ] || install -m 0644 autoqa.cron $(PREFIX)/etc/cron.d/autoqa install -d $(PREFIX)/etc/autoqa [ -f $(PREFIX)/etc/autoqa/autoqa.conf ] || install -m 0644 autoqa.conf $(PREFIX)/etc/autoqa/ - [ -f $(PREFIX)/etc/autoqa/repoinfo.conf ] || install -m 0644 repoinfo.conf $(PREFIX)/etc/autoqa/ + install -m 0644 repoinfo.conf $(PREFIX)/etc/autoqa/ install -d $(PREFIX)$(HOOK_DIR) for h in hooks/*; do cp -a $$h $(PREFIX)$(HOOK_DIR); done install -d $(PREFIX)$(TEST_DIR) diff --git a/autoqa.spec b/autoqa.spec index 540fb00..8e60be8 100644 --- a/autoqa.spec +++ b/autoqa.spec @@ -76,7 +76,7 @@ rm -rf $RPM_BUILD_ROOT %doc README LICENSE TODO %config(noreplace) %{_sysconfdir}/cron.d/autoqa %config(noreplace) %{_sysconfdir}/autoqa/autoqa.conf -%config(noreplace) %{_sysconfdir}/autoqa/repoinfo.conf +%config %{_sysconfdir}/autoqa/repoinfo.conf %config(noreplace) %{testdir}/rats_sanity/irb.cfg %dir %attr(0775,root,autotest) %{_localstatedir}/cache/autoqa %{_sysconfdir}/autoqa -- 1.7.1.1

2 2

[AutoQA] #164: Initscript review: cups
by fedora-badges 12 Jul '10

12 Jul '10

#164: Initscript review: cups ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: minor | Milestone: Initscripts review Component: tests | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- Check the test according to the <https://fedorahosted.org/autoqa/wiki/initscripts> -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/164> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

Re: Priority discussion / PUATP "sprint"
by Kamil Paral 09 Jul '10

09 Jul '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > > > The remaining milestones and all the other work we're doing is > definitely all still important. But I think it's time that we sat > down > and really tried to finish off the PUATP work - or at least get a > version 1.0 up and running. >From the last prioritization meeting I got the idea that we wanted to prioritize ResultsDB first...? I understood that we would just finish some semifinished parts - testing depcheck, implementing autotest labels - and then focusing on having ResultsDB up and running. As for myself I'm finishing my part now and am prepared for jumping onto the ResultsDB train very soon. But we can surely discuss other possibilities of prioritization. The only important thing is, I think, that we move along together, so it progresses faster. We can have it as a Monday topic or discuss it here.

2 1

Priority discussion / PUATP "sprint"
by Will Woods 08 Jul '10

08 Jul '10

Hi, all. So you probably noticed me mucking about with milestones and tickets in AutoQA trac. I'm trying to rearrange things to give us a clearer idea of what needs to be done to get the Package Update Acceptance Test Plan automation up and running - especially the depcheck test, since we keep hitting situations it should have prevented. I've broken it down into a few milestones, the key ones being: * Multi-hook tests This is a milestone for work to allow tests that run for multiple hooks (e.g. rpmguard/rpmlint). I think we might need to have this working in order to have all the tests we want as part of the Package Update Acceptance Test Plan (see below). * Package Update Acceptance Test Plan This is the main milestone for the PUATP work as a whole. Some further sub-milestones follow. * Package Update Acceptance Test Plan - depcheck This milestone is for the depcheck test, which is possibly the most complex part of the PUATP. * Package Update Acceptance Test Plan - package sanity tests This milestone tracks the progress of the package sanity tests, which are another important part of the PUATP. * Packaging, Review, & Deployment This tracks the status of the packaging and deployment of AutoQA/autotest in the Fedora infrastructure - we need this to be finished for the above tests to be truly useful, since we need the test logs to go somewhere public. The remaining milestones and all the other work we're doing is definitely all still important. But I think it's time that we sat down and really tried to finish off the PUATP work - or at least get a version 1.0 up and running. So if possible, I'd like to use part of Monday's Fedora QA meeting to discuss what other work/tickets might be needed, and try to divide up the tickets and get us all working together to make the Package Update Acceptance Test Plan fully automated, as soon as possible. Does that work for everyone? Does anyone see any obvious missing pieces? Does anyone want to volunteer to take charge of any of the open tickets? Please let me know - here or in the meeting on Monday. Thanks! -w

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel July 2010