autoqa-devel November 2010

autoqa-devel@lists.fedorahosted.org

8 participants
85 discussions

[AutoQA] #150: "artificial ignorance": suppress errors and warnings that aren't important
by fedora-badges 16 Nov '10

16 Nov '10

#150: "artificial ignorance": suppress errors and warnings that aren't important -------------------------+-------------------------------------------------- Reporter: dmalcolm | Owner: Type: enhancement | Status: new Priority: major | Milestone: Package Sanity Tests Component: tests | Version: 1.0 Keywords: | -------------------------+-------------------------------------------------- There are some interesting ideas in here about suppressing a logfile to show up the information that's truly important: http://www.ranum.com/security/computer_security/papers/ai/index.html I think that the rpmlint test needs to have a set of filters, and should suppress output matching certain patterns. For example https://fedorahosted.org/pipermail/autoqa-results/2010-April/013903.html shows: {{{ gnome-python2-gnome.i686: W: spelling-error Summary(en_US) libgnome -> lib gnome, lib-gnome, cognomen }}} i.e. it's complaining about a reference to "libgnome" within gnome- python2-gnome, which is clearly a false-positive. For this case we might have a python filtering process that says something like this (caveat: untested): {{{ patterns = [ ": W: spelling-error \S+ libgnome .*", ] def should_ignore(line): for pat in patterns # May want to precompile the patterns: m = re.match(pat, line) if m: return True # No match: return False for line in rpmlint_output if not should_ignore(line): print(line) }}} and then gradually grow patterns. The idea is to make it nice and lightweight, and Fedora QA can then easily take ownership of the suppresssions, without having to get patches into rpmlint. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/150> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 4

Re: Any further discussion on depcheck & the Glorious Future?
by Kamil Paral 16 Nov '10

16 Nov '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > Hey all, > > I've finished the series of blog posts about depcheck and the > Glorious > Future of Fedora QA (as I see it): > > [1/3] http://qa-rockstar.livejournal.com/10187.html > [2/3] http://qa-rockstar.livejournal.com/10368.html > [3/3] http://qa-rockstar.livejournal.com/10507.html > [bonus details]: http://wwoods.fedorapeople.org/depcheck.html Thanks Will for those blogposts, really great. I have some more technical questions: 1. Do we need hosts with different architectures for running depcheck? Let's imagine we have these packages proposed as updates: foo-1.2-3.fc13 |-----foo-1.2-3.i686.rpm |-----foo-1.2-3.x86_64.rpm |-----foo-data-1.2-3.noarch.rpm bar-4.5-6.fc13 |-----bar-4.5-6.fc13.noarch.rpm How do I test those packages? Should I get two machines and test i686 rpm files on i686 hardware, x86_64 rpm files on x86_64 hardware, and noarch rpm files on any hardware? Or should I test the whole set of RPMs twice, once on each hardware arch? 2. I have imagined a situation where two updates are accepted into the -pending tag in quick succession. Thus we can end up with two depcheck tests running simultaneously. And the one that was started earlier can finish up later. It could then report incorrect result. Do you have an idea what we can we do about that? Maybe we can somehow write down the state of the -pending tag in the time of starting the test, and if it is different from when the test is finished, we can just throw away the results? Just a wild idea.

1 1

Re: Update to autoqa-0.4.3-1
by Kamil Paral 16 Nov '10

16 Nov '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > Greetings gang, > > Per previous discussion on the list > (https://fedorahosted.org/pipermail/autoqa-devel/2010-November/001286.html) > and of course using the roadmap Kamil posted earlier > (https://fedoraproject.org/wiki/AutoQA_Roadmap) I updated to > autoqa-0.4.3-1. > In reply to this mail, I sent three patches that I included when > tagging > autoqa-0.4.3-1. I've applied these to master already. Hopefully that > wasn't > in poor form, we can easily adjust if there are any concerns regarding > the > patches. Two of the patches should be familiar (autoqa.conf and > repoinfo.conf). The third patch resolves ticket#239. Thanks, James, patches look completely fine. My quick smoke test succeeded. Let's deploy! :)

1 0

[AutoQA] #239: rpmlint prints "I: enchant-dictionary-not-found en_US" once for each run
by fedora-badges 16 Nov '10

16 Nov '10

#239: rpmlint prints "I: enchant-dictionary-not-found en_US" once for each run -----------------------+---------------------------------------------------- Reporter: notting | Owner: Type: defect | Status: new Priority: major | Milestone: Component: docs/wiki | Keywords: -----------------------+---------------------------------------------------- I believe this can be fixed by installing hunspell-en in the environment. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/239> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

Update to autoqa-0.4.3-1
by James Laska 15 Nov '10

15 Nov '10

Greetings gang, Per previous discussion on the list (https://fedorahosted.org/pipermail/autoqa-devel/2010-November/001286.html) and of course using the roadmap Kamil posted earlier (https://fedoraproject.org/wiki/AutoQA_Roadmap) I updated to autoqa-0.4.3-1. In reply to this mail, I sent three patches that I included when tagging autoqa-0.4.3-1. I've applied these to master already. Hopefully that wasn't in poor form, we can easily adjust if there are any concerns regarding the patches. Two of the patches should be familiar (autoqa.conf and repoinfo.conf). The third patch resolves ticket#239. More details on the changes since autoqa-0.4.2-1 are listed below. cb8c8fd 2010-11-15 Bump to autoqa-0.4.3-1 (jlaska(a)redhat.com) ff03bab 2010-11-15 Ticket#239 - install hunspell-en during rpmlint setup() (jlaska(a)redhat.com) f3cb33f 2010-11-15 Remove ppc from arches. f14 is now officially released, update path appropriately. Remove unsupported Fedora releases (jlaska(a)redhat.com) 399d703 2010-11-15 only library modules with main loop should be executable, others should not (kparal(a)redhat.com) 4e6bdc6 2010-11-10 remove unused imports, call parent initialize() method when overriding it (kparal(a)redhat.com) 1462e47 2010-11-09 upgradepath: improve error message clarity (kparal(a)redhat.com) 6670ee2 2010-10-14 allow tests to distinguish CRASHED and ABORTED (kparal(a)redhat.com) a5ce337 2010-10-13 Reviewed. (mkrizek(a)redhat.com) 821f162 2010-10-11 Added reviewed initscript tests. (jskladan(a)redhat.com) e0f77a1 2010-10-06 Include traceback in email output (kparal(a)redhat.com) 82baf5f 2010-10-06 Removed individual XYZ_failed methods in test.py (jskladan(a)redhat.com) 78cacd5 2010-10-05 guard setup() with ExceptionCatcher (kparal(a)redhat.com) 8e4e894 2010-10-05 allow self.outputs and self.highlights to be lists of strings (kparal(a)redhat.com) 03ca1e8 2010-10-04 Automatically log test output (kparal(a)redhat.com) c6e5d3a 2010-10-01 Correct missing config_loader import (jlaska(a)redhat.com) a2bd1b1 2010-10-01 Tell post-tree-compose that composes are enabled for F14 (jlaska(a)redhat.com) 885e0e1 2010-09-30 tidy up upgradepath test code (kparal(a)redhat.com) 993e394 2010-09-29 upgradepath doesn't support -testing repos yet, disable scheduling (kparal(a)redhat.com) Thanks, James

1 3

AutoQA Roadmap
by Kamil Paral 15 Nov '10

15 Nov '10

Hello all, I have created https://fedoraproject.org/wiki/AutoQA_Roadmap . Since our manpower decreased (best wishes to wwoods) we really need to get more focused and selective in our future goals. I would like us to state some near-future goals that are doable, that have visible benefits and then concentrate the team on achieving that. Sometimes the tasks can be divided easily among several people, sometimes they may not. But as much as possible we should avoid working on several projects each, as was often the case now. Let's finish up one task and then go to another, what do you say? In the roadmap, I'm partially duplicating our Trac Roadmap: https://fedorahosted.org/autoqa/roadmap Still, it seemed to me that on wiki page one could easily see the most important features we want to have in a release. Better than in Trac, where it is split up more in a functional units rather than time schedule units. I believe that the best part to concentrate upon right now are depcheck and upgradepath tests. They bring a huge benefit to the Fedora project, we just need to finish them and hook them up. Therefore I propose Bodhi comments support in depcheck and upgradepath as the nearest important feature. Rewriting bodhi watcher is related to that. I would also like to see AutoQA fully prepared for a staging environment (it almost is now) - we will get hardware in the meantime. That would be 0.4.4 release. After we have this work done, I think it's about time to finally set up ResultDB (we can have some preview in 0.4.4). Having it production ready would be a next major step, 0.5.0 release. Please look at the wiki page and tell us what you think. Thanks, Kamil

2 1

Re: autotest_lib can't be used outside autotest process
by Kamil Paral 10 Nov '10

10 Nov '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > On Tue, 2010-11-09 at 12:59 -0500, Kamil Paral wrote: > > Martin played a little today with AutoQA and discovered that he > can't > > run our watchers if he uses autotest_lib module inside them. > > > > The reason wasn't clear for me immediately, so I look at it a > little > > closer. > > > > The reason is that autotest libraries are not in a standard Python > PATH. > > The only way to gain access to autotest_lib module is to import > > /usr/share/autotest/client/bin/common.py. And that happens only > when > > you run autotest tool, like this: > > > > # autoqa post-koji-build -k dist-f14 foo -t helloworld --local > > -- or (subsequently) like this -- > > # ~autotest/client/bin/autotest /tmp/autoqa-control.9xlSGi > > Definitely something specific to the peculiar way that autotest is > packaged. I can see why, it makes sense for their project to be > portable to any directory. Just import the common module, and it > will > do the dirty work of finding the remaining libraries and setting up > sys.path. Since most of our work is specific to a single distro, we > obviously like things to be in their standard locations. You're right, that works. I can do something like: # PYTHONPATH=/usr/share/autotest/ python >>> import client.bin.common >>> import autotest_lib >>> So if we have the right PYTHONPATH (which we can solve by installing a .pth file into /usr/lib/python/site-packages) and if we keep the right order of imports (common.py first), we can use autotest libraries also outside the autotest process itself. > > > For those scripts that we want to run outside autotest environment > > (like watchers, autoqa script itself, stand-alone parts of our > tests) > > we can't use autotest_lib module, we have to rely just on our > autoqa > > library (and more specifically on those parts of our autoqa library > > that don't rely on autotest_lib themselves). > > > > So I just want to point out that if you see this kind of problem, > > this is the reason. > > > > The question remains - is this desirable or not desirable? Would we > > > like to see autotest modules be accessible as a standard Python > module? > > Do we need it? > > I may not fully understand the problem, but I don't think we should > need > to integrate watchers with autotest. Was there a specific need to > integrate the watcher with autotest? I don't think there was a strong need for it and I don't think it would be wise. We just found it by coincidence. > > Additionally, I get nervous when we integrate out test scripts with > autotest. Obviously, we need to integrate with the current test > harness > to some degree. However, I want to make sure we understand the links > and they are documented well in the event we need to swap out for a > different back-end harness/scheduler. > > This does help clarify though which files can link to autotest and > which > cannot. Obviously, the control file and the test object file can > link > against autotest. But any test script should be stand-alone. For > example, tests/conflicts/potential_conflict.py and > tests/rpmguard/rpmguard. They were designed to be run stand-alone on > a > system and shouldn't link with autotest at all. Am I correct? Yes, I have the same opinion. If it is doable and reasonable we really should have a standalone test script that doesn't rely on autotest. In some cases, however, we have put the logic directly into the test object (e.g. upgradepath test), because it seems like a best approach in those scenarios (although it has some disadvantages). > > > I don't think we need it right now, just some food for thought. > > Definitely, thanks for pointing this out! > > While researching packaging autotest, the packager will have to > answer > this question of why aren't some files in standard system locations. > I've been assuming that answering that question is a distinct effort > from anything going on with our autoqa work. Did I get that wrong? I am not sure. I think the current state (non-standard location) is sufficient for us, at least for now.

2 1

Any further discussion on depcheck & the Glorious Future?
by Will Woods 09 Nov '10

09 Nov '10

Hey all, I've finished the series of blog posts about depcheck and the Glorious Future of Fedora QA (as I see it): [1/3] http://qa-rockstar.livejournal.com/10187.html [2/3] http://qa-rockstar.livejournal.com/10368.html [3/3] http://qa-rockstar.livejournal.com/10507.html [bonus details]: http://wwoods.fedorapeople.org/depcheck.html So, yay! Now: are there any problems with the ideas or plans I proposed, or any suggestions? Are there any topics I missed? Any questions like "when do we get to [some goal]" or "where does [some feature/project] fit in"? Is there anything else you'd like me to write about? Did anyone actually read all the way to the end of those long, long posts? Let's hear it. -w

3 2

autotest_lib can't be used outside autotest process
by Kamil Paral 09 Nov '10

09 Nov '10

Martin played a little today with AutoQA and discovered that he can't run our watchers if he uses autotest_lib module inside them. The reason wasn't clear for me immediately, so I look at it a little closer. The reason is that autotest libraries are not in a standard Python PATH. The only way to gain access to autotest_lib module is to import /usr/share/autotest/client/bin/common.py. And that happens only when you run autotest tool, like this: # autoqa post-koji-build -k dist-f14 foo -t helloworld --local -- or (subsequently) like this -- # ~autotest/client/bin/autotest /tmp/autoqa-control.9xlSGi For those scripts that we want to run outside autotest environment (like watchers, autoqa script itself, stand-alone parts of our tests) we can't use autotest_lib module, we have to rely just on our autoqa library (and more specifically on those parts of our autoqa library that don't rely on autotest_lib themselves). So I just want to point out that if you see this kind of problem, this is the reason. The question remains - is this desirable or not desirable? Would we like to see autotest modules be accessible as a standard Python module? Do we need it? I don't think we need it right now, just some food for thought. Cheers, Kamil

2 1

Re: upgradepath: FAIL; 0 OK, 1 FAILED for GMT-4.5.5-2.fc13
by Kamil Paral 09 Nov '10

09 Nov '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > On Tue, 2010-11-09 at 07:44 -0500, Kamil Paral wrote: > > ----- "James Laska" <jlaska(a)redhat.com> wrote: > > > > > > Aaah, I understand the difference and why I was confused by the > > > result. > > > Thanks for explaining. > > > > What do you think about this one? > > Definitely helps me understand the output better. SOme comments > below. > > > =============================================== > > >From f9bcda33b036e7df25a50d2b1a4a72c6ef1a0619 Mon Sep 17 00:00:00 > 2001 > > From: =?UTF-8?q?Kamil=20P=C3=A1ral?= <kparal(a)redhat.com> > > Date: Tue, 9 Nov 2010 13:39:41 +0100 > > Subject: [PATCH] upgradepath: improve error message clarity > > > > Use human sentence in the error message instead of a symbolic > notation > > --- > > tests/upgradepath/upgradepath.py | 18 ++++++++++++++---- > > 1 files changed, 14 insertions(+), 4 deletions(-) > > > > diff --git a/tests/upgradepath/upgradepath.py > b/tests/upgradepath/upgradepath.py > > index edbe4ea..d42e39d 100755 > > --- a/tests/upgradepath/upgradepath.py > > +++ b/tests/upgradepath/upgradepath.py > > @@ -50,9 +50,19 @@ class upgradepath(AutoQATest): > > self.outputs = [] > > self.highlights = [] > > > > - def compare(self, matching_build, tags, op, opstr): > > + def compare(self, matching_build, tags, op): > > koji = autoqa.koji_utils.SimpleKojiClientSession() > > result = 'PASSED' > > + > > + # convert 'op' to human language for error message clarity > > + opstr = {operator.eq: 'equal to', > > + operator.ge: 'greater or equal than', > > Maybe rephrase the last part as "... equal to' > > > + operator.gt: 'greater than', > > + operator.le: 'lesser or equal than', > > Maybe rephrase as "less than or equal to' > > > + operator.lt: 'lesser than', > > Same, maybe rephrase the last part as "... equal to' Thanks for your correction. > > > + operator.ne: 'not equal to'} > > + assert op in opstr, 'Used unsupported operator: %s' % op > > + > > Heh, that's funny. I was looking to see if the operator module > itself > had some text for each of those functions, so I like your opstr > dictionary above. Unfortunately it doesn't. > > > for tag in tags: > > latest_build = koji.latest_by_tag(tag, > matching_build['name']) > > if latest_build is None: > > @@ -80,7 +90,7 @@ class upgradepath(AutoQATest): > > print msg > > self.outputs.append(msg) > > if not ok: > > - msg = '\tFailed condition: Requested package %s > Latest package' % opstr > > + msg = '\tError: Requested package must be %s > the latest package' % opstr[op] > > Future-proofing ... are there any operators we aren't adding to the > opstr dictionary that might result in a traceback at some future > date? > Should we convert the access to opstr.get(op, "UNKNOWN")? I would rather see it crashed than silently doing anything. I don't suppose we will need more operators, and if we do, we can amend the dictionary. > > Or should we stuff an assert in there somewhere (assert > optsr.has_key(op))? Oh yes, the assert is right after the dictionary declaration. > > > print msg > > self.outputs.append(msg) > > result = 'FAILED' > > @@ -138,9 +148,9 @@ class upgradepath(AutoQATest): > > self.highlights.append(msg) > > > > # compare with lower or equal tags, so version has to > be greater or equal > > - result = self.compare(matching_build, low_eq_tags, > operator.ge, '>=') > > + result = self.compare(matching_build, low_eq_tags, > operator.ge) > > # compare with higher tags, so version has to be lower > or equal > > - result2 = self.compare(matching_build, hi_tags, > operator.le, '<=') > > + result2 = self.compare(matching_build, hi_tags, > operator.le) > > > > # compute pkg result > > if self.result_order.index(result2) > > self.result_order.index(result): > > -- > > 1.7.3.2 > > > > ==================================================== > > > > # autoqa post-bodhi-update --title xxx --target-tag dist-f13-updates > GMT-4.5.5-2.fc13 -t upgradepath --local > > 13:37:27 INFO | ======================================== > > 13:37:27 INFO | GMT-4.5.5-2.fc13 into dist-f13-updates > > 13:37:27 INFO | ======================================== > > 13:37:29 INFO | [ OK ] dist-f10 > > 13:37:29 INFO | Latest package: GMT-4.3.1-2.fc10 > > 13:37:29 INFO | [ OK ] dist-f10-updates > > 13:37:29 INFO | Latest package: GMT-4.4.0-1.fc10 > > 13:37:30 INFO | [ OK ] dist-f11 > > 13:37:30 INFO | Latest package: GMT-4.4.0-2.fc11 > > 13:37:30 INFO | [ OK ] dist-f11-updates > > 13:37:30 INFO | Latest package: GMT-4.5.0-4.fc11 > > 13:37:31 INFO | [ OK ] dist-f12 > > 13:37:31 INFO | Latest package: GMT-4.5.1-1.fc12 > > 13:37:32 INFO | [ OK ] dist-f12-updates > > 13:37:32 INFO | Latest package: GMT-4.5.3-3.fc12 > > 13:37:32 INFO | [ OK ] dist-f13 > > 13:37:32 INFO | Latest package: GMT-4.5.2-1.fc13 > > 13:37:33 INFO | [ OK ] dist-f13-updates > > 13:37:33 INFO | Latest package: GMT-4.5.3-3.fc13 > > 13:37:34 INFO | [FAIL] dist-f14 > > 13:37:34 INFO | Latest package: GMT-4.5.3-3.fc14 > > 13:37:34 INFO | Error: Requested package must be lesser or equal > than the latest package > > 13:37:37 INFO | [ OK ] dist-f14-updates > > 13:37:37 INFO | No such package: GMT > > 13:37:37 INFO | [ OK ] dist-f15 > > 13:37:37 INFO | Latest package: GMT-4.5.5-2.fc15 > > 13:37:37 INFO | RESULT: FAILED > > 13:37:37 INFO | SUMMARY: 1 FAILED > > 13:37:37 INFO | **************************************** > > Looks good to me ... definitely addresses my confusion. I'll commit to master soon. Thanks for feedback.

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel November 2010