autoqa-devel February 2012

autoqa-devel@lists.fedorahosted.org

9 participants
44 discussions

[AutoQA] #361: improve pretty log output for anaconda tests
by fedora-badges 05 Mar '12

05 Mar '12

#361: improve pretty log output for anaconda tests --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: minor | Milestone: Finger Food Component: tests | Keywords: --------------------+------------------------------------------------------- With the advent of the super awesome html pretty logs, some of the less frequented tests are in need of some care and feeding. While reviewing installer test results with the anaconda team, I noticed that the installer tests don't all look good with pretty logs. Sample output for existing anaconda test results ... * [http://autoqa.fedoraproject.org/results/143510-autotest/qa02.qa.fedoraproje… anaconda_storage] - Doesn't look bad ... I don't think I'd recommend changes here. * [http://autoqa.fedoraproject.org/results/143512-autotest/qa07.qa.fedoraproje… compose_tree] - Missing some info * [http://autoqa.fedoraproject.org/results/143577-apache/10.5.124.249/anaconda… anaconda_checkbot] - Yuck, no info at all :( This ticket is intended to address improving the html result output for the above installer tests. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/361> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 11

require maintainer defined for each test & clean up current tests
by Kamil Paral 01 Mar '12

01 Mar '12

Before releasing new version of AutoQA, I want to solve one more issue, and that is the current mess in tests we manage. Honestly stated, we have some tests that we have no idea whether they work, what exactly they do, and how to fix them if needed. Some of them are probably not being used at all, and we still need to care about them when re-factoring our framework. Worst of all, for many of them we have no idea who should be responsible for keeping them in shape. I have spent some time and executed all of them. I also tried to provide my best guess about their current maintainer. This is the result: Test Maintainer Works Currently useful ==== ========== ===== ================ anaconda_checkbot clumens ? no no anaconda_storage clumens ? no no compose_tree ? no no conflicts ? yes unlikely depcheck jskladan ? yes yes helloworld kparal yes yes rats_install hongqing ? yes yes rats_sanity hongqing ? yes probably repoclosure ? yes unlikely rpmguard kparal yes somewhat rpmlint kparal yes somewhat upgradepath kparal yes yes My proposal is: 1. Every test will have a maintainer defined. In its 'control' file we will change AUTHOR line to MAINTAINER line (patch is ready). This will ensure that we always know who to talk to when the test seems broken. It doesn't mean you can't work on a test you don't maintain, no. But at least the contact point will always be defined. I tried to place my best guess in the matrix. Please speak up who wants to maintain some test. 2. Tests without maintainer will be archived and deleted. They can stay in a separate git branch and wait for their future revival (if any), but they won't be in master. 3. To save resources, we should also archive tests that don't seem to be currently much useful. We can re-enable them once required architecture is in place. More specifically: * rpmlint, rpmguard - the results are sent to opted-in maintainers, some of them said it's useful. I'd keep them enabled. * repoclosure, conflicts - this lists potential dependency problems and file conflicts for the whole repository. Until now no one cared. With resultsdb frontend we can finally have a page that lists all the results day by day. It means someone can go through the results occasionally and file some bugs. The question is: who? It's nice to have some results, but if we just *hope* someone will do something about it, that seems too uncertain for me. They are also somewhat obsoleted by depcheck. I'm sitting on the fence here. * compose_tree - this tries to build boot.iso and pxeboot images. This is basically a releng test that we execute. Previous maintainer was James, but I doubt we should ask him for future maintenance. Currently it is broken. Even if it worked, was there someone reporting the issues? It's easier to look into releng logs online: http://kojipkgs.fedoraproject.org/mash/branched-20120227/logs/ Does the above link render compose_tree test useless? * anaconda_* - anaconda build, unit test, and automated installation using various test cases. Great work, but currently broken. I was talking to clumens some time ago and asked whether he received results. He said he did not. I fixed the opt-in emails and told him. No response since, so I guess he just doesn't care. And I'm not surprised. With the speed of anaconda development, it's much easier for anaconda devs to execute the test on their own machines (at least I suppose they do). They can't send us patches all the time. Since there hasn't been any drive from anaconda devs, I propose to obsolete this tests until we have something better to offer. * the rest of the tests stay enabled For the next autoqa release I'd like to focus mainly on easy deployment of tests, so that we (for our tests) and some other teams (like anaconda) can easily update tests without tedious process of "new autoqa release". After that I expect some of the tests to return. What do you think? And please, put your name next to the test you want to maintain.

5 10

Review Request: RestulsDB Integration
by Josef Skladanka 29 Feb '12

29 Feb '12

https://fedorahosted.org/reviewboard/r/258/

3 2

[AutoQA] #409: make test does not work
by fedora-badges 29 Feb '12

29 Feb '12

#409: make test does not work -----------------------+------------------------ Reporter: mkrizek | Owner: tflink Type: defect | Status: new Priority: critical | Milestone: Hot issues Component: core | Keywords: Blocked By: | Blocking: -----------------------+------------------------ make test does not work any more, fails with: {{{ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pip/runner.py", line 16, in <module> exit = run() File "/usr/lib/python2.7/site-packages/pip/runner.py", line 11, in run import pip File "/usr/lib/python2.7/site-packages/pip/__init__.py", line 10, in <module> from pip.basecommand import command_dict, load_command, load_all_commands, command_names File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 11, in <module> from pip.baseparser import parser, ConfigOptionParser, UpdatingDefaultsHelpFormatter File "/usr/lib/python2.7/site-packages/pip/baseparser.py", line 5, in <module> import pkg_resources File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2727, in <module> add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 700, in subscribe callback(dist) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2727, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2230, in activate map(declare_namespace, self._get_metadata('namespace_packages.txt')) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1815, in declare_namespace path = sys.modules[parent].__path__ KeyError: 'peak' }}} It fails on: pip-python -E test_env/ install ''package'' I found out that it is possibly somehow connected to -E option, since running it without that option works fine. This issue is kind of critical as it blocks building AutoQA package (we run make test in %check section). -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/409> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 6

[AutoQA] #410: prettylog: autoqa logo overlaps text
by fedora-badges 29 Feb '12

29 Feb '12

#410: prettylog: autoqa logo overlaps text --------------------------+------------------------- Reporter: kparal | Owner: Type: enhancement | Status: new Priority: trivial | Milestone: Finger Food Component: core | Keywords: Blocked By: | Blocking: --------------------------+------------------------- AutoQA logo hides text: http://autoqa- stg.fedoraproject.org/results/29355-autotest/qa06.qa.fedoraproject.org/upgr… /mingw-antlr-2.7.7-8..html Adjust it so that the text wraps around it. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/410> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

[AutoQA] #398: depcheck is sometimes scheduled as noarch
by fedora-badges 24 Feb '12

24 Feb '12

#398: depcheck is sometimes scheduled as noarch ---------------------+------------------------ Reporter: kparal | Owner: Type: defect | Status: new Priority: minor | Milestone: Hot issues Component: tests | Keywords: Blocked By: | Blocking: ---------------------+------------------------ Generally the koji-bodhi watcher schedules both i386 and x86_64 architectures of depcheck: {{{ # autoqa post-bodhi-update-batch --targettag f16-updates --arch x86_64 --arch i386 openssl-1.0.0f-1.fc16 --dryrun /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *noarch -f /tmp/autoqa-control.le3exd post-bodhi-update- batch:upgradepath.noarch Keeping /tmp/autoqa-control.le3exd at user request /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *x86_64 -d fc16 -f /tmp/autoqa-control.FscGAS post-bodhi-update- batch:depcheck.x86_64 /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *i386 -d fc16 -f /tmp/autoqa-control._QiAVV post-bodhi-update- batch:depcheck.i386 Keeping /tmp/autoqa-control._QiAVV at user request }}} But sometimes it might happen that the only new updated package is a noarch package. Then the watcher "optimized" the arguments by leaving out "--arch i386" and "--arch x86_64", therefore using the argument "--arch noarch" that is implied by default: {{{ # autoqa post-bodhi-update-batch --targettag f16-updates rpmlint-1.0.0f-1.fc16 --dryrun /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *noarch -f /tmp/autoqa-control.mlUFRG post-bodhi-update- batch:upgradepath.noarch Keeping /tmp/autoqa-control.mlUFRG at user request /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *noarch -d fc16 -f /tmp/autoqa-control.zpet6Z post-bodhi-update- batch:depcheck.noarch Keeping /tmp/autoqa-control.zpet6Z at user request }}} Unfortunately this doesn't work for depcheck. Depcheck is then scheduled as noarch test, therefore run on arbitrary machine, and the test result is also reported as "depcheck on noarch" into Bodhi. We need to study this problem in detail and decide whether we should change the watcher (schedule both architectures?) or implement new features in the scheduling process (create a way for depcheck's control.autoqa to insist on running both architectures every time?). This is not super-critical because even though we provide some invalid comments in Bodhi, we will probably schedule correct depcheck tests in the following runs and test the package again. But it is unpleasant nevertheless. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/398> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

Re: RATS: now in production
by Kamil Paral 21 Feb '12

21 Feb '12

> On 02/21/2012 10:48 AM, Adam Williamson wrote: > > > > > the rats_install test runs an entirely automated test of the Fedora > > installer daily, against the daily installer composes provided by > > release engineering. It provides detailed logs and pinpoints where > > failure occurs if the installation is not successful, and preserves > > the > > complete anaconda logs for debugging. So now we can know, daily, > > whether > > the Branched tree is in an installable state or not. > > That's awesome. Is it possible to test upgrades? > > Rahul Do you mean F16 -> F17 upgrades? No. This is just a minimal-install-over-serial-console test. But we plan to expand it in the future.

1 0

[AutoQA] #363: depcheck: don't require specific release
by fedora-badges 15 Feb '12

15 Feb '12

#363: depcheck: don't require specific release -------------------------+-------------------------------------------------- Reporter: kparal | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Finger Food Component: tests | Keywords: -------------------------+-------------------------------------------------- Depcheck currently requires machine with a specific Fedora release installed to test matching repositories. From tests/depcheck/control.autoqa: {{{ # because we simulate installing packages, the autotest label of the correct distribution # must be present (like 'fc13') If proper label is not found the test will not execute if autoqa_args.has_key('nvrs'): from autoqa.util import get_distro distro = get_distro(autoqa_args['nvrs'][0]) labels.append(distro) }}} Josef says it's possible to cancel this constraint. That would allow us to test arbitrary repository on arbitrary machine (improving performance and maybe allowing us to have less machines). There is one thing worth mentioning: Depcheck is tightly tied to yum, and any weird behavior might be caused by yum. If we cancel above-mentioned constraint, we can no longer suppose that "depcheck crashed on f15-updates, therefore I should try to reproduce it on my F15 machine". It will be necessary to look which machine the test was executed on and try to reproduce it in the same environment if possible. The goal of this ticket is test properly whether cancelling the requirement works correctly. That means to test executing depcheck for F15 repos on F14 machine, or vice versa, and comparing the results with behavior prior to this change. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/363> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

Re: [AutoQA] #407: Update repoinfo.conf - Add branch 17
by fedora-badges 15 Feb '12

15 Feb '12

#407: Update repoinfo.conf - Add branch 17 ------------------------+------------------------- Reporter: kparal | Owner: kparal Type: task | Status: closed Priority: critical | Milestone: Hot issues Component: production | Resolution: fixed Keywords: | Blocked By: Blocking: | ------------------------+------------------------- Changes (by kparal): * status: new => closed * resolution: => fixed Comment: Deployed to production. We shouldn't need f17 clients because we canceled that requirement for depcheck. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/407#comment:2> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #407: Update repoinfo.conf - Add branch 17
by fedora-badges 14 Feb '12

14 Feb '12

#407: Update repoinfo.conf - Add branch 17 ------------------------+------------------------- Reporter: kparal | Owner: kparal Type: task | Status: new Priority: critical | Milestone: Hot issues Component: production | Resolution: Keywords: | Blocked By: Blocking: | ------------------------+------------------------- Changes (by kparal): * owner: => kparal * reporter: ausil => kparal Comment: Thank you. I fixed that in commit 7a8df13. I will copy it to staging, but we need to create f17 clients there for depcheck. If everything goes well, we will push that to production, but again we need f17 clients. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/407#comment:1> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel February 2012