autoqa-devel April 2011

autoqa-devel@lists.fedorahosted.org

10 participants
99 discussions

Questions about porting/writing tests to AutoQA
by Athmane Madjoudj 21 Apr '11

21 Apr '11

Hello, I've some tests written in Python (all under 50 lines), used to test for the following software: Dovecot : 1 test Apache HTTPD: 3 + 1 not yet finished vsFTPd : 1 test MySQL : 1 test SSHD : 1 not yet finished After reading about autotest (big picture) and 'Writing AutoQA Tests' [1], I need to have the following info: 1. Should I use TEST_TYPE = 'CLIENT', because actually the tests run on 'localhost' or I need to modify the tests to be network-aware and use TEST_TYPE = 'SERVER'. 2. I'm little confused about the directory structure, should I group all tests into one or split the according the service (tests/httpd, tests/httpd_vhost, tests/vsftpd, etc...) 3. The tests uses some no-standards Python modules like pexpect and 'tcpcat' (I wrote this module to emulate netcat, because tests have failed after a netcat bug[2] ), Is that OK ? [1] https://fedoraproject.org/wiki/Writing_AutoQA_Tests [2] https://bugzilla.redhat.com/show_bug.cgi?id=695108 Best regards. -- Athmane Madjoudj RHCE

2 5

Tapper - new test automation framework
by Kamil Paral 20 Apr '11

20 Apr '11

Just out of curiosity, AMD just open-sourced a tool called Tapper. It should be able to automatically set up computer, execute tests and evaluate them via web environment. Sounds pretty similar to what we use. https://github.com/amd

2 2

Hello
by Peter Tylka 18 Apr '11

18 Apr '11

Hello, my name is Peter Tylka and I am 26. 5 years I studied informatics and information technologies. For 3 years I work as RedHat and Java application server administrator. I would like to improve my basic knowledge of python scripting and development process (, maybe english :) . I would like to experience FOSS environment and help community. I hope you find some task for me. Thanks Peter

2 1

Depcheck Workflow Note
by Josef Skladanka 18 Apr '11

18 Apr '11

= Depcheck workflow = After discussing some issues with Kamil, we decided, that it would be for the best, if we wrote down the intended Depcheck results usage & workflow. The problem is tightly connected to #306. The problem described in #306 means this: == Premise == "Updates can be changed in Bodhi." - the evidence is clear <https://admin.fedoraproject.org/updates/control-center-3.0.0-1.fc15,gnome-s…>, if you look at the bodhi comment from 2011-04-06 15:05:38 "hadess has edited this update..." == Implications == === Timeline === 1) All updates in -pending passed depcheck, and have "PASSED" comment. 2) Update XYZ is changed by the developer === Consequences === Because one of the "already passed" updates is changed, we should discard the whole "passed" subset, because we can not be sure, that some errors were not introduced by the change. == Level 1 Solution == #306 describes the 'simple' solution. When we search for the 'accepted' updates, we look for all the updates, which do have "PASSED depcheck" comment in bodhi. We want to detect, whether the update was changed after the "PASSED depcheck" comment, and if so, then remove it from accepted set, and basicaly retest it. == Level 2 Solution == Level 1 solution does not really solve the problem, as described, even though it is certainly needed to detect the 'passed but changed', and will sort out some of the problems. The problem with the fact, that change in one update can possibly break some 'previously passed' dependency problems is still there. Because it would be incredibly hard to detect just the updates influenced by changes in the respective update, we decided to define a workflow, which bypasses the overall need for this. === RelEng workflow === At the moment, rel-eng is pushing updates manualy few times a week [citation needed, this is informal information which has not been audited]. The thing is, that we can provide the _at the moment 'pushable'_ subset of packages. When rel-eng will want to push from -pending to {stable, updates, ...}, they'll just 'list' the pushable subset (yes, resultsdb will help ;-)). "The tool" will detect if there were any changes in the pushable subset, and "the tool" will inform rel-eng that they either need to wait for the results of the next depcheck run, or (if we're able to do it) request new depcheck run.

4 12

[AutoQA] #305: upgradepath: CRASHED; KeyError: 'gnome-settings-daemon-3.0.0.1-1.fc15'
by fedora-badges 18 Apr '11

18 Apr '11

#305: upgradepath: CRASHED; KeyError: 'gnome-settings-daemon-3.0.0.1-1.fc15' ----------------------+----------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.4.7 Component: tests | Keywords: ----------------------+----------------------------------------------------- {{{ Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71, in newf f_result = f(*args, **kwargs) #call the decorated function File "/usr/share/autotest/tests/upgradepath/upgradepath.py", line 236, in run_once if self.result_order.index(self.nvr_results[build['nvr']]) > self.result_order.index(update_result[update['title']]): KeyError: 'gnome-settings-daemon-3.0.0.1-1.fc15' }}} https://fedorahosted.org/pipermail/autoqa-results/2011-April/103704.html -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/305> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

[AutoQA] #303: depcheck: error: Incorrect number of arguments
by fedora-badges 18 Apr '11

18 Apr '11

#303: depcheck: error: Incorrect number of arguments ----------------------+----------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: critical | Milestone: 0.4.7 Component: tests | Keywords: ----------------------+----------------------------------------------------- Our old friend CmdError, but this time seems a different one: https://fedorahosted.org/pipermail/autoqa-results/2011-April/103714.html -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/303> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 18

Re: AutoQA: distro congestion?
by Kamil Paral 18 Apr '11

18 Apr '11

> On Mon, 2011-04-18 at 04:35 -0400, Kamil Paral wrote: > > > I built two sets of security updates for f13/f14/f15 and autoqa > > > rejected > > > the f13/f14 packages. It looks like autoqa is waiting for the > > packages > > > to be properly pushed to f<N+1> stable before green-lighting the > > > matching package for f<N>. > > > > I suppose you're talking about upgradepath test. Yes, that's exactly > > its behavior. > > This seems wrong to me. If I as a packager create an upgrade in bodhi, > autoqa should consider what is present in f<N+1> not only as stable > package but also as proposed upgrade. It should then impose a > constraint > that the upgrade for f<N> can only be pushed to stable if f<N+1> is > pushed to stable at the same time (or obviously independently before). It is just a best effort for now. Once we are able to create those "safe to push" update sets, we will be able to consider not only packages in stable, but also packages in other currently proposed updates. We want to reach the state you're talking about (while still staying informative). It's just not a few hours work :)

1 1

Scheduling (or lack thereof) for non-F13 tests
by Tim Flink 18 Apr '11

18 Apr '11

When I was looking into the depcheck/mediawiki issue before, I noticed that there have been no depcheck runs (or at least email results) for anything other than F13 since Friday. I can start looking into it tomorrow if someone else doesn't get to it first but figured that I would at least mention it here. Tim

3 2

Re: AutoQA: distro congestion?
by Kamil Paral 18 Apr '11

18 Apr '11

> Hi, > > I built two sets of security updates for f13/f14/f15 and autoqa > rejected > the f13/f14 packages. It looks like autoqa is waiting for the packages > to be properly pushed to f<N+1> stable before green-lighting the > matching package for f<N>. I suppose you're talking about upgradepath test. Yes, that's exactly its behavior. > Do I have to wait until the packages for > f15 > are pushed to repush the packages for f14 and then wait again for > pushing into f13? Or is there some automatism that reevaluates and > repushes packages w/o any further intervention from the packagers? No need to do that, upgradepath re-evaluates all proposed updates again and again (every time a new update is proposed into bodhi). If your update failed upgradepath, a new comment will appear once it passes (or every 3 days if it still fails). > I guess this is also slowing down the people that grant the package > push. Previously they could evaluate the package updates in one sweep, > now they will see and approve the "same" package in three cycles. This should be improved in the future.

1 0

Re: AutoQA: distro congestion?
by Tim Flink 17 Apr '11

17 Apr '11

On 04/16/2011 01:52 PM, Axel Thimm wrote: > Hi, > > I built two sets of security updates for f13/f14/f15 and autoqa rejected > the f13/f14 packages. It looks like autoqa is waiting for the packages > to be properly pushed to f<N+1> stable before green-lighting the > matching package for f<N>. Do I have to wait until the packages for f15 > are pushed to repush the packages for f14 and then wait again for > pushing into f13? Or is there some automatism that reevaluates and > repushes packages w/o any further intervention from the packagers? > > I guess this is also slowing down the people that grant the package > push. Previously they could evaluate the package updates in one sweep, > now they will see and approve the "same" package in three cycles. Others have already commented on this, but I wanted to re-state that AutoQA is only in an informative mode right now. Any bodhi comments made by AutoQA have no karma and are not used for anything other than information. This will change sometime in the future, but not until at least the F16 timeframe. I took a look at your packages in bodhi [1] and was only able to find one update with FAILED comments from AutoQA. Are there updates that I'm missing here? Looking at the depcheck output for mediawiki-1.16.4-57.fc13, however, I see: mediawiki-nomath-1.16.4-57.fc13.x86_64 from pending has depsolving problems --> mediawiki-nomath conflicts with php-common This stems from the addition of the "Conflicts: php-common = 5.3.1" that was added to the spec file [2] for 1.16.2-56. I assume that this isn't the only package with a "Conflicts:" in the spec and have filed a ticket against autoqa to better handle these issues [3]. Tim PS - you will generally get a faster response to AutoQA related questions by posting to autoqa-devel(a)lists.fedorahosted.org instead of just devel@ [1] https://admin.fedoraproject.org/updates/user/athimm [2] http://pkgs.fedoraproject.org/gitweb/?p=mediawiki.git;a=blob;f=mediawiki.sp… [3] https://fedorahosted.org/autoqa/ticket/308

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel April 2011