autoqa-devel April 2011

autoqa-devel@lists.fedorahosted.org

10 participants
99 discussions

[AutoQA] #271: Create autotestd systemd.service file for F15 (and beyond)
by fedora-badges 01 Jun '11

01 Jun '11

#271: Create autotestd systemd.service file for F15 (and beyond) -----------------------+---------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: major | Milestone: Package Update Acceptance Test Plan Component: packaging | Keywords: -----------------------+---------------------------------------------------- 07:41:40 Viking-Ice: jlaska: nr.1 upstream man pages nr.2 comparing converting https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/compatability ( The Fedora autotest-server package includes a sysVinit script. A systemd.service file will be needed for f15 and beyond. Guidance from Viking-Ice included below ... {{{ 07:41:40 Viking-Ice: jlaska: nr.1 upstream man pages nr.2 comparing converting https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/compatability ( you can side by side view them ) nr.3 scim through Lennart's blog 07:42:36 jlaska: Viking-Ice: perfect, just the bread-crumb trail I needed. Thank you 07:43:04 * jlaska goes to create a ticket with these instructions (need to eventually create an autotestd.service) 07:44:02 Viking-Ice: I deliberately designed the wiki page so those wanting to convert could side by side compare what already had been converted for you know ah.. they solve this problem that way moment.. 07:45:14 <-- juhp [~petersen(a)66.187.239.10] has quit (Ping timeout: 240 seconds) 07:46:51 Viking-Ice: jlaska: throw this link into your pool http://0pointer.de/blog/projects/systemd-for-admins-3.html 07:48:21 Viking-Ice: jlaska: what he mentioned there is the skeleton to be built upon as inn the first example of the service file. Always start like that then gradually build on top of that if needed }}} -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/271> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

RHEL5 in AutoQA and python 2.4 compatibility
by Tim Flink 23 May '11

23 May '11

This came up in the review of #293 and I thought that I would bring it up on the list. Since our AutoQA server runs RHEL5, it is also running python 2.4 which doesn't support "finally" blocks. What is the minimum version of python we want to support? Should all of our code be 2.4 compatible or just the stuff that will likely run on the autotest server ( events, watchers etc.)? We already have several finally blocks in our tests in stable and a couple in the libs, so breaking 2.4 compatibility wouldn't be a huge deal in the tests since we haven't seen too many problems so far. Thoughts? Tim

5 9

[AutoQA] #208: update minimon to transport logs between guest and host via virtio
by fedora-badges 20 May '11

20 May '11

#208: update minimon to transport logs between guest and host via virtio -------------------+-------------------------------------------------------- Reporter: liam | Owner: Type: task | Status: new Priority: major | Milestone: Automate installation test plan Component: tests | Version: 1.0 Keywords: | -------------------+-------------------------------------------------------- Since minimon has to use network during test, some test cases will not activate network during test,in this case, minimon can not transport logs to host, test also be identified fail even it was successfully run. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/208> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 4

Decreasing Bodhi Comment Emails From AutoQA
by Tim Flink 11 May '11

11 May '11

Since this has almost nothing to do with the log output of depcheck upgradepath, I'm splitting this conversation off into another thread. As Kamil pointed out, my initial proposal was naive and needs more thought before implementing anything. We will soon have the ability to disable emails for comments in Bodhi and we need to figure out how to best make use of that ability. Lots of questions here, I'll save my thoughts for replying to myself. Tim How would we deal with tests that change state between PASS and FAIL? - Either 'PASS -> FAIL' or 'FAIL -> PASS' What emails do we want to get rid of? - Just certain 'PASS' messages? - All results messages? How do we expect maintainers to learn of AutoQA results if we turn off email notifications? Can we do this in a way such that it is configurable by maintainer? - If so, should we attempt this? - What is the benefit? How big is it? Can we just send a single email after all tests have passed? - Again, if so, should we? Is there another approach that would be better?

3 10

[PATCH] koji_utils: allow package download caching
by Kamil Paral 10 May '11

10 May '11

Please review: https://fedorahosted.org/reviewboard/r/143/

2 3

How do you test AutoQA?
by Tim Flink 02 May '11

02 May '11

I really can't tell if this is a particularly bad pain point for me since I work remotely and my internet connection isn't great (frequent SSL timeouts on koji, package downloads take forever) but I was wondering how you all went about testing AutoQA. For me, it really depends on what I'm trying to poke at. Most of the time, I'll run 'watch-koji-builds.py --verbose' if I'm trying to do some general testing. I keep track of the events that are called if I want to run something again ('autoqa post-bodhi-update-batch --targettag dist-f13-updates --arch x86_64 --arch i386 oxygen-gtk-1.0.4-1.fc13' as an example). If I'm trying to poke at something very specific, I find myself manually cobbling together some one-off script that runs something specific, like depcheck or upgradepath. Testing the interaction with Koji or Bodhi? I still haven't figured out a good way to do that. Thus far, I have been hacking in print statements into bodhi_utils or koji_utils but that doesn't quite cover everything. I ask because speaking for myself, I'm human. The more of a PITA it is to test something, the more likely I am to not do it or limit the number of times that I do test it. I think that I've been pretty good about testing stuff before pushing to master or stable, but I'm bothered by the amount of time I'm wasting on setting up tests and figuring out ways that I can trick AutoQA into going down code paths I want to test. I'm not saying that testing is a waste of time, just thinking that some of this test setup time could be much better spent on coding or additional testing. This isn't meant to be aimless complaining. I have some ideas on how to make the testing of AutoQA easier but I wanted to know if I was missing something obvious before I went too far down that road. Thanks, Tim

3 9

Proposed Change in Focus for 0.5.0
by Tim Flink 29 Apr '11

29 Apr '11

This ended up being rather long, so as an executive summary: I'm proposing that we change the 0.5.0 release of AutoQA to focus on getting the best information and the least noise to package maintainers. The focus would be on decreasing the number of emails that maintainers are receiving and improving the understandability of our logs (focusing on depcheck and upgradepath). This is a proposal, and I'm hoping to spark a discussion with this thread; not dictate a change to our roadmap. ------------------------------------------------------------ After the thread in devel@ about the volume of email coming out of bodhi for AutoQA 'PASSED' comments [1], I started thinking more about some of my past experiences with user complaints about the information they're presented with. My major concern is this: low signal to noise ratio (SNR) in output leads to users ignoring or refusing to use the tool as a whole. In my experience, once users start ignoring the tool it is a very difficult, uphill battle to get them to stop ignoring/hating/distrusting it. At the moment, I think that we have two major SNR ratio issues in AutoQA: comment emails coming out of bodhi and log files (especially depcheck and upgradepath). I can't seem to find the book I have that discusses it right now, but one of the things that I believe strongly is: testing output that is difficult for a human (with sufficient background knowledge) to understand isn't much better than not testing at all and is actually worse in some cases (we're not at that point, though). As I heard James Bach [2] put it; building software is like building a house. The developers are the construction workers and the testers are responsible for shining light on the places of the house that need work. The people with the light can't directly build the house but the construction workers can build a better house when the light is shining on the most important things to fix. When we have a low SNR, our output is muddled which isn't very far from putting tissue paper over the lights in the building analogy. The light might be in the right places but it's really hard for the builders to tell where exactly the light is pointing and have a harder time fixing those issues. So, what is the point of all this? Basically, I'm proposing that we hijack our current plans for 0.5.0 and re-focus on improving the SNR and usability of our current tests. Specifically, I would like to see us focus on two things: 1) Stop spamming maintainers with not-needed comment updates from bodhi - The current proposal is covered in #314 [3] 2) Improve our logging - focusing on depcheck and upgradepath - Goal 1: maintainers should be able to find the information they need about why their package failed within 30 seconds of opening the log file. - Goal 2: users should be able to easily find documentation on what a test is supposed to do and examples on how to triage a failure using our logging output. How could we get there? 1) This seems pretty straight forward to me. I didn't think that the bodhi side of things would get implemented quite that quickly or I would have said something earlier. From my conversations with Kamil and James, I don't think that there is going to be much resistance to this solution, though. 2) This would take a bit more work. I'm not sure if the exact goal of 30 seconds is possible but I'm of the opinion that it is a good place to start. We could start by looking at what we want the output to look like, determine if that is possible and do some user testing to get input from actual maintainers (not directly involved with AutoQA). I know why things are the way that they are and I don't disagree with those design decisions. Blamestorming is pointless and counter-productive, anyways. I'm interested in finding a solution :) And now, discussion time! Thoughts, suggestions, complaints? Tim [1] http://lists.fedoraproject.org/pipermail/devel/2011-April/150901.html [2] http://www.satisfice.com/aboutjames.shtml [3] https://fedorahosted.org/autoqa/ticket/314

4 6

AutoQA 0.4.7 released
by Kamil Paral 29 Apr '11

29 Apr '11

Please read: http://kparal.wordpress.com/2011/04/29/autoqa-0-4-7-released/

2 1

[AutoQA] #238: Move ResultsDB to Turbogears2 application
by fedora-badges 28 Apr '11

28 Apr '11

#238: Move ResultsDB to Turbogears2 application ----------------------------+----------------------------------------------- Reporter: jskladan | Owner: jskladan Type: task | Status: new Priority: major | Milestone: Resultdb Component: infrastructure | Keywords: ----------------------------+----------------------------------------------- Because the current implementation has noticeable disadvantages (speed, database management, ...), I'd like to rewrite the resultsdb backend using TG2. This will also provide a json-rpc, added to the xml-rpc interface. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/238> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[PUSHED] depcheck and koji_utils changes
by Kamil Paral 27 Apr '11

27 Apr '11

For those interested, I just pushed to master a few changes in depcheck test and koji_utils library: commit 53aa452e605b61944d6b657bc09813972be2b262 Author: Kamil Páral <kparal(a)redhat.com> Date: Wed Apr 27 17:12:56 2011 +0200 depcheck: download only necessary archs If depcheck is run on i686, don't download x86_64 RPMs because we don't need them. commit 549517418759080f18b6e37ae3801b72adbfc879 Author: Kamil Páral <kparal(a)redhat.com> Date: Wed Apr 27 14:41:48 2011 +0200 depcheck: unittests require x86_64 machine, abort for other archs commit c7ef84ea635b5c862fc7c087f504eea230786116 Author: Kamil Páral <kparal(a)redhat.com> Date: Wed Apr 27 13:11:41 2011 +0200 koji_utils: rename ensure_connection() to check_connection() The new name fits better. Also use check_connection() mainly in tests' setup() phase to check whether Koji is running. It is not necessary to execute it then again in run_once() stage, that doesn't help anything.

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel April 2011