autoqa-devel April 2011

autoqa-devel@lists.fedorahosted.org

10 participants
99 discussions

Re: Avoiding conditioning ignorance towards AutoQA
by Kamil Paral 27 Apr '11

27 Apr '11

> Hi fellow Fedorans. > > Recently, AutoQA has been introduced to catch typical problems early > in > the update process. In general, I appreciate that effort, but > currently > I find myself in a phase of conditioning ignorance towards AutoQA, > essentially because it is drowning me in irrelevant information. The > current case why I'm writing is > https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15. > > Here are two ideas to make AutoQA relevant, less time-consuming, and > more helpful. In short: good QA is always quiet, only if there is a > problem it communicates. > > - Post only errors > It is common, for example, in automated build or continuous > integration > systems to send out emails only on errors. Similar goes for Unix > tools, > which tend to be quiet if everything is ok, and only bother you with > output if something is not. Therefore, I propose to have AutoQA > messages > posted only in case that there has been an error. How can you then distinguish an update for which the tests have passed from an update for which the tests haven't yet been executed? Moreover, currently not all updates are tested. Sometimes our tests simply don't work properly. Not just the updates are being tested, the whole AutoQA is being tested (and developed) in this whole effort. > > - Accumulate error messages > An email is sent for every single comment to Bodhi. In the case of > AutoQA, it causes one email per platform. It increases the load of > email > tremendously to deal with, which in turn makes me ignore it. > Therefore, > I propose to accumulate messages for all platforms. We have that in plan, believe me. > Combined with the > earlier proposal, the states for all platforms should be collected by > an > intermediate node, and if and only if a test failed on any of the > platforms, one message with all status messages is posted to the > update. Sending Bodhi comments is just a quick way how to inform the maintainers. We are working on a results database with API that other Fedora services (Koji, Bodhi) could query and use the results as they seem fit. For most tests I expect it will be similar to what you describe. But that's future. Until that's implemented we can only either send comments to Bodhi or send no comments at all. > > On a related note: it'd be much appreciated if Bodhi would provide an > option to get a daily digest with all comments of all the packages I'm > involved with. Great idea, you can ask lmacken about that (or create ticket in its Trac). Or, you can filter your emails and check the relevant folder once a day :) > > I hope the fine folks of the AutoQA effort take these proposals into > account when proceeding in the development of the system and help me > to > stop ignorance from taking over. It will take some time, but we see the deficiencies, same as you do. We try to improve as fast as possible. > > Regards, > Tim Thanks, Kamil PS: We have a special mailing list for AutoQA: https://fedorahosted.org/mailman/listinfo/autoqa-devel

1 0

Re: AutoQA upgrade path failure makes no sense to me
by Kamil Paral 27 Apr '11

27 Apr '11

> Here is the update: > > https://admin.fedoraproject.org/updates/libguestfs-1.6.2-1.fc13.7 > > Here's the failed test result: > > http://autoqa.fedoraproject.org/results/87921-autotest/qa03.c.fedoraproject… > > This is pretty confusing. > > Why does it mention other packages != libguestfs? We currently test all pending packages at once. We're working on better presentation of the results. Of course only libguestfs results are relevant to you. > > It seems to be saying after much head-scratching and decoding that an > update to dist-f14 (ie. the original ancient released version without > updates) would fail, but how could anyone do that? I'm sorry, this is a bug in AutoQA (depcheck test): https://fedorahosted.org/autoqa/ticket/309 It will be solved in AutoQA 0.4.7, which will be deployed any day now. Thanks, Kamil PS: If you write directly to autoqa-devel list [1] next time, you'll get faster response. [1] https://fedorahosted.org/mailman/listinfo/autoqa-devel

1 0

AutoQA 0.4.7 coming soon - please test
by Kamil Paral 27 Apr '11

27 Apr '11

All the tickets from our 0.4.7 milestone have been closed: https://fedorahosted.org/autoqa/milestone/0.4.7 The changeset is here: $ git log v0.4.6-1..stable --oneline --reverse fed73fe bodhi_utils: don't place dot after hyperlink 6c08990 Upgradepath: 294 - Included link to help maintainers resolve upgradepath failures b79fd06 fix for #308 - keeps depcheck from crashing or scheduling tests when there are no builds to test d224d9a upgradepath: don't crash when Bodhi update is broken d7e830f depcheck: detect broken Bodhi updates 423f750 bodhi_utils: emphasize in comments that AutoQA is not enforcing anything f266d6d 306 - bodhi_utils: don't report changed updates as tested 32ffb22 upgradepath: fix the algorithm when pushing to older -updates repo Please spend some time running a few tests. If no problems are found, we could release 0.4.7 tomorrow.

1 0

[AutoQA] #230: upgradepath: current implementation is broken when pushing older packages
by fedora-badges 27 Apr '11

27 Apr '11

#230: upgradepath: current implementation is broken when pushing older packages --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Package Update Acceptance Test Plan Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- Currently upgradepath takes into account only the given NVR and compares it with other repositories. This approach works, except for one use case - when you want to push an older package version into the repository. Technically you can have multiple package versions in a single repository. Yum prefers the most recent one, but you can have more of them there. It's hard to imagine a use case where you want to push an older version, but let's say for example an older kernel that is needed for hardware compatibility reasons. In this case, our current upgradepath fails - it supposes the given NVR is always the latest. We need to modify upgradepath in such a way, that is supposes that given NVR has been pushed into the repository and then traverses all repositories from bottom up and checks for upgradepath constraint everywhere - including the repo you pushed into, including all package versions which might be there. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/230> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #309: Upgradepath test checks for stable repository only even if updates repository exists
by fedora-badges 27 Apr '11

27 Apr '11

#309: Upgradepath test checks for stable repository only even if updates repository exists --------------------+------------------------------------------------------- Reporter: ppisar | Owner: Type: defect | Status: new Priority: major | Milestone: Future tasks Component: core | Keywords: --------------------+------------------------------------------------------- http://autoqa.fedoraproject.org/results/83753-autotest/qa03.c.fedoraproject… {{{ ======================================== 3:perl-version-0.82-2.fc13 into dist-f13-updates ======================================== [ OK ] dist-f13 No such package: perl-version [ OK ] dist-f13-updates Latest package: 3:perl-version-0.82-1.fc13 [FAIL] dist-f14 Latest package: 3:perl-version-0.82-1.fc14 Error: Requested package must be less than or equal to the latest package [ OK ] dist-f14-updates Latest package: 3:perl-version-0.88-1.fc14 [ OK ] dist-f15 Latest package: 3:perl-version-0.88-3.fc15 [ OK ] dist-f15-updates No such package: perl-version [ OK ] dist-f16 Latest package: 3:perl-version-0.88-3.fc16 RESULT: FAILED }}} This is normal situation allowed by current packaging guidelines: You can step over EVRA in dist-f14 by dist-f13-updates package if there is higher in dist-f14-updates. IOW, you should merge dist-fN and dist-fN-updates before instead of checking sole dist-fN if dist-fN-updates exists. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/309> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 4

[AutoQA] #306: bodhi_utils: don't report changed updates as tested
by fedora-badges 27 Apr '11

27 Apr '11

#306: bodhi_utils: don't report changed updates as tested --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: 0.4.7 Component: core | Keywords: --------------------+------------------------------------------------------- Depcheck currently uses bodhi comment detection (bodhi_already_commented()) to get list of previously accepted updates. The problem is that the update might have changed in the meantime. E.g. a new build might have been added to that update, removed, etc. When we search for past AutoQA comments, we have to detect whether the update was changed after the last matching comment. If it was, we should claim that this update wasn't tested at all. Example of a changed update: https://admin.fedoraproject.org/updates/control-center-3.0.0-1.fc15,gnome- settings-daemon-3.0.0.1-1.fc15 One obvious way is to detect "xxx has edited this update" strings. A better way could be possible by looking into the bodhi update object and searching for last-modified timestamps or something like that. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/306> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 6

Re: AutoQA future task - rebuilding Rawhide from scratch?
by Kamil Paral 23 Apr '11

23 Apr '11

> I do not have any other questions. I suppose you should ask Matt. I asked him whether he could lend his machines to Fedora Infrastructure. The reply is below: ==== Unfortunately, I can't do that. The systems I've been using are past end of service life (out of warranty), just what I could cobble together. Given that it cakes 4 days on 10 servers, if you had say 4 servers it could realistically run every two weeks, which, IMHO, would be "regular enough" for starters. Seth was looking at using cloud resources too. ==== (I didn't even imagine it should be run _that_ often. I supposed once a month or so.) The only possible option then seems to be the cloud resources Seth is looking into.

1 0

AutoQA future task - rebuilding Rawhide from scratch?
by Kamil Paral 22 Apr '11

22 Apr '11

Hello, I have been spoken to by Marcela Mašláňová about "The Future of FTBFS". See this thread: http://lists.fedoraproject.org/pipermail/devel/2011-April/150310.html IIUC (Is there an abbreviation for "I'm not a developer"?) the problem is as follows: * Matt Domsch from Dell used to rebuild *all* packages from Rawhide periodically (so-called "mass rebuild"). When some package failed to build, he reported errors against that package. * This testing ensured we often find build problems early in the release process. Without it there is a chance that we discover the build failures only when a new build of that package is required, which may be shortly before final release or even after that. That's a problem. * Mass-rebuilds in Koji are not done frequently (maybe once a year), so they can't cover this issue. * Matt can't do this testing anymore. Marcela asked me whether AutoQA could be used for that. Matt's tools (scripts, etc) should be available. * I asked Marcela to inquire more about some details. I have attached the discussion below (read from bottom up). What are your thoughts? Is that something AutoQA can and should handle? Do we (will we) have enough hardware to be able to do that? According to our current priorities, is that even something we are able to implement in some reasonable time (under a year)? As for the last question, I think it clearly fits our current effort to provide generic Fedora-related tests. OTOH we still have many generic tests to finish (either un-started or semi-finished) and before that we need to concentrate on architecture first (ResultDB etc.). I'm afraid to have complex tests running without solid architecture basis beneath it. In that respect unless we all agree this is a top-priority next-to-work-on test (and provided that we have enough hardware for it) I don't think we're able to run it soon. Do we need some more information I should ask Matt for? Thanks, Kamil ----- Forwarded Message ----- From: "Matt Domsch" <Matt_Domsch(a)Dell.com> To: mmaslano(a)redhat.com Cc: kparal(a)redhat.com, skvidal(a)fedoraproject.org Sent: Tuesday, April 12, 2011 7:30:14 PM Subject: RE: future of FTBFS Seth was asking me the same question. My environment consists of: Builders: 10 PowerEdge 1955 servers, each with 2x4core 3.0GHz CPUs, 8GB RAM, 2x144GB disks. Disk space on the builders is mostly used for the buildroots, and swap for large (or several parallel) jobs that can exceed 8GB in a tmpfs buildroot. http and NFS server with space for the current rawhide tree (daily rsync), a hardlinked copy of the rawhide tree from the day the build starts (initially zero space, but growing to the size of the full rawhide tree as rawhide moves on), and space for the newly built tree results to land. ~250GB total. One more server as the "master" that kicks off all the jobs to the builders. This can be anything, technically even one of the builders. It takes this setup ~30 hours to build all 10,000 SRPMs twice, once for each of i386 and x86_64. That was before the tmpfs change in F14, which prevents mock from using tmpfs for its buildroots. Now it takes ~96 hours with disk-backed buildroots. In my setup, each builder runs 4 jobs concurrently, two for each architecture. They're mostly I/O-bound, hence the disk-backed buildroots are so much slower. There is often plenty of memory and CPU left over; not always (depends on the size of the jobs that happen to be handed to each builder concurrently) - sometimes they're CPU-bound, but not mostly. Seth was going to look into using cloud-based builders. I think this is a great idea, provided you have a place to store the build results outside of the builders themselves, and have network-local copy of the SRPM tree you're starting from and a copy of the buildroot repositories network-local too. Thanks for your interest in taking this on! -Matt -- Matt Domsch Technology Strategist Dell | Office of the CTO -----Original Message----- From: Marcela Mašláňová [mailto:mmaslano@redhat.com] Sent: Tuesday, April 12, 2011 4:02 AM To: Domsch, Matt Cc: kparal(a)redhat.com Subject: future of FTBFS Hello Matt, I was speaking with Fedora QA (Kamil Páral) about FTBFS. It might be possible to run it as one of project of QA, but they'd like to know some details. What are the hw requirements (disk space, number of machines used to run it, how long it takes (days?), is it needed to have installed rawhide)? Best regards, Marcela -- Marcela Mašláňová BaseOS team Brno

4 5

[AutoQA] #310: Improve Bodhi comment wording
by fedora-badges 22 Apr '11

22 Apr '11

#310: Improve Bodhi comment wording --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.7 Component: core | Keywords: --------------------+------------------------------------------------------- Our Bodhi comments cause confusion for some package maintainers: http://lists.fedoraproject.org/pipermail/devel/2011-April/150639.html We should change the comments a little to stress that AutoQA does not block any updates in any way. Proposed new version of the comment: ''AutoQA: depcheck test PASSED ... Result: http://.... '''(This result is purely informational, AutoQA never blocks your update.)''''' -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/310> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

[AutoQA] #214: conflicts: fails with parent repositories
by fedora-badges 22 Apr '11

22 Apr '11

#214: conflicts: fails with parent repositories --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- These two commands work (although the package counts are somewhat weirdly same): {{{ # repoclosure --tempcache --newest --repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os --repoid=target Added target repo from /root/http:/download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os Reading in repository metadata - please wait.... Checking Dependencies /usr/lib/python2.6/site-packages/yum/packages.py:401: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if prcotuple in self.prco[prcotype]: Repos looked at: 1 target Num Packages in Repos: 20806 }}} {{{ # repoclosure --tempcache --newest --repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64 --repoid=target Added target repo from /root/http:/download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64 Reading in repository metadata - please wait.... Checking Dependencies /usr/lib/python2.6/site-packages/yum/packages.py:401: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if prcotuple in self.prco[prcotype]: Repos looked at: 1 target Num Packages in Repos: 20806 }}} But if you combine it: {{{ # repoclosure --tempcache --newest --repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64 --repoid=target --repofrompath=parent1,http://download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os --repoid=parent1 Added target repo from /root/http:/download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64 Added parent1 repo from /root/http:/download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os Reading in repository metadata - please wait.... Cannot retrieve repository metadata (repomd.xml) for repository: parent1. Please verify its path and try again Some dependencies may not be complete for this repository Run as root to get all dependencies or use -t to enable a user temp cache Checking Dependencies Cannot retrieve repository metadata (repomd.xml) for repository: parent1. Please verify its path and try again }}} Something is broken here. {{{ yum-utils-1.1.28-1.fc13.noarch yum-metadata-parser-1.1.4-1.fc13.x86_64 yum-3.2.27-4.fc13.noarch }}} -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/214> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 6

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel April 2011