Re: Avoiding conditioning ignorance towards AutoQA
by Kamil Paral
> Hi fellow Fedorans.
>
> Recently, AutoQA has been introduced to catch typical problems early
> in
> the update process. In general, I appreciate that effort, but
> currently
> I find myself in a phase of conditioning ignorance towards AutoQA,
> essentially because it is drowning me in irrelevant information. The
> current case why I'm writing is
> https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15.
>
> Here are two ideas to make AutoQA relevant, less time-consuming, and
> more helpful. In short: good QA is always quiet, only if there is a
> problem it communicates.
>
> - Post only errors
> It is common, for example, in automated build or continuous
> integration
> systems to send out emails only on errors. Similar goes for Unix
> tools,
> which tend to be quiet if everything is ok, and only bother you with
> output if something is not. Therefore, I propose to have AutoQA
> messages
> posted only in case that there has been an error.
How can you then distinguish an update for which the tests have passed from an update for which the tests haven't yet been executed?
Moreover, currently not all updates are tested. Sometimes our tests simply don't work properly. Not just the updates are being tested, the whole AutoQA is being tested (and developed) in this whole effort.
>
> - Accumulate error messages
> An email is sent for every single comment to Bodhi. In the case of
> AutoQA, it causes one email per platform. It increases the load of
> email
> tremendously to deal with, which in turn makes me ignore it.
> Therefore,
> I propose to accumulate messages for all platforms.
We have that in plan, believe me.
> Combined with the
> earlier proposal, the states for all platforms should be collected by
> an
> intermediate node, and if and only if a test failed on any of the
> platforms, one message with all status messages is posted to the
> update.
Sending Bodhi comments is just a quick way how to inform the maintainers. We are working on a results database with API that other Fedora services (Koji, Bodhi) could query and use the results as they seem fit. For most tests I expect it will be similar to what you describe. But that's future. Until that's implemented we can only either send comments to Bodhi or send no comments at all.
>
> On a related note: it'd be much appreciated if Bodhi would provide an
> option to get a daily digest with all comments of all the packages I'm
> involved with.
Great idea, you can ask lmacken about that (or create ticket in its Trac).
Or, you can filter your emails and check the relevant folder once a day :)
>
> I hope the fine folks of the AutoQA effort take these proposals into
> account when proceeding in the development of the system and help me
> to
> stop ignorance from taking over.
It will take some time, but we see the deficiencies, same as you do.
We try to improve as fast as possible.
>
> Regards,
> Tim
Thanks,
Kamil
PS: We have a special mailing list for AutoQA:
https://fedorahosted.org/mailman/listinfo/autoqa-devel
13 years
AutoQA 0.4.7 coming soon - please test
by Kamil Paral
All the tickets from our 0.4.7 milestone have been closed:
https://fedorahosted.org/autoqa/milestone/0.4.7
The changeset is here:
$ git log v0.4.6-1..stable --oneline --reverse
fed73fe bodhi_utils: don't place dot after hyperlink
6c08990 Upgradepath: 294 - Included link to help maintainers resolve upgradepath failures
b79fd06 fix for #308 - keeps depcheck from crashing or scheduling tests when there are no builds to test
d224d9a upgradepath: don't crash when Bodhi update is broken
d7e830f depcheck: detect broken Bodhi updates
423f750 bodhi_utils: emphasize in comments that AutoQA is not enforcing anything
f266d6d 306 - bodhi_utils: don't report changed updates as tested
32ffb22 upgradepath: fix the algorithm when pushing to older -updates repo
Please spend some time running a few tests. If no problems are found, we could release 0.4.7 tomorrow.
13 years
[AutoQA] #230: upgradepath: current implementation is broken when pushing older packages
by fedora-badges
#230: upgradepath: current implementation is broken when pushing older packages
--------------------+-------------------------------------------------------
Reporter: kparal | Owner:
Type: defect | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: tests | Version: 1.0
Keywords: |
--------------------+-------------------------------------------------------
Currently upgradepath takes into account only the given NVR and compares
it with other repositories. This approach works, except for one use case -
when you want to push an older package version into the repository.
Technically you can have multiple package versions in a single repository.
Yum prefers the most recent one, but you can have more of them there. It's
hard to imagine a use case where you want to push an older version, but
let's say for example an older kernel that is needed for hardware
compatibility reasons. In this case, our current upgradepath fails - it
supposes the given NVR is always the latest.
We need to modify upgradepath in such a way, that is supposes that given
NVR has been pushed into the repository and then traverses all
repositories from bottom up and checks for upgradepath constraint
everywhere - including the repo you pushed into, including all package
versions which might be there.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/230>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years
[AutoQA] #309: Upgradepath test checks for stable repository only even if updates repository exists
by fedora-badges
#309: Upgradepath test checks for stable repository only even if updates
repository exists
--------------------+-------------------------------------------------------
Reporter: ppisar | Owner:
Type: defect | Status: new
Priority: major | Milestone: Future tasks
Component: core | Keywords:
--------------------+-------------------------------------------------------
http://autoqa.fedoraproject.org/results/83753-autotest/qa03.c.fedoraproje...
{{{
========================================
3:perl-version-0.82-2.fc13 into dist-f13-updates
========================================
[ OK ] dist-f13
No such package: perl-version
[ OK ] dist-f13-updates
Latest package: 3:perl-version-0.82-1.fc13
[FAIL] dist-f14
Latest package: 3:perl-version-0.82-1.fc14
Error: Requested package must be less than or equal to the latest
package
[ OK ] dist-f14-updates
Latest package: 3:perl-version-0.88-1.fc14
[ OK ] dist-f15
Latest package: 3:perl-version-0.88-3.fc15
[ OK ] dist-f15-updates
No such package: perl-version
[ OK ] dist-f16
Latest package: 3:perl-version-0.88-3.fc16
RESULT: FAILED
}}}
This is normal situation allowed by current packaging guidelines: You can
step over EVRA in dist-f14 by dist-f13-updates package if there is higher
in dist-f14-updates.
IOW, you should merge dist-fN and dist-fN-updates before instead of
checking sole dist-fN if dist-fN-updates exists.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/309>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years
[AutoQA] #306: bodhi_utils: don't report changed updates as tested
by fedora-badges
#306: bodhi_utils: don't report changed updates as tested
--------------------+-------------------------------------------------------
Reporter: kparal | Owner:
Type: defect | Status: new
Priority: major | Milestone: 0.4.7
Component: core | Keywords:
--------------------+-------------------------------------------------------
Depcheck currently uses bodhi comment detection
(bodhi_already_commented()) to get list of previously accepted updates.
The problem is that the update might have changed in the meantime. E.g. a
new build might have been added to that update, removed, etc.
When we search for past AutoQA comments, we have to detect whether the
update was changed after the last matching comment. If it was, we should
claim that this update wasn't tested at all.
Example of a changed update:
https://admin.fedoraproject.org/updates/control-center-3.0.0-1.fc15,gnome-
settings-daemon-3.0.0.1-1.fc15
One obvious way is to detect "xxx has edited this update" strings. A
better way could be possible by looking into the bodhi update object and
searching for last-modified timestamps or something like that.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/306>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years
Re: AutoQA future task - rebuilding Rawhide from scratch?
by Kamil Paral
> I do not have any other questions. I suppose you should ask Matt.
I asked him whether he could lend his machines to Fedora Infrastructure. The reply is below:
====
Unfortunately, I can't do that. The systems I've been using are past end of service life (out of warranty), just what I could cobble together.
Given that it cakes 4 days on 10 servers, if you had say 4 servers it could realistically run every two weeks, which, IMHO, would be "regular enough" for starters. Seth was looking at using cloud resources too.
====
(I didn't even imagine it should be run _that_ often. I supposed once a month or so.)
The only possible option then seems to be the cloud resources Seth is looking into.
13 years
AutoQA future task - rebuilding Rawhide from scratch?
by Kamil Paral
Hello,
I have been spoken to by Marcela Mašláňová about "The Future of FTBFS". See this thread:
http://lists.fedoraproject.org/pipermail/devel/2011-April/150310.html
IIUC (Is there an abbreviation for "I'm not a developer"?) the problem is as follows:
* Matt Domsch from Dell used to rebuild *all* packages from Rawhide periodically (so-called "mass rebuild"). When some package failed to build, he reported errors against that package.
* This testing ensured we often find build problems early in the release process. Without it there is a chance that we discover the build failures only when a new build of that package is required, which may be shortly before final release or even after that. That's a problem.
* Mass-rebuilds in Koji are not done frequently (maybe once a year), so they can't cover this issue.
* Matt can't do this testing anymore. Marcela asked me whether AutoQA could be used for that. Matt's tools (scripts, etc) should be available.
* I asked Marcela to inquire more about some details. I have attached the discussion below (read from bottom up).
What are your thoughts? Is that something AutoQA can and should handle? Do we (will we) have enough hardware to be able to do that? According to our current priorities, is that even something we are able to implement in some reasonable time (under a year)?
As for the last question, I think it clearly fits our current effort to provide generic Fedora-related tests. OTOH we still have many generic tests to finish (either un-started or semi-finished) and before that we need to concentrate on architecture first (ResultDB etc.). I'm afraid to have complex tests running without solid architecture basis beneath it. In that respect unless we all agree this is a top-priority next-to-work-on test (and provided that we have enough hardware for it) I don't think we're able to run it soon.
Do we need some more information I should ask Matt for?
Thanks,
Kamil
----- Forwarded Message -----
From: "Matt Domsch" <Matt_Domsch(a)Dell.com>
To: mmaslano(a)redhat.com
Cc: kparal(a)redhat.com, skvidal(a)fedoraproject.org
Sent: Tuesday, April 12, 2011 7:30:14 PM
Subject: RE: future of FTBFS
Seth was asking me the same question.
My environment consists of:
Builders: 10 PowerEdge 1955 servers, each with 2x4core 3.0GHz CPUs, 8GB RAM, 2x144GB disks. Disk space on the builders is mostly used for the buildroots, and swap for large (or several parallel) jobs that can exceed 8GB in a tmpfs buildroot.
http and NFS server with space for the current rawhide tree (daily rsync), a hardlinked copy of the rawhide tree from the day the build starts (initially zero space, but growing to the size of the full rawhide tree as rawhide moves on), and space for the newly built tree results to land. ~250GB total.
One more server as the "master" that kicks off all the jobs to the builders. This can be anything, technically even one of the builders.
It takes this setup ~30 hours to build all 10,000 SRPMs twice, once for each of i386 and x86_64. That was before the tmpfs change in F14, which prevents mock from using tmpfs for its buildroots. Now it takes ~96 hours with disk-backed buildroots.
In my setup, each builder runs 4 jobs concurrently, two for each architecture. They're mostly I/O-bound, hence the disk-backed buildroots are so much slower. There is often plenty of memory and CPU left over; not always (depends on the size of the jobs that happen to be handed to each builder concurrently) - sometimes they're CPU-bound, but not mostly.
Seth was going to look into using cloud-based builders. I think this is a great idea, provided you have a place to store the build results outside of the builders themselves, and have network-local copy of the SRPM tree you're starting from and a copy of the buildroot repositories network-local too.
Thanks for your interest in taking this on!
-Matt
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
-----Original Message-----
From: Marcela Mašláňová [mailto:mmaslano@redhat.com]
Sent: Tuesday, April 12, 2011 4:02 AM
To: Domsch, Matt
Cc: kparal(a)redhat.com
Subject: future of FTBFS
Hello Matt,
I was speaking with Fedora QA (Kamil Páral) about FTBFS. It might be possible to run it as one of project of QA, but they'd like to know some details.
What are the hw requirements (disk space, number of machines used to run it, how long it takes (days?), is it needed to have installed rawhide)?
Best regards,
Marcela
--
Marcela Mašláňová
BaseOS team Brno
13 years
[AutoQA] #214: conflicts: fails with parent repositories
by fedora-badges
#214: conflicts: fails with parent repositories
--------------------+-------------------------------------------------------
Reporter: kparal | Owner:
Type: defect | Status: new
Priority: major | Milestone: Hot issues
Component: tests | Version: 1.0
Keywords: |
--------------------+-------------------------------------------------------
These two commands work (although the package counts are somewhat weirdly
same):
{{{
# repoclosure --tempcache --newest
--repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os
--repoid=target
Added target repo from
/root/http:/download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os
Reading in repository metadata - please wait....
Checking Dependencies
/usr/lib/python2.6/site-packages/yum/packages.py:401: UnicodeWarning:
Unicode equal comparison failed to convert both arguments to Unicode -
interpreting them as being unequal
if prcotuple in self.prco[prcotype]:
Repos looked at: 1
target
Num Packages in Repos: 20806
}}}
{{{
# repoclosure --tempcache --newest
--repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64
--repoid=target
Added target repo from
/root/http:/download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64
Reading in repository metadata - please wait....
Checking Dependencies
/usr/lib/python2.6/site-packages/yum/packages.py:401: UnicodeWarning:
Unicode equal comparison failed to convert both arguments to Unicode -
interpreting them as being unequal
if prcotuple in self.prco[prcotype]:
Repos looked at: 1
target
Num Packages in Repos: 20806
}}}
But if you combine it:
{{{
# repoclosure --tempcache --newest
--repofrompath=target,http://download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64
--repoid=target
--repofrompath=parent1,http://download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os
--repoid=parent1
Added target repo from
/root/http:/download.fedoraproject.org/pub/fedora/linux/updates/13/x86_64
Added parent1 repo from
/root/http:/download.fedoraproject.org/pub/fedora/linux/releases/13/Everything/x86_64/os
Reading in repository metadata - please wait....
Cannot retrieve repository metadata (repomd.xml) for repository: parent1.
Please verify its path and try again
Some dependencies may not be complete for this repository
Run as root to get all dependencies or use -t to enable a user temp cache
Checking Dependencies
Cannot retrieve repository metadata (repomd.xml) for repository: parent1.
Please verify its path and try again
}}}
Something is broken here.
{{{
yum-utils-1.1.28-1.fc13.noarch
yum-metadata-parser-1.1.4-1.fc13.x86_64
yum-3.2.27-4.fc13.noarch
}}}
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/214>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years