Production out of disk space [again]
by James Laska
Hey gang,
Instead of resolving this issue manually by removing older test results,
I thought others would might have some better ideas for resolving the
immediate issue of no free disk space.
I'm checking in with infrastructure to see if there are any free disks
that can be assigned to our guest to alleviate matters. In the
meantime, any recommendations to resolve this issue in the short-term
(today/tomorrow) so we can get jobs processing again?
# of jobs scheduled per day ...
20110502 - 38 # small since this is when we fixed the
tmpwatch arg
20110503 - 536
20110504 - 223
20110505 - 0 # out of disk space
Size of results directory (18G available on system)
14G /usr/share/autotest/results
Jobs with *obscenely* large logs ...
1.2G /usr/share/autotest/results/91523-autotest
3.0G /usr/share/autotest/results/91524-autotest
1.9G /usr/share/autotest/results/91550-autotest
1.5G /usr/share/autotest/results/91604-autotest
2.0G /usr/share/autotest/results/91605-autotest
967M /usr/share/autotest/results/91549-autotest
252M /usr/share/autotest/results/91705-autotest
933M /usr/share/autotest/results/91635-autotest
685M /usr/share/autotest/results/91634-autotest
536M /usr/share/autotest/results/92318-autotest
200M /usr/share/autotest/results/92317-autotest
Thanks,
James
13 years
2011-05-04 AutoQA Discussion Minutes
by Tim Flink
While a little disorganized, I think that our phone meeting earlier
today was useful. Hopefully everyone else is of the same mind even
though it was a little longer than initially planned :)
Rather than regurgitating all of the information covered in the
discussion, I have started documenting our planning in Trac [1]. It's
not done but I will finish the initial pages later today or tomorrow and
start creating more tickets to reflect our discussion today.
Feedback on formatting and/or usefullness would be appreciated. The raw
etherpad document produced is also available [2].
Tim
Discussed:
- What is the important feedback information for maintainers?
- What could the test logs look like?
- What are our design goals for test log output?
- When should we be sending email notifications to maintainers?
- What are our next steps?
General Agreed:
1. Evaluate release ready-ness on a regular basis (every 2 weeks to
start)
2. Write code for email defaults with an eye towards configurability
- Only as far as this makes sense to do
- Per-user email configuration will not be a part of 0.5.x
3. For log formatting, plaintext logs are a must-have. HTML logs would
be nice to have, but will likely not be a part of 0.5.x
General Todo:
1. Create mockups for plaintext logs (specifically the "pretty" logs)
2. Determine timebox for mockups, planning
Action Items:
1. tflink to reach out to lmacken, infra on deployment timeline for
Bodhi enhancements for email notification control
2. jlaska to reach out to dmalcolm and esantiago for thoughts on
maintainer result notification frequency
[1] https://fedorahosted.org/autoqa/wiki/Planning050series
[2]
http://tflink.fedorapeople.org/autoqa/planning/05x/2011-05-04_meeting_ope...
13 years
2011-05-04 @ **13:30 UTC** AutoQA Discussion: Making Results More Maintainer-Friendly
by Tim Flink
# What: AutoQA discussion - Making Results More Maintainer-Friendly
# Who: AutoQA Developers and Other Interested Parties
# Where: #fedora-qa (IRC) and teleconference
# When: 2011-05-04 @ 13:30 UTC (09:30 EDT, 15:30 CEST)
# Why: Because we care
As the theme for the 0.5.0 release has changed to "Making AutoQA More
Maintainer-Friendly", we need to start figuring out what that means
exactly. To start figuring more of this out, we'll be holding a
brainstorming session over phone, IRC and gobby tomorrow @ 13:30 UTC.
I made some quick mockups of ideas that I have for making our logs
easier to use. In the interest of getting the most possible ideas (and
not influencing other ideas before the brainstorm), I'm holding off
until the meeting to show them. Other mockups/demos are very much
encouraged!
Tim
Dial-in information:
Participant Code: 684-645-9442
Czech Republic Dial-In: 800701035
United States Dial-In: 2127295016
Let us know if you need a different local dial-in number
Other needed info:
http://fedoraproject.org/wiki/Communicate/GobbyHowTo
Proposed Agenda:
* What is the most important triage information for maintainers?
* What could our logs look like?
- Depcheck
- Upgradepath
* When should we be sending emails to maintainers?
13 years
[AutoQA] #323: Protect against extremely large log files
by fedora-badges
#323: Protect against extremely large log files
-------------------------+--------------------------------------------------
Reporter: kparal | Owner:
Type: enhancement | Status: new
Priority: major | Milestone: Nice to have soon
Component: production | Keywords:
-------------------------+--------------------------------------------------
See the log tflink posted in
https://fedorahosted.org/autoqa/ticket/321#comment:2 .
It is 1 GB in size (and there are several copies in the results
directory). We need to limit that.
We could probably watch the log being created and if it exceeds certain
quota (like 100 MB) we could crash the test.
Another (not so good, since it wastes bandwidth) approach is to let the
log be created in any size, but have a cron job on the server to
periodically go through all log files and truncate any large ones. It
could also compress them and symlink the original file to the compressed
one (that could be combined with the first approach).
Or maybe there is support for this kind of watchdog in autotest? That
would be great, let's have a look.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/323>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years
[AutoQA] #320: Koji uses also i386 arch, not just i686
by fedora-badges
#320: Koji uses also i386 arch, not just i686
--------------------+-------------------------------------------------------
Reporter: kparal | Owner:
Type: defect | Status: new
Priority: major | Milestone: Future tasks
Component: core | Keywords:
--------------------+-------------------------------------------------------
Today I discovered that although Koji uses "i686" arch suffix for almost
all its packages (like "Miro-3.5.1-1.fc15.i686.rpm"), there are a few
packages that have "i386" (and no "i686") suffix.
{{{
$ koji list-tagged dist-f15 --rpms --latest > f15-rpms-latest
$ grep i686 f15-rpms-latest | wc -l
17519
$ grep i386 f15-rpms-latest
chealpix-2.13a-2.fc14.i386
chealpix-devel-2.13a-2.fc14.i386
elilo-3.6-9.i386
elilo-debuginfo-3.6-9.i386
frysk-0.4-30.fc15.i386
frysk-debuginfo-0.4-30.fc15.i386
frysk-devel-0.4-30.fc15.i386
frysk-gnome-0.4-30.fc15.i386
gnome-applet-cpufire-1.6-3.fc14.i386
gnome-applet-cpufire-debuginfo-1.6-3.fc14.i386
healpix-2.13a-2.fc14.i386
healpix-c++-2.13a-2.fc14.i386
healpix-c++-devel-2.13a-2.fc14.i386
healpix-debuginfo-2.13a-2.fc14.i386
healpix-devel-2.13a-2.fc14.i386
}}}
That very possibly breaks some of our koji_utils code (if not other). We
need to
1. find out whether the aforementioned rpms are broken builds and should
be rebuilt
2. or whether this is acceptable state and we should fix our code
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/320>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years
How do you test AutoQA?
by Tim Flink
I really can't tell if this is a particularly bad pain point for me
since I work remotely and my internet connection isn't great (frequent
SSL timeouts on koji, package downloads take forever) but I was
wondering how you all went about testing AutoQA.
For me, it really depends on what I'm trying to poke at. Most of the
time, I'll run 'watch-koji-builds.py --verbose' if I'm trying to do some
general testing. I keep track of the events that are called if I want to
run something again ('autoqa post-bodhi-update-batch --targettag
dist-f13-updates --arch x86_64 --arch i386 oxygen-gtk-1.0.4-1.fc13' as
an example).
If I'm trying to poke at something very specific, I find myself manually
cobbling together some one-off script that runs something specific, like
depcheck or upgradepath.
Testing the interaction with Koji or Bodhi? I still haven't figured out
a good way to do that. Thus far, I have been hacking in print statements
into bodhi_utils or koji_utils but that doesn't quite cover everything.
I ask because speaking for myself, I'm human. The more of a PITA it is
to test something, the more likely I am to not do it or limit the number
of times that I do test it. I think that I've been pretty good about
testing stuff before pushing to master or stable, but I'm bothered by
the amount of time I'm wasting on setting up tests and figuring out ways
that I can trick AutoQA into going down code paths I want to test.
I'm not saying that testing is a waste of time, just thinking that some
of this test setup time could be much better spent on coding or
additional testing.
This isn't meant to be aimless complaining. I have some ideas on how to
make the testing of AutoQA easier but I wanted to know if I was missing
something obvious before I went too far down that road.
Thanks,
Tim
13 years
