8:37 a.m.
Greetings,
As noted earlier, I deployed autoqa-0.4.1-1 yesterday. It has been quietly
running jobs for about a day now. After reviewing test results, I noticed
several issues. This mail attempts to summarize the issues and propose patches
to correct the problems.
= conflicts test failures =
1) Failing to find packages or mirrors
This appears to be a result of running the test in a different environment.
When run previously, download.fedoraproject.org would result to an internal Red
Hat mirror which was (for the most part) always updated and accurate. With our
tests now running in Fedora infrastructure, we are at the mercy of
MirrorManager to provide an appropriate mirror. It seems that we rarely get an
updated mirror. I have 2 proposed solutions to this problem.
1. Force our deployed systems to use the infrastructure mirro by updating
repoinfo.conf appropriately (tested and works)
2. Request MirrorManager configuration change so that any requests from
infrastructure systems are always routed to the private infrastructure
mirror
2) Sloooooow....
This test seems to run for a *long* time now (~15 minutes). I seem to remember
this taking, maybe 30 seconds tops. Perhaps something with Fedora 13 changed
how this test runs. Not urgent, but perhaps something to follow-up with
skvidal for guidance/ideas on.
Option#2 is my preference as this mirrors our internal setup, and keeps our
test+configuration consistent inside and outside of Fedora infrastructure.
I'll raise this idea with the infrastructure team shortly.
UPDATE: mmcgrath updated MirrorManager so option#2 seems like the way forward.
This change should propogate sometime today.
= upgradepath test failures =
https://fedorahosted.org/pipermail/autoqa-results/2010-September/035783.html
Traceback (most recent call last):
File "/usr/share/autotest/common_lib/test.py", line 570, in
_call_test_function
return func(*args, **dargs)
File "/usr/share/autotest/common_lib/test.py", line 279, in execute
postprocess_profiled_run, args, dargs)
File "/usr/share/autotest/common_lib/test.py", line 201, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71, in
newf
f_result = f(*args, **kwargs) #call the decorated function
File "/usr/share/autotest/tests/upgradepath/upgradepath.py", line 92, in
run_once
assert kojitag in repotags, 'Requested unsupported kojitag: %s' % kojitag
AssertionError: Requested unsupported kojitag: dist-f14-updates-testing
See proposed patch. I've adjusted upgradepath.py so that it ...
1) Uses autoqa.repoinfo directly, and not through autoqa.koji_utils.repoinfo
2) Includes '-testing' repos when attempting to find applicable tags
= autoqa script =
1) Do not reboot before or after running a job
I'd like to include a previously posted (but not accepted) patch that prevents
test clients from rebooting before or after tests. We have a *lot* of tests
running now, and reboots are expensive as jobs queue up while a system is out
of action.
I've been manually applying this patch to our "production" instance for
months
now. It's safe to include, the only question I have is .. "is there a better,
more appropriate way, to make this change?"
2) Restrict known_hooks to sub-directories of hookdir
Since I moved /etc/cron.d/autoqa to /usr/share/autoqa/autoqa.cron, it was being
listed as a valid hook. I've changed the known_hooks filter to only list
sub-directories. I've also moved autoqa.cron into %{_docdir} in another patch.
Moving it there is in line with several other packages that include recommended
cron files.
= rats_sanity =
This seems to be straightforward, corrected a missing import. I've tested this
fix and it's in production now.
With this fix, the test proceeds but encounters another problem. I believe
this may be a valid failure discovered by the test (see
https://fedorahosted.org/pipermail/autoqa-results/2010-September/035811.html)
= autoqa.spec =
1) Move autoqa.cron into the %{_docdir}
2) Bump the release to autoqa-0.4.2-1
Thanks,
James
8:37 a.m.
New subject: [PATCH 1/5] Use repoinfo module directly (not through autoqa.koji_utils). Don't exclude '-testing' repos.
---
tests/upgradepath/upgradepath.py | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py
index 1811aba..51cdf44 100755
--- a/tests/upgradepath/upgradepath.py
+++ b/tests/upgradepath/upgradepath.py
@@ -24,6 +24,7 @@ import operator
import autoqa.koji_utils
import autoqa.util
import rpmUtils.miscutils
+from autoqa.repoinfo import repoinfo
import autoqa.util
from autoqa.test import AutoQATest
@@ -86,8 +87,10 @@ class upgradepath(AutoQATest):
def run_once(self, envrs, kojitag, **kwargs):
update_id = kwargs['name'] or kwargs['id']
- reponames = [reponame for reponame in autoqa.koji_utils.repoinfo.repos() if not
reponame.endswith('-testing')]
- repotags = [autoqa.koji_utils.repoinfo.getrepo(reponame)['tag'] for
reponame in reponames]
+ # Get a list of koji-tags for all repos
+ # FIXME - perhaps we should only query for 'active' repos
+ reponames = [reponame for reponame in repoinfo.repos()]
+ repotags = [repoinfo.getrepo(reponame)['tag'] for reponame in reponames]
repotags.sort()
assert kojitag in repotags, 'Requested unsupported kojitag: %s' %
kojitag
current_tag = repotags.index(kojitag)
--
1.7.2.3
8:37 a.m.
New subject: [PATCH 2/5] Correct missing import.
---
tests/rats_sanity/rats_sanity.py | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/tests/rats_sanity/rats_sanity.py b/tests/rats_sanity/rats_sanity.py
index 3f1e915..a5248a5 100644
--- a/tests/rats_sanity/rats_sanity.py
+++ b/tests/rats_sanity/rats_sanity.py
@@ -21,6 +21,7 @@ import os
from autoqa import rats, util
from autotest_lib.client.bin import utils
from autotest_lib.client.common_lib import error
+from autotest_lib.client.bin.test_config import config_loader
from autoqa.test import AutoQATest
from autoqa.decorators import ExceptionCatcher
--
1.7.2.3
8:37 a.m.
New subject: [PATCH 3/5] Restrict known_hooks to only sub-directories of hookdir
---
autoqa | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/autoqa b/autoqa
index e6bd8c9..56aea8e 100755
--- a/autoqa
+++ b/autoqa
@@ -181,7 +181,8 @@ if not os.path.isdir(conf['hookdir']):
print "Can't find hooks in %s. Check your installation." %
conf['hookdir']
sys.exit(1)
# known hooks = dirs in hookdir
-known_hooks = [d for d in os.listdir(conf['hookdir'])]
+# FIXME - would be nice to have some way to determine what makes a hook valid
+known_hooks = [d for d in os.listdir(conf['hookdir']) if
os.path.isdir(os.path.join(conf['hookdir'], d))]
# Set up the option parser
parser = optparse.OptionParser(usage="%prog HOOKNAME [options] ...",
--
1.7.2.3
3:05 p.m.
New subject: [PATCH 3/5] Restrict known_hooks to only sub-directories of hookdir
----- "James Laska" <jlaska(a)redhat.com> wrote:
---
autoqa | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/autoqa b/autoqa
index e6bd8c9..56aea8e 100755
--- a/autoqa
+++ b/autoqa
@@ -181,7 +181,8 @@ if not os.path.isdir(conf['hookdir']):
print "Can't find hooks in %s. Check your installation." %
conf['hookdir']
sys.exit(1)
# known hooks = dirs in hookdir
-known_hooks = [d for d in os.listdir(conf['hookdir'])]
+# FIXME - would be nice to have some way to determine what makes a
hook valid
What about creating /usr/share/autoqa/hooks and putting all hooks there?
That way we can be sure that just hooks are there and nothing else.
+known_hooks = [d for d in os.listdir(conf['hookdir']) if
os.path.isdir(os.path.join(conf['hookdir'], d))]
# Set up the option parser
parser = optparse.OptionParser(usage="%prog HOOKNAME [options] ...",
--
1.7.2.3
3:59 p.m.
New subject: [PATCH 3/5] Restrict known_hooks to only sub-directories of hookdir
On Tue, 2010-09-21 at 16:05 -0400, Kamil Paral wrote:
----- "James Laska" <jlaska(a)redhat.com> wrote:
> ---
> autoqa | 3 ++-
> 1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/autoqa b/autoqa
> index e6bd8c9..56aea8e 100755
> --- a/autoqa
> +++ b/autoqa
> @@ -181,7 +181,8 @@ if not os.path.isdir(conf['hookdir']):
> print "Can't find hooks in %s. Check your installation." %
> conf['hookdir']
> sys.exit(1)
> # known hooks = dirs in hookdir
> -known_hooks = [d for d in os.listdir(conf['hookdir'])]
> +# FIXME - would be nice to have some way to determine what makes a
> hook valid
What about creating /usr/share/autoqa/hooks and putting all hooks there?
That way we can be sure that just hooks are there and nothing else.
I like that idea. I'll work up something and send to the list later
this week.
Thanks,
James
8:37 a.m.
New subject: [PATCH 4/5] Don't reboot test clients before or after the test runs
---
autoqa | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/autoqa b/autoqa
index 56aea8e..a4261ec 100755
--- a/autoqa
+++ b/autoqa
@@ -135,6 +135,8 @@ def schedule_job(controlfile, required_arch=None, email=None,
name=None, dryrun=
cmd = ['/usr/bin/atest', 'job', 'create']
if email:
cmd += ['-e', email]
+ # Don't reboot the client before or after the test runs
+ cmd += ["--reboot_before=never", "--reboot_after=never"]
# some hooks/tests may require special machines
# autotest currently doesn't support 'noarch' tests, so execute them on
x86_64
if not required_arch or required_arch == 'noarch':
--
1.7.2.3
8:37 a.m.
New subject: [PATCH 5/5] Update to autoqa-0.4.2-1
---
autoqa.spec | 10 +++++++---
1 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/autoqa.spec b/autoqa.spec
index 6e038d5..40a18e0 100644
--- a/autoqa.spec
+++ b/autoqa.spec
@@ -75,7 +75,7 @@ rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
-%doc README LICENSE TODO
+%doc README LICENSE TODO autoqa.cron
%dir %{_sysconfdir}/autoqa
%config(noreplace) %{_sysconfdir}/autoqa/autoqa.conf
%config %{_sysconfdir}/autoqa/repoinfo.conf
@@ -85,7 +85,6 @@ rm -rf $RPM_BUILD_ROOT
%{testdir}
%{hookdir}
%{_datadir}/autotest/client/bin/site_utils.py*
-%{_datadir}/%{name}/autoqa.cron
%{python_sitelib}/autoqa*
@@ -102,7 +101,12 @@ rm -rf $RPM_BUILD_ROOT
%changelog
-* Tue Sep 16 2010 James Laska <jlaska(a)redhat.com> - 0.4.1-1
+* Fri Sep 17 2010 James Laska <jlaska(a)redhat.com> - 0.4.2-1
+- Minor bug fixes (rats_sanity, upgradepath, autoqa) (jlaska)
+- Move watcher cronjob into documentation directory (jlaska)
+- Don't reboot clients before or after test run (jlaska)
+
+* Thu Sep 16 2010 James Laska <jlaska(a)redhat.com> - 0.4.1-1
- Corrected several multihook test issues (kparal+jskladan)
- Enable assertions for autoqa tests (kparal)
- Updated rats_sanity comps.rng URL (wwoods)
--
1.7.2.3
2:27 p.m.
New subject: [PATCH 5/5] Update to autoqa-0.4.2-1
On Fri, 2010-09-17 at 09:37 -0400, James Laska wrote:
---
autoqa.spec | 10 +++++++---
1 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/autoqa.spec b/autoqa.spec
index 6e038d5..40a18e0 100644
--- a/autoqa.spec
+++ b/autoqa.spec
@@ -75,7 +75,7 @@ rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
-%doc README LICENSE TODO
+%doc README LICENSE TODO autoqa.cron
%dir %{_sysconfdir}/autoqa
%config(noreplace) %{_sysconfdir}/autoqa/autoqa.conf
%config %{_sysconfdir}/autoqa/repoinfo.conf
@@ -85,7 +85,6 @@ rm -rf $RPM_BUILD_ROOT
%{testdir}
%{hookdir}
%{_datadir}/autotest/client/bin/site_utils.py*
-%{_datadir}/%{name}/autoqa.cron
%{python_sitelib}/autoqa*
@@ -102,7 +101,12 @@ rm -rf $RPM_BUILD_ROOT
%changelog
-* Tue Sep 16 2010 James Laska <jlaska(a)redhat.com> - 0.4.1-1
+* Fri Sep 17 2010 James Laska <jlaska(a)redhat.com> - 0.4.2-1
+- Minor bug fixes (rats_sanity, upgradepath, autoqa) (jlaska)
+- Move watcher cronjob into documentation directory (jlaska)
+- Don't reboot clients before or after test run (jlaska)
+
+* Thu Sep 16 2010 James Laska <jlaska(a)redhat.com> - 0.4.1-1
- Corrected several multihook test issues (kparal+jskladan)
- Enable assertions for autoqa tests (kparal)
- Updated rats_sanity comps.rng URL (wwoods)
Note, I missed actually updating the %{version} in this patch. I'm not
sure how. Regardless, I've corrected that in my local branch.
Thanks,
James
9:06 a.m.
New subject: [PATCH 6/6] Don't overwrite target repo name
---
tests/repoclosure/repoclosure.py | 3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
diff --git a/tests/repoclosure/repoclosure.py b/tests/repoclosure/repoclosure.py
index c0e4838..26f5ac5 100644
--- a/tests/repoclosure/repoclosure.py
+++ b/tests/repoclosure/repoclosure.py
@@ -39,8 +39,7 @@ class repoclosure(AutoQATest):
cmd += ' --repofrompath=target,%s --repoid=target' % baseurl
count = 1
for parent in parents:
- name = 'parent-%u' % count
- cmd += ' --repofrompath=%s,%s --repoid=%s' % (name, parent, name)
+ cmd += ' --repofrompath=parent-%u,%s --repoid=parent-%u' % (count,
parent, count)
count += 1
# TODO set cachedir etc. to e.g. test.scratchdir?
out = utils.system_output(cmd, retain_output=True)
--
1.7.2.3
10:46 a.m.
New subject: [PATCH 7/7] Fix typo
---
tests/conflicts/conflicts.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/tests/conflicts/conflicts.py b/tests/conflicts/conflicts.py
index b2d003c..4328728 100644
--- a/tests/conflicts/conflicts.py
+++ b/tests/conflicts/conflicts.py
@@ -74,6 +74,6 @@ class conflicts(AutoQATest):
self.summary = "%u packages with file conflicts in %s" %
(len(conflicting_pkgs), name)
self.highlights = "= Conflicting pkgs =\n%s" %
"\n".join(sorted(conflicting_pkgs))
- self.highlights += "\n\n= File conflicts =\n%s" %
"\n".jouin(sorted(file_conflicts))
+ self.highlights += "\n\n= File conflicts =\n%s" %
"\n".join(sorted(file_conflicts))
self.outputs = out
--
1.7.2.3
2:52 p.m.
New subject: [PATCH 7/7] Fix typo
On Fri, 2010-09-17 at 11:46 -0400, James Laska wrote:
---
tests/conflicts/conflicts.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
All 7 patches look reasonable and clear. And I guess they've gotten some
testing already on the public autoqa instance. So go for it.
-w
3:54 p.m.
New subject: [PATCH 7/7] Fix typo
On Fri, 2010-09-17 at 15:52 -0400, Will Woods wrote:
On Fri, 2010-09-17 at 11:46 -0400, James Laska wrote:
> ---
> tests/conflicts/conflicts.py | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
All 7 patches look reasonable and clear. And I guess they've gotten some
testing already on the public autoqa instance. So go for it.
Thanks for the quick feedback Will. I've corrected a few goofs and
pushed the changes to master. I've also built autoqa-0.4.2-1 for F-14,
F-13 and EPEL-5 and updated "production".
The only CRASHED test results at that I'm aware of will be from the
conflicts test when download.fedoraproject.org directs to a old/stale
mirror, resulting in a panic.
Thanks,
James
3 p.m.
New subject: [PATCH 7/7] Use repoinfo module directly (not through autoqa.koji_utils). Don't exclude the requested tag even if it is in -testing
---
tests/upgradepath/upgradepath.py | 9 +++++++--
1 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py
index 1811aba..cecf439 100755
--- a/tests/upgradepath/upgradepath.py
+++ b/tests/upgradepath/upgradepath.py
@@ -24,6 +24,7 @@ import operator
import autoqa.koji_utils
import autoqa.util
import rpmUtils.miscutils
+from autoqa.repoinfo import repoinfo
import autoqa.util
from autoqa.test import AutoQATest
@@ -86,8 +87,12 @@ class upgradepath(AutoQATest):
def run_once(self, envrs, kojitag, **kwargs):
update_id = kwargs['name'] or kwargs['id']
- reponames = [reponame for reponame in autoqa.koji_utils.repoinfo.repos() if not
reponame.endswith('-testing')]
- repotags = [autoqa.koji_utils.repoinfo.getrepo(reponame)['tag'] for
reponame in reponames]
+ # Get a list of all repos we monitor
+ # FIXME - perhaps we should only query for 'active' repos
+ reponames = [reponame for reponame in repoinfo.repos()]
+ # Get a list of koji-tags For all stable repos (includes the requested kojitag)
+ repotags = [repoinfo.getrepo(reponame)['tag'] for reponame in reponames
\
+ if not reponame.endswith('-testing') or
repoinfo.getrepo(reponame)['tag'] == kojitag]
repotags.sort()
assert kojitag in repotags, 'Requested unsupported kojitag: %s' %
kojitag
current_tag = repotags.index(kojitag)
--
1.7.2.3
2:44 p.m.
----- "James Laska" <jlaska(a)redhat.com> wrote:
= conflicts test failures =
1) Failing to find packages or mirrors
This appears to be a result of running the test in a different
environment.
When run previously, download.fedoraproject.org would result to an
internal Red
Hat mirror which was (for the most part) always updated and accurate.
With our
tests now running in Fedora infrastructure, we are at the mercy of
MirrorManager to provide an appropriate mirror. It seems that we
rarely get an
updated mirror. I have 2 proposed solutions to this problem.
I have also noticed problems with MirrorManager, I get an outdated
mirror quite frequently when testing. The question is - how is that
possible? Isn't that a bug? Isn't a sole purpose of the MirrorManager
to ensure I actually always get an up-to-date mirror? If that's the
case, we should simply request a fix for MirrorManager. Does anyone
know how it *should* work?
Anyway, for performance reasons, it would be nice to download from
our fast internal mirror, sure.
(But I'm still puzzled about the MirrorManager giving outdated mirrors.)
3:59 p.m.
On Tue, 2010-09-21 at 15:44 -0400, Kamil Paral wrote:
----- "James Laska" <jlaska(a)redhat.com> wrote:
> = conflicts test failures =
>
> 1) Failing to find packages or mirrors
>
> This appears to be a result of running the test in a different
> environment.
> When run previously, download.fedoraproject.org would result to an
> internal Red
> Hat mirror which was (for the most part) always updated and accurate.
> With our
> tests now running in Fedora infrastructure, we are at the mercy of
> MirrorManager to provide an appropriate mirror. It seems that we
> rarely get an
> updated mirror. I have 2 proposed solutions to this problem.
>
I have also noticed problems with MirrorManager, I get an outdated
mirror quite frequently when testing. The question is - how is that
possible? Isn't that a bug? Isn't a sole purpose of the MirrorManager
to ensure I actually always get an up-to-date mirror? If that's the
case, we should simply request a fix for MirrorManager. Does anyone
know how it *should* work?
Anyway, for performance reasons, it would be nice to download from
our fast internal mirror, sure.
(But I'm still puzzled about the MirrorManager giving outdated mirrors.)
Great questions Kamil. We've temporarily resolved the problem by
ensuring we get a known-good mirror. But your questions remain. I'd
need to check-in with folks from the infrastructure team for
thoughts/guidance.
Thanks,
James
1:18 a.m.
----- "James Laska" <jlaska(a)redhat.com> wrote:
On Tue, 2010-09-21 at 15:44 -0400, Kamil Paral wrote:
> ----- "James Laska" <jlaska(a)redhat.com> wrote:
>
> > = conflicts test failures =
> >
> > 1) Failing to find packages or mirrors
> >
> > This appears to be a result of running the test in a different
> > environment.
> > When run previously, download.fedoraproject.org would result to
an
> > internal Red
> > Hat mirror which was (for the most part) always updated and
accurate.
> > With our
> > tests now running in Fedora infrastructure, we are at the mercy
of
> > MirrorManager to provide an appropriate mirror. It seems that we
> > rarely get an
> > updated mirror. I have 2 proposed solutions to this problem.
> >
>
> I have also noticed problems with MirrorManager, I get an outdated
> mirror quite frequently when testing. The question is - how is that
> possible? Isn't that a bug? Isn't a sole purpose of the
MirrorManager
> to ensure I actually always get an up-to-date mirror? If that's the
> case, we should simply request a fix for MirrorManager. Does anyone
> know how it *should* work?
>
> Anyway, for performance reasons, it would be nice to download from
> our fast internal mirror, sure.
>
> (But I'm still puzzled about the MirrorManager giving outdated
mirrors.)
Great questions Kamil. We've temporarily resolved the problem by
ensuring we get a known-good mirror. But your questions remain. I'd
need to check-in with folks from the infrastructure team for
thoughts/guidance.
I just realized that maybe MirrorManager just gives you any mirror from
the list and only yum itself checks whether it is up-to-date or not, and
switches to another repo in the latter case.
Since we do a lot of stuff without yum, then we obviously don't have this
feature.
I have to consult someone from releng.
5023
days inactive
5028
days old
autoqa-devel@lists.fedorahosted.org
16 comments
3 participants
participants (3)
-
James Laska -
Kamil Paral -
Will Woods