[AutoQA] #232: Outdated documentation tracking ticket
by fedora-badges
#232: Outdated documentation tracking ticket
-----------------------+----------------------------------------------------
Reporter: kparal | Owner:
Type: task | Status: new
Priority: major | Milestone:
Component: docs/wiki | Version: 1.0
Keywords: |
-----------------------+----------------------------------------------------
I would like us to use this ticket for tracking outdated documentation.
Anytime there is a source code change that requires some documentation to
be updated (and you won't update it right that moment), change this
description and add a bullet point describing the update. When we have
time we can go through the list and update the documentation, removing
those bullet points. This is a good way how to ensure we won't forget
about something.
Outdated documentation:
* '''Use of exceptions''' - AutoQA tests may no longer raise exceptions
arbitrarily (like !TestFail etc). Any exception raised means that the test
will end immediately and will be reported as CRASHED. If the tests
finishes fine (it doesn't matter if it passed or failed, it just matters
it has been executed fully), no exception must be raised.
* '''Self variables may be set any time''' - We have recommended to set
self.result and similar variables only at the end of the test (because of
our old exception handling). It is no longer the case. Self.* variables
may be set any time.
* '''Crontab file no longer installed''' - We no longer install crontab
file into /etc/cron.d/autoqa. Therefore there is no longer need to comment
out this file on clients. Also, this file must be created on servers (we
install a template as a documentation file).
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/232>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 7 months
[PATCH] Include traceback in email output
by Kamil Paral
If the test crashes, provide the full traceback at the end of the
self.outputs. This way we can quickly examine the cause from the body of
the email and don't have to search for full autoqa logs on the server.
---
lib/python/decorators.py | 6 +++---
lib/python/test.py | 8 ++++++--
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/python/decorators.py b/lib/python/decorators.py
index 7db78ad..0e504fc 100644
--- a/lib/python/decorators.py
+++ b/lib/python/decorators.py
@@ -76,12 +76,12 @@ class ExceptionCatcher(object):
#if the 'on_exception' is a name of 'class member function'
#then call the f_self.on_exception(oe_args, oe_kwargs)
if self.oe_is_self_method:
- f_self.__getattribute__(self.oe)(exc = exc_info[1], *self.oe_a, **self.oe_k)
+ f_self.__getattribute__(self.oe)(exc = exc_info, *self.oe_a, **self.oe_k)
#'on_exception' is pointer to function
else:
- self.oe(exc = exc_info[1], *self.oe_a, **self.oe_k)
+ self.oe(exc = exc_info, *self.oe_a, **self.oe_k)
#re-raise the stored exception
- raise exc_info[1], None, exc_info[2]
+ raise exc_info[0], exc_info[1], exc_info[2]
return f_result
return newf
diff --git a/lib/python/test.py b/lib/python/test.py
index f04ad10..eccbb09 100644
--- a/lib/python/test.py
+++ b/lib/python/test.py
@@ -49,14 +49,18 @@ class AutoQATest(test.test, object):
pass
def process_exception(self, exc = None):
+ self._convert_list_variables()
+
self.result = "CRASHED"
if exc is not None:
- self.summary = "%s: %s" % (exc.__class__.__name__, exc)
+ self.summary = "%s: %s" % (exc[1].__class__.__name__, exc[1])
+ self.outputs += '\n%s\n%s' % ('-'*70,
+ ''.join(traceback.format_exception(exc[0], exc[1], exc[2])))
else:
self.summary = "Exception: Unknown exception"
try:
self.postprocess_iteration()
- except Exception, e:
+ except Exception:
traceback.print_exc(file=sys.stderr)
def postprocess_iteration(self):
--
1.7.2.3
13 years, 7 months
[AutoQA] #134: ResultsDB: create resultsdb prototype
by fedora-badges
#134: ResultsDB: create resultsdb prototype
----------------------+-----------------------------------------------------
Reporter: jskladan | Owner:
Type: task | Status: new
Priority: major | Milestone: Resultdb
Component: tests | Version: 1.0
Keywords: |
----------------------+-----------------------------------------------------
Create prototype of database and software which will implement dbSchema
from #133 and API from #131
This prototype should also include some test suite, which will be used to
determine, if all the functionality is OK.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/134>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 7 months
[PATCH] Removed individual XYZ_failed methods in test.py
by Josef Skladanka
According to the <https://fedorahosted.org/pipermail/autoqa-devel/2010-October/001214.html>
I've removed the individual {setup,initialize,run_once}_failed()
methods and replaced these with one generic process_exception().
I also set the self.process_exception as default method for
the ExceptionCatcher decorator, so we can use it without arguments.
The possibility to provide the name of some particular method
is, of course, still there.
Will change the wiki documentation, once this patch is in master.
---
lib/python/decorators.py | 2 +-
lib/python/test.py | 30 ++++--------------------------
tests/conflicts/conflicts.py | 2 +-
tests/helloworld/helloworld.py | 2 +-
tests/initscripts/initscripts.py | 6 +++---
tests/rats_install/rats_install.py | 6 +++---
tests/rats_sanity/rats_sanity.py | 6 +++---
tests/repoclosure/repoclosure.py | 4 ++--
tests/rpmguard/rpmguard.py | 6 +++---
tests/rpmlint/rpmlint.py | 6 +++---
tests/upgradepath/upgradepath.py | 4 ++--
11 files changed, 26 insertions(+), 48 deletions(-)
diff --git a/lib/python/decorators.py b/lib/python/decorators.py
index 0b920b4..7db78ad 100644
--- a/lib/python/decorators.py
+++ b/lib/python/decorators.py
@@ -46,7 +46,7 @@ class ExceptionCatcher(object):
Exception
"""
- def __init__(self, on_exception, *args, **kwargs):
+ def __init__(self, on_exception = "self.process_exception", *args, **kwargs):
"""
on_exception - string with class function name, or pointer to function
*args, **kwargs - args for the on_exception function
diff --git a/lib/python/test.py b/lib/python/test.py
index 61e44e7..f04ad10 100644
--- a/lib/python/test.py
+++ b/lib/python/test.py
@@ -35,42 +35,20 @@ class AutoQATest(test.test, object):
self.outputs = "" #second part of mail_body
self.mail_to = [] #list of emails to send results to. Mail is always sent to results list
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
pass
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
self.config = config_loader(config, self.tmpdir)
self.autotest_url = make_autotest_url(self.config)
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, **kwargs):
pass
- def setup_failed(self, exc = None):
- self.result = "CRASHED"
- if exc is not None:
- self.summary = "Setup failed: %s: %s" % (exc.__class__.__name__, exc)
- else:
- self.summary = "Setup failed: Exception: Unknown exception"
- try:
- self.postprocess_iteration()
- except Exception, e:
- traceback.print_exc(file=sys.stderr)
-
- def initialize_failed(self, exc = None):
- self.result = "CRASHED"
- if exc is not None:
- self.summary = "Initialize failed: %s: %s" % (exc.__class__.__name__, exc)
- else:
- self.summary = "Initialize failed: Exception: Unknown exception"
- try:
- self.postprocess_iteration()
- except Exception, e:
- traceback.print_exc(file=sys.stderr)
-
- def run_once_failed(self, exc = None):
+ def process_exception(self, exc = None):
self.result = "CRASHED"
if exc is not None:
self.summary = "%s: %s" % (exc.__class__.__name__, exc)
diff --git a/tests/conflicts/conflicts.py b/tests/conflicts/conflicts.py
index 4328728..5a93a2b 100644
--- a/tests/conflicts/conflicts.py
+++ b/tests/conflicts/conflicts.py
@@ -32,7 +32,7 @@ class conflicts(AutoQATest):
f.write(logdata)
f.close()
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, baseurl, parents, name, **kwargs):
if name:
name = "%s-%s" % (name, autoqa.util.get_basearch())
diff --git a/tests/helloworld/helloworld.py b/tests/helloworld/helloworld.py
index 9f6ac50..0af1c52 100644
--- a/tests/helloworld/helloworld.py
+++ b/tests/helloworld/helloworld.py
@@ -25,7 +25,7 @@ from autoqa.decorators import ExceptionCatcher
class helloworld(AutoQATest):
version = 1 # increment if setup() changes
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, *args, **kwargs):
self.summary = 'Hello, World!'
self.outputs = "===Printing passed params===\n"
diff --git a/tests/initscripts/initscripts.py b/tests/initscripts/initscripts.py
index 133d058..e0f95bb 100644
--- a/tests/initscripts/initscripts.py
+++ b/tests/initscripts/initscripts.py
@@ -33,7 +33,7 @@ from autoqa.decorators import ExceptionCatcher
class initscripts(AutoQATest):
version = 1 # increment if setup() changes
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
"""
adds beakerlib repository and installs beakerlib
@@ -45,7 +45,7 @@ class initscripts(AutoQATest):
f.close()
utils.system('yum -y install beakerlib')
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
super(initscripts, self).initialize(config)
self.rpmdir = os.path.join(self.tmpdir, 'rpms')
@@ -99,7 +99,7 @@ class initscripts(AutoQATest):
return (result, outputs)
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, kojitag, **kwargs):
if kwargs['hook'] == 'post-koji-build':
envrs = [kwargs['envr']]
diff --git a/tests/rats_install/rats_install.py b/tests/rats_install/rats_install.py
index 360be3f..df23451 100644
--- a/tests/rats_install/rats_install.py
+++ b/tests/rats_install/rats_install.py
@@ -30,12 +30,12 @@ from autoqa.decorators import ExceptionCatcher
class rats_install(AutoQATest):
version = 1
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
super(rats_install, self).initialize(config)
self.irb_conf = os.path.join(self.bindir, 'irb.cfg')
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
if os.path.exists(self.irb_conf):
utils.system('yum -y install python-fedora')
@@ -51,7 +51,7 @@ class rats_install(AutoQATest):
# use the big hammer until we figure out the right thing to do
utils.system('lokkit --disabled')
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, baseurl, name, image_url="", boot_args="", **kwargs):
if name:
name = "%s-%s" % (name, util.get_basearch())
diff --git a/tests/rats_sanity/rats_sanity.py b/tests/rats_sanity/rats_sanity.py
index 5a2a4f1..d6a6357 100644
--- a/tests/rats_sanity/rats_sanity.py
+++ b/tests/rats_sanity/rats_sanity.py
@@ -28,17 +28,17 @@ from autoqa.decorators import ExceptionCatcher
class rats_sanity(AutoQATest):
version = 1
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
super(rats_sanity, self).initialize(config)
self.irb_conf = os.path.join(self.bindir, 'irb.cfg')
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
if os.path.exists(self.irb_conf):
utils.system('yum -y install python-fedora')
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, baseurl, parents, name, **kwargs):
if name:
name = "%s-%s" % (name, util.get_basearch())
diff --git a/tests/repoclosure/repoclosure.py b/tests/repoclosure/repoclosure.py
index 5c8752e..b27b684 100644
--- a/tests/repoclosure/repoclosure.py
+++ b/tests/repoclosure/repoclosure.py
@@ -25,11 +25,11 @@ from autoqa.decorators import ExceptionCatcher
class repoclosure(AutoQATest):
version = 1
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
utils.system('yum -y install yum-utils')
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, baseurl, parents='', name='', **kwargs):
if name:
name = "%s-%s" % (name, autoqa.util.get_basearch())
diff --git a/tests/rpmguard/rpmguard.py b/tests/rpmguard/rpmguard.py
index 4e4ee04..588a58f 100644
--- a/tests/rpmguard/rpmguard.py
+++ b/tests/rpmguard/rpmguard.py
@@ -30,18 +30,18 @@ from autoqa.decorators import ExceptionCatcher
class rpmguard(AutoQATest):
version = 1 # increment if setup() changes
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
utils.system('yum -y install rpmlint')
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
super(rpmguard, self).initialize(config)
self.rpmguard = os.path.join(self.bindir, 'rpmguard')
self.rpmdir = os.path.join(self.tmpdir, 'rpms')
os.makedirs(self.rpmdir)
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, kojitag, **kwargs):
if kwargs['hook'] == 'post-koji-build':
envrs = [kwargs['envr']]
diff --git a/tests/rpmlint/rpmlint.py b/tests/rpmlint/rpmlint.py
index ce92a1d..bb25dbc 100644
--- a/tests/rpmlint/rpmlint.py
+++ b/tests/rpmlint/rpmlint.py
@@ -30,17 +30,17 @@ from autoqa.decorators import ExceptionCatcher
class rpmlint(AutoQATest):
version = 1 # increment if setup() changes
- @ExceptionCatcher("self.setup_failed")
+ @ExceptionCatcher()
def setup(self, **kwargs):
utils.system('yum -y install rpmlint')
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
super(rpmlint, self).initialize(config)
self.rpmdir = os.path.join(self.tmpdir, 'rpms')
os.makedirs(self.rpmdir)
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, kojitag, **kwargs):
if kwargs['hook'] == 'post-koji-build':
envrs = [kwargs['envr']]
diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py
index 4f298b1..edbe4ea 100755
--- a/tests/upgradepath/upgradepath.py
+++ b/tests/upgradepath/upgradepath.py
@@ -37,7 +37,7 @@ from autotest_lib.client.bin.test_config import config_loader
class upgradepath(AutoQATest):
version = 1 # increment if setup() changes
- @ExceptionCatcher("self.initialize_failed")
+ @ExceptionCatcher()
def initialize(self, config, **kwargs):
self.config = config_loader(config, self.tmpdir)
#URL of logs/results stored on autotest-server
@@ -86,7 +86,7 @@ class upgradepath(AutoQATest):
result = 'FAILED'
return result
- @ExceptionCatcher("self.run_once_failed")
+ @ExceptionCatcher()
def run_once(self, envrs, kojitag, **kwargs):
update_id = kwargs['name'] or kwargs['id']
--
1.7.2.3
13 years, 7 months
[PATCH 3/2] guard setup() with ExceptionCatcher
by Kamil Paral
Not only run_once() and initialize(), but setup() is now also guarded
with ExceptionCatcher (and therefore we will receive email if some test
crashed in setup() phase).
---
doc/test_class.py.template | 3 ++-
lib/python/test.py | 15 +++++++++++++++
tests/helloworld/helloworld.py | 3 ---
tests/initscripts/initscripts.py | 3 ++-
tests/rats_install/rats_install.py | 3 ++-
tests/rats_sanity/rats_sanity.py | 3 ++-
tests/repoclosure/repoclosure.py | 3 ++-
tests/rpmguard/rpmguard.py | 3 ++-
tests/rpmlint/rpmlint.py | 3 ++-
tests/upgradepath/upgradepath.py | 6 ++----
10 files changed, 31 insertions(+), 14 deletions(-)
diff --git a/doc/test_class.py.template b/doc/test_class.py.template
index b064e2e..00aef9d 100644
--- a/doc/test_class.py.template
+++ b/doc/test_class.py.template
@@ -43,7 +43,8 @@ class testclassname(AutoQATest): # <-- UPDATE Classname
# This is where you make sure that any required packages are installed,
# services are started, your test code is compiled, and so on. This is
# an optional method.
- #def setup(self, some_params):
+ #@ExceptionCatcher("self.setup_failed")
+ #def setup(self, some_params, **kwargs):
# utils.system('yum -y install yum-utils')
# This does any pre-test initialization that needs to happen. AutoQA tests
diff --git a/lib/python/test.py b/lib/python/test.py
index ea788f4..a658f53 100644
--- a/lib/python/test.py
+++ b/lib/python/test.py
@@ -35,6 +35,10 @@ class AutoQATest(test.test, object):
self.outputs = "" #second part of mail_body
self.mail_to = [] #list of emails to send results to. Mail is always sent to results list
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
+ pass
+
@ExceptionCatcher("self.initialize_failed")
def initialize(self, config, **kwargs):
self.config = config_loader(config, self.tmpdir)
@@ -44,6 +48,17 @@ class AutoQATest(test.test, object):
def run_once(self, **kwargs):
pass
+ def setup_failed(self, exc = None):
+ self.result = "CRASHED"
+ if exc is not None:
+ self.summary = "Setup failed: %s: %s" % (exc.__class__.__name__, exc)
+ else:
+ self.summary = "Setup failed: Exception: Unknown exception"
+ try:
+ self.postprocess_iteration()
+ except Exception, e:
+ traceback.print_exc(file=sys.stderr)
+
def initialize_failed(self, exc = None):
self.result = "CRASHED"
if exc is not None:
diff --git a/tests/helloworld/helloworld.py b/tests/helloworld/helloworld.py
index 7644019..9f6ac50 100644
--- a/tests/helloworld/helloworld.py
+++ b/tests/helloworld/helloworld.py
@@ -25,9 +25,6 @@ from autoqa.decorators import ExceptionCatcher
class helloworld(AutoQATest):
version = 1 # increment if setup() changes
- def setup(self):
- pass
-
@ExceptionCatcher("self.run_once_failed")
def run_once(self, *args, **kwargs):
self.summary = 'Hello, World!'
diff --git a/tests/initscripts/initscripts.py b/tests/initscripts/initscripts.py
index 2d10cbf..133d058 100644
--- a/tests/initscripts/initscripts.py
+++ b/tests/initscripts/initscripts.py
@@ -33,7 +33,8 @@ from autoqa.decorators import ExceptionCatcher
class initscripts(AutoQATest):
version = 1 # increment if setup() changes
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
"""
adds beakerlib repository and installs beakerlib
"""
diff --git a/tests/rats_install/rats_install.py b/tests/rats_install/rats_install.py
index 4896149..360be3f 100644
--- a/tests/rats_install/rats_install.py
+++ b/tests/rats_install/rats_install.py
@@ -35,7 +35,8 @@ class rats_install(AutoQATest):
super(rats_install, self).initialize(config)
self.irb_conf = os.path.join(self.bindir, 'irb.cfg')
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
if os.path.exists(self.irb_conf):
utils.system('yum -y install python-fedora')
# XXX maybe just install @virtualization + pax?
diff --git a/tests/rats_sanity/rats_sanity.py b/tests/rats_sanity/rats_sanity.py
index a5248a5..5a2a4f1 100644
--- a/tests/rats_sanity/rats_sanity.py
+++ b/tests/rats_sanity/rats_sanity.py
@@ -33,7 +33,8 @@ class rats_sanity(AutoQATest):
super(rats_sanity, self).initialize(config)
self.irb_conf = os.path.join(self.bindir, 'irb.cfg')
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
if os.path.exists(self.irb_conf):
utils.system('yum -y install python-fedora')
diff --git a/tests/repoclosure/repoclosure.py b/tests/repoclosure/repoclosure.py
index 26f5ac5..5c8752e 100644
--- a/tests/repoclosure/repoclosure.py
+++ b/tests/repoclosure/repoclosure.py
@@ -25,7 +25,8 @@ from autoqa.decorators import ExceptionCatcher
class repoclosure(AutoQATest):
version = 1
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
utils.system('yum -y install yum-utils')
@ExceptionCatcher("self.run_once_failed")
diff --git a/tests/rpmguard/rpmguard.py b/tests/rpmguard/rpmguard.py
index efd9626..4e4ee04 100644
--- a/tests/rpmguard/rpmguard.py
+++ b/tests/rpmguard/rpmguard.py
@@ -30,7 +30,8 @@ from autoqa.decorators import ExceptionCatcher
class rpmguard(AutoQATest):
version = 1 # increment if setup() changes
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
utils.system('yum -y install rpmlint')
@ExceptionCatcher("self.initialize_failed")
diff --git a/tests/rpmlint/rpmlint.py b/tests/rpmlint/rpmlint.py
index bd67766..ce92a1d 100644
--- a/tests/rpmlint/rpmlint.py
+++ b/tests/rpmlint/rpmlint.py
@@ -30,7 +30,8 @@ from autoqa.decorators import ExceptionCatcher
class rpmlint(AutoQATest):
version = 1 # increment if setup() changes
- def setup(self):
+ @ExceptionCatcher("self.setup_failed")
+ def setup(self, **kwargs):
utils.system('yum -y install rpmlint')
@ExceptionCatcher("self.initialize_failed")
diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py
index 46c6632..4f298b1 100755
--- a/tests/upgradepath/upgradepath.py
+++ b/tests/upgradepath/upgradepath.py
@@ -37,7 +37,8 @@ from autotest_lib.client.bin.test_config import config_loader
class upgradepath(AutoQATest):
version = 1 # increment if setup() changes
- def initialize(self, config):
+ @ExceptionCatcher("self.initialize_failed")
+ def initialize(self, config, **kwargs):
self.config = config_loader(config, self.tmpdir)
#URL of logs/results stored on autotest-server
self.autotest_url = autoqa.util.make_autotest_url(self.config)
@@ -49,9 +50,6 @@ class upgradepath(AutoQATest):
self.outputs = []
self.highlights = []
- def setup(self):
- pass
-
def compare(self, matching_build, tags, op, opstr):
koji = autoqa.koji_utils.SimpleKojiClientSession()
result = 'PASSED'
--
1.7.2.3
13 years, 7 months
Re: rpmlint improvements
by Kamil Paral
----- "Alexander Todorov" <atodorov(a)redhat.com> wrote:
> Hello folks,
> I want to discuss topics in tickets:
> https://fedorahosted.org/autoqa/ticket/149
> https://fedorahosted.org/autoqa/ticket/150
>
> I did a test run of rpmlint against the latest RHEL6 tree and filed
> some bugs.
> Some were legitimate but others turned out to be false negatives and
> developers
> got angry.
>
> Then it turned out that rpmlint needs more features in order to be
> used reliably
> without making developers unhappy. So I talked to twoerner who is
> rpmlint
> maintainer for RHEL6 and exchanged some ideas.
>
> What we think is a good idea and needs to be present in upstream is
> having a
> rpmlint-data package which will provide rules.$pkg-name files for
> packages that
> need special handling. Then rpmlint will take into account those files
> and act
> accordingly. The rules files will be maintained by a separate
> maintainer and not
> by the packagers. Also any change into those rules/exceptions needs to
> be
> approved with a review process to avoid pkg maintainers manipulating
> the rules
> simply to silence out rpmlint output. This will guarantee accurate
> results.
>
> This data package will be optional to rpmlint and not affect core
> functionality.
> It will most likely be different for Fedora and RHEL as well. This is
> more or
> less the same as proposed in ticket 149 with lintian tool for Debian.
>
> What do you think? Any drawbacks/gotchas you may think of?
Hello Alexander,
thanks for looking into this and sorry for late response.
The big difference between lintian and your proposed solution is that
lintian allows to specify "overrides" inside the affected packages [1]
(files located at /usr/share/lintian/overrides/<package> or at
debian/source/lintian-overrides), but your solution centralizes all those
files in a single package rpmlint-data.
I wonder what the better approach is. Per-package solution obviously gives
more freedom (and trust) to individual package maintainers and is easier
to manage (for them). Centralized approach is stricter (you have to ask
for inclusion, review process can be easily enforced), therefore quality
may be higher, on the other hand it may annoy and discourage a lot of people.
("Just another set of policies to follow.")
Originally I was thinking about the per-package (lintian-like) approach.
My idea is that we can still apply some policies on top of it, but it is
much more convenient for package maintainers in the rest of the cases.
The policy might look like: Every line whitelisting an Error message must
be sponsored by some different package maintainer. On the other hand,
Warning and Info messages might be whitelisted without restrictions.
That's only one example of a possible policy, certainly there is plenty
of solutions.
As Michal Scherer mentioned [2], rpmlint already supports "config files"
in which we can filter out some output (based on regexp I think). So we are
already able to do this inside autoqa - just read the per-package definition
file, create custom rpmlint config and run rpmlint with this config.
Of course, having this feature upstream would be much better for us. The
package maintainers should see the same output when they run rpmlint locally
as they see when rpmlint is run inside autoqa. That is possible only when
rpmlint upstream supports these whitelists definitions directly.
[1] http://lintian.debian.org/manual/ch2.html#s2.4
[2] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000861.html
>
>
> The rules files format need to support the following (as identified so
> far):
>
> * List file paths/directories with special permissions and uid/gid
> settings. For
> example CUPS has such. audit also has such (in RHEL for certification
> purposes)
An important note here - our proposed solution talked just about whitelisting
some warning/error messages. Also lintian overrides files provide just
whitelisting of output messages.
What you talk about here (and what you sent in your patch [3]) is a little
different. You just don't whitelist some output, you "fix" (override) the
input. For example you don't just define "don't show me the line that warns
me about incorrect permissions on /etc/passwd", but you also define *what
the correct permissions are*. Per package.
That is a little different (and somewhat harder task). I don't know whether
rpmlint developers will be inclined to so big change (redefining what the
correct input is). But allowing output whitelisting per-package could be
easier to convince them to adopt.
>From my point of view, I would rather start with simple whitelisting too.
Your approach seems to bring more challenging tasks and I think it's better
to start small.
What do you think?
[3] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000860.html
>
> * Ignore some test cases for particular package(or file). For
> example:
> gcc.x86_64: E: devel-dependency glibc-devel
> gcc.x86_64: W: devel-file-in-non-devel-package
>
> This looks normal since gcc is a compiler but will be an error for
> another
> package (e.g. an application).
>
> jlaska told me that Fedora QA already uses rpmlint. Have you guys
> identified
> other areas/tests that need special handling or exclusion ? What are
> those?
>
> Have you thought about the file format? It needs to be easy to
> maintain and easy
> to read - probably just plain text. It also needs to be flexible so
> that we can
> add to it without breaking compatibility. We can probably have a
> separate file
> (format) for every test/package that rpmlint performs.
>
> Just to let you know that I'm happy to work on the rpmlint code and
> push it
> upstream if noone else has spare cycles.
Your effort is greatly appreciated, thanks.
>
> Regards,
> Alexander.
>
>
> _______________________________________________
> autoqa-devel mailing list
> autoqa-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/autoqa-devel
13 years, 7 months
Re: [PATCH 2/2] allow self.outputs and self.highlights to be lists of strings
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> > + def _convert_list_variables(self):
> > + '''
> > + We allow some variables to be as lists. We now need them as
> strings,
> > + so let's just join them with newlines.
> > + '''
> > + for var in ('self.highlights', 'self.outputs'):
> > + exec(r'if isinstance(%(var)s, list): %(var)s =
> "\n".join(%(var)s)' % {'var': var})
> > +
>
> Is 'exec' needed? I'm accustomed to 'exec' being generally bad
> thing,
> but I could be wrong. There probably is a reason, but why not
> something
> like the following instead of 'exec()' ...
>
> if isinstance(var, list):
> var = "\n".join(var)
>
The reason is that I wanted to be smart ^^ and not repeat exactly
the same code several times for different variables. Generally it
is highly problematic to change immutable variables in a loop, as
I consulted with jskladan. One way is to repeat the code for every
variable (i.e. no loop):
if isinstance(self.outputs, list):
self.outputs = '\n'.join(self.outputs)
if isinstance(self.highlights, list):
self.highlights = '\n'.join(self.highlights)
Second way is to use the exec above (then you have just the just
once and all variables listed just once).
for var in ('self.highlights', 'self.outputs'):
exec(r'if isinstance(%(var)s, list): %(var)s = "\n".join(%(var)s)' % {'var': var})
Third way is something like this:
helper = [self.highlights, self.outputs]
for i in range(len(helper)):
if isinstance(helper[i], list):
helper[i] = '\n'.join(helper[i])
self.highlights, self.outputs = helper
Now you have one code, but several mentions of the variables.
You can imagine that some variables may be added, or renamed,
and less code and less variable names usage means less bugs.
For 10 variables, you wouldn't really use the first way. For
2 variables though... yeah, it's overkill.
We can do it any way that you deem appropriate, I won't feel
offended :) There seems to be no "clean" way in Python to do
this, unfortunately.
For readability (and until we have more variables), I support
the first way now.
13 years, 7 months
[PATCH 1/2] Automatically log test output
by Kamil Paral
This patch ensures that all test output is logged into output.log file
after test has finished execution. Please note that this concerns the
actual output which is sent (via email or to resultsdb) and which may be
quite different from the output printed to stdout and stderr during test
execution. Autotest automatically logs the output sent to stdout and
stderr, but this patch is needed to collect the output that is actually
transmitted.
---
lib/python/test.py | 34 ++++++++++++++++++++++------------
tests/initscripts/initscripts.py | 4 ----
tests/rpmguard/rpmguard.py | 4 ----
tests/rpmlint/rpmlint.py | 5 -----
tests/upgradepath/upgradepath.py | 5 -----
5 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/lib/python/test.py b/lib/python/test.py
index 28359f5..ea5d6c3 100644
--- a/lib/python/test.py
+++ b/lib/python/test.py
@@ -22,6 +22,7 @@ from autotest_lib.client.bin.test_config import config_loader
from decorators import ExceptionCatcher
from util import make_autotest_url
+import os
class AutoQATest(test.test, object):
version = 1 # increment if setup() changes
@@ -83,23 +84,32 @@ class AutoQATest(test.test, object):
send_to.append(email)
except:
pass
+ mail_body = "Stored logs available at <%s>\n\n" % self.autotest_url
+ if self.highlights:
+ mail_body += "%s\n" % self.highlights
+ mail_body += "-"*70 + "\n"
+ mail_body += self.outputs
# print results to the screen
- print '*'*40
- print '* RESULT: %s' % self.result
- print '* SUMMARY: %s' % self.summary
- print '* HIGHLIGHTS: %d lines' % len(self.highlights.splitlines() if self.highlights else '')
- print '* OUTPUTS: %d lines' % len(self.outputs.splitlines() if self.outputs else '')
- print '* EMAIL RECIPIENTS: %s' % ', '.join(send_to)
- print '*'*40
+ summary_table = '\n'.join([
+ '*'*40,
+ '* RESULT: %s' % self.result,
+ '* SUMMARY: %s' % self.summary,
+ '* HIGHLIGHTS: %d lines' % len(self.highlights.splitlines() if self.highlights else ''),
+ '* OUTPUTS: %d lines' % len(self.outputs.splitlines() if self.outputs else ''),
+ '* EMAIL RECIPIENTS: %s' % ', '.join(send_to),
+ '*'*40])
+ print summary_table
+
+ # store the whole output in a log file
+ log = open(os.path.join(self.resultsdir,'output.log'),'wb')
+ log.write(summary_table)
+ log.write('\n\n')
+ log.write(mail_body)
+ log.close()
# send email
if send_to:
- mail_body = "Stored logs available at <%s>\n\n" % self.autotest_url
- if self.highlights:
- mail_body += "%s\n" % self.highlights
- mail_body += "-"*70 + "\n"
- mail_body += self.outputs
utils.send_email(mail_to=send_to,
mail_from=self.config.get('test','mail_from'),
mail_server=self.config.get('test','smtpserver'),
diff --git a/tests/initscripts/initscripts.py b/tests/initscripts/initscripts.py
index c7b322a..72ea96a 100644
--- a/tests/initscripts/initscripts.py
+++ b/tests/initscripts/initscripts.py
@@ -113,7 +113,6 @@ class initscripts(AutoQATest):
self.tests_failed = []
self.outputs = []
self.highlights = []
- self.log = open(os.path.join(self.resultsdir,'initscripts.log'),'wb') # where to log output
for envr in envrs:
# add header
@@ -196,6 +195,3 @@ class initscripts(AutoQATest):
self.outputs = '\n'.join(self.outputs)
self.highlights = '\n'.join(self.highlights)
- # log outputs
- self.log.write(self.outputs)
- self.log.close()
diff --git a/tests/rpmguard/rpmguard.py b/tests/rpmguard/rpmguard.py
index 41032df..7493078 100644
--- a/tests/rpmguard/rpmguard.py
+++ b/tests/rpmguard/rpmguard.py
@@ -55,7 +55,6 @@ class rpmguard(AutoQATest):
self.envr_results = {} # results for invidual packages
self.outputs = []
self.highlights = []
- self.log = open(os.path.join(self.resultsdir,'rpmguard.log'),'wb') # where to log output
for envr in envrs:
# add header
@@ -109,9 +108,6 @@ class rpmguard(AutoQATest):
self.outputs = '\n'.join(self.outputs)
self.highlights = '\n'.join(self.highlights)
- # log outputs
- self.log.write(self.outputs)
- self.log.close()
def test_envr(self, envr, kojitag):
'''
diff --git a/tests/rpmlint/rpmlint.py b/tests/rpmlint/rpmlint.py
index 9b36e5c..3b8841d 100644
--- a/tests/rpmlint/rpmlint.py
+++ b/tests/rpmlint/rpmlint.py
@@ -54,7 +54,6 @@ class rpmlint(AutoQATest):
self.envr_results = {} # results for invidual packages
self.outputs = []
self.highlights = []
- self.log = open(os.path.join(self.resultsdir,'rpmlint.log'),'wb') # where to log output
koji = autoqa.koji_utils.SimpleKojiClientSession()
@@ -143,7 +142,3 @@ class rpmlint(AutoQATest):
self.outputs = '\n'.join(self.outputs)
self.highlights = '\n'.join(self.highlights)
- # log outputs
- self.log.write(self.outputs)
- self.log.close()
-
diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py
index bc382d3..b680da8 100755
--- a/tests/upgradepath/upgradepath.py
+++ b/tests/upgradepath/upgradepath.py
@@ -48,7 +48,6 @@ class upgradepath(AutoQATest):
self.envr_results = {} # results for invidual packages
self.outputs = []
self.highlights = []
- self.log = open(os.path.join(self.resultsdir,'upgradepath.log'),'wb') # where to log output
def setup(self):
pass
@@ -178,7 +177,3 @@ class upgradepath(AutoQATest):
self.outputs = '\n'.join(self.outputs)
self.highlights = '\n'.join(self.highlights)
- # log outputs
- self.log.write(self.outputs)
- self.log.close()
-
--
1.7.2.3
13 years, 7 months
Several small fixes
by James Laska
Greetings,
Nothing urgent, just a few small patches I'd like to apply to master. Nothing
noteworthy with the first two, but the third might be more interesting given
recent upgradepath discussion. Kamil might already have this in his pending
upgradepath changes.
Thoughts/concerns welcome.
> 0001-Tell-post-tree-compose-that-composes-are-enabled-for.patch
The repoinfo.conf configuration value 'composes' was enabled so that
post-tree-compose would know which entries to monitor for installation images.
This patch turns on composes for the Branched release (F14). I've also updated
the wiki documentation as well
(https://fedoraproject.org/wiki/How_to_update_AutoQA_repoinfo.conf).
> 0002-Correct-missing-config_loader-import.patch
Not sure why this never surfaced until now, but this patch corrects a missing
import from the rats_install.py script.
> 0003-Enable-owner-fedoraproject.org-cc-for-upgradepath-re.patch
I think this might make more sense to move out of the tests, and into
lib/python/test.py, but until we do that, this patch turns on cc of test
results for upgradepath. Josef also pointed out that we may want a way for
tests to enable/disable sending output to maintainers. I don't have any great
ideas on that, so for now, this patch just mimics our rpmlint and rpmguard by
including $package-owner(a)fedoraproject.org.
Thanks,
James
13 years, 7 months