On Wed, 2010-09-08 at 07:41 -0400, Kamil Paral wrote:
Hi,
currently many tests use this code in its test objects:
raise error.TestFail
-or-
raise error.TestWarn
to end a test that "failed" (from our perspective). It then creates a few
ugly exceptions in the output, see here, even if everything went just fine
(from autotest perspective):
http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject...
and finally it reports:
09/08 12:48:13 INFO | job:1254|
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
09/08 12:48:13 INFO | job:1254| END
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
(which is also visible in autotest web frontend).
However, since we now use jskladan's patchset defining AutoQATest parent
class (using self.result variable), I no longer see a reason to raise those
exceptions. Quite the contrary.
Those exception don't even match our result states, which we have a much richer
set (passed, failed, info, aborted, crashed and maybe some other).
I'd like to propose that we stop using those autotest-internal TestFail
and TestWarn exceptions.
What will change:
1. We no longer see red/purple boxes in autotest frontend.
Hah, I was thinking the same thing when inspecting the autotest tko
results viewer. It's hard to distinguish between a successful test run
that found problems, vs a test case failure.
2. The AutoQATest parent class will be adjusted to automatically
output
the test result after a test has finished.
3. Tests will just need to store proper result in self.result variable,
that's all. The methods may be then ended in a standard manner (return
or no statement).
What will be the benefits:
1. We will know that if we see a traceback it's a problem. We won't have to
examine in detail whether this traceback is "good" or "bad".
+1 agreed
2. It will allow us to report any crashed test object (throwing an
exception)
to the mailing list. We have already started to do that and it's great:
https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html
But currently it works only for those tests that haven't filled self.result
before the crash occurred. We need to catch all exceptions.
What do you think?
This is a good improvement, thanks for taking the time to propose.
Thanks,
James