On Wed, 2010-09-08 at 07:41 -0400, Kamil Paral wrote:
Hi,
currently many tests use this code in its test objects:
raise error.TestFail -or- raise error.TestWarn
to end a test that "failed" (from our perspective). It then creates a few ugly exceptions in the output, see here, even if everything went just fine (from autotest perspective):
http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject.or...
and finally it reports:
09/08 12:48:13 INFO | job:1254| FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13 09/08 12:48:13 INFO | job:1254| END FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
(which is also visible in autotest web frontend).
However, since we now use jskladan's patchset defining AutoQATest parent class (using self.result variable), I no longer see a reason to raise those exceptions. Quite the contrary. Those exception don't even match our result states, which we have a much richer set (passed, failed, info, aborted, crashed and maybe some other).
I'd like to propose that we stop using those autotest-internal TestFail and TestWarn exceptions.
What will change:
- We no longer see red/purple boxes in autotest frontend.
Hah, I was thinking the same thing when inspecting the autotest tko results viewer. It's hard to distinguish between a successful test run that found problems, vs a test case failure.
- The AutoQATest parent class will be adjusted to automatically output
the test result after a test has finished. 3. Tests will just need to store proper result in self.result variable, that's all. The methods may be then ended in a standard manner (return or no statement).
What will be the benefits:
- We will know that if we see a traceback it's a problem. We won't have to
examine in detail whether this traceback is "good" or "bad".
+1 agreed
- It will allow us to report any crashed test object (throwing an exception)
to the mailing list. We have already started to do that and it's great: https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html But currently it works only for those tests that haven't filled self.result before the crash occurred. We need to catch all exceptions.
What do you think?
This is a good improvement, thanks for taking the time to propose.
Thanks, James