6:41 a.m.
Hi,
currently many tests use this code in its test objects:
raise error.TestFail
-or-
raise error.TestWarn
to end a test that "failed" (from our perspective). It then creates a few
ugly exceptions in the output, see here, even if everything went just fine
(from autotest perspective):
http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject...
and finally it reports:
09/08 12:48:13 INFO | job:1254|
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
09/08 12:48:13 INFO | job:1254| END
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
(which is also visible in autotest web frontend).
However, since we now use jskladan's patchset defining AutoQATest parent
class (using self.result variable), I no longer see a reason to raise those
exceptions. Quite the contrary.
Those exception don't even match our result states, which we have a much richer
set (passed, failed, info, aborted, crashed and maybe some other).
I'd like to propose that we stop using those autotest-internal TestFail
and TestWarn exceptions.
What will change:
1. We no longer see red/purple boxes in autotest frontend.
2. The AutoQATest parent class will be adjusted to automatically output
the test result after a test has finished.
3. Tests will just need to store proper result in self.result variable,
that's all. The methods may be then ended in a standard manner (return
or no statement).
What will be the benefits:
1. We will know that if we see a traceback it's a problem. We won't have to
examine in detail whether this traceback is "good" or "bad".
2. It will allow us to report any crashed test object (throwing an exception)
to the mailing list. We have already started to do that and it's great:
https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html
But currently it works only for those tests that haven't filled self.result
before the crash occurred. We need to catch all exceptions.
What do you think?
6:48 a.m.
Hi,
I'd support this changes - seems reasonable to me, especially once we'll use
resultsdb.
At the moment, the ExceptionCatcher decorator only sets the self.result to CRASHED,
if the variable is still unset - because of the error.Test{Fail,Warn} exceptions being
thrown when the test 'fails'.
If we omit these, every exception thrown will then represent an error (as Kamil
mentioned),
and we'll be able to go through the 'CRASHED' tests with more ease.
Joza
----- Original Message -----
From: "Kamil Paral" <kparal(a)redhat.com>
To: "List for development discussion of the AutoQA project"
<autoqa-devel(a)lists.fedorahosted.org>
Sent: Wednesday, September 8, 2010 1:41:09 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome
/ Stockholm / Vienna
Subject: proposal: deprecate using raise error.TestFail
Hi,
currently many tests use this code in its test objects:
raise error.TestFail
-or-
raise error.TestWarn
to end a test that "failed" (from our perspective). It then creates a few
ugly exceptions in the output, see here, even if everything went just fine
(from autotest perspective):
http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject...
and finally it reports:
09/08 12:48:13 INFO | job:1254|
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
09/08 12:48:13 INFO | job:1254| END
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
(which is also visible in autotest web frontend).
However, since we now use jskladan's patchset defining AutoQATest parent
class (using self.result variable), I no longer see a reason to raise those
exceptions. Quite the contrary.
Those exception don't even match our result states, which we have a much richer
set (passed, failed, info, aborted, crashed and maybe some other).
I'd like to propose that we stop using those autotest-internal TestFail
and TestWarn exceptions.
What will change:
1. We no longer see red/purple boxes in autotest frontend.
2. The AutoQATest parent class will be adjusted to automatically output
the test result after a test has finished.
3. Tests will just need to store proper result in self.result variable,
that's all. The methods may be then ended in a standard manner (return
or no statement).
What will be the benefits:
1. We will know that if we see a traceback it's a problem. We won't have to
examine in detail whether this traceback is "good" or "bad".
2. It will allow us to report any crashed test object (throwing an exception)
to the mailing list. We have already started to do that and it's great:
https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html
But currently it works only for those tests that haven't filled self.result
before the crash occurred. We need to catch all exceptions.
What do you think?
_______________________________________________
autoqa-devel mailing list
autoqa-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/autoqa-devel
2:03 p.m.
On Wed, 2010-09-08 at 07:41 -0400, Kamil Paral wrote:
Hi,
currently many tests use this code in its test objects:
raise error.TestFail
-or-
raise error.TestWarn
to end a test that "failed" (from our perspective). It then creates a few
ugly exceptions in the output, see here, even if everything went just fine
(from autotest perspective):
http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject...
and finally it reports:
09/08 12:48:13 INFO | job:1254|
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
09/08 12:48:13 INFO | job:1254| END
FAIL rpmlint rpmlint timestamp=1283950093 localtime=Sep 08 12:48:13
(which is also visible in autotest web frontend).
However, since we now use jskladan's patchset defining AutoQATest parent
class (using self.result variable), I no longer see a reason to raise those
exceptions. Quite the contrary.
Those exception don't even match our result states, which we have a much richer
set (passed, failed, info, aborted, crashed and maybe some other).
I'd like to propose that we stop using those autotest-internal TestFail
and TestWarn exceptions.
What will change:
1. We no longer see red/purple boxes in autotest frontend.
Hah, I was thinking the same thing when inspecting the autotest tko
results viewer. It's hard to distinguish between a successful test run
that found problems, vs a test case failure.
2. The AutoQATest parent class will be adjusted to automatically
output
the test result after a test has finished.
3. Tests will just need to store proper result in self.result variable,
that's all. The methods may be then ended in a standard manner (return
or no statement).
What will be the benefits:
1. We will know that if we see a traceback it's a problem. We won't have to
examine in detail whether this traceback is "good" or "bad".
+1 agreed
2. It will allow us to report any crashed test object (throwing an
exception)
to the mailing list. We have already started to do that and it's great:
https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html
But currently it works only for those tests that haven't filled self.result
before the crash occurred. We need to catch all exceptions.
What do you think?
This is a good improvement, thanks for taking the time to propose.
Thanks,
James
1:50 p.m.
On Wed, 2010-09-08 at 15:03 -0400, James Laska wrote:
On Wed, 2010-09-08 at 07:41 -0400, Kamil Paral wrote:
>
> I'd like to propose that we stop using those autotest-internal TestFail
> and TestWarn exceptions.
>
> What will change:
> 1. We no longer see red/purple boxes in autotest frontend.
Hah, I was thinking the same thing when inspecting the autotest tko
results viewer. It's hard to distinguish between a successful test run
that found problems, vs a test case failure.
Yeah - I think this would clean things up significantly for us, and is
therefore a pretty great idea.
But I wish I knew why the autotest developers originally decided to use
these exceptions rather than return values from the tests - surely there
was some good reason for that design, and I'd like to know what we're
losing by discarding it.
-w
5005
days inactive
5006
days old
autoqa-devel@lists.fedorahosted.org
3 comments
4 participants
participants (4)
-
James Laska -
Josef Skladanka -
Kamil Paral -
Will Woods