7:32 a.m.
There is this discussion in test list [1] where Steve Grubb posted some security testing
scripts. He asked these scripts to be included in AutoQA and executed regularly. I would
like to discuss it with the whole team because I believe we don't have a clear idea
what our current stance is.
I'd like to propose that we currently don't accept any third-party tests. It is
tempting and I'd love to have more tests inside AutoQA. But we don't have the
manpower to maintain them. Every single extra test case drains power from implementing the
AutoQA infrastructure. We already have some test cases we (I) don't understand. And it
would get worse.
I believe we need to implement proper AutoQA infrastructure and then allow people to run
and maintain their tests, without our babysitting. Different approach will consume our
resources.
What do you think?
PS: If you believe some of the tests should be included now, I expect a volunteer to step
up and take care of the integration work. Don't be shy :)
[1] http://lists.fedoraproject.org/pipermail/test/2011-August/101624.html
8:32 a.m.
+1 from me on "not now", I'm all for the third-party tests, but disposable,
and automatically created/destroyed, guests are IMHO hard prerequisity for this.
J.
----- Original Message -----
From: "Kamil Paral" <kparal(a)redhat.com>
To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org>
Sent: Tuesday, August 16, 2011 2:32:28 PM
Subject: third-party tests *now* ?
There is this discussion in test list [1] where Steve Grubb posted
some security testing scripts. He asked these scripts to be included
in AutoQA and executed regularly. I would like to discuss it with the
whole team because I believe we don't have a clear idea what our
current stance is.
I'd like to propose that we currently don't accept any third-party
tests. It is tempting and I'd love to have more tests inside AutoQA.
But we don't have the manpower to maintain them. Every single extra
test case drains power from implementing the AutoQA infrastructure. We
already have some test cases we (I) don't understand. And it would get
worse.
I believe we need to implement proper AutoQA infrastructure and then
allow people to run and maintain their tests, without our babysitting.
Different approach will consume our resources.
What do you think?
PS: If you believe some of the tests should be included now, I expect
a volunteer to step up and take care of the integration work. Don't be
shy :)
[1]
http://lists.fedoraproject.org/pipermail/test/2011-August/101624.html
_______________________________________________
autoqa-devel mailing list
autoqa-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/autoqa-devel
8:50 a.m.
On Tue, 2011-08-16 at 09:32 -0400, Josef Skladanka wrote:
+1 from me on "not now", I'm all for the third-party
tests, but
disposable, and automatically created/destroyed, guests are IMHO hard
prerequisity for this.
Can you provide more details on the prerequisites for testing these
pkgs?
Do you just need to spin up an instance of Fedora <something> and then
execute specific scripts/commands?
-sv
9:54 a.m.
On Tue, 16 Aug 2011 09:50:36 -0400
seth vidal <skvidal(a)fedoraproject.org> wrote:
On Tue, 2011-08-16 at 09:32 -0400, Josef Skladanka wrote:
> +1 from me on "not now", I'm all for the third-party tests, but
> disposable, and automatically created/destroyed, guests are IMHO
> hard prerequisity for this.
>
Can you provide more details on the prerequisites for testing these
pkgs?
Do you just need to spin up an instance of Fedora <something> and then
execute specific scripts/commands?
Spinning up a version of fedora for a test client is part of what we
would need for this.
We would also need to configure that version of fedora to work with our
test runner, determine when to schedule the tests, figure out how to
handle the results and destroy the test client when we're done.
While I can think of ways to spin up test clients on demand and run
the scripts, I think that the biggest obstacles is making the tests play
nice with what we already have without more test-specific hacking. Just
handling the results would be a challenge until we have a central place
to put them. The amount of unsorted information coming out of AutoQA is
almost unmanageable as it is.
Tim
9:47 a.m.
On Tue, 16 Aug 2011 08:32:28 -0400 (EDT)
Kamil Paral <kparal(a)redhat.com> wrote:
There is this discussion in test list [1] where Steve Grubb posted
some security testing scripts. He asked these scripts to be included
in AutoQA and executed regularly. I would like to discuss it with the
whole team because I believe we don't have a clear idea what our
current stance is.
I'd like to propose that we currently don't accept any third-party
tests. It is tempting and I'd love to have more tests inside AutoQA.
But we don't have the manpower to maintain them. Every single extra
test case drains power from implementing the AutoQA infrastructure.
We already have some test cases we (I) don't understand. And it would
get worse.
I definitely see where you're coming from here and I agree that we're
not ready to start taking in third party tests.
What do we need to do in order to be ready to take third-party tests?
Where does this fit in the priority list?
The first things that come to my mind for stuff we would need are:
- Separate the tests from the framework for packaging and
distribution purposes
- Define and implement a common external API
* For test execution
* For reporting results
- Support disposable or at least separate test clients
* having unknown scripts run on the same clients that we're using
for our existing tests seems like a bad idea to me.
I believe we need to implement proper AutoQA infrastructure and then
allow people to run and maintain their tests, without our
babysitting. Different approach will consume our resources.
I think that there might be more of a middle ground at first but
overall, I agree that there aren't enough of us to support an
ever-growing battery of tests.
What do you think?
I'm thinking a compromise for now. AutoQA isn't ready for third party
tests but these sound worth running. I think that working with the QA
community to start running them manually for now would be a good step
forward.
4680
days inactive
4680
days old
autoqa-devel@lists.fedorahosted.org
4 comments
4 participants
participants (4)
-
Josef Skladanka -
Kamil Paral -
Seth Vidal -
Tim Flink