Re: Future Requirements/Desires for Fedora Test Automation
On Wed, 16 Jan 2013 00:06:49 -0200
Lucas Meneghel Rodrigues <lmr(a)redhat.com> wrote:
On 01/15/2013 09:25 PM, Tim Flink wrote:
> I don't think this particular conversation ever made it very far and
> IIRC, hasn't been on the list yet but I want to get it started
> before FUDCon NA this weekend. I've cc'd David and Matthew because
> I've talked with them about Fedora test automation recently and
> they might have input.
>
> In my opinion (I suspect that other people feel similarly), AutoQA
> in it's current form is not capable of meeting the test automation
> needs for Fedora; mostly because we don't have a clear path towards
> external tests and it seems pretty clear that the current devs
> (myself included) don't have the bandwidth to add any more tests to
> the current set.
>
> There has been some casual conversation about looking into switching
> over to using Beaker [1] so that we can leverage some of the tests
> currently being used by various groups within Red Hat instead of
> having to rewrite them for AutoQA/Autotest. However, I don't want
> this to sound like 'autotest is bad'. I sincerely doubt that there
> is a single framework/runner out there which will 100% satisfy all
> of our needs and I'm just looking to re-evaluate what we want from
> test automation before deciding how we get there.
As one would expect, I'd jump right away into the discussion.
Despite the fact that I said that I wanted to _not_ get into framework
and implementation specifics in this thread twice in the part of the
email you didn't quote. Oh well, I got an email from a beaker dev, too
so I can't fault you too much :-P
I just wanted to start the conversation with what we wanted to have for
Fedora test automation so that there was a basis for comparison before
we dove into 'autotest does X', 'beaker does Y' and so on.
One thing about beaker is that the test harness (Beah) has some
limitations that drive people to look into autotest frequently (I
talk to people inside the company interested in it). The folks at
beaker made some experiments using the autotest client as one of the
possible harnesses inside their framework, but I haven't heard much
from them after the initial patches.
It is true that we have scarce resources. My team has in fact 4
people (not all of us allocated full time) and we spend much of the
time writing and reviewing virtualization tests, our main
attribution. So it's understandable that you want to change.
In my mind, it's not so much of a "autotest won't work for us" as a
"we're not using much of what autotest can do and I'm not sure our
needs are all that compatible without quite a bit of work". I sincerely
doubt that we're going to get all of what we want/need in fedora
without at least submitting code to an existing framework, though so
the desired direction of any upstream would certainly be a factor.
I'm not sure whether I was actually called into this discussion,
so
I'll refrain to comment on the requirements. I still think that
autotest provides at least a part of what is required.
Yeah, I was hoping to talk to you about autotest this weekend since
we're both going to be at FUDCon NA. I don't pretend to understand all
of what autotest can do and figured you would have more insight.
Tim
Attachments:
- signature.asc (application/pgp-signature — 490 bytes)