6:51 a.m.
Hi Tim & all,
in Cloud WG we have 'Automatic Smoketests on Image Build' task:
https://fedorahosted.org/cloud/ticket/38
AFAIK you have somehting similar among future Taskotron goals. It would
be nice if we can collaborate here. By writing this message I just want
to start the conversation :-)
For RHEL cloud image checks we have special tool called 'valid':
https://github.com/RedHatQE/valid , it can be reused for Fedora as
well. The questions are: how do you see the integration and how can
someone (me?) work on this (non-existent atm) part of Taskotron.
I can see two possible approaches:
1) 'Easy'. We create special type of task (sorry if I'm not that precise
with terminology) which is being executed on static slave node which
runs valid (in black-box mode) on newly uploaded image, grabs the result
and returns it back.
2) 'Hard'. We work on 'ephemeral task clients' (term from
http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here is
that in that case we can run any test on a cloud VM. This approach will
require some optimization from the very beginning, at least:
1. One VM should be user for multiple tests (so several dozens of image
tests won't require several dozens of VMs created)
2. Tests should be able to 'control' VM (e.g. use cloud-specific
features: attach device, reboot VM,...) - that can be workarounded by
providing cloud credential inside the VM itself but that doesn't sound
that attractive.
Yesterday I've tried setting up Taskotron (according to
http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and
actually I've failed (some ansible playbooks are out-of-sync with some
git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot
stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory'
in 'TASK: [copy fedmsg config]'. I totally understand these instructions
and repos are rather proof-of-concept and can be slightly outdated but
I want to ask about 'minimalistic one-host dev environment' setup
anyway. Can 'Host' be eliminated from the setup completely and can we
merge Master and Slave (in case we'll need it for 'cloud') on one host?
That would help a lot.
I'm looking forward to hearing from you.
--
Vitaly Kuznetsov
10:01 a.m.
Vitaly Kuznetsov <vitty(a)redhat.com> writes:
Hi Tim & all,
in Cloud WG we have 'Automatic Smoketests on Image Build' task:
https://fedorahosted.org/cloud/ticket/38
AFAIK you have somehting similar among future Taskotron goals. It would
be nice if we can collaborate here. By writing this message I just want
to start the conversation :-)
For RHEL cloud image checks we have special tool called 'valid':
https://github.com/RedHatQE/valid , it can be reused for Fedora as
well. The questions are: how do you see the integration and how can
someone (me?) work on this (non-existent atm) part of Taskotron.
I can see two possible approaches:
1) 'Easy'. We create special type of task (sorry if I'm not that precise
with terminology) which is being executed on static slave node which
runs valid (in black-box mode) on newly uploaded image, grabs the result
and returns it back.
2) 'Hard'. We work on 'ephemeral task clients' (term from
http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here is
that in that case we can run any test on a cloud VM. This approach will
require some optimization from the very beginning, at least:
1. One VM should be user for multiple tests (so several dozens of image
tests won't require several dozens of VMs created)
2. Tests should be able to 'control' VM (e.g. use cloud-specific
features: attach device, reboot VM,...) - that can be workarounded by
providing cloud credential inside the VM itself but that doesn't sound
that attractive.
Yesterday I've tried setting up Taskotron (according to
http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and
actually I've failed (some ansible playbooks are out-of-sync with some
git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot
stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory'
in 'TASK: [copy fedmsg config]'. I totally understand these instructions
and repos are rather proof-of-concept and can be slightly outdated but
I want to ask about 'minimalistic one-host dev environment' setup
anyway. Can 'Host' be eliminated from the setup completely and can we
merge Master and Slave (in case we'll need it for 'cloud') on one host?
That would help a lot.
I'm looking forward to hearing from you.
Sorry to bug you again about this, am I completely out of sync with
whatever is going on? I would really appreciate your comments..
--
Vitaly Kuznetsov
9:43 a.m.
On Mon, 14 Apr 2014 17:01:49 +0200
Vitaly Kuznetsov <vitty(a)redhat.com> wrote:
Vitaly Kuznetsov <vitty(a)redhat.com> writes:
> Hi Tim & all,
<snip>
>
> Yesterday I've tried setting up Taskotron (according to
> http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and
> actually I've failed (some ansible playbooks are out-of-sync with
> some git repos, e.g. 'master playbook' is failing with 'stderr: cp:
> cannot stat '/home/qaadmin/trigger/jobtrigger.py': No such file or
> directory' in 'TASK: [copy fedmsg config]'. I totally understand
> these instructions and repos are rather proof-of-concept and can be
> slightly outdated but I want to ask about 'minimalistic one-host
> dev environment' setup anyway. Can 'Host' be eliminated from the
> setup completely and can we merge Master and Slave (in case we'll
> need it for 'cloud') on one host? That would help a lot.
>
> I'm looking forward to hearing from you.
Sorry to bug you again about this, am I completely out of sync with
whatever is going on? I would really appreciate your comments..
Hey Vitaly,
That tutorial I wrote is pretty out of date since most of the ansible
playbooks have been heavily refactored. One of the things I'm going to
be working on is updating the tutorial to reflect the changes. As for
removing 'Host' from the setup - that could be done (AIUI, but I don't
know what the value add would be. Tim would be in a better position to
answer about consolidation between Host, Master and Slave.
// Mike
9:36 a.m.
On Tue, 08 Apr 2014 13:51:10 +0200
Vitaly Kuznetsov <vitty(a)redhat.com> wrote:
Hi Tim & all,
in Cloud WG we have 'Automatic Smoketests on Image Build' task:
https://fedorahosted.org/cloud/ticket/38
AFAIK you have somehting similar among future Taskotron goals. It
would be nice if we can collaborate here. By writing this message I
just want to start the conversation :-)
I'm all for collaboration. I do have some ideas on how things would
work but the whole point of this is to make something that's useful -
if that means deviating from what I originally had in mind, that's fine
with me.
For RHEL cloud image checks we have special tool called
'valid':
https://github.com/RedHatQE/valid , it can be reused for Fedora as
well. The questions are: how do you see the integration and how can
someone (me?) work on this (non-existent atm) part of Taskotron.
For now, I think that the best place to start would be to figure out
what's missing from taskotron in order for the integration to happen.
- would the cloud image startup be done by valid or in taskotron?
- where would the cloud images be run?
- what other image work would be done in the task?
Our documentation is pretty much non-existent at the moment and I was
hoping to get depcheck written as a task before writing too much of that
in case we run into problems that require syntax changes to the yaml or
interface changes to the python code. However, if you're looking to
start soon, I can try to get some of the docs done sooner. In the
meantime, there are a couple examples on how to interface with python
code:
- https://bitbucket.org/fedoraqa/task-examplebodhi
- https://bitbucket.org/fedoraqa/task-examplelong
- https://bitbucket.org/fedoraqa/task-rpmlint
Outside of the python task work that would be needed to get valid
working, there are a couple of things I can think of that would be
likely be needed to do cloud image testing:
- write a directive for finding and dealing with images
* likely some combination of "find the new image", "upload it to a
cloud system", "launch the image with a given configuration"
- figure out how to record results
* this should be pretty simple because I think cloud image results
will be similar but not quite the same as package results.
- determine what fedmsgs to trigger off of
I can see two possible approaches:
1) 'Easy'. We create special type of task (sorry if I'm not that
precise with terminology) which is being executed on static slave
node which runs valid (in black-box mode) on newly uploaded image,
grabs the result and returns it back.
I suspect that this will be the best way to get started. The
ephemeral client work you mention below is going to be a lot of work
and there are a bunch of unknowns (use openstack or not, how isolated
should the clients be etc.). If we want to have a shot at having some
of the automated cloud tests done for f21, I think that starting with
the easiest route would likely be best
2) 'Hard'. We work on 'ephemeral task clients' (term
from
http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here
is that in that case we can run any test on a cloud VM. This approach
will require some optimization from the very beginning, at least:
1. One VM should be user for multiple tests (so several dozens of
image tests won't require several dozens of VMs created)
2. Tests should be able to 'control' VM (e.g. use cloud-specific
features: attach device, reboot VM,...) - that can be workarounded by
providing cloud credential inside the VM itself but that doesn't sound
that attractive.
I suspect that some variation on this is going to be a better solution
in the long run but as I mentioned above, I think that there is little
chance of getting it working in time to be useful for f21. With the
easier option, there is a decent chance of getting something running in
the not-too-distant future.
Yesterday I've tried setting up Taskotron (according to
http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and
actually I've failed (some ansible playbooks are out-of-sync with some
git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot
stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory'
in 'TASK: [copy fedmsg config]'. I totally understand these
instructions and repos are rather proof-of-concept and can be
slightly outdated but I want to ask about 'minimalistic one-host dev
environment' setup anyway. Can 'Host' be eliminated from the setup
completely and can we merge Master and Slave (in case we'll need it
for 'cloud') on one host? That would help a lot.
Yeah, that documentation is really out of date at this point. It was
mostly written as a proof of concept and I've been holding off on
updating it until the new playbooks are done (and they're starting to
get close - you can see them in the 'rolerefactoring' branch)
For a small setup, master and slave should be able to co-exist on the
same machine. I know buildbot supports it but I'm not sure that the
playbooks are written in a way that eliminating the 'Host' like that
would actually work.
To make things even easier, you can write tasks without doing the
entire taskotron setup process by using the runner in libtaskotron
(almost identical to how it's executed in production, just without all
the extra systems for scheduling, delegation and reporting). The
reporting functionality isn't quite done yet but the python execution
module is solid and as far as I know, it's ready to be used.
I'm looking forward to hearing from you.
Apologies for the delayed response. Looking forward to working with you
on cloud image testing.
Tim
7:25 a.m.
Tim Flink <tflink(a)redhat.com> writes:
> For RHEL cloud image checks we have special tool called
'valid':
> https://github.com/RedHatQE/valid , it can be reused for Fedora as
> well. The questions are: how do you see the integration and how can
> someone (me?) work on this (non-existent atm) part of Taskotron.
For now, I think that the best place to start would be to figure out
what's missing from taskotron in order for the integration to happen.
- would the cloud image startup be done by valid or in taskotron?
- where would the cloud images be run?
- what other image work would be done in the task?
Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml
file with all tests, their steps and results + logs). I think we can
easily kick off such testing tasks from taskotron. The only missing
piece in the picture is an 'image uploader': after image was built in
koji someone needs to upload it as an AMI to EC2 (afaict Dennis
does that manually now).
2mattdm: Matt, I remember you telling me that someone is working on such
uploader. Can you please shed some light on what's the status now and
what can be done in before f21?
> I can see two possible approaches:
> 1) 'Easy'. We create special type of task (sorry if I'm not that
> precise with terminology) which is being executed on static slave
> node which runs valid (in black-box mode) on newly uploaded image,
> grabs the result and returns it back.
I suspect that this will be the best way to get started. The
ephemeral client work you mention below is going to be a lot of work
and there are a bunch of unknowns (use openstack or not, how isolated
should the clients be etc.). If we want to have a shot at having some
of the automated cloud tests done for f21, I think that starting with
the easiest route would likely be best
Ok, I think that having something ready in f21 timeframe is really
doable in case releng can automate AMIs uploading process. I'll take a
look at the runner in libtaskotron and existing tasks examples.
Apologies for the delayed response. Looking forward to working with you
on cloud image testing.
No prob)
--
Vitaly Kuznetsov
10:07 a.m.
On Thu, Apr 17, 2014 at 02:25:03PM +0200, Vitaly Kuznetsov wrote:
2mattdm: Matt, I remember you telling me that someone is working on
such
uploader. Can you please shed some light on what's the status now and
what can be done in before f21?
Yeah. Still manual process. It feels like a very long time ago that we were
talking about this in Brno! But David Gay (oddshocks) has been hired as a
summer intern and will be working on this. Some related tickets:
https://fedorahosted.org/cloud/ticket/35
https://fedorahosted.org/cloud/ticket/38
Ok, I think that having something ready in f21 timeframe is really
doable in case releng can automate AMIs uploading process. I'll take a
look at the runner in libtaskotron and existing tasks examples.
Awesome!
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
"Tepid change for the somewhat better!"
10:11 a.m.
On Thu, 17 Apr 2014 14:25:03 +0200
Vitaly Kuznetsov <vitty(a)redhat.com> wrote:
Tim Flink <tflink(a)redhat.com> writes:
>> For RHEL cloud image checks we have special tool called 'valid':
>> https://github.com/RedHatQE/valid , it can be reused for Fedora as
>> well. The questions are: how do you see the integration and how can
>> someone (me?) work on this (non-existent atm) part of Taskotron.
>
> For now, I think that the best place to start would be to figure out
> what's missing from taskotron in order for the integration to
> happen.
> - would the cloud image startup be done by valid or in taskotron?
> - where would the cloud images be run?
> - what other image work would be done in the task?
Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml
file with all tests, their steps and results + logs). I think we can
easily kick off such testing tasks from taskotron. The only missing
piece in the picture is an 'image uploader': after image was built in
koji someone needs to upload it as an AMI to EC2 (afaict Dennis
does that manually now).
The concern I have with this is around EC2 credentials and accounts. If
I'm understanding you correctly, we would need to have those
credentials stored on the clients and we'd be using EC2 time for
testing Fedora.
Does someone have budget or account time that could be used for this
in an account that we could use? I wonder if trying to use the fedora
openstack setup be a better route (assuming that valid can work with
openstack).
Another possible snag is that I want to start locking down network
access on most if not all of the test clients so that it's less
possible for user-submitted tasks to go awry and do things they
shouldn't. This hasn't been done yet, though and it's something that we
can discuss going forward.
2mattdm: Matt, I remember you telling me that someone is working on
such uploader. Can you please shed some light on what's the status
now and what can be done in before f21?
>> I can see two possible approaches:
>> 1) 'Easy'. We create special type of task (sorry if I'm not that
>> precise with terminology) which is being executed on static slave
>> node which runs valid (in black-box mode) on newly uploaded image,
>> grabs the result and returns it back.
>
> I suspect that this will be the best way to get started. The
> ephemeral client work you mention below is going to be a lot of work
> and there are a bunch of unknowns (use openstack or not, how
> isolated should the clients be etc.). If we want to have a shot at
> having some of the automated cloud tests done for f21, I think that
> starting with the easiest route would likely be best
Ok, I think that having something ready in f21 timeframe is really
doable in case releng can automate AMIs uploading process. I'll take a
look at the runner in libtaskotron and existing tasks examples.
Cool, let us know if you have any questions or if anything isn't clear.
Tim
10:17 a.m.
On Thu, Apr 17, 2014 at 09:11:10AM -0600, Tim Flink wrote:
Does someone have budget or account time that could be used for this
in an account that we could use? I wonder if trying to use the fedora
openstack setup be a better route (assuming that valid can work with
openstack).
I think we can use the Fedora community cloud EC2 account for this, but I
will double check. If that's not possible, I'll beg for the required budget.
We _should_ test on both OpenStack and EC2.
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
"Tepid change for the somewhat better!"
10:35 a.m.
Tim Flink <tflink(a)redhat.com> writes:
On Thu, 17 Apr 2014 14:25:03 +0200
Vitaly Kuznetsov <vitty(a)redhat.com> wrote:
> Tim Flink <tflink(a)redhat.com> writes:
>
> >> For RHEL cloud image checks we have special tool called 'valid':
> >> https://github.com/RedHatQE/valid , it can be reused for Fedora as
> >> well. The questions are: how do you see the integration and how can
> >> someone (me?) work on this (non-existent atm) part of Taskotron.
> >
> > For now, I think that the best place to start would be to figure out
> > what's missing from taskotron in order for the integration to
> > happen.
> > - would the cloud image startup be done by valid or in taskotron?
> > - where would the cloud images be run?
> > - what other image work would be done in the task?
>
> Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml
> file with all tests, their steps and results + logs). I think we can
> easily kick off such testing tasks from taskotron. The only missing
> piece in the picture is an 'image uploader': after image was built in
> koji someone needs to upload it as an AMI to EC2 (afaict Dennis
> does that manually now).
The concern I have with this is around EC2 credentials and accounts. If
I'm understanding you correctly, we would need to have those
credentials stored on the clients and we'd be using EC2 time for
testing Fedora.
Does someone have budget or account time that could be used for this
in an account that we could use? I wonder if trying to use the fedora
openstack setup be a better route (assuming that valid can work with
openstack).
It can't right now but switching from boto to something like libcloud is
already in my backlog. In case we do have such requirement (and Matt's
letter says we do) I can increase the priority for the task.
Another possible snag is that I want to start locking down network
access on most if not all of the test clients so that it's less
possible for user-submitted tasks to go awry and do things they
shouldn't. This hasn't been done yet, though and it's something that we
can discuss going forward.
For valid we'll require two things:
1) Access to Cloud's (AWS, Openstack, ...) endpoint
2) SSH to running VM
Can we have special dedicated test client for valid? That would make
sense from securitty pov as we need to store cloud access credentials
there.
> 2mattdm: Matt, I remember you telling me that someone is working
on
> such uploader. Can you please shed some light on what's the status
> now and what can be done in before f21?
>
> >> I can see two possible approaches:
> >> 1) 'Easy'. We create special type of task (sorry if I'm not
that
> >> precise with terminology) which is being executed on static slave
> >> node which runs valid (in black-box mode) on newly uploaded image,
> >> grabs the result and returns it back.
> >
> > I suspect that this will be the best way to get started. The
> > ephemeral client work you mention below is going to be a lot of work
> > and there are a bunch of unknowns (use openstack or not, how
> > isolated should the clients be etc.). If we want to have a shot at
> > having some of the automated cloud tests done for f21, I think that
> > starting with the easiest route would likely be best
>
> Ok, I think that having something ready in f21 timeframe is really
> doable in case releng can automate AMIs uploading process. I'll take a
> look at the runner in libtaskotron and existing tasks examples.
Cool, let us know if you have any questions or if anything isn't clear.
Tim
--
Vitaly Kuznetsov, Cloud QE
11:50 a.m.
On Thu, 17 Apr 2014 17:35:13 +0200
Vitaly Kuznetsov <vkuznets(a)redhat.com> wrote:
<snip>
> Another possible snag is that I want to start locking down
network
> access on most if not all of the test clients so that it's less
> possible for user-submitted tasks to go awry and do things they
> shouldn't. This hasn't been done yet, though and it's something
> that we can discuss going forward.
For valid we'll require two things:
1) Access to Cloud's (AWS, Openstack, ...) endpoint
2) SSH to running VM
Interfacing with EC2 wasn't a use-case that I was thinking of for
network isolation of the taskotron clients, so those plans may change
somewhat. The clients aren't isolated yet, so this won't be a problem
immediately.
Can we have special dedicated test client for valid? That would make
sense from securitty pov as we need to store cloud access credentials
there.
I suppose that we could but I'd really prefer to avoid that if at all
possible. Having one "special" client isn't an issue but it does open
the door to other task authors asking for the same thing and that will
get unmanageable pretty quick.
That being said, I'm not sure how to go about managing credentials like
that in a secure fashion. This'll require some more thought but
suggestions are certainly welcome :)
Tim
3679
days inactive
3688
days old
qa-devel@lists.fedoraproject.org
9 comments
4 participants
participants (4)
-
Matthew Miller -
Mike Ruckman -
Tim Flink -
Vitaly Kuznetsov