3:04 p.m.
Condor Cloud for Aeolus
-----------------------
I wrote up a quick summary of the current issues/ideas around this
project so we can get things rolling. If you don't want to read
the whole document please at least look at the section you are
interested in. :) I CC'd a few condor guys as well.
Goal
----
The idea here is to create a simple cloud provider driven through
deltacloud that we can use in Aeolus. The first iteration will be
extremely simple indeed and basically just make use of the facilities
offered by condor with some glue to make things work.
- VMs will use KVM via Condor.
- Condor will keep track of all instances for us and offer scalability.
- Instances will be stateless. Condor will copy VM images before running
them and NOT save state when done.
- Simple mechanism for uploading images to machine running condor.
- Deltacloud driver to interface directly with condor or through simple
agent (to handle MAC/IP mappings, authentication etc).
Image Wharehouse Integration
----------------------------
So far the general consensus is to use something simple like NFS or scp
to copy images to the Condor central manager server. This will consist
of an upload directory and then a staging directory. While uploading
files go into the uploading directory and then are moved to staging once
upload is complete. This prevents trying to start a half-copied image,
although that is generally not possible from the aeolus UI.
Only question here is that we may need some info on what OS this is
targeting so we can do the drive mapping correctly etc. The iwhd guys
may know more issues as well.
Deltacloud Driver
-----------------
The deltacloud driver can interact directly with condor commands
to start, stop, and query the state of instances running in condor.
Ruby code to do everything but the query is already in conductor, but
generally we use condor_submit to start a new job/instance, condor_q -xml
to get the state of jobs on the system (parsing xml output), and
condor_rm to stop an instance.
Now, the difficulty comes in when we start looking at doing things like
mapping MACs to IPs, authentication etc. If we do the condor commands
directly in the driver then the driver would have to handle these tasks
as well. I'm not sure if that is desirable.
The other idea is to put some kind of agent in between the driver and
condor which can handle MAC/IP translation, authentication, perhaps
image registration etc. The benefit of this is that then we can work
on adding features to the 'cloud' without having to change the deltacloud
driver itself as often. I'm starting to think this is the way to go
even tho it may be slightly more work up front.
I also looked into having the IP address set in the job via condor but
I do not see a clean/easy way to do this.
IP Discovery
------------
There are many possible solutions to this problem but none of them are
obviously better than any other. The three that seem the most promising
are:
1) Using an agent in the VMs to register with a central server thereby
letting us know its IP.
2) Having control of the DHCP server so we can see which leases are out
for which MACs.
3) Using a config file which maps MACs to IPs and then configuring dhcpd
with the same mapping.
Of these I'm thinking we should implement #3 first. I think this should
get us up and running quickly and it can be configured in almost any
environment. Second option would be #1. I think in the end we will
need to implement all of these and have it be configurable by the admin.
Using the config file to map MACs to IPs means having to keep track of
which macs are in use. We can do this easily by querying the jobs which
contain this information. However this lookup would have to be done before
the job is submitted to condor and so this duty would fall on either
the deltacloud driver itself or the condor cloud agent.
Installation/Configuration
--------------------------
I'm hoping I can get some help on using a puppet script to ease installation
similar to what we are doing with Aeolus Conductor now. While I don't have
all the details yet, in general we will need it to:
- Configure condor.
- Set up deltacloud correctly with the right driver etc.
- Set up a network bridge for VMs.
- Set up the MAC/IP address table for admins to edit.
Other Docs
----------
Please also see Mathews document on Condor as a cloud provider:
http://spinningmatt.files.wordpress.com/2010/04/matthewfarrelleeopensourc...
Summary
-------
I think I've touched on the major issues/talking points here. Hopefully we
can come up with the best solutions quickly and get to writing the parts
soon.
Really the only parts that are still up in the air are the IP/MAC mapping
(possibly tho I'm fine with the config file thing for now), and how to
architect the driver - eg whether we need an agent in between the driver
and condor. I think the rest we can work out as we go.
Ian
3:17 p.m.
From: Ian Main <imain(a)redhat.com>
Date: Fri, 11 Mar 2011 16:04:00 -0500
[...]
IP Discovery
------------
There are many possible solutions to this problem but none of them are
obviously better than any other. The three that seem the most promising
are:
1) Using an agent in the VMs to register with a central server thereby
letting us know its IP.
2) Having control of the DHCP server so we can see which leases are out
for which MACs.
3) Using a config file which maps MACs to IPs and then configuring dhcpd
with the same mapping.
Of these I'm thinking we should implement #3 first.
3 is relatively easy. I would have thought 2 was easy, but I'm told
that we can't count on it.
My observation is that, immediately after you prove that you can
launch an instance, you're going to need to care about configuring
it. That means you need the config server, which is door number 1.
So I advocate that we just bite the bullet and do that one right
away.
I think this should
get us up and running quickly and it can be configured in almost any
environment. Second option would be #1. I think in the end we will
need to implement all of these and have it be configurable by the admin.
You always need 1, regardless of what other things you do.
Other tidbit that I didn't see: We have the ability to inject some
small snippet of data at launch time, correct? For instance as a karg
or something? I'm pretty sure we talked about this offline, but I
want to make sure (a) the answer is yes, and (b) we build it into the
api.
This looks great so far. Thanks for writing this up!
2:50 p.m.
On Fri, Mar 11, 2011 at 04:17:18PM -0500, John R. Dunning wrote:
From: Ian Main <imain(a)redhat.com>
Date: Fri, 11 Mar 2011 16:04:00 -0500
[...]
IP Discovery
------------
There are many possible solutions to this problem but none of them are
obviously better than any other. The three that seem the most promising
are:
1) Using an agent in the VMs to register with a central server thereby
letting us know its IP.
2) Having control of the DHCP server so we can see which leases are out
for which MACs.
3) Using a config file which maps MACs to IPs and then configuring dhcpd
with the same mapping.
Of these I'm thinking we should implement #3 first.
3 is relatively easy. I would have thought 2 was easy, but I'm told
that we can't count on it.
Well it is relatively easy for us to implement but it is not always easy
to set up. Not for developers at least which is our initial target
audience.
My observation is that, immediately after you prove that you can
launch an instance, you're going to need to care about configuring
it. That means you need the config server, which is door number 1.
So I advocate that we just bite the bullet and do that one right
away.
I think this should
get us up and running quickly and it can be configured in almost any
environment. Second option would be #1. I think in the end we will
need to implement all of these and have it be configurable by the admin.
You always need 1, regardless of what other things you do.
Yes and we will get there but my understanding is that it is still a ways
off in the grand scheme of things.. at least longer than I want to wait.
It's also my understanding that this would integrate with Aeolus and not
necessarily allow a condor cloud to stand on its own (by giving the user
of condor cloud only the IP of the instance).
Other tidbit that I didn't see: We have the ability to inject some
small snippet of data at launch time, correct? For instance as a karg
or something? I'm pretty sure we talked about this offline, but I
want to make sure (a) the answer is yes, and (b) we build it into the
api.
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
The other thing I noticed is that there is no way to override the
architecture via condor. Eg run an i686 guest on x86_64 hardware.
Ian
2:57 p.m.
From: Ian Main <imain(a)redhat.com>
Date: Mon, 14 Mar 2011 15:50:07 -0400
You always need 1, regardless of what other things you do.
Yes and we will get there but my understanding is that it is still a ways
off in the grand scheme of things..
Audrey is in prototype right now today, using such a facility. The
ability to configure an image post-boot is DOA without it.
Separate from that, since we *know* we need the config server, in the
not-too-distant future, and the config server solves the IP discovery
problem, what's the benefit in building something else, just to throw
it away?
Other tidbit that I didn't see: We have the ability to inject some
small snippet of data at launch time, correct? For instance as a karg
or something? I'm pretty sure we talked about this offline, but I
want to make sure (a) the answer is yes, and (b) we build it into the
api.
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
I would expect that it would be useful upstream. But whether or not
that's the case, it's a pretty hard requirement for our usage of this
stuff.
3:46 p.m.
On Mon, Mar 14, 2011 at 03:57:40PM -0400, John R. Dunning wrote:
From: Ian Main <imain(a)redhat.com>
Date: Mon, 14 Mar 2011 15:50:07 -0400
[snip]
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
I would expect that it would be useful upstream. But whether or not
that's the case, it's a pretty hard requirement for our usage of this
stuff.
OK, I stand corrected, you can in fact invent whatever XML you want to
hand to virsh via whatever params you want in the job. So, we are
golden on that front.
Ian
10:17 a.m.
From: Ian Main <imain(a)redhat.com>
Date: Mon, 14 Mar 2011 16:46:55 -0400
On Mon, Mar 14, 2011 at 03:57:40PM -0400, John R. Dunning wrote:
From: Ian Main <imain(a)redhat.com>
Date: Mon, 14 Mar 2011 15:50:07 -0400
[snip]
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
I would expect that it would be useful upstream. But whether or not
that's the case, it's a pretty hard requirement for our usage of this
stuff.
OK, I stand corrected, you can in fact invent whatever XML you want to
hand to virsh via whatever params you want in the job. So, we are
golden on that front.
Ok.
So to recap:
1. We have a scheme for using condor to handle the resource
allocation, ie when a request comes in, decide where to place it, and
cause an instance to be launched.
2. We know how to make that scheme interact with the process of
copying images around when needed, so that the iwhd will be able to
provision images into it once, and let the "backend" deal with horsing
things around as needed at launch time.
3. We believe we can arrange to pass a small snippet of config data
in at launch time, to point a booting instance at some source of
further config.
4. [I assert] that we have established the need for some kind of
config server to exist at launch time. So the strawman implementation
of 3 is to point the instance at the config server.
5. Assuming I'm correct about 4, that server can also communicate the
IP address of the booting instance back to the mother ship
(conductor). So there's no need to build any other scheme for
managing IP addresses.
6. We understand what's needed to build a DC API interface layer to
the condor instance referenced in 1. Michal is all over that.
7. We will need to work out the admin details of standing up the
stateful server piece, presumably including the condor instance, the
config server, and the interface piece. I believe David has
volunteered to take a look at this piece.
So I think that's all the functional pieces we need.
TBD:
What do we document about standing up the actual servers which
are running the VMs?
Do we need to do anything about mac address management? Is that part
of 1?
What else have I forgot to mention?
If this is all about right, is it time to divvy up tasks and talk
about timeframes and stuff?
12:35 p.m.
On Tue, Mar 15, 2011 at 11:17:32AM -0400, John R. Dunning wrote:
From: Ian Main <imain(a)redhat.com>
Date: Mon, 14 Mar 2011 16:46:55 -0400
On Mon, Mar 14, 2011 at 03:57:40PM -0400, John R. Dunning wrote:
> From: Ian Main <imain(a)redhat.com>
> Date: Mon, 14 Mar 2011 15:50:07 -0400
[snip]
>
> Actually I was just looking through the code and AFAICT this is only
> supported for Xen based VMs, not KVM. It would be easy enough to add to
> condor though.. but then we are getting into needing a modified condor
> again. Perhaps this line of development would be useful to upstream
> though?
>
> I would expect that it would be useful upstream. But whether or not
> that's the case, it's a pretty hard requirement for our usage of this
> stuff.
OK, I stand corrected, you can in fact invent whatever XML you want to
hand to virsh via whatever params you want in the job. So, we are
golden on that front.
Ok.
So to recap:
1. We have a scheme for using condor to handle the resource
allocation, ie when a request comes in, decide where to place it, and
cause an instance to be launched.
Correct.
2. We know how to make that scheme interact with the process of
copying images around when needed, so that the iwhd will be able to
provision images into it once, and let the "backend" deal with horsing
things around as needed at launch time.
Correct.
3. We believe we can arrange to pass a small snippet of config data
in at launch time, to point a booting instance at some source of
further config.
Kernel boot params, yes.
4. [I assert] that we have established the need for some kind of
config server to exist at launch time. So the strawman implementation
of 3 is to point the instance at the config server.
Yes.
5. Assuming I'm correct about 4, that server can also
communicate the
IP address of the booting instance back to the mother ship
(conductor). So there's no need to build any other scheme for
managing IP addresses.
So I may be wrong about this, but what I think I'm seeing here is that
the config server is NOT the same piece of software as the agent we are
making, and it is the agent which should know the IP. Generally the
flow of information is from deltacloud/condor to conductor, you are
talking about going around deltacloud here and injecting data directly
into conductor. I'm not saying this is impossible but it's not how it's
currently designed, and would need some thought.
That said I guess there's no reason we couldn't do the registration
twice. Basically ec2 supports reporting its public IPs and I would like
to do that in this condor cloud as well. The config server thing and
reporting IPs directly to conductor is, as far as I understand, a
separate issue.
Basically my plan is to support several schemes and let the admin
choose.
6. We understand what's needed to build a DC API interface layer
to
the condor instance referenced in 1. Michal is all over that.
yes.
7. We will need to work out the admin details of standing up the
stateful server piece, presumably including the condor instance, the
config server, and the interface piece. I believe David has
volunteered to take a look at this piece.
I was not aware of that. I have been working on getting it all stood up
too and have condor working running VMs and we're starting to build
infrastructure around it.
In the end I think we want a puppet script like we use in conductor to
stand it all up for us easily.
So I think that's all the functional pieces we need.
Yes.
TBD:
What do we document about standing up the actual servers which
are running the VMs?
Everything needed, and then hopefully puppetize it.
Do we need to do anything about mac address management? Is that
part
of 1?
That will be up to the 'agent' that is between the deltacloud driver and
condor.
What else have I forgot to mention?
If this is all about right, is it time to divvy up tasks and talk
about timeframes and stuff?
Yes I think it is. Michal and I are working on getting the
deltacloud/condor bit up and running. We need someone to start working
on image wharehouse integration, the first step being to hash out the
details a bit more.
I'll have a git repo up soon I hope. Ticket is in to fedorahosted.
Once things are more solidified we will need someone to do puppetizing.
Ian
12:45 p.m.
On 03/15/2011 01:35 PM, Ian Main wrote:
On Tue, Mar 15, 2011 at 11:17:32AM -0400, John R. Dunning wrote:
> 5. Assuming I'm correct about 4, that server can also
communicate the
> IP address of the booting instance back to the mother ship
> (conductor). So there's no need to build any other scheme for
> managing IP addresses.
So I may be wrong about this, but what I think I'm seeing here is that
the config server is NOT the same piece of software as the agent we are
making, and it is the agent which should know the IP. Generally the
flow of information is from deltacloud/condor to conductor, you are
talking about going around deltacloud here and injecting data directly
into conductor. I'm not saying this is impossible but it's not how it's
currently designed, and would need some thought.
That said I guess there's no reason we couldn't do the registration
twice. Basically ec2 supports reporting its public IPs and I would like
to do that in this condor cloud as well. The config server thing and
reporting IPs directly to conductor is, as far as I understand, a
separate issue.
Basically my plan is to support several schemes and let the admin
choose.
This stuck out for me too. I'd strongly recommend a return path of
information that goes through the condor_schedd. So, job (VM in this
case) -> schedd -> driver -> conductor. VM -> Conductor feels like a
layer violation.
Best,
matt
1:46 p.m.
On 03/15/11 - 01:35:53PM, Ian Main wrote:
> 3. We believe we can arrange to pass a small snippet of config
data
> in at launch time, to point a booting instance at some source of
> further config.
Kernel boot params, yes.
Note that this won't work. In general, you can't add kernel boot params to
a guest, because we don't direct-boot the kernel. We boot the emulated BIOS,
which calls grub, which calls the kernel.
Luckily there is a workaround. We can stuff some data into the SMBIOS tables
during launch time, and use that as our injection mechanism instead.
--
Chris Lalancette
7:05 p.m.
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > 3. We believe we can arrange to pass a small snippet of config data
> > in at launch time, to point a booting instance at some source of
> > further config.
>
> Kernel boot params, yes.
Note that this won't work. In general, you can't add kernel boot params to
a guest, because we don't direct-boot the kernel. We boot the emulated BIOS,
which calls grub, which calls the kernel.
Luckily there is a workaround. We can stuff some data into the SMBIOS tables
during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
SMBIOS is fine for internal instrumentation (e.g., where to report IP
address), but not for generic user data.
David
7:54 a.m.
From: David Lutterkort <lutter(a)redhat.com>
Date: Tue, 15 Mar 2011 17:05:24 -0700
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > 3. We believe we can arrange to pass a small snippet of config data
> > in at launch time, to point a booting instance at some source of
> > further config.
>
> Kernel boot params, yes.
Note that this won't work. In general, you can't add kernel boot params to
a guest, because we don't direct-boot the kernel. We boot the emulated BIOS,
which calls grub, which calls the kernel.
Luckily there is a workaround. We can stuff some data into the SMBIOS tables
during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
Do you mean pry open the image a few ms before launch, frob something
in it, then launch?
That's ok if you've got a backend which doesn't do, for instance, COW,
while running multiple instances from a single image file.
I think you want to do something which ties to the instance, not the
image.
Besides, if we're arguing from the API point of view, who cares what
the underlying implementation is? One of the main points of having an
API in the first place is to allow for hiding wacky implementation
details, especially when there are multiple implementations involved.
SMBIOS is fine for internal instrumentation (e.g., where to report IP
address), but not for generic user data.
I think all that's being proposed to go in smbios is a 32-bit dingus
(ip address of server) and a somewhat longer dingus for instance UID.
If we need 128 bits for that, it's 160 bits total. Is that acceptable
to stash in smbios at boot time?
7:59 a.m.
On 03/16/2011 08:54 AM, John R. Dunning wrote:
From: David Lutterkort<lutter(a)redhat.com>
Date: Tue, 15 Mar 2011 17:05:24 -0700
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > 3. We believe we can arrange to pass a small snippet of config
data
> > > in at launch time, to point a booting instance at some source of
> > > further config.
> >
> > Kernel boot params, yes.
>
> Note that this won't work. In general, you can't add kernel boot
params to
> a guest, because we don't direct-boot the kernel. We boot the emulated
BIOS,
> which calls grub, which calls the kernel.
>
> Luckily there is a workaround. We can stuff some data into the SMBIOS tables
> during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
Do you mean pry open the image a few ms before launch, frob something
in it, then launch?
That's ok if you've got a backend which doesn't do, for instance, COW,
while running multiple instances from a single image file.
I think you want to do something which ties to the instance, not the
image.
I imagine you would crack open the instance's disk, not the shared COW,
forcing a little C'ing.
Best,
matt
Besides, if we're arguing from the API point of view, who cares
what
the underlying implementation is? One of the main points of having an
API in the first place is to allow for hiding wacky implementation
details, especially when there are multiple implementations involved.
SMBIOS is fine for internal instrumentation (e.g., where to report IP
address), but not for generic user data.
I think all that's being proposed to go in smbios is a 32-bit dingus
(ip address of server) and a somewhat longer dingus for instance UID.
If we need 128 bits for that, it's 160 bits total. Is that acceptable
to stash in smbios at boot time?
8:16 a.m.
From: Matthew Farrellee <matt(a)redhat.com>
Date: Wed, 16 Mar 2011 08:59:28 -0400
That's ok if you've got a backend which doesn't do, for instance, COW,
while running multiple instances from a single image file.
I think you want to do something which ties to the instance, not the
image.
I imagine you would crack open the instance's disk, not the shared COW,
forcing a little C'ing.
I probably don't know enough of the underlying technical mechanics to
speak sensibly on the topic. My overall point is that as long as
we're not gratuitously copying things, ie copying solely for the sake
of injecting data, then I don't see that it makes a lot of difference
what the underlying implementation is for injecting per-instance
data. If we have to copy things for other reasons (which we expect to
have to do on some backends) then libguestfs ought to be fine.
1:20 p.m.
On Wed, 2011-03-16 at 08:59 -0400, Matthew Farrellee wrote:
On 03/16/2011 08:54 AM, John R. Dunning wrote:
> From: David Lutterkort<lutter(a)redhat.com>
> Date: Tue, 15 Mar 2011 17:05:24 -0700
>
> On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> > On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > > 3. We believe we can arrange to pass a small snippet of
config data
> > > > in at launch time, to point a booting instance at some source
of
> > > > further config.
> > >
> > > Kernel boot params, yes.
> >
> > Note that this won't work. In general, you can't add kernel boot
params to
> > a guest, because we don't direct-boot the kernel. We boot the
emulated BIOS,
> > which calls grub, which calls the kernel.
> >
> > Luckily there is a workaround. We can stuff some data into the SMBIOS
tables
> > during launch time, and use that as our injection mechanism instead.
>
> There's also door #3, which would be hugely preferrable from an API POV:
> using kpartx^W libguestfs to directly inject files into the image. This
> is the approach Rackspace takes; IOW, we wouldn't introduce yet another
> paradigm of how to inject user data into an instance.
>
> Do you mean pry open the image a few ms before launch, frob something
> in it, then launch?
>
> That's ok if you've got a backend which doesn't do, for instance, COW,
> while running multiple instances from a single image file.
>
> I think you want to do something which ties to the instance, not the
> image.
I imagine you would crack open the instance's disk, not the shared COW,
forcing a little C'ing.
Yes, what Matt said. libguestfs takes care of the nasty bits of doing
this.
David
1:12 p.m.
On Wed, 2011-03-16 at 08:54 -0400, John R. Dunning wrote:
From: David Lutterkort <lutter(a)redhat.com>
Date: Tue, 15 Mar 2011 17:05:24 -0700
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > 3. We believe we can arrange to pass a small snippet of config data
> > > in at launch time, to point a booting instance at some source of
> > > further config.
> >
> > Kernel boot params, yes.
>
> Note that this won't work. In general, you can't add kernel boot params
to
> a guest, because we don't direct-boot the kernel. We boot the emulated
BIOS,
> which calls grub, which calls the kernel.
>
> Luckily there is a workaround. We can stuff some data into the SMBIOS tables
> during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
Do you mean pry open the image a few ms before launch, frob something
in it, then launch?
Yes, since we already have to have a copy (or a COW snapshot) before
launching the instance, we'd just use virt-edit[1] to modify files in
the instance's image. In reality, this would boil down to using the
libguestfs Ruby bindings [2]
This should not be much of an effort to implement.
That's ok if you've got a backend which doesn't do, for
instance, COW,
while running multiple instances from a single image file.
We absolutely have to either have COW or a per-instance copy of the
image.
Besides, if we're arguing from the API point of view, who cares
what
the underlying implementation is? One of the main points of having an
API in the first place is to allow for hiding wacky implementation
details, especially when there are multiple implementations involved.
Not quite - the two existing ways of userdata injection, EC2-style via a
special webserver, and RAX-style by scribbling in the FS, can not really
be exposed in the same way in the Deltacloud API. So for API-level
user-data injection, I'd rather stick with one of those two approaches
than introduce yet a third method.
SMBIOS is fine for internal instrumentation (e.g., where to
report IP
address), but not for generic user data.
I think all that's being proposed to go in smbios is a 32-bit dingus
(ip address of server) and a somewhat longer dingus for instance UID.
If we need 128 bits for that, it's 160 bits total. Is that acceptable
to stash in smbios at boot time?
That's fine for the bits of data that Falcon needs for its IP-sniffing
purposes. It's probably not enough for what we'd offer users through the
API.
David
[1] http://libguestfs.org/virt-edit.1.html
[2] http://libguestfs.org/guestfs-ruby.3.html
6:47 p.m.
From: David Lutterkort <lutter(a)redhat.com>
Date: Wed, 16 Mar 2011 11:12:22 -0700
On Wed, 2011-03-16 at 08:54 -0400, John R. Dunning wrote:
From: David Lutterkort <lutter(a)redhat.com>
Date: Tue, 15 Mar 2011 17:05:24 -0700
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > 3. We believe we can arrange to pass a small snippet of config data
> > > in at launch time, to point a booting instance at some source of
> > > further config.
> >
> > Kernel boot params, yes.
>
> Note that this won't work. In general, you can't add kernel boot params
to
> a guest, because we don't direct-boot the kernel. We boot the emulated
BIOS,
> which calls grub, which calls the kernel.
>
> Luckily there is a workaround. We can stuff some data into the SMBIOS tables
> during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
Do you mean pry open the image a few ms before launch, frob something
in it, then launch?
Yes, since we already have to have a copy (or a COW snapshot) before
launching the instance, we'd just use virt-edit[1] to modify files in
the instance's image. In reality, this would boil down to using the
libguestfs Ruby bindings [2]
This should not be much of an effort to implement.
That's ok if you've got a backend which doesn't do, for
instance, COW,
while running multiple instances from a single image file.
We absolutely have to either have COW or a per-instance copy of the
image.
Sure.
Besides, if we're arguing from the API point of view, who cares
what
the underlying implementation is? One of the main points of having an
API in the first place is to allow for hiding wacky implementation
details, especially when there are multiple implementations involved.
Not quite - the two existing ways of userdata injection, EC2-style via a
special webserver, and RAX-style by scribbling in the FS, can not really
be exposed in the same way in the Deltacloud API. So for API-level
user-data injection, I'd rather stick with one of those two approaches
than introduce yet a third method.
Well, ok, but please recall that the reason why we're going to the
trouble to build a stateful layer in front of the falcon backend is
precisely to avoid showing vagaries of backends through. Instead, we
are opting to present a uniform model, and hide the details behind the
api.
If we can't do that similarly for userdata, or a subset of it, I
submit that we're doing something wrong. Thats not to say that we
might not opt to allow the full glory of individual backends to show
through in backend-specific ways, but we ought to be able to define a
subset which we present in a common fashion.
This particular point is off topic for this conversation. If you'd
like to discuss further, let's take it offline.
SMBIOS is fine for internal instrumentation (e.g., where to
report IP
address), but not for generic user data.
I think all that's being proposed to go in smbios is a 32-bit dingus
(ip address of server) and a somewhat longer dingus for instance UID.
If we need 128 bits for that, it's 160 bits total. Is that acceptable
to stash in smbios at boot time?
That's fine for the bits of data that Falcon needs for its IP-sniffing
purposes. It's probably not enough for what we'd offer users through the
API.
Again, the purpose of the falcon backend is to serve the needs of
cloud engine. It's much more interesting, to me, and I suspect to
Hugh's group, to make that work first. If, after that, we decide that
there's value in making it available separate from cloud engine, and
if we decide further that it makes sense to present DC as the front
end for it, and if we decide further that in that standalone mode we
want to make available facilities which aren't relevant when it's used
as part of CE, that will be a worthy discussion to have. For now, I
think we should focus on the simple cases which alloc CE to do what it
needs.
7:01 a.m.
On Wed, Mar 16, 2011 at 07:47:40PM -0400, John R. Dunning wrote:
From: David Lutterkort <lutter(a)redhat.com>
Date: Wed, 16 Mar 2011 11:12:22 -0700
On Wed, 2011-03-16 at 08:54 -0400, John R. Dunning wrote:
> From: David Lutterkort <lutter(a)redhat.com>
> Date: Tue, 15 Mar 2011 17:05:24 -0700
>
> On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> > On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > > 3. We believe we can arrange to pass a small snippet of
config data
> > > > in at launch time, to point a booting instance at some source
of
> > > > further config.
> > >
> > > Kernel boot params, yes.
> >
> > Note that this won't work. In general, you can't add kernel
boot params to
> > a guest, because we don't direct-boot the kernel. We boot the
emulated BIOS,
> > which calls grub, which calls the kernel.
> >
> > Luckily there is a workaround. We can stuff some data into the SMBIOS
tables
> > during launch time, and use that as our injection mechanism instead.
>
> There's also door #3, which would be hugely preferrable from an API
POV:
> using kpartx^W libguestfs to directly inject files into the image. This
> is the approach Rackspace takes; IOW, we wouldn't introduce yet another
> paradigm of how to inject user data into an instance.
>
> Do you mean pry open the image a few ms before launch, frob something
> in it, then launch?
Yes, since we already have to have a copy (or a COW snapshot) before
launching the instance, we'd just use virt-edit[1] to modify files in
the instance's image. In reality, this would boil down to using the
libguestfs Ruby bindings [2]
This should not be much of an effort to implement.
> That's ok if you've got a backend which doesn't do, for instance,
COW,
> while running multiple instances from a single image file.
We absolutely have to either have COW or a per-instance copy of the
image.
Sure.
> Besides, if we're arguing from the API point of view, who cares what
> the underlying implementation is? One of the main points of having an
> API in the first place is to allow for hiding wacky implementation
> details, especially when there are multiple implementations involved.
Not quite - the two existing ways of userdata injection, EC2-style via a
special webserver, and RAX-style by scribbling in the FS, can not really
be exposed in the same way in the Deltacloud API. So for API-level
user-data injection, I'd rather stick with one of those two approaches
than introduce yet a third method.
Well, ok, but please recall that the reason why we're going to the
trouble to build a stateful layer in front of the falcon backend is
precisely to avoid showing vagaries of backends through. Instead, we
are opting to present a uniform model, and hide the details behind the
api.
If we can't do that similarly for userdata, or a subset of it, I
submit that we're doing something wrong. Thats not to say that we
might not opt to allow the full glory of individual backends to show
through in backend-specific ways, but we ought to be able to define a
subset which we present in a common fashion.
This particular point is off topic for this conversation. If you'd
like to discuss further, let's take it offline.
I agree with John here -- unless there's a really good reason to
expose to the user the details of how we do userdata injection, why
wouldn't we want to hide them? What benefit does exposing the details
grant to the user?
> SMBIOS is fine for internal instrumentation (e.g., where
to report IP
> address), but not for generic user data.
>
> I think all that's being proposed to go in smbios is a 32-bit dingus
> (ip address of server) and a somewhat longer dingus for instance UID.
> If we need 128 bits for that, it's 160 bits total. Is that acceptable
> to stash in smbios at boot time?
That's fine for the bits of data that Falcon needs for its IP-sniffing
purposes. It's probably not enough for what we'd offer users through the
API.
Again, the purpose of the falcon backend is to serve the needs of
cloud engine. It's much more interesting, to me, and I suspect to
Hugh's group, to make that work first. If, after that, we decide that
there's value in making it available separate from cloud engine, and
if we decide further that it makes sense to present DC as the front
end for it, and if we decide further that in that standalone mode we
want to make available facilities which aren't relevant when it's used
as part of CE, that will be a worthy discussion to have. For now, I
think we should focus on the simple cases which alloc CE to do what it
needs.
Here's to focusing on the simple case!
--Hugh
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
== http://aeolusproject.org ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey
1:23 p.m.
On Wed, 2011-03-16 at 19:47 -0400, John R. Dunning wrote:
Not quite - the two existing ways of userdata injection,
EC2-style via a
special webserver, and RAX-style by scribbling in the FS, can not really
be exposed in the same way in the Deltacloud API. So for API-level
user-data injection, I'd rather stick with one of those two approaches
than introduce yet a third method.
Well, ok, but please recall that the reason why we're going to the
trouble to build a stateful layer in front of the falcon backend is
precisely to avoid showing vagaries of backends through. Instead, we
are opting to present a uniform model, and hide the details behind the
api.
If we can't do that similarly for userdata, or a subset of it, I
submit that we're doing something wrong.
We can try and produce a LCD version of these facilities, though there's
a very high danger that that forces us to break API compatibility:
imagine we wallpapered over the data injection differences between EC2
and RAX by saying 'you can give us a userblob to inject into the
instance, up to 10kB; in EC2 it will be posted at 169.254.169.254, for
RAX it will wind up in /var/lib/userblob' Now we add Falcon's SMBIOS
based data injection - we immediately broke API because we can't inject
10kB of data anymore into any cloud that supports the 'userblob'.
In any event, these simplifications are really application policy.
This particular point is off topic for this conversation. If
you'd
like to discuss further, let's take it offline.
Sure - but it's an important point in making sure Deltacloud remains a
versatile API.
That's fine for the bits of data that Falcon needs for its
IP-sniffing
purposes. It's probably not enough for what we'd offer users through the
API.
Again, the purpose of the falcon backend is to serve the needs of
cloud engine. It's much more interesting, to me, and I suspect to
Hugh's group, to make that work first. If, after that, we decide that
there's value in making it available separate from cloud engine, and
if we decide further that it makes sense to present DC as the front
end for it
Isn't DC the abstraction that CE uses to talk to backend clouds ? Why
would CE talk to Falcon with anything else than DC ? At the very least,
it would force CE to do a lot more special casing than what we discussed
above for user-data.
David
3:43 p.m.
On Thu, Mar 17, 2011 at 11:23:32AM -0700, David Lutterkort wrote:
On Wed, 2011-03-16 at 19:47 -0400, John R. Dunning wrote:
> Not quite - the two existing ways of userdata injection, EC2-style via a
> special webserver, and RAX-style by scribbling in the FS, can not really
> be exposed in the same way in the Deltacloud API. So for API-level
> user-data injection, I'd rather stick with one of those two approaches
> than introduce yet a third method.
>
> Well, ok, but please recall that the reason why we're going to the
> trouble to build a stateful layer in front of the falcon backend is
> precisely to avoid showing vagaries of backends through. Instead, we
> are opting to present a uniform model, and hide the details behind the
> api.
>
> If we can't do that similarly for userdata, or a subset of it, I
> submit that we're doing something wrong.
We can try and produce a LCD version of these facilities, though there's
a very high danger that that forces us to break API compatibility:
imagine we wallpapered over the data injection differences between EC2
and RAX by saying 'you can give us a userblob to inject into the
instance, up to 10kB; in EC2 it will be posted at 169.254.169.254, for
RAX it will wind up in /var/lib/userblob' Now we add Falcon's SMBIOS
based data injection - we immediately broke API because we can't inject
10kB of data anymore into any cloud that supports the 'userblob'.
In any event, these simplifications are really application policy.
I have a feeling we're closer to agreement than we think here.
Here are some things I'm pretty sure are true:
1. Right now, in the absence of our config server, we need to be able
to inject a relatively large amount of data into the instance (I mean,
a few kb, anyway) for configuration to work right. My guess is that
over time we want to continue supporting that "we don't have a config
server" case, although I'm not at all sure about that.
2. Once we do have a config server, we really only need two very small
pieces of information: a UUID for the guest, and a DNS name or IP
address for the config server. (We'll leave out the possibility of
ipv6 zeroconf or DNS SRV records for now just to keep things under
control.) All the other config for an instance will come from the
config server. (Again somebody correct me if I'm wrong here.)
It seems to me this probably argues for two different things in the
API: an "inject a userblob" API that is generic, maybe the max
userblob size varies from cloud to cloud but the semantics are
identical, and we support it everywhere; and an "inject an identity"
API that we support whenever the underlying cloud supports it. I don't
see a major problem with Conductor understanding the difference
between these two cases, since it seems to me we'll need to know
whether the cloud we're talking to has a config server present and
therefore which kind of data we need to inject, *anyway*. (JRD correct
me if I'm wrong here.) So at a glance I'm not sure I have a problem
with having these two different APIs, since they seem to serve two
different purposes.
> This particular point is off topic for this conversation. If
you'd
> like to discuss further, let's take it offline.
Sure - but it's an important point in making sure Deltacloud remains a
versatile API.
> That's fine for the bits of data that Falcon needs for its IP-sniffing
> purposes. It's probably not enough for what we'd offer users through
the
> API.
>
> Again, the purpose of the falcon backend is to serve the needs of
> cloud engine. It's much more interesting, to me, and I suspect to
> Hugh's group, to make that work first. If, after that, we decide that
> there's value in making it available separate from cloud engine, and
> if we decide further that it makes sense to present DC as the front
> end for it
Isn't DC the abstraction that CE uses to talk to backend clouds ? Why
would CE talk to Falcon with anything else than DC ? At the very least,
it would force CE to do a lot more special casing than what we discussed
above for user-data.
Certainly Conductor wants to use DC API to talk to Falcon, I don't
think that's at issue. I guess the question is how much additional
logic do we need to add (if any) for the Falcon cloud case that isn't
already in the DC API. Frankly, I'm going to let you guys figure that
one out -- my standpoint is that I want to have to do as little work
as possible to support this.
I hope this hasn't muddied the waters more. Let me know,
Take care,
--H
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
== http://aeolusproject.org ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey
8:12 a.m.
On 03/15/11 - 05:05:24PM, David Lutterkort wrote:
On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > 3. We believe we can arrange to pass a small snippet of config data
> > > in at launch time, to point a booting instance at some source of
> > > further config.
> >
> > Kernel boot params, yes.
>
> Note that this won't work. In general, you can't add kernel boot params to
> a guest, because we don't direct-boot the kernel. We boot the emulated BIOS,
> which calls grub, which calls the kernel.
>
> Luckily there is a workaround. We can stuff some data into the SMBIOS tables
> during launch time, and use that as our injection mechanism instead.
There's also door #3, which would be hugely preferrable from an API POV:
using kpartx^W libguestfs to directly inject files into the image. This
is the approach Rackspace takes; IOW, we wouldn't introduce yet another
paradigm of how to inject user data into an instance.
Not to go too far off into the weeds, but I was under the impression that
Rackspace does it by passing kargs while booting (presumably they are doing
Xen PV direct-kernel boot, so this should be possible for them).
Regardless, I actually think that "crack open image, insert file" is a tactic
that works, but should only be used if there is no other way to inject data.
That is, I don't see why it would be preferable to modify the image rather than
use something like SMBIOS (as John points out elsewhere, we'll only need some
number of bytes to be passed along).
--
Chris Lalancette
1:18 p.m.
On Wed, 2011-03-16 at 09:12 -0400, Chris Lalancette wrote:
On 03/15/11 - 05:05:24PM, David Lutterkort wrote:
> On Tue, 2011-03-15 at 14:46 -0400, Chris Lalancette wrote:
> > On 03/15/11 - 01:35:53PM, Ian Main wrote:
> > > > 3. We believe we can arrange to pass a small snippet of config data
> > > > in at launch time, to point a booting instance at some source of
> > > > further config.
> > >
> > > Kernel boot params, yes.
> >
> > Note that this won't work. In general, you can't add kernel boot
params to
> > a guest, because we don't direct-boot the kernel. We boot the emulated
BIOS,
> > which calls grub, which calls the kernel.
> >
> > Luckily there is a workaround. We can stuff some data into the SMBIOS tables
> > during launch time, and use that as our injection mechanism instead.
>
> There's also door #3, which would be hugely preferrable from an API POV:
> using kpartx^W libguestfs to directly inject files into the image. This
> is the approach Rackspace takes; IOW, we wouldn't introduce yet another
> paradigm of how to inject user data into an instance.
Not to go too far off into the weeds, but I was under the impression that
Rackspace does it by passing kargs while booting (presumably they are doing
Xen PV direct-kernel boot, so this should be possible for them).
Nope - the API lets you provide 5 paths to files, and their contents;
each file can be up to 10k [1]. They get scribbled into the instance
(they actually used kpartx last time I looked)
Regardless, I actually think that "crack open image, insert
file" is a tactic
that works, but should only be used if there is no other way to inject data.
That is, I don't see why it would be preferable to modify the image rather than
use something like SMBIOS (as John points out elsewhere, we'll only need some
number of bytes to be passed along).
For Falcon-internal purposes, I don't really care what mechanism is
used; for user data injection exposed via the Deltacloud API, we should
stick with an existing approach. Scribbling into the instance before
boot seems much easier to implement than EC2's web server.
David
[1] http://www.rackspace.com/cloud/cloud_hosting_products/servers/api/
section 4.1.2 in the 'cloud developer guide'
6:57 p.m.
On Tue, 2011-03-15 at 13:35 -0400, Ian Main wrote:
On Tue, Mar 15, 2011 at 11:17:32AM -0400, John R. Dunning wrote:
> 5. Assuming I'm correct about 4, that server can also communicate the
> IP address of the booting instance back to the mother ship
> (conductor). So there's no need to build any other scheme for
> managing IP addresses.
So I may be wrong about this, but what I think I'm seeing here is that
the config server is NOT the same piece of software as the agent we are
making, and it is the agent which should know the IP. Generally the
flow of information is from deltacloud/condor to conductor, you are
talking about going around deltacloud here and injecting data directly
into conductor.
I guess I don't understand enough what is meant by the config server
here, and who controls it. For just a basic cloud, I would keep this
piece as simple as possible - whether people want to have instances talk
to a puppet server, cfengine or whatnot shouldn't be part of this cloud.
If we are really convinced that the only way we can get IP information
is by instrumenting images (I am not), we should keep this callback to
just that.
In that case, since the instance/IP map resulting from the callbacks is
internal to the cloud, there's no layer violation; the Deltacloud driver
will just consult the IP map when generating details about an instance.
That said I guess there's no reason we couldn't do the
registration
twice. Basically ec2 supports reporting its public IPs and I would like
to do that in this condor cloud as well. The config server thing and
reporting IPs directly to conductor is, as far as I understand, a
separate issue.
Absolutely agreed.
> 7. We will need to work out the admin details of standing up
the
> stateful server piece, presumably including the condor instance, the
> config server, and the interface piece. I believe David has
> volunteered to take a look at this piece.
I was not aware of that.
Neither was I ;) Suffice it to say that you are not duplicating existing
work.
> Do we need to do anything about mac address management? Is that
part
> of 1?
That will be up to the 'agent' that is between the deltacloud driver and
condor.
We have two options: (1) generate random MAC's whenever we start a VM or
(2) assign a known MAC when we start a VM. I'd be surprised if (1) would
fly in many environments. As such, there needs to be a way to configure
a MAC address range from which we pick.
David
8:01 a.m.
From: David Lutterkort <lutter(a)redhat.com>
Date: Tue, 15 Mar 2011 16:57:58 -0700
> 7. We will need to work out the admin details of standing up
the
> stateful server piece, presumably including the condor instance, the
> config server, and the interface piece. I believe David has
> volunteered to take a look at this piece.
I was not aware of that.
Neither was I ;) Suffice it to say that you are not duplicating existing
work.
Oops, I probably misspoke. I thought when you and I talked on the
phone, we said that you were going to think about what the components
would need to be for the stateful chunk of code, interfacing between
DC API and the buzzard^H^H^H^H^H^H^Hcondor backend. If you've already
got enough on your plate that's fine.
Perhaps it will make sense for Greg to look at that?
3:48 p.m.
On 03/14/2011 03:50 PM, Ian Main wrote:
On Fri, Mar 11, 2011 at 04:17:18PM -0500, John R. Dunning wrote:
> From: Ian Main<imain(a)redhat.com>
> Date: Fri, 11 Mar 2011 16:04:00 -0500
>
> [...]
>
> IP Discovery
> ------------
>
> There are many possible solutions to this problem but none of them are
> obviously better than any other. The three that seem the most promising
> are:
>
> 1) Using an agent in the VMs to register with a central server thereby
> letting us know its IP.
> 2) Having control of the DHCP server so we can see which leases are out
> for which MACs.
> 3) Using a config file which maps MACs to IPs and then configuring dhcpd
> with the same mapping.
>
> Of these I'm thinking we should implement #3 first.
>
> 3 is relatively easy. I would have thought 2 was easy, but I'm told
> that we can't count on it.
Well it is relatively easy for us to implement but it is not always easy
to set up. Not for developers at least which is our initial target
audience.
Also, presents a high barrier to entry for end users (admins), who may
not be allowed to run DHCP themselves or wall off a portion of addresses.
> My observation is that, immediately after you prove that you can
> launch an instance, you're going to need to care about configuring
> it. That means you need the config server, which is door number 1.
> So I advocate that we just bite the bullet and do that one right
> away.
>
> I think this should
> get us up and running quickly and it can be configured in almost any
> environment. Second option would be #1. I think in the end we will
> need to implement all of these and have it be configurable by the admin.
>
> You always need 1, regardless of what other things you do.
Yes and we will get there but my understanding is that it is still a ways
off in the grand scheme of things.. at least longer than I want to wait.
It's also my understanding that this would integrate with Aeolus and not
necessarily allow a condor cloud to stand on its own (by giving the user
of condor cloud only the IP of the instance).
>
> Other tidbit that I didn't see: We have the ability to inject some
> small snippet of data at launch time, correct? For instance as a karg
> or something? I'm pretty sure we talked about this offline, but I
> want to make sure (a) the answer is yes, and (b) we build it into the
> api.
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
The other thing I noticed is that there is no way to override the
architecture via condor. Eg run an i686 guest on x86_64 hardware.
Ian
The last hop on the user -> guest path through Condor hits libvirt. So
anything you can do with libvirt you can do here. If you can pass kargs
via libvirt domain XML you're ok. Same for arch etc.
Best,
matt
2:33 p.m.
On Mar 14, 2011, at 2:50 PM, Ian Main wrote:
On Fri, Mar 11, 2011 at 04:17:18PM -0500, John R. Dunning wrote:
>
> Other tidbit that I didn't see: We have the ability to inject some
> small snippet of data at launch time, correct? For instance as a karg
> or something? I'm pretty sure we talked about this offline, but I
> want to make sure (a) the answer is yes, and (b) we build it into the
> api.
Actually I was just looking through the code and AFAICT this is only
supported for Xen based VMs, not KVM. It would be easy enough to add to
condor though.. but then we are getting into needing a modified condor
again. Perhaps this line of development would be useful to upstream
though?
Does libvirt provide a mechanism to do this? We'd be interested in adding this to
Condor.
The other thing I noticed is that there is no way to override the
architecture via condor. Eg run an i686 guest on x86_64 hardware.
You should be able to do this in your Condor submit file:
requirements = Arch =!= undefined
But does it ever make sense for Condor to restrict the possible execute machines to ones
with matching architecture? I think I should take that behavior out of Condor.
+--------------------------------+-----------------------------------+
| Jaime Frey | I used to be a heavy gambler. |
| jfrey(a)cs.wisc.edu | But now I just make mental bets. |
| | That's how I lost my mind. |
+--------------------------------+-----------------------------------+
2:53 p.m.
On Tue, Mar 15, 2011 at 02:33:07PM -0500, Jaime Frey wrote:
On Mar 14, 2011, at 2:50 PM, Ian Main wrote:
> On Fri, Mar 11, 2011 at 04:17:18PM -0500, John R. Dunning wrote:
>>
>
>> Other tidbit that I didn't see: We have the ability to inject some
>> small snippet of data at launch time, correct? For instance as a karg
>> or something? I'm pretty sure we talked about this offline, but I
>> want to make sure (a) the answer is yes, and (b) we build it into the
>> api.
>
> Actually I was just looking through the code and AFAICT this is only
> supported for Xen based VMs, not KVM. It would be easy enough to add to
> condor though.. but then we are getting into needing a modified condor
> again. Perhaps this line of development would be useful to upstream
> though?
Does libvirt provide a mechanism to do this? We'd be interested in adding this to
Condor.
It does, but I've been corrected since.. you can basically pass in
whatever libvirt/virsh XML you want so it's not necessary that we do it
via a classad variable. At least there is a workaround by doing it this
way. As Chris mentioned in a separate email you want to do this via
SMBIOS extensions. You can see them here:
http://libvirt.org/formatdomain.html#elementsSysinfo
> The other thing I noticed is that there is no way to override
the
> architecture via condor. Eg run an i686 guest on x86_64 hardware.
You should be able to do this in your Condor submit file:
requirements = Arch =!= undefined
But does it ever make sense for Condor to restrict the possible execute machines to ones
with matching architecture? I think I should take that behavior out of Condor.
Not sure I follow what you are saying, but the issue is that it will
restrict the architecture of the machines but it's generally perfectly
valid to start an i386 guest up in an x86_64 host, not the other way
around though.
Ian
3:02 p.m.
On Mar 15, 2011, at 2:53 PM, Ian Main wrote:
On Tue, Mar 15, 2011 at 02:33:07PM -0500, Jaime Frey wrote:
> On Mar 14, 2011, at 2:50 PM, Ian Main wrote:
>
>> On Fri, Mar 11, 2011 at 04:17:18PM -0500, John R. Dunning wrote:
>>>
>>
>>> Other tidbit that I didn't see: We have the ability to inject some
>>> small snippet of data at launch time, correct? For instance as a karg
>>> or something? I'm pretty sure we talked about this offline, but I
>>> want to make sure (a) the answer is yes, and (b) we build it into the
>>> api.
>>
>> Actually I was just looking through the code and AFAICT this is only
>> supported for Xen based VMs, not KVM. It would be easy enough to add to
>> condor though.. but then we are getting into needing a modified condor
>> again. Perhaps this line of development would be useful to upstream
>> though?
>
> Does libvirt provide a mechanism to do this? We'd be interested in adding this to
Condor.
It does, but I've been corrected since.. you can basically pass in
whatever libvirt/virsh XML you want so it's not necessary that we do it
via a classad variable. At least there is a workaround by doing it this
way. As Chris mentioned in a separate email you want to do this via
SMBIOS extensions. You can see them here:
http://libvirt.org/formatdomain.html#elementsSysinfo
If there's a standard way that most people use to introduce instance-specific data, we
might want to introduce an easy way to use it in the Condor submit file. It sounds like
that's not the case here.
>> The other thing I noticed is that there is no way to override
the
>> architecture via condor. Eg run an i686 guest on x86_64 hardware.
>
>
> You should be able to do this in your Condor submit file:
>
> requirements = Arch =!= undefined
>
> But does it ever make sense for Condor to restrict the possible execute machines to
ones with matching architecture? I think I should take that behavior out of Condor.
Not sure I follow what you are saying, but the issue is that it will
restrict the architecture of the machines but it's generally perfectly
valid to start an i386 guest up in an x86_64 host, not the other way
around though.
I may be confused. Are you talking about an architecture restriction in what execute
machine Condor picks for a vm job, or the architecture configured in the vm xml
description given to libvirt?
+--------------------------------+-----------------------------------+
| Jaime Frey | I used to be a heavy gambler. |
| jfrey(a)cs.wisc.edu | But now I just make mental bets. |
| | That's how I lost my mind. |
+--------------------------------+-----------------------------------+
3:34 p.m.
On Tue, Mar 15, 2011 at 03:02:06PM -0500, Jaime Frey wrote:
On Mar 15, 2011, at 2:53 PM, Ian Main wrote:
> On Tue, Mar 15, 2011 at 02:33:07PM -0500, Jaime Frey wrote:
[snip]
> Not sure I follow what you are saying, but the issue is that it
will
> restrict the architecture of the machines but it's generally perfectly
> valid to start an i386 guest up in an x86_64 host, not the other way
> around though.
I may be confused. Are you talking about an architecture restriction in what execute
machine Condor picks for a vm job, or the architecture configured in the vm xml
description given to libvirt?
hehe, we're confusing each other.. I'm talking about the architecture
configured in the vm xml description given to libvirt.
Ian
3:47 p.m.
On Tue, 2011-03-15 at 15:53 -0400, Ian Main wrote:
> > Actually I was just looking through the code and AFAICT
this is
only
> > supported for Xen based VMs, not KVM.
Not true, it supports both Xen and KVM. If you have questions feel free
to poke me on #grid or freenode #distcomp
Cheers,
Tim
5:07 a.m.
On Mar 11, 2011, at 10:04 PM, Ian Main wrote:
Condor Cloud for Aeolus
-----------------------
I wrote up a quick summary of the current issues/ideas around this
project so we can get things rolling. If you don't want to read
the whole document please at least look at the section you are
interested in. :) I CC'd a few condor guys as well.
Goal
----
The idea here is to create a simple cloud provider driven through
deltacloud that we can use in Aeolus. The first iteration will be
extremely simple indeed and basically just make use of the facilities
offered by condor with some glue to make things work.
- VMs will use KVM via Condor.
- Condor will keep track of all instances for us and offer scalability.
- Instances will be stateless. Condor will copy VM images before running
them and NOT save state when done.
- Simple mechanism for uploading images to machine running condor.
- Deltacloud driver to interface directly with condor or through simple
agent (to handle MAC/IP mappings, authentication etc).
Image Wharehouse Integration
----------------------------
So far the general consensus is to use something simple like NFS or scp
to copy images to the Condor central manager server. This will consist
of an upload directory and then a staging directory. While uploading
files go into the uploading directory and then are moved to staging once
upload is complete. This prevents trying to start a half-copied image,
although that is generally not possible from the aeolus UI.
Only question here is that we may need some info on what OS this is
targeting so we can do the drive mapping correctly etc. The iwhd guys
may know more issues as well.
Deltacloud Driver
-----------------
The deltacloud driver can interact directly with condor commands
to start, stop, and query the state of instances running in condor.
Ruby code to do everything but the query is already in conductor, but
generally we use condor_submit to start a new job/instance, condor_q -xml
to get the state of jobs on the system (parsing xml output), and
condor_rm to stop an instance.
In Deltacloud API I *need* to have this collections defined:
* realms
I though it should be Condor location (or in other terms machine running Condor).
For start it can be something like 'default' and just report state (AVAILABLE,
UNAVAILABLE)
* hardware_profiles
Dunno how we get this informations. We can get it from Libvirt. Like:
- maximum amount of memory which can be used for VM
- maximum number of CPU cores available
- storage?
* images
Image Warehouse here?
- There informations are required:
- id and name (could be the same)
- description?
- state (UP, PENDING/BUILDING...)
- owner_id (!)
* instances
- These informations are required:
- state
- id and name (could be the same)
- image_id, realm_id, hardware_profile_id
- public_addresses (MAC->IP translation here)
- authentication (password/SSh keys? for Linux, administrator password for Windows)
Also there are some optional collections like storage_snapshots and storage_volumes.
Now, the difficulty comes in when we start looking at doing things
like
mapping MACs to IPs, authentication etc. If we do the condor commands
directly in the driver then the driver would have to handle these tasks
as well. I'm not sure if that is desirable.
Let me do some 'brainstorming' here. My idea is to have something middle, between
DC API and Condor. Something that will do:
POST /condor -> Pass command to Condor using condor_q -xml.
Reason for this is to have more 'clean' Ruby code in driver. I'm a little
bit aware about
doing `condor_q -xml ...` in every method or some client library. Just because that
driver
or client lib will look terribly messy.
Another point is that with this 'thing' you don't need to have Condor on the
same machine
as Deltacloud API is and DC API will not need to store anything driver specific.
Implementation of this call in Ruby could be a 10 lines of Sinatra code like this:
post '/condor' do
content_type 'application/xml'
`condor_q -xml #{params[:q]}`
end
This 'middleware' can also do IP translations for driver (simple key->value DB)
and
also image registration and other things needed besides driver.
The other idea is to put some kind of agent in between the driver
and
condor which can handle MAC/IP translation, authentication, perhaps
image registration etc. The benefit of this is that then we can work
on adding features to the 'cloud' without having to change the deltacloud
driver itself as often. I'm starting to think this is the way to go
even tho it may be slightly more work up front.
I also looked into having the IP address set in the job via condor but
I do not see a clean/easy way to do this.
IP Discovery
------------
There are many possible solutions to this problem but none of them are
obviously better than any other. The three that seem the most promising
are:
1) Using an agent in the VMs to register with a central server thereby
letting us know its IP.
RHEV-M is doing this but I don't like this idea at all. Just because it complicate
things, you need to install something in templates and this agent need to know where
IP translation app resides (to report IP).
2) Having control of the DHCP server so we can see which leases are
out
for which MACs.
3) Using a config file which maps MACs to IPs and then configuring
dhcpd
with the same mapping.
Of these I'm thinking we should implement #3 first. I think this should
get us up and running quickly and it can be configured in almost any
environment. Second option would be #1. I think in the end we will
need to implement all of these and have it be configurable by the admin.
Using the config file to map MACs to IPs means having to keep track of
which macs are in use. We can do this easily by querying the jobs which
contain this information. However this lookup would have to be done before
the job is submitted to condor and so this duty would fall on either
the deltacloud driver itself or the condor cloud agent.
Installation/Configuration
--------------------------
I'm hoping I can get some help on using a puppet script to ease installation
similar to what we are doing with Aeolus Conductor now. While I don't have
all the details yet, in general we will need it to:
- Configure condor.
- Set up deltacloud correctly with the right driver etc.
- Set up a network bridge for VMs.
- Set up the MAC/IP address table for admins to edit.
Other Docs
----------
Please also see Mathews document on Condor as a cloud provider:
http://spinningmatt.files.wordpress.com/2010/04/matthewfarrelleeopensourc...
Summary
-------
I think I've touched on the major issues/talking points here. Hopefully we
can come up with the best solutions quickly and get to writing the parts
soon.
Really the only parts that are still up in the air are the IP/MAC mapping
(possibly tho I'm fine with the config file thing for now), and how to
architect the driver - eg whether we need an agent in between the driver
and condor. I think the rest we can work out as we go.
Ian
_______________________________________________
aeolus-devel mailing list
aeolus-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/aeolus-devel
------------------------------------------------------
Michal Fojtik, mfojtik(a)redhat.com
Deltacloud API: http://deltacloud.org
2:57 p.m.
On Mon, Mar 14, 2011 at 11:07:53AM +0100, Michal Fojtik wrote:
On Mar 11, 2011, at 10:04 PM, Ian Main wrote:
>
> Condor Cloud for Aeolus
> -----------------------
>
> I wrote up a quick summary of the current issues/ideas around this
> project so we can get things rolling. If you don't want to read
> the whole document please at least look at the section you are
> interested in. :) I CC'd a few condor guys as well.
>
> Goal
> ----
>
> The idea here is to create a simple cloud provider driven through
> deltacloud that we can use in Aeolus. The first iteration will be
> extremely simple indeed and basically just make use of the facilities
> offered by condor with some glue to make things work.
>
> - VMs will use KVM via Condor.
> - Condor will keep track of all instances for us and offer scalability.
> - Instances will be stateless. Condor will copy VM images before running
> them and NOT save state when done.
> - Simple mechanism for uploading images to machine running condor.
> - Deltacloud driver to interface directly with condor or through simple
> agent (to handle MAC/IP mappings, authentication etc).
>
>
> Image Wharehouse Integration
> ----------------------------
>
> So far the general consensus is to use something simple like NFS or scp
> to copy images to the Condor central manager server. This will consist
> of an upload directory and then a staging directory. While uploading
> files go into the uploading directory and then are moved to staging once
> upload is complete. This prevents trying to start a half-copied image,
> although that is generally not possible from the aeolus UI.
>
> Only question here is that we may need some info on what OS this is
> targeting so we can do the drive mapping correctly etc. The iwhd guys
> may know more issues as well.
>
>
> Deltacloud Driver
> -----------------
>
> The deltacloud driver can interact directly with condor commands
> to start, stop, and query the state of instances running in condor.
> Ruby code to do everything but the query is already in conductor, but
> generally we use condor_submit to start a new job/instance, condor_q -xml
> to get the state of jobs on the system (parsing xml output), and
> condor_rm to stop an instance.
In Deltacloud API I *need* to have this collections defined:
* realms
I though it should be Condor location (or in other terms machine running Condor).
For start it can be something like 'default' and just report state (AVAILABLE,
UNAVAILABLE)
I think for now we just stick with giving the whole cloud one realm
name? I suspect that is how it will go anyway. If you wanted another
installation someplace else you could install another condor and change
that realm name and conductor could figure out what to do there.
* hardware_profiles
Dunno how we get this informations. We can get it from Libvirt. Like:
- maximum amount of memory which can be used for VM
- maximum number of CPU cores available
- storage?
I think we're going to have to just define some reasonable sizes. There
is a condor config for setting the maximums though. Probably a
configuration option again eventually.
* images
Image Warehouse here?
- There informations are required:
- id and name (could be the same)
- description?
- state (UP, PENDING/BUILDING...)
- owner_id (!)
Yes we will have image wharehouse. I can get you all the other stuff as
well I think. Is owner id really important? This looks like we'd need
a bit of metadata to go along with our images.
* instances
- These informations are required:
- state
- id and name (could be the same)
- image_id, realm_id, hardware_profile_id
- public_addresses (MAC->IP translation here)
- authentication (password/SSh keys? for Linux, administrator password for Windows)
Yes, I can get you all that stuff given an intermediate agent.
Everything is available through condor except IP and authentication.
Also there are some optional collections like storage_snapshots and
storage_volumes.
Don't think we need those for now.
> Now, the difficulty comes in when we start looking at doing
things like
> mapping MACs to IPs, authentication etc. If we do the condor commands
> directly in the driver then the driver would have to handle these tasks
> as well. I'm not sure if that is desirable.
Let me do some 'brainstorming' here. My idea is to have something middle,
between
DC API and Condor. Something that will do:
POST /condor -> Pass command to Condor using condor_q -xml.
Reason for this is to have more 'clean' Ruby code in driver. I'm a little
bit aware about
doing `condor_q -xml ...` in every method or some client library. Just because that
driver
or client lib will look terribly messy.
Another point is that with this 'thing' you don't need to have Condor on
the same machine
as Deltacloud API is and DC API will not need to store anything driver specific.
Implementation of this call in Ruby could be a 10 lines of Sinatra code like this:
post '/condor' do
content_type 'application/xml'
`condor_q -xml #{params[:q]}`
end
This 'middleware' can also do IP translations for driver (simple key->value
DB) and
also image registration and other things needed besides driver.
Yes, this is my thinking now too. The best part about this is that it
lets me develop on the condor cloud without having to mess with
the libdeltacloud driver.
Let us pursue this avenue then. This is probably the first piece to get
working on so lets get together and see how we can do this.
Ian
5:30 a.m.
On Mar 14, 2011, at 8:57 PM, Ian Main wrote:
On Mon, Mar 14, 2011 at 11:07:53AM +0100, Michal Fojtik wrote:
> On Mar 11, 2011, at 10:04 PM, Ian Main wrote:
>
>>
>> Condor Cloud for Aeolus
>> -----------------------
>>
>> I wrote up a quick summary of the current issues/ideas around this
>> project so we can get things rolling. If you don't want to read
>> the whole document please at least look at the section you are
>> interested in. :) I CC'd a few condor guys as well.
>>
>> Goal
>> ----
>>
>> The idea here is to create a simple cloud provider driven through
>> deltacloud that we can use in Aeolus. The first iteration will be
>> extremely simple indeed and basically just make use of the facilities
>> offered by condor with some glue to make things work.
>>
>> - VMs will use KVM via Condor.
>> - Condor will keep track of all instances for us and offer scalability.
>> - Instances will be stateless. Condor will copy VM images before running
>> them and NOT save state when done.
>> - Simple mechanism for uploading images to machine running condor.
>> - Deltacloud driver to interface directly with condor or through simple
>> agent (to handle MAC/IP mappings, authentication etc).
>>
>>
>> Image Wharehouse Integration
>> ----------------------------
>>
>> So far the general consensus is to use something simple like NFS or scp
>> to copy images to the Condor central manager server. This will consist
>> of an upload directory and then a staging directory. While uploading
>> files go into the uploading directory and then are moved to staging once
>> upload is complete. This prevents trying to start a half-copied image,
>> although that is generally not possible from the aeolus UI.
>>
>> Only question here is that we may need some info on what OS this is
>> targeting so we can do the drive mapping correctly etc. The iwhd guys
>> may know more issues as well.
>>
>>
>> Deltacloud Driver
>> -----------------
>>
>> The deltacloud driver can interact directly with condor commands
>> to start, stop, and query the state of instances running in condor.
>> Ruby code to do everything but the query is already in conductor, but
>> generally we use condor_submit to start a new job/instance, condor_q -xml
>> to get the state of jobs on the system (parsing xml output), and
>> condor_rm to stop an instance.
>
> In Deltacloud API I *need* to have this collections defined:
>
> * realms
> I though it should be Condor location (or in other terms machine running Condor).
> For start it can be something like 'default' and just report state
(AVAILABLE, UNAVAILABLE)
I think for now we just stick with giving the whole cloud one realm
name? I suspect that is how it will go anyway. If you wanted another
installation someplace else you could install another condor and change
that realm name and conductor could figure out what to do there.
Yes this make sense to me, we can keep it like 'default' or just use some
hostname
where Condor is reside.
> * hardware_profiles
> Dunno how we get this informations. We can get it from Libvirt. Like:
> - maximum amount of memory which can be used for VM
> - maximum number of CPU cores available
> - storage?
I think we're going to have to just define some reasonable sizes. There
is a condor config for setting the maximums though. Probably a
configuration option again eventually.
I played with libvirt Ruby gem recently and I found that you can easily get
this sizes (limits) via libvirt API:
<snip>
result = Nokogiri::XML(lvirt.capabilities)
profiles <<
HardwareProfile.new((result/'/capabilities/host/cpu/model').text) do
architecture lvirt.node_get_info.model
cpu (1..lvirt.node_get_info.cpus.to_i)
memory (1..lvirt.node_get_info.memory.to_i)
storage 1000
end
</snip>
Magic thing there is 'capabilities', which produce nice XML from where we can get
all this informations.
It would be nice to pass this to Condor driver, so client will know how much memory
left from 'pool' in hypervisor.
> * images
> Image Warehouse here?
> - There informations are required:
> - id and name (could be the same)
> - description?
> - state (UP, PENDING/BUILDING...)
> - owner_id (!)
Yes we will have image wharehouse. I can get you all the other stuff as
well I think. Is owner id really important? This looks like we'd need
a bit of metadata to go along with our images.
Owner: I realized that in Condor/libvirt is not really important because I think
we will not mess with user authentication anyway. So we can set it to 'root'
IMHO.
Metadata: Example representation of Deltacloud API Image model looks like:
<image href="http://localhost:3001/api/images/img1" id="img1">
<name>Fedora 10</name>
<owner_id>fedoraproject</owner_id>
<description>Fedora 10</description>
<architecture>x86_64</architecture>
<state>AVAILABLE</state>
<actions>
<link href="http://localhost:3001/api/instances;image_id=img1"
method="post" rel="create_instance"/>
</actions>
</image>
> * instances
> - These informations are required:
> - state
> - id and name (could be the same)
> - image_id, realm_id, hardware_profile_id
> - public_addresses (MAC->IP translation here)
> - authentication (password/SSh keys? for Linux, administrator password for
Windows)
Yes, I can get you all that stuff given an intermediate agent.
Everything is available through condor except IP and authentication.
I bet that ImageWarehouse can produce images with built-in SSH key authentication.
Maybe we can do authentication in the 'middle-layer' app as well as IP
management.
This application will have pool of MAC->IP and we can also associate some SSH key
with IP. So ImageWarehouse can connect to this application and fetch SSH key and then
bundle it to Image. (I assume that ImageWarehouse will get MAC address for final VM)
-- Michal
------------------------------------------------------
Michal Fojtik, mfojtik(a)redhat.com
Deltacloud API: http://deltacloud.org
8:58 a.m.
On 03/15/2011 06:30 AM, Michal Fojtik wrote:
On Mar 14, 2011, at 8:57 PM, Ian Main wrote:
> On Mon, Mar 14, 2011 at 11:07:53AM +0100, Michal Fojtik wrote:
>>
>> In Deltacloud API I *need* to have this collections defined:
>>
>> * realms
>> I though it should be Condor location (or in other terms machine running
Condor).
>> For start it can be something like 'default' and just report state
(AVAILABLE, UNAVAILABLE)
> I think for now we just stick with giving the whole cloud one realm
> name? I suspect that is how it will go anyway. If you wanted another
> installation someplace else you could install another condor and change
> that realm name and conductor could figure out what to do there.
Yes this make sense to me, we can keep it like 'default' or just use some
hostname
where Condor is reside.
I think we should keep it as 'default' -- there's no need to add the
hostname that Condor is on, since separate condor instances would be
separate providers anyway, so you don't need a different realm name to
distinguish them. Mapping to front end (conductor) realms can be done at
the provider level in addition to at the individual realm level, so
simply making it 'default' in the driver for all condor provider
instances would be sufficient here.
Scott
4850
days inactive
4856
days old
aeolus-devel@lists.fedorahosted.org
32 comments
10 participants
participants (10)
-
Chris Lalancette -
David Lutterkort -
Hugh O. Brock -
Ian Main -
Jaime Frey -
jrd＠redhat.com -
Matthew Farrellee -
Michal Fojtik -
Scott Seago -
Tim St. Clair