I'm currently working on getting file injection into the RimuHosting API. After that I will work on some patches for the deltacloud api to add this functionality in. -Ivan On Tue, Dec 8, 2009 at 1:46 PM, Michael Neale <michael.neale(a)gmail.com&gt; wrote:

On Wed, Dec 16, 2009 at 1:34 PM, David Lutterkort <lutter(a)redhat.com&gt; wrote: Yes you would be able to do that

On Tue, Dec 8, 2009 at 10:06 AM, Bob McWhirter <bmcwhirt(a)redhat.com&gt; wrote: What about using ace/puppet or something similar for this kind of configuration? Then we don't have to depend on the provider of the cloud, just that we have some basic components installed on our instance(s). Rightscale has a similar concept to this with their server templates and 'rightScripts', and we did the same thing in the oVirt project to configure/setup an 'appliance' server. -- -j

On Wed, Dec 9, 2009 at 7:12 AM, Adrian Cole <adrian(a)jclouds.org&gt; wrote: I guess it depends on what people are happy having deltacloud do/be. Is it acceptable to require features that are not (yet) available be "emulated" via other means ? I think Thorsten's comments make sense: you need some way to give the server enough identity to boot strap from there, that is often what it comes down to, so perhaps it isn't about arbitrary file injection but about identity/security so it can then run bootstrap scripts etc. -- Michael D Neale home: www.michaelneale.net blog: michaelneale.blogspot.com

On Thu, 2010-01-14 at 13:15 +0300, Roman wrote: Nice, I see you even have a patch for supporting user data on that post. Before committing that, I'd like to see if we can expand that mechanism to other clouds than just EC2. I agree with that - I believe people are in general better off if they use some central cloud-independent service for configuring their instances. Of course, we still need to provide them with enough of a hook to bootstrap that process. Not attached, but I saw it on your blog ;) I think longer term, we'd also want some sort of cloud-compat package that people can install inside their instances so that you can get user data through the exact same mechanism, regardless of cloud. I want to get some clarity on how we can simulate that in other clouds, but I think your patch will be the way to go as a generic mechanism. David

On Sun, 2010-01-17 at 09:14 +1100, Michael Neale wrote: mechanism the backend cloud supports. Anything else, for example, simulating EC2's user-data mechanism on rackspace by injecting a fixed file will lead to a lowest-common denominator API. What I am thinking is that we indicate such variations in the toplevel api.xml. For example, instead of just saying <link href="http://localhost:3000/api/instances" rel="instances"/> for EC2 we'd say <link href="http://localhost:3000/api/instances" rel="instances"> <operation name="create"> <user-data> <maxsize unit="kB">16</maxsize> </user-data> </operation> </link> and for Rack we'd say <link href="http://localhost:3000/api/instances" rel="instances"> <operation name="create"> <user-files max="5"> <maxsize="10" unit="kB">10</maxsize> <path-length unit="byte">255</path-length> </user-files> </operation> </link> That way, applications can discover which mechanism for file injection the cloud supports, and act accordingly. David

On 01/19/2010 09:20 PM, Michael Neale wrote: Masking irrelevant differences is a key goal for the API, enabling developers to write once and interact with many clouds. At the same time, exposing differentiating behaviour, or methods which cannot be masked via capability discovery is also important. File injection is perhaps somewhere in the middle. b

On Wed, 2010-01-20 at 13:20 +1100, Michael Neale wrote: I think you're disappointed with the messenger ;) To stick with file/user-data injection: today, only EC2 and Rackspace give you some capability to do that as far as I can tell. EC2 does the user-data thing that you can access via HTTP inside the image, Rackspace lets you write up to five small files anywhere in the file system. The only way in which we could make that difference disappear is to simulate EC2's user-data on Rackspace by putting user-data into a fixed file in the filesystem - that's unsatisfactory for people who want to use deltacloud only for Rackspace, since we've removed a feature. Both mechanisms are reasonable, and I would hope that other cloud providers will follow one of these instead of inventing yet another one. That in itself means that you can port applications from one cloud to another as long as they support the same injection mechanism - deltacloud will take care of minor differences of how exactly you inject that data. If you don't want to bother with discovery, ensure in some other way that all the clouds you're targeting support the features you need, e.g. restrict yourself to a subset of drivers. To get portability to a wider range of clouds, an application needs to be aware to some extent of the differences between clouds - there's no way we can pretend in deltacloud that GoGrid (or most other clouds) support any sort of data injection. To help those, more complex, applications, deltacloud should tell them about features that only some of the drivers support. So, I see adding capabilities metadata and allowing discovery not as a restriction of what people can do, but a help for people who want or need to do more complicated things. David

I beleive there is a consensus over the fact that the data injection feature is a must (or at least very useful) for deltacloud. Please correct me if I'm wrong. These are my comments. 2010/1/20 David Lutterkort <lutter(a)redhat.com&gt;: Deltacloud is a provider-independent API. if somebody chooses it, he will expect it to be closer to a lowest-common denominator API. Not all features of underlying providers will be supported. In exchange there is no need to make multiple cloud management implementations. If deltacloud API comes with optional extension features, which are not supported across all providers, such features should be clearly marked in API and documentation as an extension so that developers can avoid using them if they want to stay provider-independent. I think that the data injection feature discussed in this thread is a basic one thus it should not be implemented as a kind of discoverable extension (if possible). In worst case, make the data injection feature an extension, but do not offer different ways of injecting data in deltacloud API. --------------------------------------------- 2010/1/15 David Lutterkort <lutter(a)redhat.com&gt;: I think this would be a good way to go. Wonder if there any other way for deltacloud API to stay cloud provider-independent? --------------------------------------------- 2010/1/17 Michael Neale <michael.neale(a)gmail.com&gt;: I think that single file enough, and the rackspace way of injecting multiple files looks equally good. Indeed, some developers will probably prefer a richer-style rackspace way of injecting multiple files. In API, can we agree that the additional instance model field (named client_data in my patch) should contain an array of File objects? Each file can have name and content properties. Let's put no any hard limits on the number of files. Let's put limits on a total file size. I think that it would be better for implementation to pass a single file to the instance and let the cloud-compat package to expand it on first instance startup into multiple files specified in deltacloud create instance API call. Would this way be sufficiently good to go? Probably it worth to start from trivial single-file injection and then extend it to injecting multiple files mixed with bootstrap scripts, etc. --------------------------------------------- 2010/1/20 David Lutterkort <lutter(a)redhat.com&gt;: I agree that for other providers it might be missing, still it is a useful feature. It will make a sense. If my application needs data injection and it is supported only by a subset of drivers, I will limit it to the subset of drivers, and that would be absolutely OK for me. I can think about some, a kind of last resort, workaround ways for achieving data injection for providers which do not support data injection. E.g., something along EC2-style. In certain conditions it is possible to set up a service for distributing injected data to the instances. Upon instance boot the deltacloud-compat will contact the service over the internet and the service will retrieve the injected data from deltacloud based on instance IP address and push it to the instance. Downsides are security issues and the need to set up a corresponding network infrastructure.

On Thu, Jan 21, 2010 at 5:45 AM, David Lutterkort <lutter(a)redhat.com&gt; wrote: Right, so perhaps I was getting things confused in my mind. So what you are suggesting is that some file injection feature is part of the api - but in an optional section. And the api would provide a way to query what in the optional section is available? What would be nice would be to pick either the Rackspace or the EC2 way of injection, and make that the "normal" way - and emulate it if really needed (in the driver if necessary) for the other (I think that would be possible to do the EC2 way in Rackspace without too much trouble). In time cloud providers would possibly provide it anyway. If its discovering 2 different ways to do file injection, I don't think I would like that (unless 1 type is a super set of the other type - if you follow what I mean).

On Fri, Jan 22, 2010 at 5:28 AM, David Lutterkort <lutter(a)redhat.com&gt; wrote: http://code.google.com/p/jclouds/wiki/ComputeGuide#Adding_a_post-boot_script This is a single file injection of a script. If not supported, fallback to SSH will be required so it can do it.

hmm, well, it could make sense to move the runscript option into the Template. We are currently guarding all changes to the template based on discoverable, or statically coded features of the cloud. With the runscript inside the template, we can check support on the image so that the API call doesn't fall flat on its face in an unsupported scenario ;) anyway, this is more fodder for the jclouds-dev group than deltacloud. I'll leave you to it. thanks for the feedback, David. Cheers, -Adrian On Thu, Jan 21, 2010 at 3:46 PM, David Lutterkort <lutter(a)redhat.com&gt; wrote:

It's nice to see discussion switches back to a simplier, more realistic way of doing things. Feature discovery is cool and not a problem itself. I affraid only that keeping both user-data and rackspace file-injection in API can lead to issues. These are essentially the same feature, it is just implemented by providers in different ways. Currently there are two different providers offering data injection, and they offer it in different ways. I just think, there are can be multiple other ways for other providers to choose. E.g. bootstrap scripts are often used. And who have said it is not possible to do data injection via provisioning systems? E.g. pass data via kickstart file sections. What about configuration systems? E.g. passing data as a puppet node definition. There can be other ways or variations as well. If every way of doing the same thing will be a separate extension to the API it will risk to become bloated with provider-specific features. Thus applications (and application developers) will still struggle heavily from provider dependencies and have to implement multiple ways of dong the same thing. I agree with Michael that viable options are a single feature or have a superset feature included in impl. 1) a single feature (data-injection) to be an extension. Or 2) EC2 to offer data-injection and user-data features + Rackspace to offer data-injection and file-injection features. Or 3) EC2 to offer data-injection + Rackspace to offer data-injection and file-injection features. To repeat myself, I strongly believe that deltacloud should offer a provider-independent way of doing things as a highest priority. There is nothing wrong with support for provider-specific ways, but, as long as they are backed with provider-independent ones. Roman blog: http://it-result.me

On Tue, 2009-12-08 at 11:46 +1100, Michael Neale wrote: What I am having trouble with for this is that it's not clear to me how to express that in a cross-cloud way. For clouds that do not support file injection (e.g., EC2), we'd need to make sure that the image is built in a certain way; should we carry additional image metadata to tell us which images have been prepped with a 'file injector' for EC2 ? Or should we assume that file injection will eventually be available on any cloud ? That heavily influences how we would express file injection on the API level. David