Hey, Apologies for another long email, but here's my proposal for how to proceed with the image building CLI etc. Cheers, Mark. = Terminology = - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2 region. - Target/Provider Type - the cloud type, irrespective of the specific instance e.g. RHEV-M vs EC2. - Provider Account - a set of user credentials for a specific provider. - Provider Image - these are images which have been pushed to, or imported from, a provider and are available for use by instances. - Target Image - these are images which have been built for a specific target and are available to be pushed to multiple providers. - Image Build - these are objects which track all target and provider images built from a template at a certain time, all of which should be equivalent to each other.

- Image - these are metadata objects which describe the purpose of the image, the parameters it takes, the set of available builds of the image and the latest build. = Overview = As described previously[1], the short/medium term plan is that Aeolus will: - Have well documented deployable and template description formats with plenty of examples. Deployable authors will create these descriptions manually or with the help of simple command line tools and store them as they fit (e.g. in their own git repo) - Support launching deployables in conductor simply by allowing the user to supply a URL to a deployable description, or choosing a URL from a list populated by an admin. - Include a set of command line tools for building images from a template description. This tools will also allow deployable authors to list available images available to be referenced in their deployables. - Store images and their related metadata in IWHD. Conductor will use IWHD queries to resolve deployables' image references and allow users to launch individual images directly. - Use image factory to build and import images, including the updating of IWHD metadata about the images.

= IHWD Metadata = The following IWHD buckets will be used, each one for a different object type: images: - object_type == "image" - object_body: XML document with image description and parameters, for deployable authors and simple instance launch from UI - uuid: UUID - latest_build: build UUID builds: - object_type == "build" - object_body: empty - uuid: UUID - image: UUID of the image object - parent: UUID of previous build

target_images: - object_type == "target_image" - object_body: target specific image, for upload builds; empty for snapshot builds - uuid: UUID - build: UUID of the build object - icicle: UUID of icicle object - template: UUID of template object - target: ec2, rhev-m, etc. - target_parameters: target specific data stashed here for the push stage

provider_images: - object_type == "provider_image" - object_body: empty - uuid: UUID - target_image: UUID of image - provider: e.g. ec2-us-east-1, rhev-m site - icicle == "none" - target_identifier: target specific ID for the image

templates: - object_type == "template" - object_body:<template/> document supplied for image build - uuid: UUID

icicles: - object_type == "icicle" - object_body:<icicle/> document - uuid: UUID This fairly closely reflects the metadata currently stored by image factory in IWHD, with the following changes: - the current "image" type is renamed to "target_type", along with the bucket and image reference on provider images[2] - new image and build object types and buckets are introduced - a build reference is added to the target image type = Image Factory APIs = Image factory currently supports building target images and pushing provider images, along with the appropriate updating of image warehouse metadata. In order to allow the CLI to fire-and-forget, we will need to add new build and push APIs which can handle multiple images at once. These would look like: - image(image_id, template, targets[]) + builds an image for the supplied targets + image_id should be ommitted, if a new image is required + template and targets may be ommitted if a previous build exists and the previous template and targets will be used - push(image_id, providers, credentials) + push the image the supplied providers + assumes factory can figure out which target image is appropriate for each provider + credentials argument is a<provider_credentials/> document

Also, we'll need to rename image factory's current concept of an image as a target image. Some initial work on that here[3]. = Conductor APIs = Conductor needs APIs to support the following: 1) List the available provider types in an XML doc 2) List the available providers in an XML doc 3) Dump a<provider_credentials/> document encapsulating all of the available provider accounts

The latter API would be subject to authentication and access control.

= Conductor IWHD Queries = Conductor has basically two use cases for querying IWHD: 1) As a user, I want to launch a deployable: 2) As a user, I want to launch an instance of an image For (1), conductor needs to resolve each deployable image reference to a set of provider images. This can be done with something like: $> $build_id = curl http://iwhd/$image_id/latest_build $> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query $> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done For (2), conductor needs to list all images available for launching a standalone instance and, when the user launches the image, it needs to list the parameters for the image. Both are easily achieved by querying the images bucket. = Image Building CLI = The image building CLI is used by Aeolus users to build and upload images from templates. It is also used by deployable authors to list the available images. The use cases are: 1) As an image builder or deployable author, I want to list all images 2) As an image builder or deployable author, I want to list all builds of an image 3) As an image builder or deployable author, I want to list all the targets and providers an image has been built for 4) As an image builder, I want to build an image 5) As an image builder, I want to push an image to a provider 6) As an image builder, I want to import an image 7) As an image builder, I want to delete an image 8) As an image builder, I want to delete old versions of an image 9) As an image builder, I want to delete a provider image 10) As an image builder, I want to delete an image The tool needs to interact with IWHD for listing, image factory for building and conductor for provider/account details. It might look like: $> aeolus-image images # list available images $> aeolus-image builds $image_id # list the builds of an image $> aeolus-image target-images $build_id # list the target images from a build $> aeolus-image provider-images $target_image_id # list the provider images from a target image $> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template $> aeolus-image build --image $image_id # rebuild the image template and targets from latest build $> aeolus-image build --target ec2 --target rackspace \ # --image $image_id \ # rebuild the image with a new template and set of targets --template my.tmpl # $> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider $> aeolus-image push $build_id # push all target images for a build, to same providers as previously $> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account $> aeolus-image push $image_id # push all the target images for the latest build $> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider $> aeolus-image delete --image $image_id # deletes all builds, target images and provider images $> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate $> aeolus-image delete --target-image $target_image # deletes a target image and its provider images $> aeolus-image delete --provider-image $provider_image # deletes a provider image $> aeolus-image targets # list the values available for the --target parameter $> aeolus-image providers # list the values available for the --provider parameter $> aeolus-image accounts # list the values available for the --account parameter Some other notes: - The tool will need to authenticate at least against conductor, so --user/--pass arguments will be needed. We may also want to support supplying these via environment variables and/or a dotfile. - The build/push/import commands are long running, so we should support displaying the progress of the builds and have a --background parameter for fire-and-forget. = Footnotes = [1] - https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001640.html [2] - to migrate existing data, you can do: $> cat> update.js<<EOF db.main.update({ _bucket : "images", object_type : "image" }, { \$set: { _bucket : "target_images", object_type : "target_image" } }, false, true) db.main.find({ _bucket : "provider_images", object_type : "provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image; db.main.save(pi); }) db.main.update({ _bucket : "provider_images", object_type : "provider_image" }, { \$unset: { image : 1 } }, false, true) EOF $> mongo --port 27018 repo update.js $> mv _fs/images _fs/provider_images This clearly only works for IWHD with the fs_autostart, but should be easy to adjust. [3] - https://github.com/markmc/image_factory/commits/master _______________________________________________ aeolus-devel mailing list aeolus-devel(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/aeolus-devel

Hey, Apologies for another long email, but here's my proposal for how to proceed with the image building CLI etc. Cheers, Mark. = Terminology = - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2 region. - Target/Provider Type - the cloud type, irrespective of the specific instance e.g. RHEV-M vs EC2. - Provider Account - a set of user credentials for a specific provider. - Provider Image - these are images which have been pushed to, or imported from, a provider and are available for use by instances. - Target Image - these are images which have been built for a specific target and are available to be pushed to multiple providers. - Image Build - these are objects which track all target and provider images built from a template at a certain time, all of which should be equivalent to each other. - Image - these are metadata objects which describe the purpose of the image, the parameters it takes, the set of available builds of the image and the latest build. = Overview = As described previously[1], the short/medium term plan is that Aeolus will: - Have well documented deployable and template description formats with plenty of examples. Deployable authors will create these descriptions manually or with the help of simple command line tools and store them as they fit (e.g. in their own git repo) - Support launching deployables in conductor simply by allowing the user to supply a URL to a deployable description, or choosing a URL from a list populated by an admin. - Include a set of command line tools for building images from a template description. This tools will also allow deployable authors to list available images available to be referenced in their deployables. - Store images and their related metadata in IWHD. Conductor will use IWHD queries to resolve deployables' image references and allow users to launch individual images directly. - Use image factory to build and import images, including the updating of IWHD metadata about the images. = IHWD Metadata = The following IWHD buckets will be used, each one for a different object type: images: - object_type == "image" - object_body: XML document with image description and parameters, for deployable authors and simple instance launch from UI - uuid: UUID - latest_build: build UUID builds: - object_type == "build" - object_body: empty - uuid: UUID - image: UUID of the image object - parent: UUID of previous build target_images: - object_type == "target_image" - object_body: target specific image, for upload builds; empty for snapshot builds - uuid: UUID - build: UUID of the build object - icicle: UUID of icicle object - template: UUID of template object - target: ec2, rhev-m, etc. - target_parameters: target specific data stashed here for the push stage provider_images: - object_type == "provider_image" - object_body: empty - uuid: UUID - target_image: UUID of image - provider: e.g. ec2-us-east-1, rhev-m site - icicle == "none" - target_identifier: target specific ID for the image templates: - object_type == "template" - object_body: <template/> document supplied for image build - uuid: UUID icicles: - object_type == "icicle" - object_body: <icicle/> document - uuid: UUID This fairly closely reflects the metadata currently stored by image factory in IWHD, with the following changes: - the current "image" type is renamed to "target_type", along with the bucket and image reference on provider images[2] - new image and build object types and buckets are introduced - a build reference is added to the target image type = Image Factory APIs = Image factory currently supports building target images and pushing provider images, along with the appropriate updating of image warehouse metadata. In order to allow the CLI to fire-and-forget, we will need to add new build and push APIs which can handle multiple images at once. These would look like: - image(image_id, template, targets[]) + builds an image for the supplied targets + image_id should be ommitted, if a new image is required + template and targets may be ommitted if a previous build exists and the previous template and targets will be used - push(image_id, providers, credentials) + push the image the supplied providers + assumes factory can figure out which target image is appropriate for each provider + credentials argument is a <provider_credentials/> document Also, we'll need to rename image factory's current concept of an image as a target image. Some initial work on that here[3]. = Conductor APIs = Conductor needs APIs to support the following: 1) List the available provider types in an XML doc 2) List the available providers in an XML doc 3) Dump a <provider_credentials/> document encapsulating all of the available provider accounts The latter API would be subject to authentication and access control. = Conductor IWHD Queries = Conductor has basically two use cases for querying IWHD: 1) As a user, I want to launch a deployable: 2) As a user, I want to launch an instance of an image For (1), conductor needs to resolve each deployable image reference to a set of provider images. This can be done with something like: $> $build_id = curl http://iwhd/$image_id/latest_build $> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query $> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done For (2), conductor needs to list all images available for launching a standalone instance and, when the user launches the image, it needs to list the parameters for the image. Both are easily achieved by querying the images bucket. = Image Building CLI = The image building CLI is used by Aeolus users to build and upload images from templates. It is also used by deployable authors to list the available images. The use cases are: 1) As an image builder or deployable author, I want to list all images 2) As an image builder or deployable author, I want to list all builds of an image 3) As an image builder or deployable author, I want to list all the targets and providers an image has been built for 4) As an image builder, I want to build an image 5) As an image builder, I want to push an image to a provider 6) As an image builder, I want to import an image 7) As an image builder, I want to delete an image 8) As an image builder, I want to delete old versions of an image 9) As an image builder, I want to delete a provider image 10) As an image builder, I want to delete an image The tool needs to interact with IWHD for listing, image factory for building and conductor for provider/account details. It might look like: $> aeolus-image images # list available images $> aeolus-image builds $image_id # list the builds of an image $> aeolus-image target-images $build_id # list the target images from a build $> aeolus-image provider-images $target_image_id # list the provider images from a target image $> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template $> aeolus-image build --image $image_id # rebuild the image template and targets from latest build $> aeolus-image build --target ec2 --target rackspace \ # --image $image_id \ # rebuild the image with a new template and set of targets --template my.tmpl # $> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider $> aeolus-image push $build_id # push all target images for a build, to same providers as previously $> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account $> aeolus-image push $image_id # push all the target images for the latest build $> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider $> aeolus-image delete --image $image_id # deletes all builds, target images and provider images $> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate $> aeolus-image delete --target-image $target_image # deletes a target image and its provider images $> aeolus-image delete --provider-image $provider_image # deletes a provider image $> aeolus-image targets # list the values available for the --target parameter $> aeolus-image providers # list the values available for the --provider parameter $> aeolus-image accounts # list the values available for the --account parameter Some other notes: - The tool will need to authenticate at least against conductor, so --user/--pass arguments will be needed. We may also want to support supplying these via environment variables and/or a dotfile. - The build/push/import commands are long running, so we should support displaying the progress of the builds and have a --background parameter for fire-and-forget. = Footnotes = [1] - https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001640.html [2] - to migrate existing data, you can do: $> cat > update.js <<EOF db.main.update({ _bucket : "images", object_type : "image" }, { \$set: { _bucket : "target_images", object_type : "target_image" } }, false, true) db.main.find({ _bucket : "provider_images", object_type : "provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image; db.main.save(pi); }) db.main.update({ _bucket : "provider_images", object_type : "provider_image" }, { \$unset: { image : 1 } }, false, true) EOF $> mongo --port 27018 repo update.js $> mv _fs/images _fs/provider_images This clearly only works for IWHD with the fs_autostart, but should be easy to adjust. [3] - https://github.com/markmc/image_factory/commits/master _______________________________________________ aeolus-devel mailing list aeolus-devel(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/aeolus-devel

Hey, Apologies for another long email, but here's my proposal for how to proceed with the image building CLI etc. Cheers, Mark. = Terminology = - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2 region.

- Target/Provider Type - the cloud type, irrespective of the specific instance e.g. RHEV-M vs EC2. - Provider Account - a set of user credentials for a specific provider. - Provider Image - these are images which have been pushed to, or imported from, a provider and are available for use by instances. - Target Image - these are images which have been built for a specific target and are available to be pushed to multiple providers. - Image Build - these are objects which track all target and provider images built from a template at a certain time, all of which should be equivalent to each other. - Image - these are metadata objects which describe the purpose of the image, the parameters it takes, the set of available builds of the image and the latest build. = Overview = As described previously[1], the short/medium term plan is that Aeolus will: - Have well documented deployable and template description formats with plenty of examples. Deployable authors will create these descriptions manually or with the help of simple command line tools and store them as they fit (e.g. in their own git repo) - Support launching deployables in conductor simply by allowing the user to supply a URL to a deployable description, or choosing a URL from a list populated by an admin. - Include a set of command line tools for building images from a template description. This tools will also allow deployable authors to list available images available to be referenced in their deployables. - Store images and their related metadata in IWHD. Conductor will use IWHD queries to resolve deployables' image references and allow users to launch individual images directly. - Use image factory to build and import images, including the updating of IWHD metadata about the images. = IHWD Metadata = The following IWHD buckets will be used, each one for a different object type: images: - object_type == "image" - object_body: XML document with image description and parameters, for deployable authors and simple instance launch from UI - uuid: UUID - latest_build: build UUID builds: - object_type == "build" - object_body: empty - uuid: UUID - image: UUID of the image object - parent: UUID of previous build target_images: - object_type == "target_image" - object_body: target specific image, for upload builds; empty for snapshot builds - uuid: UUID - build: UUID of the build object - icicle: UUID of icicle object - template: UUID of template object - target: ec2, rhev-m, etc. - target_parameters: target specific data stashed here for the push stage provider_images: - object_type == "provider_image" - object_body: empty - uuid: UUID - target_image: UUID of image - provider: e.g. ec2-us-east-1, rhev-m site - icicle == "none" - target_identifier: target specific ID for the image templates: - object_type == "template" - object_body: <template/> document supplied for image build - uuid: UUID icicles: - object_type == "icicle" - object_body: <icicle/> document - uuid: UUID This fairly closely reflects the metadata currently stored by image factory in IWHD, with the following changes: - the current "image" type is renamed to "target_type", along with the bucket and image reference on provider images[2] - new image and build object types and buckets are introduced - a build reference is added to the target image type = Image Factory APIs = Image factory currently supports building target images and pushing provider images, along with the appropriate updating of image warehouse metadata. In order to allow the CLI to fire-and-forget, we will need to add new build and push APIs which can handle multiple images at once. These would look like: - image(image_id, template, targets[]) + builds an image for the supplied targets + image_id should be ommitted, if a new image is required + template and targets may be ommitted if a previous build exists and the previous template and targets will be used - push(image_id, providers, credentials) + push the image the supplied providers + assumes factory can figure out which target image is appropriate for each provider + credentials argument is a <provider_credentials/> document Also, we'll need to rename image factory's current concept of an image as a target image. Some initial work on that here[3]. = Conductor APIs = Conductor needs APIs to support the following: 1) List the available provider types in an XML doc 2) List the available providers in an XML doc 3) Dump a <provider_credentials/> document encapsulating all of the available provider accounts The latter API would be subject to authentication and access control. = Conductor IWHD Queries = Conductor has basically two use cases for querying IWHD: 1) As a user, I want to launch a deployable: 2) As a user, I want to launch an instance of an image For (1), conductor needs to resolve each deployable image reference to a set of provider images. This can be done with something like: $> $build_id = curl http://iwhd/$image_id/latest_build $> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query $> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done For (2), conductor needs to list all images available for launching a standalone instance and, when the user launches the image, it needs to list the parameters for the image. Both are easily achieved by querying the images bucket. = Image Building CLI = The image building CLI is used by Aeolus users to build and upload images from templates. It is also used by deployable authors to list the available images. The use cases are: 1) As an image builder or deployable author, I want to list all images 2) As an image builder or deployable author, I want to list all builds of an image 3) As an image builder or deployable author, I want to list all the targets and providers an image has been built for 4) As an image builder, I want to build an image 5) As an image builder, I want to push an image to a provider 6) As an image builder, I want to import an image 7) As an image builder, I want to delete an image 8) As an image builder, I want to delete old versions of an image 9) As an image builder, I want to delete a provider image 10) As an image builder, I want to delete an image The tool needs to interact with IWHD for listing, image factory for building and conductor for provider/account details. It might look like: $> aeolus-image images # list available images $> aeolus-image builds $image_id # list the builds of an image $> aeolus-image target-images $build_id # list the target images from a build $> aeolus-image provider-images $target_image_id # list the provider images from a target image $> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template $> aeolus-image build --image $image_id # rebuild the image template and targets from latest build $> aeolus-image build --target ec2 --target rackspace \ # --image $image_id \ # rebuild the image with a new template and set of targets --template my.tmpl # $> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider $> aeolus-image push $build_id # push all target images for a build, to same providers as previously $> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account $> aeolus-image push $image_id # push all the target images for the latest build $> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider $> aeolus-image delete --image $image_id # deletes all builds, target images and provider images $> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate $> aeolus-image delete --target-image $target_image # deletes a target image and its provider images $> aeolus-image delete --provider-image $provider_image # deletes a provider image $> aeolus-image targets # list the values available for the --target parameter $> aeolus-image providers # list the values available for the --provider parameter $> aeolus-image accounts # list the values available for the --account parameter Some other notes: - The tool will need to authenticate at least against conductor, so --user/--pass arguments will be needed. We may also want to support supplying these via environment variables and/or a dotfile. - The build/push/import commands are long running, so we should support displaying the progress of the builds and have a --background parameter for fire-and-forget. = Footnotes = [1] - https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001640.html [2] - to migrate existing data, you can do: $> cat > update.js <<EOF db.main.update({ _bucket : "images", object_type : "image" }, { \$set: { _bucket : "target_images", object_type : "target_image" } }, false, true) db.main.find({ _bucket : "provider_images", object_type : "provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image; db.main.save(pi); }) db.main.update({ _bucket : "provider_images", object_type : "provider_image" }, { \$unset: { image : 1 } }, false, true) EOF $> mongo --port 27018 repo update.js $> mv _fs/images _fs/provider_images This clearly only works for IWHD with the fs_autostart, but should be easy to adjust. [3] - https://github.com/markmc/image_factory/commits/master _______________________________________________ aeolus-devel mailing list aeolus-devel(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/aeolus-devel

On Wed, 2011-05-25 at 13:39 -0400, Joseph VLcek wrote: > On Tue, 2011-05-24 at 15:52 +0100, Mark McLoughlin wrote: > > Hey, > > > > Apologies for another long email, but here's my proposal for how to > > proceed with the image building CLI etc. > > > > Cheers, > > Mark. > > > > = Terminology = > > > > - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2 > > region. > > I find using the term Provider here to be confusing. To me the > unqualified term indicates a high level concept of a tool that > "Provides" the infrastructure. This is conductor's terminology. It's even the terminology we expose in the UI. (Not really commenting on whether it's confusing or not, just pointing out that it's consistent with the project-wide terminology) Cheers, Mark.

On Tue, 2011-05-24 at 15:52 +0100, Mark McLoughlin wrote: > Hey, > > Apologies for another long email, but here's my proposal for how to > proceed with the image building CLI etc. > > Cheers, > Mark. > > = Terminology = > > - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2 > region. > > - Target/Provider Type - the cloud type, irrespective of the specific > instance e.g. RHEV-M vs EC2. > > - Provider Account - a set of user credentials for a specific > provider. > > - Provider Image - these are images which have been pushed to, or > imported from, a provider and are available for use by instances. > > - Target Image - these are images which have been built for a > specific target and are available to be pushed to multiple > providers. > > - Image Build - these are objects which track all target and > provider images built from a template at a certain time, all of > which should be equivalent to each other. > > - Image - these are metadata objects which describe the purpose of > the image, the parameters it takes, the set of available builds of > the image and the latest build. > > = Overview = > > As described previously[1], the short/medium term plan is that Aeolus > will: > > - Have well documented deployable and template description formats > with plenty of examples. Deployable authors will create these > descriptions manually or with the help of simple command line > tools and store them as they fit (e.g. in their own git repo) > > - Support launching deployables in conductor simply by allowing the > user to supply a URL to a deployable description, or choosing a > URL from a list populated by an admin. > > - Include a set of command line tools for building images from a > template description. This tools will also allow deployable > authors to list available images available to be referenced in > their deployables. > > - Store images and their related metadata in IWHD. Conductor will > use IWHD queries to resolve deployables' image references and > allow users to launch individual images directly. > > - Use image factory to build and import images, including the > updating of IWHD metadata about the images. > > = IHWD Metadata = > > The following IWHD buckets will be used, each one for a different > object type: > > images: > - object_type == "image" > - object_body: XML document with image description and parameters, > for deployable authors and simple instance launch from UI > - uuid: UUID > - latest_build: build UUID > > builds: > - object_type == "build" > - object_body: empty > - uuid: UUID > - image: UUID of the image object > - parent: UUID of previous build > > target_images: > - object_type == "target_image" > - object_body: target specific image, for upload builds; empty for > snapshot builds > - uuid: UUID > - build: UUID of the build object > - icicle: UUID of icicle object > - template: UUID of template object I don't get why this ^ doesnt go with Image object?

> - target: ec2, rhev-m, etc. We need to be careful of magic string here, I propose these values come from deltacloud api

> - target_parameters: target specific data stashed here for the push > stage > > provider_images: > - object_type == "provider_image" > - object_body: empty > - uuid: UUID > - target_image: UUID of image Do you actually mean uuid of image here, or of target image?

As imcleod said in his reply, this probably need to be where the icicle ref is required, and target image is an optional one (to have a ref to icicle).

> - provider: e.g. ec2-us-east-1, rhev-m site > - icicle == "none" Is this ^ really just a literal string? If so, why have it? If not, does this just mean it is an optional association?

> - target_identifier: target specific ID for the image > > templates: > - object_type == "template" > - object_body: <template/> document supplied for image build > - uuid: UUID > > icicles: > - object_type == "icicle" > - object_body: <icicle/> document > - uuid: UUID > > This fairly closely reflects the metadata currently stored by image > factory in IWHD, with the following changes: > > - the current "image" type is renamed to "target_type", along with > the bucket and image reference on provider images[2] > > - new image and build object types and buckets are introduced > > - a build reference is added to the target image type > > = Image Factory APIs = > > Image factory currently supports building target images and pushing > provider images, along with the appropriate updating of image > warehouse metadata. > > In order to allow the CLI to fire-and-forget, we will need to add new > build and push APIs which can handle multiple images at once. > Two alternatives to this come to mind: * the cli could just call the existing methods sequentially for each target passed in

* factory could provide a thin wrapper on the existing methods to do the same thing from that side.

> These would look like: > > - image(image_id, template, targets[]) > + builds an image for the supplied targets > + image_id should be ommitted, if a new image is required If image_id _is_ passed in, is that essentially a rebuild of existing, thereby creating an updated version (build) of the image?

> + template and targets may be ommitted if a previous build > exists and the previous template and targets will be used > > - push(image_id, providers, credentials) > + push the image the supplied providers > + assumes factory can figure out which target image is > appropriate for each provider > + credentials argument is a <provider_credentials/> document > > Also, we'll need to rename image factory's current concept of an image > as a target image. Some initial work on that here[3]. I agree with Scott's concerns here on the multiple account per provider thing. I don't know the answer either, so for a first pass, perhaps we just limit it to 1 account?

> = Conductor APIs = > > Conductor needs APIs to support the following: > > 1) List the available provider types in an XML doc > 2) List the available providers in an XML doc > 3) Dump a <provider_credentials/> document encapsulating all of the > available provider accounts > > The latter API would be subject to authentication and access control. > > = Conductor IWHD Queries = > > Conductor has basically two use cases for querying IWHD: > > 1) As a user, I want to launch a deployable: > > 2) As a user, I want to launch an instance of an image > > For (1), conductor needs to resolve each deployable image reference to > a set of provider images. This can be done with something like: > > $> $build_id = curl http://iwhd/$image_id/latest_build > $> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query > $> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done > > For (2), conductor needs to list all images available for launching a > standalone instance and, when the user launches the image, it needs to > list the parameters for the image. Both are easily achieved by > querying the images bucket. > > = Image Building CLI = > > The image building CLI is used by Aeolus users to build and upload > images from templates. It is also used by deployable authors to list > the available images. > > The use cases are: > > 1) As an image builder or deployable author, I want to list all > images > 2) As an image builder or deployable author, I want to list all > builds of an image > 3) As an image builder or deployable author, I want to list all the > targets and providers an image has been built for > 4) As an image builder, I want to build an image > 5) As an image builder, I want to push an image to a provider > 6) As an image builder, I want to import an image > 7) As an image builder, I want to delete an image > 8) As an image builder, I want to delete old versions of an image > 9) As an image builder, I want to delete a provider image > 10) As an image builder, I want to delete an image > > The tool needs to interact with IWHD for listing, image factory for > building and conductor for provider/account details. > > It might look like: > > $> aeolus-image images # list available images > $> aeolus-image builds $image_id # list the builds of an image > $> aeolus-image target-images $build_id # list the target images from a build > $> aeolus-image provider-images $target_image_id # list the provider images from a target image > $> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template > $> aeolus-image build --image $image_id # rebuild the image template and targets from latest build > $> aeolus-image build --target ec2 --target rackspace \ # In the interest of making the cli less verbose, would it be horrible to have target take (maybe optionally) an array? So the value is something like: --target [ec2,rackspace,rhevm]

> --image $image_id \ # rebuild the image with a new template and set of targets > --template my.tmpl # > $> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider > $> aeolus-image push $build_id # push all target images for a build, to same providers as previously > $> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account What is the expected value of $provider_account? is that a uui or name from conductor, or just the un/pw of actual provider account? Something else?

= Conductor APIs = Conductor needs APIs to support the following:

3) Dump a <provider_credentials/> document encapsulating all of the available provider accounts The latter API would be subject to authentication and access control.

On 05/26/2011 09:26 AM, Mark McLoughlin wrote: > On Tue, 2011-05-24 at 15:52 +0100, Mark McLoughlin wrote: >> = Conductor APIs = >> >> Conductor needs APIs to support the following: >> > ... >> 3) Dump a<provider_credentials/> document encapsulating all of the >> available provider accounts >> >> The latter API would be subject to authentication and access control. > We need to think about the access control rules here. > > What users would be allowed to access these credentials? > > Users who don't have access to the credentials can't build images for > those providers. > > Thoughts? > So yes, this one is more difficult w/ the separation. When it's all integrated in one app, it's easy enough to say that a user has "push" access for provider images under a provider account without direct access to the provider credentials. Now, however, we're getting credentials from conductor and using them elsewhere, so without additional security measures in place, we can't really return credentials for provider accounts unless the calling user has explicit view access on the provider account. If this restriction is OK for beta, then the solution is relatively straightforward:

From: Mark McLoughlin <markmc(a)redhat.com&gt; Date: Tue, 24 May 2011 15:52:38 +0100

- Have well documented deployable and template description formats with plenty of examples. Deployable authors will create these descriptions manually or with the help of simple command line tools and store them as they fit (e.g. in their own git repo)

In order to allow the CLI to fire-and-forget, we will need to add new build and push APIs which can handle multiple images at once. These would look like: - image(image_id, template, targets[]) + builds an image for the supplied targets + image_id should be ommitted, if a new image is required + template and targets may be ommitted if a previous build exists and the previous template and targets will be used - push(image_id, providers, credentials) + push the image the supplied providers + assumes factory can figure out which target image is appropriate for each provider + credentials argument is a <provider_credentials/> document Also, we'll need to rename image factory's current concept of an image as a target image. Some initial work on that here[3]. = Conductor APIs = Conductor needs APIs to support the following: 1) List the available provider types in an XML doc 2) List the available providers in an XML doc 3) Dump a <provider_credentials/> document encapsulating all of the available provider accounts The latter API would be subject to authentication and access control.

= Conductor IWHD Queries = Conductor has basically two use cases for querying IWHD: 1) As a user, I want to launch a deployable: 2) As a user, I want to launch an instance of an image For (1), conductor needs to resolve each deployable image reference to a set of provider images. This can be done with something like: $> $build_id = curl http://iwhd/$image_id/latest_build $> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query $> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done For (2), conductor needs to list all images available for launching a standalone instance and, when the user launches the image, it needs to list the parameters for the image. Both are easily achieved by querying the images bucket. = Image Building CLI = The image building CLI is used by Aeolus users to build and upload images from templates. It is also used by deployable authors to list the available images. The use cases are: 1) As an image builder or deployable author, I want to list all images 2) As an image builder or deployable author, I want to list all builds of an image 3) As an image builder or deployable author, I want to list all the targets and providers an image has been built for 4) As an image builder, I want to build an image 5) As an image builder, I want to push an image to a provider 6) As an image builder, I want to import an image 7) As an image builder, I want to delete an image 8) As an image builder, I want to delete old versions of an image 9) As an image builder, I want to delete a provider image 10) As an image builder, I want to delete an image The tool needs to interact with IWHD for listing, image factory for building and conductor for provider/account details. It might look like: $> aeolus-image images # list available images $> aeolus-image builds $image_id # list the builds of an image $> aeolus-image target-images $build_id # list the target images from a build $> aeolus-image provider-images $target_image_id # list the provider images from a target image $> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template $> aeolus-image build --image $image_id # rebuild the image template and targets from latest build $> aeolus-image build --target ec2 --target rackspace \ # --image $image_id \ # rebuild the image with a new template and set of targets --template my.tmpl # $> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider $> aeolus-image push $build_id # push all target images for a build, to same providers as previously $> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account $> aeolus-image push $image_id # push all the target images for the latest build $> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider $> aeolus-image delete --image $image_id # deletes all builds, target images and provider images $> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate $> aeolus-image delete --target-image $target_image # deletes a target image and its provider images $> aeolus-image delete --provider-image $provider_image # deletes a provider image $> aeolus-image targets # list the values available for the --target parameter $> aeolus-image providers # list the values available for the --provider parameter $> aeolus-image accounts # list the values available for the --account parameter

From: Carl Trieloff <cctrieloff(a)redhat.com&gt; Date: Thu, 26 May 2011 15:59:01 -0400

> - Support launching deployables in conductor simply by allowing the > user to supply a URL to a deployable description, or choosing a > URL from a list populated by an admin. > > - Include a set of command line tools for building images from a > template description. This tools will also allow deployable > authors to list available images available to be referenced in > their deployables. I would have said. "This tools will also allow deployable authors to list available images and assemblies available to be referenced in their deployables. Templates referenced by the deployable are automatically wrapped by an assembly."

> > Also, we'll need to rename image factory's current concept of an image > as a target image. Some initial work on that here[3]. What are the assumptions around push... we need to support the case where the images are cached at the resource provider and the case that we only move the image to the provider when it is going to be instantiated and then removed once we destroy it. The mail later seems to imply that the push is a user action, where in reality it needs to be policy driven.

If the user can push the image how does this affect the policy. I would not allow user push for rev one, we don't export the ability for the user to push the images, which means we don't have to unwind a complicated policy interaction mess later. Then there is no mention on services in the mail.

Hey,

The tool needs to interact with IWHD for listing, image factory for building and conductor for provider/account details.

It might look like: $> aeolus-image images # list available images

$> aeolus-image builds $image_id # list the builds of an image

$> aeolus-image target-images $build_id # list the target images from a build

$> aeolus-image provider-images $target_image_id # list the provider images from a target image

$> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template

$> aeolus-image build --image $image_id # rebuild the image template and targets from latest build

$> aeolus-image build --target ec2 --target rackspace \ # --image $image_id \ # rebuild the image with a new template and set of targets --template my.tmpl #

$> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider

$> aeolus-image push $build_id # push all target images for a build, to same providers as previously

$> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account

$> aeolus-image push $image_id # push all the target images for the latest build

$> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider

$> aeolus-image delete --image $image_id # deletes all builds, target images and provider images

$> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate

$> aeolus-image delete --target-image $target_image # deletes a target image and its provider images

$> aeolus-image delete --provider-image $provider_image # deletes a provider image

$> aeolus-image targets # list the values available for the --target parameter

$> aeolus-image providers # list the values available for the --provider parameter

$> aeolus-image accounts # list the values available for the --account parameter