### Is Heat going to be the only way to launch deployments in Conductor or will it be an optional part? ### We will need to see how stable and usable Heat is. How easy it is to deploy and keep up (it would be another dependency, after all) and how much benefit and code cleanup it brings to Aeolus.

Hey Folks, I've been contributing to the Heat project lately and I think it's gotten to a point where we (the Aeolus community) could look at it and figure out if it's something we want to use. Please forgive me the length of this email. I tried to make it as short but there's a lot to cover. Grab your favourite beverage and sit comfortably. A little background ------------------- Heat is a project that provides orchestration and high availability to OpenStack. Its API is modelled after Amazon CloudFormation. What it does in Aeolus' terms is: you create a deployable and pass it to Heat API along with any launch-time parameters. Heat will decide on the correct order of the resources (instances, databases, load balancers, remote storage, floating IP addresses, etc.) and launches and configures the whole thing to produce a running deployment. Heat started about five months ago and hopes to become one of the core OpenStack components. It's written in Python following the OpenStack development model and tools. What's this to do with Aeolus? ------------------------------ We're trying to solve some of the same problems: * launching multiple instances in correct order * passing parameters that are only known when an instance is launched * configuring and advertising the services running inside the VMs If we could outsource this to a project whose sole focus is solving these issues, we could focus on things that make Aeolus unique: building great UI, tools and APIs for managing cross-cloud deployments. As a side benefit, we would get the current and future features of Heat for free: * high availability * auto-scaling * load-balancing OpenStack? Huh? --------------- Heat is an OpenStack project and the developers did not put effort into making sure that it works with anything else. However, Ian Main and I looked into the code and it seems that we could integrate Deltacloud support with little changes in the Heat codebase. The Deltacloud backend would be an external package that would not pollute Heat's code and dependencies. The changes to add supports for a custom backend would be small so there's a good chance they'd be accepted. Proof of concept ---------------- It would be cool if we had a working prototype to demonstrate the strengths and weaknesses as a reference for the discussion. We're not there yet but Ian and I have came up with a plan that should produce a basic prototype reasonably quickly. The first step is to get Conductor to launch multi-instance deployments using Heat, nothing else. The plan is at the end of this email. Image management ---------------- Heat's image management is very similar to Aeolus'. The images are built using Oz and stored in Glance (OpenStack's image storage tool). Heat supports both installing everything at runtime from bare (JEOS) images and using preconfigured images and just launching those. So far the focus has been on the former but both approaches work and Heat developers are spending more time making the experience with image prebuilding smoother. Since Conductor supports both modes as well (prebuilt images with Image Factory, post-launch with Audrey), there shouldn't be an impedance mismatch.d We will keep building images in Image Factory, we'll pass the image IDs from Deployable XMLs to Heat that will in turn pass them to Deltacloud. What's next? ------------ You can read the plan for the proof-of-concept and the long-term phases and I would like your feedback. Please post correction on anything I've missed or said wrong and anything else that comes in your mind. Do you think integrating with Heat makes sense? I want stress that this is not a done deal. I think that that Heat can be very useful to Aeolus but it's quite possible that I've missed something and we still need to see it in action before we commit to anything. There's a real possibility that this just is not a good match in which case we'll go our separate ways (unless we decide to make it a good match). Feel free to ask anything that's unclear. Gimme teh links! ---------------- If you want to try Heat out, follow these steps: https://github.com/heat-api/heat/blob/master/docs/GettingStarted.rst Heat developers hang out at #heat on freenode and are all really nice chaps. The repo for experimental integration of Deltacloud into Heat: https://github.com/tomassedovic/heat/tree/multiple-backends The Deltacloud client for Python lives in the official Deltacloud repository: https://git-wip-us.apache.org/repos/asf?p=deltacloud.git;a=tree;f=clients... The client is very barebones at the moment. We'll be adding the necessary features here: https://github.com/tomassedovic/deltacloud/tree/python-client and work on getting them accepted upstream. The API Heat uses at the moment is based on AWS CloudFormation. You can read the documentation here: http://docs.amazonwebservices.com/AWSCloudFormation/latest/APIReference/W... Here are some Deployable descriptions that Heat accepts and is able to launch (in Heat and CloudFormation terminology, these are called "templates", the deployment is called "stack") https://github.com/heat-api/heat/tree/master/templates What follows is the short-term and long-term plan on getting this done. Take care, Thomas Proof of concept ---------------- ### Getting Deltacloud to work with Python ### The Deltacloud API is language-agnostic but we still need Python bindings. Deltacloud ships with a Python client but it hasn't been touched for over a year. I've sent a few patches to fix it but more work is required for it to be usable. ### Passing the Provider credentials from Heat API to Deltacloud ### Currently, the Heat API user can authenticate using either Keystone or the EC2-style credentials. We need a new authentication handler that will receive Deltacloud-style credentials and pass them to the Deltacloud API. Ian has written a patch that does this. ### Integrating Deltacloud client with Heat ### Heat is currently using the Nova python client to launch instances, etc. The calls are already reasonably isolated. I've written a shim module that provides the same API as the Nova client and we hooked it up to Ian's authentication code. It's not connected to Deltacloud yet (that's the next step), but as far as majority of Heat's codebase is concerned, everything works without making a single call to OpenStack. ### Getting Aeolus Conductor to generate AWS::CFN Template ### Conductor already has all the info it needs to launch the instances stored in the Deployable XML. We'll take the XML, parse it and generate the equivalent JSON template that Heat accepts. ### Passing the generate template from Conductor to Heat ### Once we have the converted deployable, Conductor will send it to Heat via the current CloudFormation API. Heat will launch the deployment/stack via its Deltacloud bindings. ### Querying Heat data from Conductor ### Heat doesn't support any callbacks. When Conductor wants to know details about the stack it launched, it will use the CloudFormation API to query the data. For the proof of concept stage, we will just issue the query to Heat upon every relevant UI action: e.g. `ListStacks` when showing deployables in the UI, `DescribeStackResource` when shoving a details of a single deployable, `DescribeStackEvents` to get deployable events, etc.

Long-term ---------- ### Dead Nova client dependencies ### Parts of the Heat code import Nova and Glance clients but they don't use the code. These should be removed. ### Nova client exceptions ### Several libraries import the exception classes from Nova client. If we could make the Deltacloud backend to raise the same exceptions without depending on the `python-novaclient` library, that would be cool. Barring that, we'll either have to wrap the backend exceptions or have the Deltacloud code bite the bullet and depend on the novaclient lib (the Heat package depends on it anyway so it may not be as bad). ### Make the backends swappable at the configuration level ### Similarly to how Heat can change between the AMQP implementations for Fedora and Ubuntu, we need a configuration option that replaces OpenStack with Deltacloud. ### Make the Heat Deltacloud backend a separate package ### Heat doesn't have to ship additional backends if they don't want to. The configuration option from previous task would take a Python package namespace. That way it doesn't care if it's within the main codebase or a third-party module. ### Template format ### Aeolus uses a different format from Heat to describe deployables. We need to settle on what's the best thing to use. Options: 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON 2) Implement Aeolus Deployable XML format in Heat 3) Adopt CloudFormation JSON in Conductor 4) Have both Aeouls and Heat adopt another (open standard) format With all likelihood Heat will reject 2), we won't switch to 3) and since *both* communities would have to accept and implement 4), we'll probably stick to 1) for some time. Still, listing it here as it needs to be decided. ### Heat-Conductor communication ### Heat is working on implementing a proper RESTful API that will 1) be more consistent with other OpenStack components 2) suck less Once that's done, Conductor should probably adopt this API. We then need to figure out how to pass data from Heat back to Conductor. If Conductor's launching instances via Heat, how does that information get updated? 1) Conductor asks Heat every time it needs the data 2) Conductor asks Heat periodically and caches the results 3) Heat allows Conductor to register a callback -- either via HTTP (webhooks) or via RPC -- and tells Conductor when stuff happens (e.g. an instance was launched, crashed, etc.) 4) Conductor continues to use DBomatic to query the instances directly via Deltacloud, ignoring any additional information Heat may provide For the proof of concept, everything other than 1) is a premature optimization and then we'll see. I think 4) would be rather brittle especially since adopting Heat might possibly let us drop DBomatic entirely. ### Is Heat going to be the only way to launch deployments in Conductor or will it be an optional part? ### We will need to see how stable and usable Heat is. How easy it is to deploy and keep up (it would be another dependency, after all) and how much benefit and code cleanup it brings to Aeolus. ### Missing parts in Deltacloud ### Deltacloud provides only a subset of the information OpenStack clients have access to. A lot of these are provider-specific. We need to identify these cases and decide what to do with them: 1) Have Deltacloud provide the access for providers that support them and advertise the support properly 2) Don't let Heat provide them for non-Openstack providers 3) Implement them in Heat for providers that support them and keep Deltacloud as the lowest common denominator This includes working with the Deltacloud people: we're going to let them know what's missing and discuss the appropriate solution. Since Deltacloud API is already focused towards discoverability (e.g. when showing an instance, it lists the actions that can be done to the instance) that may be the best approach if all parties agree.

On 08/27/2012 02:47 PM, Jan Provaznik wrote: > On 08/21/2012 06:15 PM, Tomas Sedovic wrote: >> Hey Folks, >> >> I've been contributing to the Heat project lately and I think it's >> gotten to a point where we (the Aeolus community) could look at it and >> figure out if it's something we want to use. >> >> Please forgive me the length of this email. I tried to make it as short >> but there's a lot to cover. Grab your favourite beverage and sit >> comfortably. >> >> >> A little background >> ------------------- >> >> Heat is a project that provides orchestration and high availability to >> OpenStack. Its API is modelled after Amazon CloudFormation. >> >> What it does in Aeolus' terms is: you create a deployable and pass >> it to >> Heat API along with any launch-time parameters. Heat will decide on the >> correct order of the resources (instances, databases, load balancers, >> remote storage, floating IP addresses, etc.) and launches and >> configures >> the whole thing to produce a running deployment. >> >> Heat started about five months ago and hopes to become one of the core >> OpenStack components. It's written in Python following the OpenStack >> development model and tools. >> >> >> What's this to do with Aeolus? >> ------------------------------ >> >> We're trying to solve some of the same problems: >> >> * launching multiple instances in correct order >> * passing parameters that are only known when an instance is launched >> * configuring and advertising the services running inside the VMs >> >> If we could outsource this to a project whose sole focus is solving >> these issues, we could focus on things that make Aeolus unique: >> building >> great UI, tools and APIs for managing cross-cloud deployments. >> >> As a side benefit, we would get the current and future features of Heat >> for free: >> >> * high availability >> * auto-scaling >> * load-balancing >> >> >> OpenStack? Huh? >> --------------- >> >> Heat is an OpenStack project and the developers did not put effort into >> making sure that it works with anything else. >> >> However, Ian Main and I looked into the code and it seems that we could >> integrate Deltacloud support with little changes in the Heat codebase. >> >> The Deltacloud backend would be an external package that would not >> pollute Heat's code and dependencies. The changes to add supports for a >> custom backend would be small so there's a good chance they'd be >> accepted. >> >> >> Proof of concept >> ---------------- >> >> It would be cool if we had a working prototype to demonstrate the >> strengths and weaknesses as a reference for the discussion. >> >> We're not there yet but Ian and I have came up with a plan that should >> produce a basic prototype reasonably quickly. >> >> The first step is to get Conductor to launch multi-instance deployments >> using Heat, nothing else. The plan is at the end of this email. >> >> >> Image management >> ---------------- >> >> Heat's image management is very similar to Aeolus'. >> >> The images are built using Oz and stored in Glance (OpenStack's image >> storage tool). >> >> Heat supports both installing everything at runtime from bare (JEOS) >> images and using preconfigured images and just launching those. >> >> So far the focus has been on the former but both approaches work and >> Heat developers are spending more time making the experience with image >> prebuilding smoother. >> >> Since Conductor supports both modes as well (prebuilt images with Image >> Factory, post-launch with Audrey), there shouldn't be an impedance >> mismatch.d >> >> We will keep building images in Image Factory, we'll pass the image IDs >> from Deployable XMLs to Heat that will in turn pass them to Deltacloud. >> >> >> What's next? >> ------------ >> >> You can read the plan for the proof-of-concept and the long-term phases >> and I would like your feedback. Please post correction on anything I've >> missed or said wrong and anything else that comes in your mind. Do you >> think integrating with Heat makes sense? >> >> I want stress that this is not a done deal. I think that that Heat can >> be very useful to Aeolus but it's quite possible that I've missed >> something and we still need to see it in action before we commit to >> anything. >> >> There's a real possibility that this just is not a good match in which >> case we'll go our separate ways (unless we decide to make it a good >> match). Feel free to ask anything that's unclear. >> >> >> Gimme teh links! >> ---------------- >> >> If you want to try Heat out, follow these steps: >> >> https://github.com/heat-api/heat/blob/master/docs/GettingStarted.rst >> >> Heat developers hang out at #heat on freenode and are all really nice >> chaps. >> >> The repo for experimental integration of Deltacloud into Heat: >> >> https://github.com/tomassedovic/heat/tree/multiple-backends >> >> The Deltacloud client for Python lives in the official Deltacloud >> repository: >> >> https://git-wip-us.apache.org/repos/asf?p=deltacloud.git;a=tree;f=clients... >> >> >> >> >> The client is very barebones at the moment. We'll be adding the >> necessary features here: >> >> https://github.com/tomassedovic/deltacloud/tree/python-client >> >> and work on getting them accepted upstream. >> >> The API Heat uses at the moment is based on AWS CloudFormation. You can >> read the documentation here: >> >> http://docs.amazonwebservices.com/AWSCloudFormation/latest/APIReference/W... >> >> >> >> >> Here are some Deployable descriptions that Heat accepts and is able to >> launch (in Heat and CloudFormation terminology, these are called >> "templates", the deployment is called "stack") >> >> https://github.com/heat-api/heat/tree/master/templates >> >> >> What follows is the short-term and long-term plan on getting this done. >> >> Take care, >> Thomas >> >> >> Proof of concept >> ---------------- >> >> ### Getting Deltacloud to work with Python ### >> >> The Deltacloud API is language-agnostic but we still need Python >> bindings. >> >> Deltacloud ships with a Python client but it hasn't been touched for >> over a year. I've sent a few patches to fix it but more work is >> required >> for it to be usable. >> >> >> ### Passing the Provider credentials from Heat API to Deltacloud ### >> >> Currently, the Heat API user can authenticate using either Keystone or >> the EC2-style credentials. >> >> We need a new authentication handler that will receive Deltacloud-style >> credentials and pass them to the Deltacloud API. >> >> Ian has written a patch that does this. >> >> >> ### Integrating Deltacloud client with Heat ### >> >> Heat is currently using the Nova python client to launch instances, >> etc. >> The calls are already reasonably isolated. >> >> I've written a shim module that provides the same API as the Nova >> client >> and we hooked it up to Ian's authentication code. >> >> It's not connected to Deltacloud yet (that's the next step), but as far >> as majority of Heat's codebase is concerned, everything works without >> making a single call to OpenStack. >> >> >> ### Getting Aeolus Conductor to generate AWS::CFN Template ### >> >> Conductor already has all the info it needs to launch the instances >> stored in the Deployable XML. >> >> We'll take the XML, parse it and generate the equivalent JSON template >> that Heat accepts. >> >> >> ### Passing the generate template from Conductor to Heat ### >> >> Once we have the converted deployable, Conductor will send it to Heat >> via the current CloudFormation API. >> >> Heat will launch the deployment/stack via its Deltacloud bindings. >> >> >> ### Querying Heat data from Conductor ### >> >> Heat doesn't support any callbacks. When Conductor wants to know >> details >> about the stack it launched, it will use the CloudFormation API to >> query >> the data. >> >> For the proof of concept stage, we will just issue the query to Heat >> upon every relevant UI action: e.g. `ListStacks` when showing >> deployables in the UI, `DescribeStackResource` when shoving a >> details of >> a single deployable, `DescribeStackEvents` to get deployable events, >> etc. >> > > This is OK for POC, but it would be really nice to have callback support > for real integration. Yeah I agree. > > nit: you probably meant 'deployment' instead of 'deployable' in the > paragraph above. You're right, thanks! > >> >> >> Long-term >> ---------- >> >> ### Dead Nova client dependencies ### >> >> Parts of the Heat code import Nova and Glance clients but they don't >> use >> the code. These should be removed. >> >> >> ### Nova client exceptions ### >> >> Several libraries import the exception classes from Nova client. >> >> If we could make the Deltacloud backend to raise the same exceptions >> without depending on the `python-novaclient` library, that would be >> cool. >> >> Barring that, we'll either have to wrap the backend exceptions or have >> the Deltacloud code bite the bullet and depend on the novaclient lib >> (the Heat package depends on it anyway so it may not be as bad). >> >> >> ### Make the backends swappable at the configuration level ### >> >> Similarly to how Heat can change between the AMQP implementations for >> Fedora and Ubuntu, we need a configuration option that replaces >> OpenStack with Deltacloud. >> >> >> ### Make the Heat Deltacloud backend a separate package ### >> >> Heat doesn't have to ship additional backends if they don't want to. >> The >> configuration option from previous task would take a Python package >> namespace. >> >> That way it doesn't care if it's within the main codebase or a >> third-party module. >> >> >> ### Template format ### >> >> Aeolus uses a different format from Heat to describe deployables. >> >> We need to settle on what's the best thing to use. Options: >> >> 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON >> >> 2) Implement Aeolus Deployable XML format in Heat >> >> 3) Adopt CloudFormation JSON in Conductor >> >> 4) Have both Aeouls and Heat adopt another (open standard) format >> >> With all likelihood Heat will reject 2), we won't switch to 3) and >> since >> *both* communities would have to accept and implement 4), we'll >> probably >> stick to 1) for some time. >> >> Still, listing it here as it needs to be decided. >> >> >> ### Heat-Conductor communication ### >> >> Heat is working on implementing a proper RESTful API that will >> >> 1) be more consistent with other OpenStack components >> 2) suck less >> >> Once that's done, Conductor should probably adopt this API. >> >> We then need to figure out how to pass data from Heat back to >> Conductor. >> If Conductor's launching instances via Heat, how does that information >> get updated? >> >> 1) Conductor asks Heat every time it needs the data >> >> 2) Conductor asks Heat periodically and caches the results >> >> 3) Heat allows Conductor to register a callback -- either via HTTP >> (webhooks) or via RPC -- and tells Conductor when stuff happens >> (e.g. an >> instance was launched, crashed, etc.) >> >> 4) Conductor continues to use DBomatic to query the instances directly >> via Deltacloud, ignoring any additional information Heat may provide >> >> For the proof of concept, everything other than 1) is a premature >> optimization and then we'll see. >> >> I think 4) would be rather brittle especially since adopting Heat might >> possibly let us drop DBomatic entirely. >> >> >> ### Is Heat going to be the only way to launch deployments in Conductor >> or will it be an optional part? ### >> >> We will need to see how stable and usable Heat is. How easy it is to >> deploy and keep up (it would be another dependency, after all) and how >> much benefit and code cleanup it brings to Aeolus. >> >> >> ### Missing parts in Deltacloud ### >> >> Deltacloud provides only a subset of the information OpenStack clients >> have access to. A lot of these are provider-specific. >> >> We need to identify these cases and decide what to do with them: >> >> 1) Have Deltacloud provide the access for providers that support them >> and advertise the support properly >> >> 2) Don't let Heat provide them for non-Openstack providers >> >> 3) Implement them in Heat for providers that support them and keep >> Deltacloud as the lowest common denominator >> >> This includes working with the Deltacloud people: we're going to let >> them know what's missing and discuss the appropriate solution. >> >> Since Deltacloud API is already focused towards discoverability (e.g. >> when showing an instance, it lists the actions that can be done to the >> instance) that may be the best approach if all parties agree. > > Hi Tomas, > I think it's too early to say if Conductor will switch to Heat, but it > would be nice to have this option. I suppose that we will extract > launching logic out of Conductor anyway and if it will be possible to > use Heat as this launching backend, it would be great. Yup, that sounds great. > > Here is a couple of questions/concerns: > - the deployment launch setup becomes more complicated, a request will > pass through 3 standalone services: > Conductor -> Heat -> Deltacloud -> Provider > Having rack-compatible engine on top of dc-api lib mounted directly in > Conductor would be easier. > That is a valid point. I think we'll have to see whether the benefits Heat brings in outweigh this. > - Heat is focused mainly on Openstack and Deltacloud backend will use > only subset of possible features, I wonder if it will be possible in > future to keep Heat's backend API compatible with Deltacloud? At least as far as the initial goal is (implement a fully compatible CloudFormation API), I think so. That API and concepts within are stable. There are and will be things that Deltacloud probably won't support, but since Aeolus doesn't provide them now anyway, that should not be a concern. As far as the Heat-specific API and features beyond CloudFormation go, I don't know. I think it's too early to tell. But it's a good think to watch out for. > > - What are Heat's plans about deployment launch atomicity? (what if any > of instance launch fails during deployment launch) CloudFormation's launching is atomic. If any of the instances fails to launch, the whole thing is rolled back. This can optionally be disabled per deployment launch. Heat currently doesn't support rollbacks but it will: https://github.com/heat-api/heat/issues/154 The feature does not have very a high priority now, though. > > - Heat's HA would have to be friendly to single-instance actions, it > should distinguish between instance failures and actions invoked by a > user (if a user stops single instance, Heat shouldn't try to start it) Good point. HA is optional and enabled on a per-instance basis in the deployable. If it is on, though, I'm not sure how could Heat distinguish a user action from an actual failure. In both cases the instance would be down. I guess we could temporarily disable the monitoring. Alternatively, Heat could extend its API to allow users to stop/restart instances manually.

However, from the CloudFormation (and therefore Heat) perspective, the deployments are atomic. I thought Aeolus was trying to do the same thing (though we do allow users to stop individual instances now).

> > - it's probably out of Heat's scope, but does it check availability of > other provider resources like realms or hw profiles? Except periodic > checking of deployment/instance status, we need to check also status of > providers and provider realms (and maybe also hw profiles in future), > this is now done by dbomatic too. It checks the availability when it's launching a deployment. If you specify an invalid realm or profile, the launch will fail. Heat's HA monitors individual instances, services within and deployments. I think the Heat API could be extended to notify other services about these things. Of course, since we'd still be using Deltacloud, these things could be monitored by their notification extension (once that's available).

> > Jan

On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: > On 08/21/2012 06:15 PM, Tomas Sedovic wrote: >> Hey Folks, >> [snip] >> ### Querying Heat data from Conductor ### >> >> Heat doesn't support any callbacks. When Conductor wants to know details >> about the stack it launched, it will use the CloudFormation API to query >> the data. >> >> For the proof of concept stage, we will just issue the query to Heat >> upon every relevant UI action: e.g. `ListStacks` when showing >> deployables in the UI, `DescribeStackResource` when shoving a details of >> a single deployable, `DescribeStackEvents` to get deployable events, etc. >> > > This is OK for POC, but it would be really nice to have callback > support for real integration. > > nit: you probably meant 'deployment' instead of 'deployable' in the > paragraph above. I am curious as to why you think it is necessary to use callbacks and mirror the data held in heat within aeolus? Ian

On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote: > On 08/29/2012 09:55 PM, Ian Main wrote: >> On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: >>> On 08/21/2012 06:15 PM, Tomas Sedovic wrote: >>>> Hey Folks, >>>> >> >> [snip] >> >>>> ### Querying Heat data from Conductor ### >>>> >>>> Heat doesn't support any callbacks. When Conductor wants to know details >>>> about the stack it launched, it will use the CloudFormation API to query >>>> the data. >>>> >>>> For the proof of concept stage, we will just issue the query to Heat >>>> upon every relevant UI action: e.g. `ListStacks` when showing >>>> deployables in the UI, `DescribeStackResource` when shoving a details of >>>> a single deployable, `DescribeStackEvents` to get deployable events, etc. >>>> >>> >>> This is OK for POC, but it would be really nice to have callback >>> support for real integration. >>> >>> nit: you probably meant 'deployment' instead of 'deployable' in the >>> paragraph above. >> >> I am curious as to why you think it is necessary to use callbacks and >> mirror the data held in heat within aeolus? >> >> Ian >> > > Conductor needs to know if/when a deployment or single instance > changed its state (is this what you mean by mirroring data?). W/o > notification support on Heat side, Conductor would have to poll Heat > which is painful (dbomatic-like service presence on conductor side) > and not very effective. I agree dbomatic type service is error prone. However mirroring data from one service to another is a very difficult problem to solve well and have it be reliable. Is this required for some sort of reporting? If it is just for the

user to view the states then that can be done on an as-needed basis by contacting heat. Even reporting is part of the AWS cloudformations API and events for stacks are supposed to be kept around for something like 90 days (iirc). Personally I very much question the need to mirror the data heat retains into Aeolus as these kinds of things are very error prone and difficult. Unless there is some kind of special need for reporting etc. the data could just as easily be queried directly. Ian

On Mon, Sep 03, 2012 at 12:44:05PM +0200, Jan Provazník wrote: > On 08/30/2012 10:23 PM, Ian Main wrote: >> On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote: >>> On 08/29/2012 09:55 PM, Ian Main wrote: >>>> On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: >>>>> On 08/21/2012 06:15 PM, Tomas Sedovic wrote: >>>>>> Hey Folks, >>>>>> >>>> [snip] >>>> >>>>>> ### Querying Heat data from Conductor ### >>>>>> >>>>>> Heat doesn't support any callbacks. When Conductor wants to know details >>>>>> about the stack it launched, it will use the CloudFormation API to query >>>>>> the data. >>>>>> >>>>>> For the proof of concept stage, we will just issue the query to Heat >>>>>> upon every relevant UI action: e.g. `ListStacks` when showing >>>>>> deployables in the UI, `DescribeStackResource` when shoving a details of >>>>>> a single deployable, `DescribeStackEvents` to get deployable events, etc. >>>>>> >>>>> This is OK for POC, but it would be really nice to have callback >>>>> support for real integration. >>>>> >>>>> nit: you probably meant 'deployment' instead of 'deployable' in the >>>>> paragraph above. >>>> I am curious as to why you think it is necessary to use callbacks and >>>> mirror the data held in heat within aeolus? >>>> >>>> Ian >>>> >>> Conductor needs to know if/when a deployment or single instance >>> changed its state (is this what you mean by mirroring data?). W/o >>> notification support on Heat side, Conductor would have to poll Heat >>> which is painful (dbomatic-like service presence on conductor side) >>> and not very effective. >> I agree dbomatic type service is error prone. However mirroring data > >from one service to another is a very difficult problem to solve well >> and have it be reliable. >> >> Is this required for some sort of reporting? If it is just for the > Yes, reporting and keeping history logs about instances is part of > Conductor. Conductor also uses this information when choosing a > provider when launching an instance and also for quota checking. This could be done either way, but really you are just needing a tally of instances per user and per cloud. I'm not saying it is ideal but I wouldn't say it's impossible or even unwise to consider direct querying even here.

>> user to view the states then that can be done on an as-needed basis by >> contacting heat. Even reporting is part of the AWS cloudformations API >> and events for stacks are supposed to be kept around for something like >> 90 days (iirc). >> >> Personally I very much question the need to mirror the data heat retains >> into Aeolus as these kinds of things are very error prone and difficult. >> Unless there is some kind of special need for reporting etc. the data >> could just as easily be queried directly. >> >> Ian >> > Besides the needs listed above, I'm afraid there might be > performance issues with querying Heat directly per each user > request. I think we would have to try it and see. Ultimately it all comes from a database over a network connection. Querying heat only adds one more http request/socket layer. Perhaps some time munging data but all in all it's fairly light weight stuff. I think it's important to really consider what is involved in duplicating that data. It has been difficult in the past eg with dbomatic because it is a very hard problem. Even with events this doesn't change. What happens when a service goes down for a time? You need to replay events, read logs etc.. people spend a lot of time and a lot of money trying to solve data replication. IMO if you can access the primary data source for your implementation that will be a far more robust solution. Yes, this may mean having to sort out performance issues, but I think that is ultimately an easier problem.

Ian

On Thu, Sep 06, 2012 at 03:56:44PM -0400, Scott Seago wrote: > On 09/04/2012 05:25 PM, Ian Main wrote: > >On Mon, Sep 03, 2012 at 12:44:05PM +0200, Jan Provazník wrote: > >>On 08/30/2012 10:23 PM, Ian Main wrote: > >>>On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote: > >>>>On 08/29/2012 09:55 PM, Ian Main wrote: > >>>>>On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: > >>>>>>On 08/21/2012 06:15 PM, Tomas Sedovic wrote: > >>>>>>>Hey Folks, > >>>>>>> > >>>>>[snip] > >>>>> > >>>>>>>### Querying Heat data from Conductor ### > >>>>>>> > >>>>>>>Heat doesn't support any callbacks. When Conductor wants to know details > >>>>>>>about the stack it launched, it will use the CloudFormation API to query > >>>>>>>the data. > >>>>>>> > >>>>>>>For the proof of concept stage, we will just issue the query to Heat > >>>>>>>upon every relevant UI action: e.g. `ListStacks` when showing > >>>>>>>deployables in the UI, `DescribeStackResource` when shoving a details of > >>>>>>>a single deployable, `DescribeStackEvents` to get deployable events, etc. > >>>>>>> > >>>>>>This is OK for POC, but it would be really nice to have callback > >>>>>>support for real integration. > >>>>>> > >>>>>>nit: you probably meant 'deployment' instead of 'deployable' in the > >>>>>>paragraph above. > >>>>>I am curious as to why you think it is necessary to use callbacks and > >>>>>mirror the data held in heat within aeolus? > >>>>> > >>>>> Ian > >>>>> > >>>>Conductor needs to know if/when a deployment or single instance > >>>>changed its state (is this what you mean by mirroring data?). W/o > >>>>notification support on Heat side, Conductor would have to poll Heat > >>>>which is painful (dbomatic-like service presence on conductor side) > >>>>and not very effective. > >>>I agree dbomatic type service is error prone. However mirroring data > >>>from one service to another is a very difficult problem to solve well > >>>and have it be reliable. > >>> > >>>Is this required for some sort of reporting? If it is just for the > >>Yes, reporting and keeping history logs about instances is part of > >>Conductor. Conductor also uses this information when choosing a > >>provider when launching an instance and also for quota checking. > >This could be done either way, but really you are just needing a tally > >of instances per user and per cloud. I'm not saying it is ideal but I > >wouldn't say it's impossible or even unwise to consider direct querying > >even here. > One difficulty is that if we're talking about making heat optional, > heat and any other launching infrastructure (including, perhaps, the > current/legacy one if that remains) will need to handle > quota/instance state and queries/etc in the same way. Currently > instance metadata/state/quota checking is tracked in conductor > itself. As long as heat is optional, I'm not sure how we would > change that. If heat became a complete replacement, we _could_ query > all of this stuff live (rather than caching), but there are a lot of > moving parts in the existing infrastructure that would need to be > rewritten. After the pain of handling image metadata as a separate > server (with data only available with live calls outside), we're in > the process of moving that back into conductor. We need to be > careful about deciding to do the opposite with instances and > deployments. I'm not saying we can't/shouldn't do that, but we'd > better make sure we've got answers to the various pain points -- > performance, searching, object associations, permissions, etc. For > permissions, in particular, even if all instance/deployment metadata > were in heat (and only queried live from Conductor), we'd at least > need placeholder objects on our side, so that we don't lose the > ability to manage permissions on a per-object basis. Yeah, I'm really just asking people to think about it a bit more and consider what is really involved. Ultimately as in the nature of any open source project we will just have to see how things unfold :).

On Mon, Sep 03, 2012 at 12:44:05PM +0200, Jan Provazník wrote: > On 08/30/2012 10:23 PM, Ian Main wrote: >> On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote: >>> On 08/29/2012 09:55 PM, Ian Main wrote: >>>> On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: >>>>> On 08/21/2012 06:15 PM, Tomas Sedovic wrote: >>>>>> Hey Folks, >>>>>> >>>> >>>> [snip] >>>> >>>>>> ### Querying Heat data from Conductor ### >>>>>> >>>>>> Heat doesn't support any callbacks. When Conductor wants to know details >>>>>> about the stack it launched, it will use the CloudFormation API to query >>>>>> the data. >>>>>> >>>>>> For the proof of concept stage, we will just issue the query to Heat >>>>>> upon every relevant UI action: e.g. `ListStacks` when showing >>>>>> deployables in the UI, `DescribeStackResource` when shoving a details of >>>>>> a single deployable, `DescribeStackEvents` to get deployable events, etc. >>>>>> >>>>> >>>>> This is OK for POC, but it would be really nice to have callback >>>>> support for real integration. >>>>> >>>>> nit: you probably meant 'deployment' instead of 'deployable' in the >>>>> paragraph above. >>>> >>>> I am curious as to why you think it is necessary to use callbacks and >>>> mirror the data held in heat within aeolus? >>>> >>>> Ian >>>> >>> >>> Conductor needs to know if/when a deployment or single instance >>> changed its state (is this what you mean by mirroring data?). W/o >>> notification support on Heat side, Conductor would have to poll Heat >>> which is painful (dbomatic-like service presence on conductor side) >>> and not very effective. >> >> I agree dbomatic type service is error prone. However mirroring data > >from one service to another is a very difficult problem to solve well >> and have it be reliable. >> >> Is this required for some sort of reporting? If it is just for the > > Yes, reporting and keeping history logs about instances is part of > Conductor. Conductor also uses this information when choosing a > provider when launching an instance and also for quota checking. This could be done either way, but really you are just needing a tally of instances per user and per cloud. I'm not saying it is ideal but I wouldn't say it's impossible or even unwise to consider direct querying even here. >> user to view the states then that can be done on an as-needed basis by >> contacting heat. Even reporting is part of the AWS cloudformations API >> and events for stacks are supposed to be kept around for something like >> 90 days (iirc). >> >> Personally I very much question the need to mirror the data heat retains >> into Aeolus as these kinds of things are very error prone and difficult. >> Unless there is some kind of special need for reporting etc. the data >> could just as easily be queried directly. >> >> Ian >> > > Besides the needs listed above, I'm afraid there might be > performance issues with querying Heat directly per each user > request. I think we would have to try it and see. Ultimately it all comes from a database over a network connection. Querying heat only adds one more http request/socket layer. Perhaps some time munging data but all in all it's fairly light weight stuff.

I think it's important to really consider what is involved in duplicating

that data. It has been difficult in the past eg with dbomatic because it is a very hard problem. Even with events this doesn't change. What happens when a service goes down for a time? You need to replay events, read logs etc.. people spend a lot of time and a lot of

money trying to solve data replication. IMO if you can access the primary data source for your implementation that will be a far more

robust solution. Yes, this may mean having to sort out performance issues, but I think that is ultimately an easier problem. Ian

On Fri, Sep 07, 2012 at 02:35:39PM +0200, Jan Provaznik wrote: > On 09/04/2012 11:25 PM, Ian Main wrote: >> On Mon, Sep 03, 2012 at 12:44:05PM +0200, Jan Provazník wrote: >>> On 08/30/2012 10:23 PM, Ian Main wrote: >>>> On Thu, Aug 30, 2012 at 09:21:37AM +0200, Jan Provaznik wrote: >>>>> On 08/29/2012 09:55 PM, Ian Main wrote: >>>>>> On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: >>>>>>> On 08/21/2012 06:15 PM, Tomas Sedovic wrote: [snip] >> I think we would have to try it and see. Ultimately it all comes from a >> database over a network connection. Querying heat only adds one more >> http request/socket layer. Perhaps some time munging data but all in >> all it's fairly light weight stuff. >> > > Hi, sorry for late response, there was a deadline in last days... > > Honestly I don't think that it would be light weight stuff, Heat > will have sooner or later some authentication mechanism, also it > will do another query (multiple queries) into its own DB backend. As would Aeolus.. what I'm saying is the data and querying is similar in both cases, there's just 1 extra http connection between heat and aeolus. The trick would be getting the queries fine grained enough in heat to allow aeolus to make single requests to get the info it needs.

>> I think it's important to really consider what is involved in duplicating > > As I mentioned in the first mail in this thread Conductor needs > primarily information about instance changes, from other resources > it checks also realms availability (maybe we might need check hw > profile changes in future). > >> that data. It has been difficult in the past eg with dbomatic >> because it is a very hard problem. Even with events this doesn't >> change. What happens when a service goes down for a time? You need to >> replay events, read logs etc.. people spend a lot of time and a lot of > > Well, for callback-related issues were discussed in a thread about > DC tracker and it wouldn't be so hard (it would be combination of > retry+replay). This is what dbomatic had when it was using condor FYI.

> What about this: > instead of callbacks we could use a messaging queue (an AMQP > implementation, e.g. RabbitMQ). Integrate this into Heat and > Conductor would be matter of few seconds (there are clients for most > of programming languages). This would be quite robust solution. That would probably be a better idea. You will still need replay and I'm not sure you can ever say you are getting 100% of the events, but I think that is true with any solution. > The same messaging might be used also for sending notifications > between Conductor and Imagefactory. > >> money trying to solve data replication. IMO if you can access the >> primary data source for your implementation that will be a far more > > Don't forget that Heat is *not* primary data source (only cloud > provider is), And BTW, Heat will have to do the same "data > mirroring" (if you mean instance state checking by this term) itself > and if it's going to use dc-api you will probably end up with > polling providers. Yes very true, but I what I am saying is why do it more than needed? From what I've seen the real issue is actually ActiveRecord. The data model is so tightly tied into the logic and application that using some other data source becomes very difficult. I did some searching and I think what I would propose is using something like the Repository Pattern: http://msdn.microsoft.com/en-us/library/ff649690.aspx (Sorry it's an msdn page but it's the best description I could find.) The idea would be to create a layer to allow for storage of data in multiple places, including another web service. Perhaps this would be useful for the "pluggable backend" idea? Just a thought anyway, I'm probably in over my head now.. I'm no rails pro :).

Ian

On Mon, Aug 27, 2012 at 02:47:42PM +0200, Jan Provaznik wrote: > On 08/21/2012 06:15 PM, Tomas Sedovic wrote: [heavy snipping below] > >As a side benefit, we would get the current and future features of Heat > >for free: > > > >* high availability > >* auto-scaling > >* load-balancing I think these things would be really valuable additions to Aeolus. I confess that I haven't fully wrapped my head around how some of these things can best be adapted to a cross-cloud environments -- it seems like it would introduce a bunch of new complexity. But if it works reliably, I think it would be a fantastic feature to add! > nit: you probably meant 'deployment' instead of 'deployable' in the > paragraph above. It occurs to me that if we as developers (and I'm absolutely including myself here -- I'm a repeat offender) keep mixing these terms up, it might make sense to try to change them. The terms are similar and one morphs into another at a certain point, so it's very easy to mix them up even if you know the distinction inside and out. I can't imagine users find this any easier than we do. (This is really a digression from the real topic here, but I wanted to mention it.) > - the deployment launch setup becomes more complicated, a request > will pass through 3 standalone services: > Conductor -> Heat -> Deltacloud -> Provider I was noticing the other day that we don't do a good job of accurately catching and reporting errors from other components, or making it clear which component the problem is. So if you have, say, trouble building an image, you need to poke around a bit to figure out if it's a Conductor problem or a Factory problem, and then you need to figure out what the actual error is. Adding an extra component seems like it would make this even more complicated. I'm all for adding Heat, mind you. I just think that if we do, we're going to have to commit to good error handling and reporting, without leaving people to read multiple logs to figure out where the problem began.

-- Matt

On 08/21/2012 12:15 PM, Tomas Sedovic wrote: > Hey Folks, > > I've been contributing to the Heat project lately and I think it's > gotten to a point where we (the Aeolus community) could look at it and > figure out if it's something we want to use. > > Please forgive me the length of this email. I tried to make it as > short but there's a lot to cover. Grab your favourite beverage and sit > comfortably. > > > A little background > ------------------- > > Heat is a project that provides orchestration and high availability to > OpenStack. Its API is modelled after Amazon CloudFormation. > > What it does in Aeolus' terms is: you create a deployable and pass it > to Heat API along with any launch-time parameters. Heat will decide on > the correct order of the resources (instances, databases, load > balancers, remote storage, floating IP addresses, etc.) and launches > and configures the whole thing to produce a running deployment. > > Heat started about five months ago and hopes to become one of the core > OpenStack components. It's written in Python following the OpenStack > development model and tools. Is Heat implemented as a centralized python service? How does the user currently invoke that service and how does the service in return managed the openstack components?

Is there a hard dependency of openstack in heat? Will that be removed / made optional when the deltacloud backend is in place?

> > > What's this to do with Aeolus? > ------------------------------ > > We're trying to solve some of the same problems: > > * launching multiple instances in correct order > * passing parameters that are only known when an instance is launched > * configuring and advertising the services running inside the VMs Does Heat require anything to be installed on the instances themselves?

> > If we could outsource this to a project whose sole focus is solving > these issues, we could focus on things that make Aeolus unique: > building great UI, tools and APIs for managing cross-cloud deployments. > > As a side benefit, we would get the current and future features of > Heat for free: > > * high availability > * auto-scaling > * load-balancing > <snip> > > > ### Template format ### > > Aeolus uses a different format from Heat to describe deployables. > > We need to settle on what's the best thing to use. Options: > > 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON > > 2) Implement Aeolus Deployable XML format in Heat > > 3) Adopt CloudFormation JSON in Conductor > > 4) Have both Aeouls and Heat adopt another (open standard) format > > With all likelihood Heat will reject 2), we won't switch to 3) and > since *both* communities would have to accept and implement 4), we'll > probably stick to 1) for some time. > > Still, listing it here as it needs to be decided. Curious as to why you went w/ the CloudFormation JSON as the deployable descriptor format? Is this what openstack uses?

Supporting multiple deployable formats would be a nice feature in either application but obviously puts a burden on the developers / testers / etc. I'm not 100% familiar w/ the intricacies of the formats, but if they are simple enough and map well enough to each other, a barebones utility to do the conversion may be optimal for the time being.

> > > ### Heat-Conductor communication ### > > Heat is working on implementing a proper RESTful API that will > > 1) be more consistent with other OpenStack components > 2) suck less > > Once that's done, Conductor should probably adopt this API. > > We then need to figure out how to pass data from Heat back to > Conductor. If Conductor's launching instances via Heat, how does that > information get updated? > > 1) Conductor asks Heat every time it needs the data > > 2) Conductor asks Heat periodically and caches the results > > 3) Heat allows Conductor to register a callback -- either via HTTP > (webhooks) or via RPC -- and tells Conductor when stuff happens (e.g. > an instance was launched, crashed, etc.) > > 4) Conductor continues to use DBomatic to query the instances directly > via Deltacloud, ignoring any additional information Heat may provide > > For the proof of concept, everything other than 1) is a premature > optimization and then we'll see. > > I think 4) would be rather brittle especially since adopting Heat > might possibly let us drop DBomatic entirely. Some of this reminds me of condor, we were using that for a while in sort of an orchestration manner. It was responsible for tracking the lifetimes of instances on different cloud providers and dbomatic was initially written to track events through condor logs. It seems the level of abstraction is a bit different as Heat represents resources using the openstack model.

> > > ### Is Heat going to be the only way to launch deployments in > Conductor or will it be an optional part? ### > > We will need to see how stable and usable Heat is. How easy it is to > deploy and keep up (it would be another dependency, after all) and how > much benefit and code cleanup it brings to Aeolus. I'd be in favour of keeping this optional for the time being. It sounds like Heat would be a useful add on, but I'd rather keep the required dependency set as small as possible. Even external components that are currently in place should be made modular / pluggable: image management, config servers, system engines, etc.

-Mo

Sorry for the delay been heads down prepping for the conf. On 08/28/2012 08:04 AM, Tomas Sedovic wrote: >> <snip> >> >>> >>> >>> ### Template format ### >>> >>> Aeolus uses a different format from Heat to describe deployables. >>> >>> We need to settle on what's the best thing to use. Options: >>> >>> 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON >>> >>> 2) Implement Aeolus Deployable XML format in Heat >>> >>> 3) Adopt CloudFormation JSON in Conductor >>> >>> 4) Have both Aeouls and Heat adopt another (open standard) format >>> >>> With all likelihood Heat will reject 2), we won't switch to 3) and >>> since *both* communities would have to accept and implement 4), we'll >>> probably stick to 1) for some time. >>> >>> Still, listing it here as it needs to be decided. >> >> Curious as to why you went w/ the CloudFormation JSON as the deployable >> descriptor format? Is this what openstack uses? > > > There's been a precedent for that in OpenStack (they started with EC2 > API before moving to their own, iirc). > > More importantly, CloudFormation is reasonably well documented and > people are using it. > > Heat wants to provide an easy path for people using EC2 to switch to > OpenStack. CloudFormation (both the API and the format) is one piece > of the puzzle. I was thinking about this a bit, and while supporting the CloudFormation API makes sense to allow for switching as easy as possible, doesn't it just encourage end-users to use Amazon's API? (eg proprietary and what we're trying to move away from w/ Aeolus). What happens when Amazon changes their API?

It makes sense from OpenStack's perspective to use their own API, with EC2 compatibility as an extra feature supported on the side. I'm not against it, but apprehensive about how many proprietary APIs we expose on top of our open implementations.

On Wed, Sep 26, 2012 at 06:06:19PM +0100, Steven Hardy wrote: > On Wed, Sep 26, 2012 at 08:39:11AM -0400, Mo Morsi wrote: >> Sorry for the delay been heads down prepping for the conf. >> >> >> On 08/28/2012 08:04 AM, Tomas Sedovic wrote: >>>> <snip> >>>> >>>>> >>>>> >>>>> ### Template format ### >>>>> >>>>> Aeolus uses a different format from Heat to describe deployables. >>>>> >>>>> We need to settle on what's the best thing to use. Options: >>>>> >>>>> 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON >>>>> >>>>> 2) Implement Aeolus Deployable XML format in Heat >>>>> >>>>> 3) Adopt CloudFormation JSON in Conductor >>>>> >>>>> 4) Have both Aeouls and Heat adopt another (open standard) format >>>>> >>>>> With all likelihood Heat will reject 2), we won't switch to 3) and >>>>> since *both* communities would have to accept and implement 4), we'll >>>>> probably stick to 1) for some time. >>>>> >>>>> Still, listing it here as it needs to be decided. >>>> >>>> Curious as to why you went w/ the CloudFormation JSON as the deployable >>>> descriptor format? Is this what openstack uses? >>> >>> >>> There's been a precedent for that in OpenStack (they started with EC2 >>> API before moving to their own, iirc). >>> >>> More importantly, CloudFormation is reasonably well documented and >>> people are using it. >>> >>> Heat wants to provide an easy path for people using EC2 to switch to >>> OpenStack. CloudFormation (both the API and the format) is one piece >>> of the puzzle. >> >> I was thinking about this a bit, and while supporting the CloudFormation >> API makes sense to allow for switching as easy as possible, doesn't it >> just encourage end-users to use Amazon's API? (eg proprietary and what >> we're trying to move away from w/ Aeolus). What happens when Amazon >> changes their API? > > I think people starting from scratch on openstack (rather than migrating > from AWS) would want to actively avoid the Amazon API - there are plenty > of weird/bad aspects to the API, but we support it because it's a widely > recognised specification, which is fairly well docmented, and it also has > pretty good client-library support. > > The client-library point should help manage the API-changing problem, > since the AWS API is versioned, all we need to do is state which version > of their API specification we support, then hopefully the compatibility > problems can be handled in the client libraries (e.g projects like boto > for python) > >> >> It makes sense from OpenStack's perspective to use their own API, with >> EC2 compatibility as an extra feature supported on the side. I'm not >> against it, but apprehensive about how many proprietary APIs we expose >> on top of our open implementations. > > We've already refactored our internal interfaces to provide a more > generic interal API, and efforts are underway to support multiple API > variants - we will soon have an openstack-specific ReST API, as well as > the existing AWS-compatible Cloudformation API. This should provide a > cleaner interface for users who don't care about AWS compatibility, and > is being implemented in a similar style to existing openstack ReST APIs. > > Template format is a more difficult problem; heat currently only supports > AWS CFN template format, adding support for another template format would > be hard (but not impossible). We have already made some progress > decoupling heat core logic and interfaces from AWS naming and formats, > so taken to it's logical conclusion I can imagine the template validation > and translation could take place at the API level, leaving us with a > completely generic/open internal format (we are a long way from that > point now though..) > > Here is an approximation of the sort of architecture we are moving > towards, hopefully it illustrates that we're aiming for modularity and > separation: > > https://github.com/hardys/diagrams/blob/master/PNG/architecture-overview.png This is great stuff Steve. My understanding is that the OpenStack folks will want Heat to develop support for a non-AWS-specific API in addition to CloudFormations, for exactly the reasons you cite Mo. I believe it will be most successful if we have a converter for the common templates out there (CloudFormations, App Blueprints, TOSCA, whatever else there is) so that Heat itself can be completely agnostic.

I also believe that Conductor should expose an API that is compatible with Heat for instance lifecycle management -- in other words, people should be able to take templates that work with Heat and decide to run them through the Conductor abstraction layer instead -- whereupon Conductor will do all the stuff it is good at, like checking quotas and matching hardware profiles and choosing providers and so on, and then pass the template through to Heat with the appropriate account credentials supplied. What that API is is still somewhat up for discussion, so maybe we ought to start discussing it here? --Hugh