Re: [Libcloud-list] First shot at verbs (nouns to follow)
by Bob McWhirter
For anyone external w/o access to our clearspace, here's the content.
-Bob
Authentication Sessions
* Create a new authentication session - POST /sessions
* Destroy an authentication session - DELETE /sessions/${id}
Images
* Query collection of images - GET /images
o ec2-describe-images
* Retrieve a single image - GET /images/${id}
o ec2-describe-images
* Create an image [NOT SUPPORTED]
o ec2-bundle
o ec2-register
* Modify an image [NOT SUPPORTED]
* Delete an image [NOT SUPPORTED]
o ec2-deregister
Instances
* Query collection of instances - GET /instances
o ec2-describe-instances
* Retrieve a single instance - GET /instances/${id}
o ec2-describe-instances
* Create an instance - POST /instances
o ec2-run-instances
* Modify an instance - PUT /instances/${id}
o ec2-reboot-instances
* Delete an image - DELETE /instances/${id}
o ec2-terminate-instances
Security Policies
* Query collection of policies - GET /security/policies
o ec2-describe-group
o ec2-describe-keypairs
* Retrieve a single policy - GET /security/policies/${id}
o ec2-describe-group
o ec2-describe-keypairs
o ec2-fingerprint-key
* Create a policy - POST /security/policies
o ec2-add-group
o ec2-add-keypair
* Modify a policy - PUT /security/policies/${id}
o ec2-authorize
o ec2-revoke
* Delete a policy - DELETE /security/policies/${id}
o ec2-delete-group
o ec2-delete-keypair
Addresses
* Query collection of addresses - GET /network/addresses
o ec2-describe-address
* Retrieve a single address - GET /network/addresses/${id}
o ec2-describe-address
* Create a new address - POST /network/addresses
o ec2-allocate-address
* Modify an address - PUT /network/addresses/${id}
o ec2-associate-address
o ec2-deassociate-address
* Delete an address - DELETE /network/addresses/${id}
o ec2-release-address
Storage Volumes
* Query a collection of storage volumes - GET /storage/volumes
o ec2-describe-volumes
* Query a single storage volume - GET /storage/volumes/${id}
o ec2-describe-volumes
* Create a new storage volume - POST /storage/volumes
o ec2-create-volume
* Modify a storage volume - PUT /storage/volumes/${id}
o ec2-attach-volume
o ec2-detach-volume
* Delete a storage volume - DELETE /storage/volumes/${id}
o ec2-delete-volume
Storage Volume Snapshots
* Query a collection of storage volume snapshots - GET /storage/
snapshots
o ec2-describe-snapshots
* Query a single storage volume snapshot - GET /storage/snapshots/$
{id}
o ec2-describe-snapshots
* Create a new storage volume snapshot - POST /storage/snapshots
o ec2-create-snapshot
* Delete a storage volume - DELETE /storage/snapshots/${id}
o ec2-destroy-snapshot
On Jul 14, 2009, at 2:19 PM, Bob McWhirter wrote:
> Here's a first shot at delineating the verbs for libcloud. There's
> a big lack of what the contents of the nouns are (attributes on the
> resources) so far.
>
> But, this includes a mapping to the EC2 commands, at least initially.
>
> https://docspace.corp.redhat.com/clearspace/docs/DOC-22758
>
> Still not sure about docspace, but trying it anyhow.
>
> -Bob
>
> _______________________________________________
> Libcloud-list mailing list
> Libcloud-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/libcloud-list
14 years, 9 months
[Libcloud-list] Security, stateless; discuss!
by Bob McWhirter
I know it's probably an implementation detail, but always better to
consider security sooner rather than later.
Also, given that security is one way that cloud vendors differ, it's
something we should bake into the API correctly the first time through.
And yet also, given the desire for stateless drivers where possible,
it may affect our security decisions, depending on how we define
"stateless" or "security".
Three main models if security seen by me thus far are:
a) Some login() type of call, which takes a name and a password.
Many times the "password" is not your personal password associated
with web-console logins, but another, randomly-generated (hah!) key
that you can use for just API calls. The API key can be regenerated
if it's been compromised, without affecting your web-UI login
credentials. This login() returns a session token which is then
passed through to every other call. This is basically normal web
workflow with session ID in query string or a cookie, but with a
separate password.
b) no login(), but an API key passed with each request. The ultimate
service has no concept of sessions.
c) no login(), but each request is cryptographically signed. Once
again, no concept of sessions.
Does our desire for stateless drivers also include a desire to have no
session state? Or would that be an acceptable level of state the
implementor of the API needs to concern himself with?
If we insist on no sessions, and we're using, let's say, the adapter
that talks EC2 signed SOAP out the backside, how do we communicate our
EC2 credentials to the libcloud receiver? If we're doing credential
mapping, then I can login to the proxy as "bob", and it's already in
possession of my EC2 X.509 credentials, provided through some out-of-
band method (ie, a portal).
But, if I'm using the driver directly, on port 80, over REST... Do we
still insist on the driver having a store of credentials local to it?
Or would the client be expected to ship his X.509 certificate to the
driver with every request, so that the driver can sign the ultimate
EC2 SOAP requests on his behalf?
If we allow/require stateful sessions, then at least login(...) could
take a driver-specific hash of information to set up the security
context.
ie:
login
( :x509_cert=>'.....', :access_key=>'...', :secret_access_key=>'....' )
for the EC2 driver, which would expect those 3 members.
Yes, login() would actually be a PUT to /sessions, and be RESTful...
But basically, leave the schema for the Session resource as void*, or
implementation-defined, to allow for flexibility.
For name/password, login( :name=>'...', :password=>'...'), and for API
key-based, just login( :key=>'...' ) or such.
So, is session state too much state? I think it'd be okay to store in
RAM, no guarantee on longevity, and requires no local long-term storage.
Thanks,
-Bob
14 years, 9 months
[Libcloud-list] another libcloud
by Alex Polvi
Hi,
I was excited when I found your project. Not only does it have similar
goals, but it has the same name:
http://github.com/polvi/libcloud/
We're essentially trying to do the same thing ... that is, develop a
unified interface, plus a set of drivers, so that we only have to
develop against one API. I read through the archives, and I think
we're on the same track.
Right now, my approach has been to keep things as simple as possible.
The library will give you an api client for whatever providers (only
ec2 and a dummy provider, atm) it supports. The methods (only
list_nodes, so far) on the client will be identical. The end result
will be a library, written in python, that you could expose via a REST
API or whatever you wanted.
Here is a very basic example:
http://github.com/polvi/libcloud/blob/4ea159380dfa966f15711602c47d37a37f6...
Anyway, next week I'm speaking at OSCON about this topic (cloud
standards, portability). Would love to mention this if possible. Also,
if anyone is going to be around OSCON, we should team up and try to
get some more momentum!
-Alex
--
co-founder, cloudkick.com
twitter.com/cloudkick
541 231 0624
14 years, 9 months
[Libcloud-list] libcloud as SOA
by Bob McWhirter
Further discussion in #libcloud has lead us to think (but no decisions
yet!) about instead of "drivers" being implementation details, but
rather have them be running services that directly speak libcloud-API
on one side, and the cloud-native API on the other.
In this case, Steve the Scripter would fire up the driver service he's
using today, perhaps on localhost, and work with a cloudsh or other
client that talks REST to the localhost-hosted service.
For the value-add of the previously discussed generic libcloud-
service, which might do credential mapping and cloud multiplexing, it
would be another cloud that speaks libcloud-API on the front, but also
talks libcloud-API out the back ultimately to the real driver services.
In this case, libcloud-service would act more as a proxy/aggregator,
and not directly drive the drivers.
One benefit is the RHEV-M driver would be able to run fully on the
win32 machine, colocated with RHEV-M. It'd be a separate process from
RHEV-M, but would not require a "driver" to be placed in libcloud-
service AND a shim placed on the win32 machine. The full driver would
be a single process running on the win32 box, aggregated by the
libcloud-service.
If libcloud-service is a proxy, we *could* end up chaining them
together, such as a libcloud-service for each department, which talks
to its own local clouds AND also the libcloud-service for the
corporation. The libcloud-service for the corporation would then also
talk to its own clouds.
If each driver is really a service, and we stitch them together, we're
SOA buzzword compliant. And each driver can be authored in the
language most appropriate for that driver.
In previous discussion, if libcloud-service directly spoke to the
drivers in-process, that severely restricts the implementation choices.
Here's some diagrams:
http://fnokd.com/~bob/libcloud-soa-1.png
The first represents Steve the Scripter who just wants a homogenous
API he can script against regardless of which cloud he's really
using. It would require 1 service running, possibly on localhost (but
not required!).
The bottom half of that diagram shows a more complex configuration,
where a proxying/aggregating value-add libcloud-service stands between
Don the Developer and several clouds available simultaneously through
the service.
Each driver may be on the same host, or may be exposed by the cloud
provider natively, or distributed within the organization as
appropriate (ie, RHEV-M's driver sitting directly on a win32 machine,
everything else is scattered on linux boxen).
To make life easy for folks who are happy to use whatever language
we've picked, we can also provide a libcloud-service-framework. This
would be a template implementation of a driver, lacking the actual
guts that do real work, but otherwise handling the HTTP and REST
aspects.
http://fnokd.com/~bob/libcloud-framework-1.png
Folks could simply ignore it, though, and do their own implementation
between libcloud-API and native-API. It's just an optional chunk to
help driver implementors if they so choose.
-Bob
14 years, 10 months
[Libcloud-list] Stateless v. stateful
by David Lutterkort
Hi,
I was just chatting with Hugh, and we went another round or two on the
'Is libcloud stateless or stateful?' question.
Mostly to clear my head, some thoughts on that: there's two usage
scenarios I'd want to capture with libcloud
1. Steve the Scripter writes a few Python scripts to make his
miserable, awful, pitisome existence as a developer easier; some
of these scripts need to start/stop virtual machines in the
cloud (maybe a JBoss cluster from Ivan the Integrator) - Steve
uses libcloud to make sure that he can move from cloud A to
cloud B without having to rewrite all those scripts.
2. Don the Developer is writing a cloud portal UI. He uses libcloud
in the backend of his portal so that he can claim 100%
compatibility with N (fslvo of N) popular clouds.
I think these two uses cover a large percentage of the current EC2 API
uses; they differ tremendously in the amount of deployment pain Don and
Steve are willing to go through - Steve is perfectly happy with a very
basic API, but would prefer libcloud setup be as simple as possible,
ideally just installing a library, while Don wants to enable lots of
spiffy features in his cloud portal UI (SSO, accounting, image
mgmt, ...) and is happy with a more complex setup involving lots of
auxiliary services (DB, web server, ...).
To serve Steve, we'd want a stateless, thin library; Steve might have to
use different ways to authenticate from his script for different clouds,
and therefore change that part of his script when he switches to a
different cloud backend, but he'll be happy with that as long as those
changes are localized and don't require a big rewrite of his scripts.
IOW, authentication should be the only area where he needs to be aware
of the concrete backend he's using - this usage is very close to what
libvirt does.
Ideally, libcloud deployment for Steve involves little more than
installing an RPM (and maybe starting a service) - but it should
definitely not require setting up a fullblown DB server, manually
loading schema etc.
How should such a stateless API be exposed ? I think an in-process
DSO-style API is out of the question, since it would either require
doing SOAP calls from C (yuck) or tying the API to a specific
implementation language. That means that it needs to be exposed by some
RPC mechanism, i.e. will require running a server; and since we already
want to expose a REST API, some sort of web server makes the most sense.
To make both Don and Steve happy, we should consider either two
different ways to run the server: stateless with reduced functionality
as a fairly direct API-passthrough, and stateful with value-add features
like accounting etc. 'Stateless' might actually not be completely
stateless: we could still offer simple deployment if we use a sqlite
backend to store some basic data like auth credentials for different
clouds. (Completely stateless has the disadvantage that backend cloud
credentials only live for the lifetime of the server, which leads to
interesting error handling for the client)
For both Don and Steve a very basic session to start a bunch of VM's
might look like
1. Authenticate to libcloud
2. Register credentials for backend clouds
3. List available backend clouds[1] (Steve has Alzheimer's and
forgot what happened in step (2))
4. List available templates[2] for each backend cloud
5. Run some VM's based on the available templates
6. Shut the VM's down
David
[1] At that point, a cloud is an abstraction that consists of things
like a cost/billing model and resource quotas; what specific cloud we
are talking about should be immaterial
[2] This is one place in the API where we need to decide whether we want
to expose the concept of 'VM image' as distinct from 'VM template' in
the API
14 years, 10 months
[Libcloud-list] Stateful vs Stateless VMs
by Bob McWhirter
Not to confuse with stateful vs stateless API....
As we know, EC2 has no state for the instances. If you power-off and
terminate a machine, it's gone. You can't start up the exact same VM
again. You *can* start up another copy of the same image, but it'll
lack any changes you made, of course.
And I consider that to be a Good Thing, when doing cloud computing.
If some virtualization fabric provides stateful machines (ie, RHEV-M,
VMware), I think it's still best to ignore that if you're truly
wanting to deploy using "the cloud methodology".
My view of the cloud is based upon homogeneity, particularly if you're
wanting to scale up/down over time, or have instances simply fail.
For example, in our JBoss demo, we start with a cluster of 2 AS
nodes. As demand increases, we want to add 2 more node *just like*
the existing 2 nodes. As demand falls, we want to be able to kill off
any 2 of the now 4 nodes, to conserve resources.
If we assume statefulness of the VMs, it seems we're no longer
thinking in terms of commodity/utility computing.
I do like EC2's EBS, though, for maintaining data, which should not be
ephemeral. Glue that to your DB node. If the DB node fails, mount
the storage partition on another copy of the DB node. Or mount it on
an updated version of the DB node.
Ultimately, if you develop your images (or portfolio of cooperating
images) to behave well in a stateless environment, I think you'll end
up follow more "best practices" in terms of fail-over, scalability and
management.
If you develop assuming your nodes are stateful, you're also more apt
to "park" them somewhere "just in case". Or end up being too reliant
upon a very particular instance as it's evolved over time.
Assuming statelessness helps ensure you separate compute resources
from storage resources, and have something you can spin up multiple
times in a meaningful way. Makes it more cloudy above and beyond just
virtualizationy.
But that's just me, feel free to disagree.
-Bob
14 years, 10 months
[Libcloud-list] front-end/back-end authentication
by Bob McWhirter
wrt authentication...
I think users should authenticate against libcloud, using a pluggable
auth method. CompanyOne might have it wired up to LDAP for users to
auth. CompanyTwo might use Kerb or RSA or whatever. CompanyThree
might use a libcloud-specific Users table with local password hashes
stored for use only with libcloud.
Then a strategy to map inbound consumers of libcloud to outbound
credentials for the various backend drivers. CompanyOne might have a
1:1 mapping of libcloud user to back-end user. For instance, sending
*my* particularly EC2 credentials out with commands to EC2.
CompanyOne handles 230 people sending in monthly expense reports for
their individual AWS bills.
CompanyTwo might have a many:1 mapping, where "bob" logs into
libcloud, but uses a singular corporate-wide credential to fire up
stuff on EC2. libcloud tracks the accounting for that. While A/P
pays a single bill for $2000, they know that "bob" used $42 of it.
CompanyThree might have a dept:1 mapping, where "bob" is a member of
JBoss, and uses JBoss EC2 creds, and "bryan" is a member of 'Red Hat'
and uses Red Hat creds. That results in a bill from AWS for each
department, instead of individual.
Some backends may require no authentication, where libcloud is the
sole source of accounting. For internal datacenter clouds, for
instance, running VMware ESXi of vSphere or whatever. "bob" used 400
hours of CPU time, and his department's budget is reduced by some
arbitrarily determined "cost" per CPU hour.
Depending on the proxying strategy chosen, all users may already be
enrolled in the clouds, or they may have to inform libcloud "here are
*my* credentials for a particular backend" before that provider is
enabled for them.
Next layer up, libcloud could restrict some users from using some
backends. Maybe everyone can use the EC2 backend, but only IT can use
the internal production cloud backend.
Which also brings up another point, which I think might relate to
"clusters" in RHEV-M if I understand them correctly. While we might
have "the RHEV-M driver", which may have different clouds powered by
that driver. The QA cluster, the IT production cluster, etc. That
can wait for v2 of libcloud, though. :)
-Bob
14 years, 10 months
[Libcloud-list] High Level Use Cases
by Bryan Kearney
Here are some high level use cases to get us discussing the boundaries
of libcloud. There are some things in here which may or may not be part
of libcloud, and the anbiguity should be good to help drive the boundaries.
Actor:
Jerome the Application Deployer: Owns an applications, responsible for
the SLA of the application
Use Case: Enroll with the Cloud
-------------------------------
Jerome chooses to use CloudCo as his cloud provider. Jerome enrolls with
CloudCo providing them his credentials and his billing information.
Jerome is provided with an additional set of credentials which he must
use when authenticating with CloudCo.
Use Case: Test Application
----------------------------
Jerome bundles an application with Fedora to create his test image. He
loads the image into the cloud. His application is database intensive,
so he launches the image and associates a data disk to the instance
where DB files are to be located. He sets an FQDN for the instance, and
uses this FQDN to do testing. Jerome finds an issue with the image, so
he updates the image in the cloud as well as his internal gold copy.
Once testing is complete, Jerome generates a new gold image and reloads
a second version of the image. He then performs another round of
testing. Once satisfied, Jerome leaves the instance running for 1 week
for his users to sanity test the application.
Use Case: Simple Deployment
--------------------
Jerome is satisfied with the gold copy of the instance. Jerome clones
the image to create a new image for production, and assigns a new FQDN.
Jerome associates new storage with the production image for the
production database. Jerome configures automated backup of the database,
and sets up monitoring to inform him when the image stops working.
Note: Not dealing with multi-tier applications with firewalls or
clustering.
Use Case: Reporting
-----------------------
Jerome is required to maintain a 3 nines uptime. On the first of each
month, Jerome queries for the uptime of the image in order to generate
his statistics. Jerome also queries disk, bandwidth, and memory usage in
order to generate his chargeback reports.
Use Case: Decommissioning
------------------------
CloudCo has only allowed Jerome to achieve a 95% uptime for 3 months.
Jerome decides to go with iCloud instead of CloudCo. Jerome has enrolled
with iCloud,and exports his image and data files from CloudCo and loads
them into iCloud.
Other Ideas Not covered here
----------------------------
- granting access to images.
- firewalls, LANS,
- clustering
- Recovery
-- bk
14 years, 10 months
