I still don't understand why we are using raw protocols which means I use one set of JS libraries for talking to AMQP and a second set for connector calls. JavaScript should just care about JSON and have a thin layer to request listening to topics which would come in as a json payload through a JS callback. This JS API would be the same no matter what backends we use and could even switch out transports (orbited socket or XMLHTTPRequest) depending on the server setup. AMQP just seems really low level. I would rather JS use the mechanisms it already has for communicating instead of relying on the client to encode and decode various protocols (and potentially triggering latent bugs in those protocols).
----- "Luke Macken" lmacken@redhat.com wrote:
Hey all,
I pushed out a bunch of commits last night that added a new MokshaHub, along with various AMQPHub implementations. I created the 'AMQPHub' skeleton class so we would know exactly the functionality we need/want from an AMQP module, and thus have the ability to swap them in-and-out in the future. I've attempted implementations of this using python-qpid (0.10 and 0.8) and amqplib -- the latter being the currently working version.
So, right now there is temporary hard requirement on RabbitMQ. I really liked how Orbited could utilize it's own built-in message broker out of the box, however it only speaks STOMP, not AMQP. So, I threw together some basic instructions for setting up RabbitMQ; it should Just Work out of the box.
http://lmacken.fedorapeople.org/moksha/main/RabbitMQ.html
RabbitMQ has a STOMP adapter, which is why we need it for now. Once the AMQP javascript bindings are complete, we'll be able to swap out RabbitMQ for *any* AMQP broker (most likely Qpid, in our case), and will also be able to start using version 0.10 of the AMQP spec (we're stuck with 0.8 due to RabbitMQ).
With regard to the AMQP javascript bindings, I've been talking with the Orbited/js.io guys about this for a while now. It seems that RabbitMQ already has implemented AMQP in javascript, but it's currently doing it over a JSON/HTTP RPC. Swapping out their 'JsonRpcService' for a 'TCPSocketService' should not be very difficult, and it will allow us to hopefully use any AMQP broker on the backend. For now, our widgets are stuck speaking STOMP to RabbitMQ.
luke _______________________________________________ moksha mailing list moksha@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/moksha
On Thu, Jan 22, 2009 at 01:44:32PM -0500, John Palmieri wrote:
I still don't understand why we are using raw protocols which means I use one set of JS libraries for talking to AMQP and a second set for connector calls. JavaScript should just care about JSON and have a thin layer to request listening to topics which would come in as a json payload through a JS callback. This JS API would be the same no matter what backends we use and could even switch out transports (orbited socket or XMLHTTPRequest) depending on the server setup. AMQP just seems really low level. I would rather JS use the mechanisms it already has for communicating instead of relying on the client to encode and decode various protocols (and potentially triggering latent bugs in those protocols).
So, the way I've designed the communication mechanism so far is pretty much identical to what you described. It's a thin layer of JS that hands off JSON messages to the app/widgets. Right now it speaks STOMP behind the scenes, and we can easily swap it out for AMQP in the future.
The reason for using AMQP is because of the way the browser consumes messages from the queues. Right now, widgets can consume messages from an AMQP queue over STOMP. However, they have no other capabilities, such as creating AMQP queues or exchanges, which means that we cannot properly do AMQP multicasting. Right now when a client consumes a message from a given queue -- no one else can consume that message from that particular queue. This is a huge roadblock, but there are a couple of ways around it...
The ideal solution, from my limited understanding of AMQP, is to have each client create their own personal (auto-deleting) queue, and bind it to a `fanout` exchange. This will allow for multicast AMQP messages.
In the mean time, we're stuck with the limited capabilities of STOMP->RabbitMQ from the browser side. Another potential short-term solution could be to create these unique queues for each user (with auto_delete=True, so they magically disappear when they aren't being used), bind them to the exchange, server side (within the update_params), and have their widgets listen to their own unique queues. This will allow the fanout exchange to properly multicast to multiple users.
I'm going to be sitting down with some AMQP / Qpid guys in the near future to discuss our setup, and see if they can provide some additional insight.
luke
So, I did a lot of thinking about this last night, and ended up refactoring a lot of code.
The main issue was, there are so many ways to facilitate message flow. The solution is to allow for message flows to be fully configurable.
What I started doing last night was breaking out all of the hub protocols into their own modules (amqp, stomp, etc.), and created some high-level interfaces (MessagingHub, BaseAMQPHub, etc). This gives us an idea as to what our main Hub expects in a messaging protocol.
I made the MokshaHub able to speak STOMP and AMQP. This will allow moksha to work out-of-the-box as a STOMP-only messaging hub. Then, when adding an 'amqp_host' to the configuration, the hub will automatically connect to it as well, and will proxy AMQP messages to STOMP. Eventually, down the road, you'll be able to remove the 'stomp_host', and be 100% AMQP-only. But being able to support all three use-cases seems extremely valuable to me; as some smaller deployments won't even care about AMQP.
For now, having the LiveWidget speak STOMP makes our lives much easier, as it automatically handles subscribing to topics, and multicast, etc.
Also, I'm going to be employing a high-level "Topic" concept within Moksha. This way, behind the scenes, these topics can be represented by whatever protocol-level method is best, from a stomp destination, to an AMQP queue/exchange, or whatever. This way, Widgets, Consumers, DataStreamers, Applications, etc. only have to care about sending/consuming messages from a "topic" -- without any protocol-level details getting in the way.
I'm also going to be modifying the LiveWidget today to make it protocol-agnostic. At the moment some STOMP-ish things kind of snuck in :)
Anyway, I committed a ton of stuff last night, and will continue to work on integration this weekend. My goal today is to make it trivial to hook up Qpid using the AMQP 0.10 spec.
luke