moksha January 2009

moksha@lists.fedorahosted.org

4 participants
7 discussions

Meeting Notes for Jan 26, 2009
by Tom Callaway 30 Jan '09

30 Jan '09

Attendees: Luke Macken, John "J5" Palmieri, Mo Duffy, Tom "spot" Callaway Agenda: * Status check on Moksha, where do we stand with our Feb 1 goal target? - Moksha now structured around configurability with various message brokers, does STOMP out of the box - Widget Browser still incomplete, should be done by the end of the week - Module Connectors still incomplete, should be done soon - Koji code done, needs documentation - Bodhi code done, needs documentation - PackageDB code almost done - J5 will send a follow-up email when this code is finished and committed - Extension Points are in progress, should be ported by the end of the week - Containers and tabs need to be documented in the near future - URL Rewriting is incomplete, J5 will open a trac ticket describing what needs to be done * Luke will set up moksha-commits-list and send an email to the moksha list * Mo completed some additional mockups, but still has more to do: - Package Details Mockups - User Profile Mockups - Cleanup front page mockups, both pre-login and post-login, specifically to remove old concepts such as the login box. ~spot

2 2

Message flow
by Luke Macken 26 Jan '09

26 Jan '09

I just checked in support for configurable message flows within Moksha. Out of the box, Moksha does not require an AMQP broker, and will work using the STOMP protocol through Orbited's built-in MorbidQ message broker. I created the high-level idea of a "Topic" within Moksha, which allows applications/consumers/widgets to listen and send messages to them, without having to care about the underlying message protocol or broker. To add support for an AMQP broker, all you have to do is add the following line to your configuration file (tested with Qpid and RabbitMQ): amqp_broker = guest/guest@localhost The MokshaHub will then automatically connect up to your AMQP broker, and will consume all messages from it, forwarding them to the STOMP broker, which will then be consumed by the users web browser. I'll throw together some documentation for this shortly... Cheers, luke

1 0

Re: AMQP status
by John (J5) Palmieri 24 Jan '09

24 Jan '09

I read your second mail already but I wanted to comment on some of the stuff you said in this just so when you sit down with the AMPQ guys we get a clear picture of capabilities (I would like to be in on that discussion BTW, given my experience dealing with message buses). ----- "Luke Macken" <lmacken(a)redhat.com> wrote: <snip/> > > The reason for using AMQP is because of the way the browser consumes > messages from the queues. Right now, widgets can consume messages > from > an AMQP queue over STOMP. However, they have no other capabilities, > such as creating AMQP queues or exchanges, which means that we cannot > properly do AMQP multicasting. I'm not quite sure I understand the multicast bit but in no way do we want to allow a client to create queues from JavaScript. We want them to consume from already existing queues. If they want to request something they go through a connector. AMQP is simply there for push publishing (e.g. signals that an event has happened). > Right now when a client consumes a > message from a given queue -- no one else can consume that message > from > that particular queue. This is a huge roadblock, but there are a > couple > of ways around it... That seems off. Are you sure you are using the correct portion of the spec? Any message bus should be able to distribute a message to anyone who can subscribe to it. Imagine a world where a stock gets updated and only one person can see that message. I think I see the issue looking at some client code. There are Brokers and Queues. Brokers are the bus, anytime they see a message they look to see if any queue is subscribed to that message and put it in that queue. Brokers are inherently multi-casting. More comments bellow. > The ideal solution, from my limited understanding of AMQP, is to have > each client create their own personal (auto-deleting) queue, and bind > it > to a `fanout` exchange. This will allow for multicast AMQP messages. I don't think a fanout is you think it is. It is simply an exchange that ignore routing options when publishing to a queue. e.g. every queue in the exchange gets the message whether or not they requested to listen to it. What we want is for moksha.EventService (or something to that effect) acting as an exchange with one queue per route it is listening to. In effect this would be a queue on the Fedora exchange. The event service would take requests to listen for a topic from live widgets and store their orbitted channel id in a table that associates it with a list of topics (well actually it would be a topics table with a list of orbitted channels). If the topic comes up you send the contents to all the channels which are subscribed. If the channel gets close you take it out of the table. Alternately, you create a queue for each orbitted connection and add routing criteria for listen request from the client. When a message comes into a queue, it gets shipped off to the connection associate with it. The client should only ever care about being able to subscribe to a topic and then have a callback get triggered with the message's payload as a parameter. The client code would look something like this: es = moksha.get_event_service() // perhaps we allow multiple event services with noargs = get default es.subscribe('koji.build.status_changed', status_changed_cb) es.subscribe('koji.build.new_build', new_build_cb) es.subscribe('bodhi.*', bodhi_events_cb) //get all bodhi events def status_changed_cb(json) { alert('koji sent us a status change for build ' + json.build.build_id.toString()); } def new_build_cb(json) { alert('A new build has just started for package ' + json.build.nvr); } def bodhi_events_cb(json){ alert('Bodhi just sent us the ' + json.topic + ' event'); } Our AMQP setup in Fedora would look like: koji, bodhi, fas, pkgdb, etc AMQP producer clients. _____________________________ | Fedora AMQP Exchange | | | | | |Fedora Community | | | |serverside queue | |____| |______________________| | | \|/ ____| |______________________ | | | Moksha AMQP | | | | Client | | | | FC Local Queue | | | | | \|/ | | lookup_conn_for_topic() | |_____|_______________________| / \ / \ |/ \ _________ | |live | | |widget | | |topic | | |consumer | | |#1 | | |_________| | ___\|/___ |live | |widget | |topic | |consumer | |#2 | |_________| -- John (J5) Palmieri Software Engineer Red Hat, Inc.

2 2

Re: AMQP status
by John (J5) Palmieri 23 Jan '09

23 Jan '09

I still don't understand why we are using raw protocols which means I use one set of JS libraries for talking to AMQP and a second set for connector calls. JavaScript should just care about JSON and have a thin layer to request listening to topics which would come in as a json payload through a JS callback. This JS API would be the same no matter what backends we use and could even switch out transports (orbited socket or XMLHTTPRequest) depending on the server setup. AMQP just seems really low level. I would rather JS use the mechanisms it already has for communicating instead of relying on the client to encode and decode various protocols (and potentially triggering latent bugs in those protocols). ----- "Luke Macken" <lmacken(a)redhat.com> wrote: > Hey all, > > I pushed out a bunch of commits last night that added a new > MokshaHub, > along with various AMQPHub implementations. I created the 'AMQPHub' > skeleton class so we would know exactly the functionality we > need/want > from an AMQP module, and thus have the ability to swap them > in-and-out > in the future. I've attempted implementations of this using > python-qpid > (0.10 and 0.8) and amqplib -- the latter being the currently working > version. > > So, right now there is temporary hard requirement on RabbitMQ. I > really > liked how Orbited could utilize it's own built-in message broker out > of > the box, however it only speaks STOMP, not AMQP. So, I threw > together > some basic instructions for setting up RabbitMQ; it should Just Work > out > of the box. > > http://lmacken.fedorapeople.org/moksha/main/RabbitMQ.html > > RabbitMQ has a STOMP adapter, which is why we need it for now. Once > the > AMQP javascript bindings are complete, we'll be able to swap out > RabbitMQ for *any* AMQP broker (most likely Qpid, in our case), and > will > also be able to start using version 0.10 of the AMQP spec (we're > stuck > with 0.8 due to RabbitMQ). > > With regard to the AMQP javascript bindings, I've been talking with > the > Orbited/js.io guys about this for a while now. It seems that > RabbitMQ > already has implemented AMQP in javascript, but it's currently doing > it > over a JSON/HTTP RPC. Swapping out their 'JsonRpcService' for a > 'TCPSocketService' should not be very difficult, and it will allow us > to > hopefully use any AMQP broker on the backend. For now, our widgets > are > stuck speaking STOMP to RabbitMQ. > > > luke > _______________________________________________ > moksha mailing list > moksha(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/moksha -- -- John (J5) Palmieri Software Engineer Red Hat, Inc.

2 2

AMQP status
by Luke Macken 22 Jan '09

22 Jan '09

Hey all, I pushed out a bunch of commits last night that added a new MokshaHub, along with various AMQPHub implementations. I created the 'AMQPHub' skeleton class so we would know exactly the functionality we need/want from an AMQP module, and thus have the ability to swap them in-and-out in the future. I've attempted implementations of this using python-qpid (0.10 and 0.8) and amqplib -- the latter being the currently working version. So, right now there is temporary hard requirement on RabbitMQ. I really liked how Orbited could utilize it's own built-in message broker out of the box, however it only speaks STOMP, not AMQP. So, I threw together some basic instructions for setting up RabbitMQ; it should Just Work out of the box. http://lmacken.fedorapeople.org/moksha/main/RabbitMQ.html RabbitMQ has a STOMP adapter, which is why we need it for now. Once the AMQP javascript bindings are complete, we'll be able to swap out RabbitMQ for *any* AMQP broker (most likely Qpid, in our case), and will also be able to start using version 0.10 of the AMQP spec (we're stuck with 0.8 due to RabbitMQ). With regard to the AMQP javascript bindings, I've been talking with the Orbited/js.io guys about this for a while now. It seems that RabbitMQ already has implemented AMQP in javascript, but it's currently doing it over a JSON/HTTP RPC. Swapping out their 'JsonRpcService' for a 'TCPSocketService' should not be very difficult, and it will allow us to hopefully use any AMQP broker on the backend. For now, our widgets are stuck speaking STOMP to RabbitMQ. luke

1 0

Presentation of Fedora Community available
by John (J5) Palmieri 20 Jan '09

20 Jan '09

Hello all. We just put out a high level presentation of Fedora Community which includes information on where we are going, where we are at, and where we plan to be in the short term. Grab it at http://johnp.fedorapeople.org/fedoracommunity_intro.pdf You can also get my lower level FUDCon presentation at http://johnp.fedorapeople.org/fedoracommunity_fudcon_presentation.pdf -- John (J5) Palmieri Software Engineer Red Hat, Inc.

1 0

Monday Meeting
by Tom Callaway 05 Jan '09

05 Jan '09

I've got to work from home on Monday, so if we're having a Monday meeting, we'd have to do it over the telephone. We haven't really announced a public meeting for monday, so my inclination is to push it to next Monday and have a more public announcement of the dial-in and IRC channel, along with the announcement text that J5 worked up. Thoughts? ~spot

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

moksha January 2009