11:16 a.m.
Hey folks!
Jeremy and I have been working on a proposal to migrate fedmsg from our
current brokerless architecture to a broker-based architecture.
The overview and reasons for the migration are described on this page:
https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overvie...
Head there if you want the details. The plan has the following requirements:
* No flag day.
* Don't disrupt any services or applications.
* Don't break any services outside of Fedora's infrastructure relying on
these messages.
The first step is to deploy a broker in Fedora to use. In order to avoid a
flag day, bridges from AMQP to ZeroMQ and ZeroMQ to AMQP have been
implemented and will be deployed.
In order to validate that the bridges are functioning, a small service will
be run during the transition period that connects to fedmsg and to the AMQP
queues to compare messages.
After the bridges are running, applications are free to migrate. There are
several options when migrating and each has advantages and disadvantages.
We have written a new library called fedora-messaging that has the
following features:
* A method to define message schemas and offer automatic validation of
messages using those schemas.
* Boilerplate for typical publishers and consumers.
Head over to the document for a demo!
What do you think of this proposal? Any blind spots?
Thanks!
Aurélien
3:54 p.m.
Follow-up to this, Patrick had a few questions on IRC, which I've copied
here and answered in case others are interested:
I was wondering whether you intend to continue cryptographically
(x509) signing messages, or if you were planning to enforce sender per
subject in another way?
Ultimately, no. RabbitMQ provides access controls[0] which I think will
meet our needs.
However, the bridge from AMQP back to ZeroMQ will support signing the
messages so consumers outside of Fedora Infrastructure are not broken.
My expectation is we'll run this bridge long term as the way for
external consumers to get events, even if we don't use ZeroMQ
internally. ZeroMQ is a solid library and it feels like a good fit
for the public access use-case. The bridge is ~10-20 lines of code
so it's not a huge maintenance burden, either.
I would like to eventually drop the message signing completely and
replace it with the a ZeroMQ socket with zmq-curve[1] for
authentication. That lets us stop using fedmsg completely (which is
appealing because it depends on pyOpenSSL which is not long for this
world).
is the plan to move the projects to the fedora-infra org in the long
run, or was your plan to keep them under your personal account?
Definitely planning on moving it over if people like it.
[0] https://www.rabbitmq.com/access-control.html
[1] http://api.zeromq.org/4-2:zmq-curve
--
Jeremy Cline
XMPP: jeremy(a)jcline.org
IRC: jcline
1:17 p.m.
On Thu, May 24, 2018 at 10:16 AM, Aurelien Bompard
<abompard(a)fedoraproject.org> wrote:
What do you think of this proposal? Any blind spots?
Thanks!
This sounds like a great idea. I use ActiveMQ and RabbitMQ at work,
and both are good choices for brokers. Sounds like you're going
towards RabbitMQ.
- Ken
8:31 a.m.
On Thu, May 24, 2018 at 11:16 AM, Aurelien Bompard <
abompard(a)fedoraproject.org> wrote:
What do you think of this proposal? Any blind spots?
Not that I disagree, but please add/expand a section as to why AMQP (and
RabbitMQ) was chosen over other messaging technologies.
--
Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.
2:51 p.m.
Hi,
On 05/29/2018 09:31 AM, Jeffrey Ollie wrote:
On Thu, May 24, 2018 at 11:16 AM, Aurelien Bompard <
abompard(a)fedoraproject.org> wrote:
>
> What do you think of this proposal? Any blind spots?
>
Not that I disagree, but please add/expand a section as to why AMQP (and
RabbitMQ) was chosen over other messaging technologies.
Thanks for the feedback, I've added a small section[0]. It is, perhaps,
a little wishy-washy. I don't want to give the impression that we
couldn't implement this with a different messaging protocol or a
different broker. We definitely could. AMQP has short-comings, to be
sure, but the RabbitMQ extensions (mainly pulisher acks) cover the most
important ones in my opinion.
I did some research, but I'd definitely welcome feedback on protocols
and brokers. I've read all or nearly all of the AMQP 0.9, ZeroMQ, and
STOMP protocols, and I skimmed through the MQTT protocol, but I've not
looked closely at the AMQP 1.0 protocol and I'm by no means a message
protocol expert.
As for brokers, my only experience is with Qpid and RabbitMQ and that
experience points me firmly at RabbitMQ. I don't know much about running
ActiveMQ.
[0]
https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overvie...
Thanks,
Jeremy
4:45 p.m.
On Tue, May 29, 2018 at 12:51 PM, Jeremy Cline <jeremy(a)jcline.org> wrote:
Hi,
On 05/29/2018 09:31 AM, Jeffrey Ollie wrote:
> On Thu, May 24, 2018 at 11:16 AM, Aurelien Bompard <
> abompard(a)fedoraproject.org> wrote:
>
>>
>> What do you think of this proposal? Any blind spots?
>>
>
> Not that I disagree, but please add/expand a section as to why AMQP (and
> RabbitMQ) was chosen over other messaging technologies.
Thanks for the feedback, I've added a small section[0]. It is, perhaps,
a little wishy-washy. I don't want to give the impression that we
couldn't implement this with a different messaging protocol or a
different broker. We definitely could. AMQP has short-comings, to be
sure, but the RabbitMQ extensions (mainly pulisher acks) cover the most
important ones in my opinion.
I did some research, but I'd definitely welcome feedback on protocols
and brokers. I've read all or nearly all of the AMQP 0.9, ZeroMQ, and
STOMP protocols, and I skimmed through the MQTT protocol, but I've not
looked closely at the AMQP 1.0 protocol and I'm by no means a message
protocol expert.
I think moving to a broker-based architecture is a great idea! Your
document does a great job explaining the advantages it brings, and it could
help increase the adoption of event-based workflows.
Regarding protocols, my preference would be for STOMP. It's has very wide
support, with libraries in pretty much every language, and being entirely
text-based makes it *much* easier to debug than other protocols. The
message delivery semantics are well-defined, and the protocol spec has the
nice property of being readable in one sitting. Some brokers provide the
ability to translate between protocols, so it may not be difficult to
support more than one, but I would suggest STOMP as the reference protocol.
As far as brokers, I'll mention that we use ActiveMQ internally and it has
been performing well for us. There may be some value to standardization
there, and we may be able to provide some resources and best-practices on
configuration and maintenance. I don't know much about RabbitMQ.
As for brokers, my only experience is with Qpid and RabbitMQ and that
experience points me firmly at RabbitMQ. I don't know much about running
ActiveMQ.
[0]
https://fedmsg-migration-tools.readthedocs.io/en/
latest/migration/overview.html#why-amqp-and-rabbitmq
Thanks,
Jeremy
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave@lists.
fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/
infrastructure(a)lists.fedoraproject.org/message/
2ARVA7MSRJ5NALJL2O5CGWB4BGCKJ67S/
9:30 a.m.
On 06/01/2018 05:45 PM, Michael Bonnet wrote:
On Tue, May 29, 2018 at 12:51 PM, Jeremy Cline
<jeremy(a)jcline.org> wrote:
> Hi,
>
> On 05/29/2018 09:31 AM, Jeffrey Ollie wrote:
>> On Thu, May 24, 2018 at 11:16 AM, Aurelien Bompard <
>> abompard(a)fedoraproject.org> wrote:
>>
>>>
>>> What do you think of this proposal? Any blind spots?
>>>
>>
>> Not that I disagree, but please add/expand a section as to why AMQP (and
>> RabbitMQ) was chosen over other messaging technologies.
>
> Thanks for the feedback, I've added a small section[0]. It is, perhaps,
> a little wishy-washy. I don't want to give the impression that we
> couldn't implement this with a different messaging protocol or a
> different broker. We definitely could. AMQP has short-comings, to be
> sure, but the RabbitMQ extensions (mainly pulisher acks) cover the most
> important ones in my opinion.
>
> I did some research, but I'd definitely welcome feedback on protocols
> and brokers. I've read all or nearly all of the AMQP 0.9, ZeroMQ, and
> STOMP protocols, and I skimmed through the MQTT protocol, but I've not
> looked closely at the AMQP 1.0 protocol and I'm by no means a message
> protocol expert.
>
I think moving to a broker-based architecture is a great idea! Your
document does a great job explaining the advantages it brings, and it could
help increase the adoption of event-based workflows.
Regarding protocols, my preference would be for STOMP. It's has very wide
support, with libraries in pretty much every language, and being entirely
text-based makes it *much* easier to debug than other protocols. The
message delivery semantics are well-defined, and the protocol spec has the
nice property of being readable in one sitting. Some brokers provide the
ability to translate between protocols, so it may not be difficult to
support more than one, but I would suggest STOMP as the reference protocol.
I had a hard time justifying choosing STOMP over AMQP because most
brokers just map the other protocol they focus on onto STOMP. It's true
the the spec is short, but it leaves a lot up to individual
implementations as far as I can tell (like how topic matching works, for
example).
While debuggability is important, I'm not certain we'll ever need to dig
into the wire protocol. In the unlikely event that we need to, I'd go
about it the same way (e.g. tcpdump/wireshark) and Wireshark knows how
to parse the AMQP protocol. Based on my super simple test (capture a
single message being published) it seems very easy to inspect. Have you
had a different experience here?
--
Jeremy Cline
XMPP: jeremy(a)jcline.org
IRC: jcline
5:59 p.m.
On Mon, Jun 4, 2018 at 8:30 AM, Jeremy Cline <jeremy(a)jcline.org> wrote:
I had a hard time justifying choosing STOMP over AMQP because most
brokers just map the other protocol they focus on onto STOMP. It's true
the the spec is short, but it leaves a lot up to individual
implementations as far as I can tell (like how topic matching works, for
example).
It's nice to give the flexibility to clients by exposing both. I
haven't seen a problem with topic matching in my experience so far.
One thing I found with AMQP vs STOMP is that it's possible for AMQP
clients to (accidentally) emit "binary" message bodies, and then
ActiveMQ does not translate or expose these as plaintext JSON for
STOMP clients. It just looks like an empty message body to STOMP
clients, or possibly garbage. The solution was for clients to
translate the messages to text/json prior to sending. (Of course if
you never enable STOMP on your broker at all, maybe this won't be a
problem :)
- Ken
9:54 a.m.
It's nice to give the flexibility to clients by exposing both. I
haven't seen a problem with topic matching in my experience so far.
While I like the idea of adding flexibility, it'll probably also be
harder on the debugging and maintenance side of things. We will keep
the ZeroMQ gateway for external bus users, we may also consider a
STOMP gateway that will do sanity checks on the fly if that becomes
necessary.
One thing I found with AMQP vs STOMP is that it's possible for
AMQP
clients to (accidentally) emit "binary" message bodies, and then
ActiveMQ does not translate or expose these as plaintext JSON for
STOMP clients. It just looks like an empty message body to STOMP
clients, or possibly garbage.
When using the fedora-messaging library, outgoing messages will be
validated using a JSON schema and enforced to JSON/UTF-8. That should
make it much harder to emit something broken. Received messages will
also be validated, of course.
Aurélien
12:13 p.m.
On 06/04/2018 06:59 PM, Ken Dreyer wrote:
On Mon, Jun 4, 2018 at 8:30 AM, Jeremy Cline
<jeremy(a)jcline.org> wrote:
> I had a hard time justifying choosing STOMP over AMQP because most
> brokers just map the other protocol they focus on onto STOMP. It's true
> the the spec is short, but it leaves a lot up to individual
> implementations as far as I can tell (like how topic matching works, for
> example).
It's nice to give the flexibility to clients by exposing both. I
haven't seen a problem with topic matching in my experience so far.
I read up a bit on the RabbitMQ plugin for STOMP, it sounds like it uses
the same routing key rules (e.g. '*' and '#' work) as the AMQP queue
bindings in Rabbit. I don't have a problem with them both being exposed
in the Fedora deployment.
I don't mind the fedora-messaging library using STOMP either, but people
will still need to understand the AMQP semantics because we're relying
on using two different exchanges to track what messages came from ZeroMQ
vs AMQP, and Rabbit maps all the STOMP interaction onto AMQP concepts
anyway.
I would like to make our work easily usable by Red Hat internal
infrastructure since that can only be a beneficial relationship for both
Fedora and Red Hat, but I don't have a good sense of what they're doing.
Mike, if your suggestion is driven by a desire to make this a useful
tool internally, please let me know. My current impression is that
the client and protocol it uses is of minimal interest. I imagine the
schema would be of interest, though. Am I wrong here?
One thing I found with AMQP vs STOMP is that it's possible for AMQP
clients to (accidentally) emit "binary" message bodies, and then
ActiveMQ does not translate or expose these as plaintext JSON for
STOMP clients. It just looks like an empty message body to STOMP
clients, or possibly garbage. The solution was for clients to
translate the messages to text/json prior to sending. (Of course if
you never enable STOMP on your broker at all, maybe this won't be a
problem :)
The fedora-messaging publish/subscribe API (that's just wrapping an AMQP
client) requires the message to match a jsonschema, that it is UTF-8
encoded, and that the content type and encoding is properly set so if
some publisher is misbehaving in this way the clients using
fedora-messaging will reject this sort of message.
Thanks for sharing your experience with AMQP and STOMP, it's very
helpful!
--
Jeremy Cline
XMPP: jeremy(a)jcline.org
IRC: jcline
2150
days inactive
2162
days old
infrastructure@lists.fedoraproject.org
9 comments
5 participants
participants (5)
-
Aurelien Bompard -
Jeffrey Ollie -
Jeremy Cline -
Ken Dreyer -
Michael Bonnet