On Wed, 2018-04-04 at 09:31 +0530, Dona De wrote:
From what I understand fedmsg can automate the publishing and consumption
of messages without a broker. I still don't completely understand how to
integrate the datagrepper to work with fedmsg. I went through this
<
https://apps.fedoraproject.org/datagrepper/> and understood the fetching
process of data but how can I use fedmsg to consume it.
When you're building something whose job is to sit there and listen
continuously for fedmsgs meeting certain rules, then do something when
one arrives, you don't actually need to use datagrepper at all.
Datagrepper is basically a store of past messages sent via a given
fedmsg bus; you might want to use datagrepper if you need at some point
in time to look for fedmsgs that were *sent before*, perhaps ones you
missed, or which were sent before your tool was written.
It's a relatively common pattern that if you've already written a 'live
consumer'-type thing, you write a companion script that checks in with
datagrepper for past messages of the same type, which can be run
periodically as a cron job or just manually by an admin, for the
purpose of catching events which were missed by the live consumer for
some reason (for instance, it happened to break or go down for a
while). But that's kind of an 'advanced' pattern, you don't need to
start there - you can start by just writing a thing that listens for
fedmsgs as they show up.
There's a well-established pattern for writing a simple consumer that's
actually quite easy - I've done a bunch using the same basic approach.
There's some very high-level documentation on this here:
http://www.fedmsg.com/en/stable/subscribing/
and here's an example of a consumer I wrote:
https://pagure.io/fedora_nightlies/blob/master/f/fedora_nightlies.py#_693...
The 'config_key' and 'topic' class attributes, and the 'consume'
method, are the key parts of any consumer following this pattern -
'config_key' is a fedmsg configuration key which controls whether this
consumer is actually enabled or not, 'topic' (despite the name, it's an
iterable...) defines which message topic(s) the consumer should listen
out for ('*' wildcards are allowed), and the 'consume' method is what
should actually happen when one of those messages appears. All the
stuff in _nightly_consume is the specific stuff this particular
consumer does.
The relevant bits in setup.py are important:
https://pagure.io/fedora_nightlies/blob/master/f/setup.py#_32-33
you really need the 'entry_points' bit in setup.py to make it work. You
also need a config file in /etc/fedmsg.d/ which sets the 'config_key'
for the consumer to True, so for that consumer, you could have a file
/etc/fedmsg.d/nightlies.py with this content:
config = {
'nightlyconsumer.prod.enabled': True,
}
Once you have the config file in place and you run 'python setup.py
install' for the consumer, you need to install and enable/start (or
restart, if it's already installed/started) fedmsg-hub. Then, as long
as fedmsg-hub is listening to a fedmsg hub which actually gets messages
on the topics the consumer listens for, the consumer should kick in and
run its 'consume' method any time a message on the relevant topics
shows up.
Hope that helps, a bit, anyway :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net