I came up with a draft of a nagios/nrpe check for datanommer/fedmsg
https://gist.github.com/ralphbean/5482129
It queries the datanommer DB and asks for the time since the latest message in a particular category (i.e., bodhi, buildsys/koji, askbot).
We could configure it to raise a warning if, for example:
- It hasn't seen a buildsys message in the last 30 minutes. - It hasn't seen a bodhi message in the last 6 hours. - It hasn't seen a fedoratagger message in the last 2 months.
Since nagios alerts affect lots of people, it should probably be discussed here or in the infra meeting before being rolled out.
Pierre pointed out in channel that this approach assumes that there *must* be bodhi activity for such and such amount of time or else something is wrong. This could be problematic. There are times like the holidays in December when fewer people are contributing to Fedora, in which case this plugin could throw false positives. Accordingly, we would need to set the WARN and CRIT thresholds to be generously long.