On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote:
Hello,
is it a known problem that koji notifications arrive way too late? For
yes.
Everyone. :(
The problem here is our notification system ( fmn ) has a number of
issues:
* The current version is still python2 and crashes from time to time for
difficult to debug reasons. So, sometimes it has crashed and takes a bit
to realize it's not processing and get restarted. :( We do have a
python3 port almost ready to swap in. Once thats in at least we can
hopefully debug any crashes. I hope that will happen in the first
quarter.
* The current version has issues processing when 'large' amounts of
messages flood the bus. Examples of this include:
- When a beta or final release is 'go' releng tags all the packages to
record they were part of the release, so ~23k messages in a few
minutes.
- when copr rebuilds all rubygems
- mass rebuilds, thats ~23*N messages
- Most recently, when epel9 branching opened, because say 200 packages
branched quicky and merged every commit from rawhide into the epel9
branch and each commit is a message.
So, anyhow, I hope the python3 version will help out some, but really we
need to do a re-write of the application to better handle lots of
messages. The current app is... difficult to use, but super flexible.
If we made it easier to use, simpliler, but some less flexable it could
help processing speed. There has also been talk about seeing if we can
just use rabbitmq for the heavy lifting here (ie, have queues for people
who customize messages with those topics and have rabbitmq sort them for
us), that might work or help. I hope we can work on this this year.
kevin