This is happening to me, too - but it doesn't seem to be limited to koji notifications. It happens with bugzilla notifications as well. There isn't any pattern to the delays that I can see. I regularly receive notification digests during the day, yet every now and then I receive a digest informing me of stuff that happened 2 or 3 days ago. I think once it informed me on stuff that was almost a week old. My settings in Fedora Notifications are "100 messages or 1 hour". I don't receive many notifications, so 100% of the time, the time limit kicks in first; most digests I receive have single-digit message counts. This applies to the delayed digests as well - I've never seen one actually reach 100.

It seems that there is some kind of bottleneck somewhere, because the delay often gets progressively worse if there are lots of notifications in a short period of time, and then things are only able to catch up to recent notifications multiple days later (or get lost, when something seemingly crashes and needs to be restarted).

Hello, is it a known problem that koji notifications arrive way too late? For

example, I got mame build notifications in the wee hours today (31.12) wheras the builds completed around noon on the 29th: https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 This does not seem normal. Is anybody seeing this too?

On Sun, Jan 02, 2022 at 12:03:25PM -0800, Kevin Fenzi wrote: > On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote: >> Hello, >> >> is it a known problem that koji notifications arrive way too late? For > yes. > >> example, I got mame build notifications in the wee hours today (31.12) >> wheras the builds completed around noon on the 29th: >> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 >> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 >> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 >> This does not seem normal. Is anybody seeing this too? > Everyone. :( > > The problem here is our notification system ( fmn ) has a number of > issues: > > * The current version is still python2 and crashes from time to time for > difficult to debug reasons. So, sometimes it has crashed and takes a bit > to realize it's not processing and get restarted. :( We do have a > python3 port almost ready to swap in. Once thats in at least we can > hopefully debug any crashes. I hope that will happen in the first > quarter. > > * The current version has issues processing when 'large' amounts of > messages flood the bus. Examples of this include: > - When a beta or final release is 'go' releng tags all the packages to > record they were part of the release, so ~23k messages in a few > minutes. > - when copr rebuilds all rubygems > - mass rebuilds, thats ~23*N messages > - Most recently, when epel9 branching opened, because say 200 packages > branched quicky and merged every commit from rawhide into the epel9 > branch and each commit is a message. I'm right now the unlucky receiver of this. It's really annoying. Facter has a ~15 year history so it's a lot of commits. Couldn't this be grouped? I don't see a multi commit viewer in Pagure where you can say $start..$end (like regular git, github, gitlab and others). That would save a lot of notifications and actually be effective.