This is a general 'what I am seeing in the infrastructure' email taking various comments I made in irc into a more formal email
I started looking at some email failures last week and realized that many of the mailing lists have vast amounts of 'held' email and 'held' subscriptions. In most cases these are SPAM messages and from many of the email addresses probably also spam accounts.
In viewing this, I "felt" that lists with large numbers of these seemed slower to bring up various web pages so I started to clean up the loads. My rough estimate is that there are several hundred thousand outstanding subscriptions on many lists going back to 2015 and there are approximately million held messages in the database.
Due to the age of our mailman3 implementation, I could not easily do this with any command line tool and found that the direct action given in various forums were not working either due to the schema we have in our DB not matching what is now the current version. I instead went through cleaning up various lists of old spam by hand. This takes a long time and led me to see that the general load average of the mailman was running around 10 to 12 over long periods of time. With only 4 CPU's, I thought this might be cause of my original issues so I increased the CPU count to 8.
This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are using.
At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not.