Fedora 37 Beta Freeze now in effect
by Kevin Fenzi
Greetings.
We are now in the infrastructure freeze leading up to the Fedora 37
Beta release. This is a pre release freeze.
We do this to ensure that our infrastructure is stable and ready to
release the Fedora 37 Beta when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone
https://infrastructure.fedoraproject.org/infra/ansible.git
ansible/scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2022-09-13 (or later if
release slips). Frozen hosts should have no changes made to them without
a sign-off on the change from at least 2 sysadmin-main or rel-eng
members, along with (in most cases) a patch of the exact change to be
made to this list or a pull request for review.
Thanks,
Kevin
1 year, 7 months
The state of mailing lists/mailman (not great)
by Stephen Smoogen
This is a general 'what I am seeing in the infrastructure' email taking
various comments I made in irc into a more formal email
I started looking at some email failures last week and realized that many
of the mailing lists have vast amounts of 'held' email and 'held'
subscriptions. In most cases these are SPAM messages and from many of the
email addresses probably also spam accounts.
In viewing this, I "felt" that lists with large numbers of these seemed
slower to bring up various web pages so I started to clean up the loads. My
rough estimate is that there are several hundred thousand outstanding
subscriptions on many lists going back to 2015 and there are approximately
million held messages in the database.
Due to the age of our mailman3 implementation, I could not easily do this
with any command line tool and found that the direct action given in
various forums were not working either due to the schema we have in our DB
not matching what is now the current version. I instead went through
cleaning up various lists of old spam by hand. This takes a long time and
led me to see that the general load average of the mailman was running
around 10 to 12 over long periods of time. With only 4 CPU's, I thought
this might be cause of my original issues so I increased the CPU count to
8.
This lowered the long term load average but now led to reports where the
mailman REST daemon does not respond in time due to timeouts. Looking at
what is going on in the logs, and reading through the forums, my guess was
these are issues due to the old django and early mailman we are using.
At this point I started to look at what would be needed to update mailman
to a newer version which could be supported on Fedora or RHEL9. The main
issues are setting up the django "project" and then making sure that the
needed packages are available. The upstream 'recommended' ways are to use
either venv or containers but I found that the solutions are for a) docker,
b) have a lot of security vulnerabilities in the quay listing and c) using
it looks like Debian/Ubuntu packages. There is also a lot of customization
needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout
in staging, get a copy of the database from db01 and try experiments to
make it work until we either burn it all down or not.
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
1 year, 7 months
Freeze Break Request: Increase mailman to 8 CPU
by Stephen Smoogen
I am trying to deal with some crashing issues with the mailman. It is
currently at 100% CPU usage with 4 cpus and about 12-16 waiting services at
all times. I would like to increase the number of CPUs to 8. There does not
look to be memory problems at this time so no increase is needed there.
Actions:
update ansible
update vm system via virsh
reboot vm
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
1 year, 7 months
