This is a general 'what I am seeing in the infrastructure' email taking various comments I made in irc into a more formal email
I started looking at some email failures last week and realized that many of the mailing lists have vast amounts of 'held' email and 'held' subscriptions. In most cases these are SPAM messages and from many of the email addresses probably also spam accounts.
In viewing this, I "felt" that lists with large numbers of these seemed slower to bring up various web pages so I started to clean up the loads. My rough estimate is that there are several hundred thousand outstanding subscriptions on many lists going back to 2015 and there are approximately million held messages in the database.
Due to the age of our mailman3 implementation, I could not easily do this with any command line tool and found that the direct action given in various forums were not working either due to the schema we have in our DB not matching what is now the current version. I instead went through cleaning up various lists of old spam by hand. This takes a long time and led me to see that the general load average of the mailman was running around 10 to 12 over long periods of time. With only 4 CPU's, I thought this might be cause of my original issues so I increased the CPU count to 8.
This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are using.
At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not.
On 02/09/2022 22:10, Stephen Smoogen wrote:
This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are using.
At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not.
I'd be curious on the django version and release issues. The way the project is working (regular releases, release often,...) and getting long term support does not really work well with a community-based approach ("the community will keep the version alive", "do backports", etc) with epel packages.
While there is a bug https://bugzilla.redhat.com/show_bug.cgi?id=2033064 to get django in EPEL, I'm not sure how much this would help you here?
Would be hosting on Fedora an alternative?
Matthias
On Mon, 5 Sept 2022 at 03:13, Matthias Runge mrunge@matthias-runge.de wrote:
On 02/09/2022 22:10, Stephen Smoogen wrote:
This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are
using.
At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not.
I'd be curious on the django version and release issues. The way the project is working (regular releases, release often,...) and getting long term support does not really work well with a community-based approach ("the community will keep the version alive", "do backports", etc) with epel packages.
While there is a bug https://bugzilla.redhat.com/show_bug.cgi?id=2033064 to get django in EPEL, I'm not sure how much this would help you here?
Would be hosting on Fedora an alternative?
That is possible in one of the following scenarios: 1. We install into a VM and do major updates every 6 months to the next Fedora release. I think that this is more doable now than it has been in the past. 2. We make a set of containers via Fedora which can be run in openshift. Similar set of spin up and move every couple of months. There are very out of date quay.io and docker.io ones which might work as a template to start with.
On Mon, 5 Sept 2022 at 03:13, Matthias Runge mrunge@matthias-runge.de wrote:
On 02/09/2022 22:10, Stephen Smoogen wrote:
This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are
using.
At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that.
I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not.
I'd be curious on the django version and release issues. The way the project is working (regular releases, release often,...) and getting long term support does not really work well with a community-based approach ("the community will keep the version alive", "do backports", etc) with epel packages.
I forgot to answer you on this. These are the versions being used.
django-assets-0.8-1.el7.centos.noarch hyperkitty-1.1.5-0.1.el7.centos.noarch hyperkitty-selinux-1.1.5-0.1.el7.centos.noarch mailman3-3.1.1-0.6.el7.centos.noarch mailman3-fedmsg-plugin-0.5-6.el7.noarch mailman3-hyperkitty-1.1.1-0.2.el7.centos.noarch mailman3-selinux-3.1.1-0.6.el7.centos.noarch python-django-1.8.8-2.el7.centos.noarch python-django-allauth-0.34.0-1.el7.centos.noarch python-django-appconf-1.0.2-1.el7.centos.noarch python-django-bash-completion-1.8.8-2.el7.centos.noarch python-django-browserid-2.0.2-1.el7.centos.noarch python-django-compressor-2.0-1.el7.centos.noarch python-django-extensions-1.6.1-1.el7.centos.noarch python-django-gravatar2-1.0.6-2.el7.centos.noarch python-django-haystack-2.5.0-1.el7.centos.noarch python-django-haystack-xapian-2.0.0-2.el7.centos.noarch python-django-mailman3-1.1.1-0.3.el7.centos.noarch python-django-paintstore-0.2-1.el7.centos.noarch python-django-picklefield-0.3.2-1.el7.centos.noarch python-django-q-0.7.18-1.el7.centos.noarch python-django-rest-framework-3.3.3-1.el7.centos.noarch
While there is a bug https://bugzilla.redhat.com/show_bug.cgi?id=2033064 to get django in EPEL, I'm not sure how much this would help you here?
Would be hosting on Fedora an alternative?
Matthias _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, 2 Sept 2022 at 16:10, Stephen Smoogen ssmoogen@redhat.com wrote:
This is a general 'what I am seeing in the infrastructure' email taking various comments I made in irc into a more formal email
Looking at the last month of mailing lists which archived mails, we have about 120 email out of 700+ email lists which got at least one email. The 40 most active emails in August to beginning of September were
36148 scm-commits 2178 package-announce 1737 go-sig 1729 package-review 1163 rust-sig 1145 python-sig 864 devel 786 releng-cron 676 gnome-sig 449 epel-package-announce 441 perl-devel 419 neuro-sig 344 users 318 epel-packagers-sig 317 infra-sig 313 kde-sig 313 arch-excludes 289 container-sig 229 test-reports 192 deepinde-sig 186 ruby-packagers-sig 156 virt-maint 151 kernel 141 fedoramagazine-tips 126 freeipa-users 124 perl-maint 108 epel-devel 104 rpm-software-management 102 openstack-sig 101 meetingminutes 98 test 97 i18n-bugs 97 certbot-sig 86 kexec 75 java-sig-commits 67 dns-sig 50 nodejs-sig 45 lvm2-commits 45 infrastructure 44 389-users
I am wondering if some of these lists would be better using something like public inbox https://lwn.net/Articles/748184/ like sourceware is using https://inbox.sourceware.org/. Various announcement lists and things like scm-commits do not need to have interactive webpages like mailman3 hyperkitty give since they are limited to few people.
On Tue, Sep 06, 2022 at 09:45:11AM -0400, Stephen Smoogen wrote: ....snip...
I am wondering if some of these lists would be better using something like public inbox https://lwn.net/Articles/748184/ like sourceware is using https://inbox.sourceware.org/. Various announcement lists and things like scm-commits do not need to have interactive webpages like mailman3 hyperkitty give since they are limited to few people.
I've often pondered setting up public-inbox, but then we have another thing to maintain, and it's got some...limitations... like expecting everything to be plain text.
But it's a thought...
kevin
On Tue, 6 Sept 2022 at 12:34, Kevin Fenzi kevin@scrye.com wrote:
On Tue, Sep 06, 2022 at 09:45:11AM -0400, Stephen Smoogen wrote: ....snip...
I am wondering if some of these lists would be better using something
like
public inbox https://lwn.net/Articles/748184/ like sourceware is using https://inbox.sourceware.org/. Various announcement lists and things
like
scm-commits do not need to have interactive webpages like mailman3 hyperkitty give since they are limited to few people.
I've often pondered setting up public-inbox, but then we have another thing to maintain, and it's got some...limitations... like expecting everything to be plain text.
But it's a thought...
I was thinking this might be only for some high bandwidth lists which are primarily read-only. scm-commits and maybe a list for fedora-message emails.
Dne 06. 09. 22 v 15:45 Stephen Smoogen napsal(a):
On Fri, 2 Sept 2022 at 16:10, Stephen Smoogen ssmoogen@redhat.com wrote:
This is a general 'what I am seeing in the infrastructure' email taking various comments I made in irc into a more formal emailLooking at the last month of mailing lists which archived mails, we have about 120 email out of 700+ email lists which got at least one email. The 40 most active emails in August to beginning of September were
36148 scm-commits 2178 package-announce 1737 go-sig 1729 package-review 1163 rust-sig 1145 python-sig 864 devel 786 releng-cron 676 gnome-sig 449 epel-package-announce 441 perl-devel 419 neuro-sig 344 users 318 epel-packagers-sig 317 infra-sig 313 kde-sig 313 arch-excludes 289 container-sig 229 test-reports 192 deepinde-sig 186 ruby-packagers-sig
Uh, this ^^ took me by surprise. I don't even know there would be any incoming traffic. This ML was created, probably because we needed when we established the associated group. I am quite sure that I am notified about the activity on related components by different means (FMN) then this ML.
Vít
156 virt-maint 151 kernel 141 fedoramagazine-tips 126 freeipa-users 124 perl-maint 108 epel-devel 104 rpm-software-management 102 openstack-sig 101 meetingminutes 98 test 97 i18n-bugs 97 certbot-sig 86 kexec 75 java-sig-commits 67 dns-sig 50 nodejs-sig 45 lvm2-commits 45 infrastructure 44 389-users
I am wondering if some of these lists would be better using something like public inbox https://lwn.net/Articles/748184/ like sourceware is using https://inbox.sourceware.org/. Various announcement lists and things like scm-commits do not need to have interactive webpages like mailman3 hyperkitty give since they are limited to few people. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren
infrastructure mailing list --infrastructure@lists.fedoraproject.org To unsubscribe send an email toinfrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Matthias Runge -
Stephen Smoogen -
Vít Ondruch