Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some of them I think we can reinstall anytime:
backup01.iad2.fedoraproject.org batcave13.iad2.fedoraproject.org cloud-noc-os01.rdu-cc.fedoraproject.org dl01.iad2.fedoraproject.org dl02.iad2.fedoraproject.org dl03.iad2.fedoraproject.org dl04.iad2.fedoraproject.org dl05.iad2.fedoraproject.org download-ib01.fedoraproject.org download-rdu-cc01.fedoraproject.org dedicationsolutions01.iad2.fedoraproject.org ibiblio01.iad2.fedoraproject.org ibiblio05.iad2.fedoraproject.org memcached01.iad2.fedoraproject.org noc01.iad2.fedoraproject.org noc02.fedoraproject.org ns01.iad2.fedoraproject.org ns02.iad2.fedoraproject.org ns02.fedoraproject.org ns13.rdu2.fedoraproject.org osuosl01.fedoraproject.org secondary01.iad2.fedoraproject.org smtp-mm-ib01.fedoraproject.org smtp-mm-osuosl01.fedoraproject.org smtp-mm-cc-rdu01.fedoraproject.org storinator01.rdu-cc.fedoraproject.org sundries01.iad2.fedoraproject.org sundries02.iad2.fedoraproject.org tang01.iad2.fedoraproject.org tang02.iad2.fedoraproject.org torrent02.fedoraproject.org unbound-cc-rdu01.fedoraproject.org unbound-cc-rdu01.fedoraproject.org
Some of them we can do, but we will need an outage for them:
bastion01.iad2.fedoraproject.org bastion02.iad2.fedoraproject.org bastion13.iad2.fedoraproject.org people02.fedoraproject.org db01.iad2.fedoraproject.org db03.iad2.fedoraproject.org db-datanommer.iad2.fedoraproject.org db-koji.iad2.fedoraproject.org db-openq.iad2.fedoraproject.org (we can see how long the upgrade takes in stg on the various db servers)
Some of them need applications/packages built for rhel9 and we can't do them until that is sorted out:
badges (hopefully now ongoing?) notifs (ongoing) mm- (is mirrormanager2 ready to branch/build for rhel9?) pagure (how about pagure?) pkgs (also need pagure) sign (needs the new sigul. I'll ping patrick about it again) value* - needs limnoria in epel9 and fedmsg-irc replacement somehow. The zodbot part perhaps could be moved to openshift.
Some we need to carefully save local data and restore after reinstall:
batcave01.iad2.fedoraproject.org log01.iad2.fedoraproject.org
The vmhosts need some investigation to see if we can reinstall and keep the vm data around. Otherwise we need to migrate vm's off and back on. I tried this in a early 9.0 install and it didn't seem to work on the host I was testing with, but should try again.
qvmhost bvmhost vmhost*
Some, we need to talk about:
db-fas01 - this used to have the fas db on it to be more secure (seperate from db01), now all it has is the ipsilon db. I guess we could rename it db-ipsilon01? or fold it into db01? ipa - I think, but want to confirm we can just reinstall one replica with rhel9, get it all synced and then do another and another until the entire cluster is rhel9. rabbitmq - currently we are using rhel8 + openstack 16 repo. I suppose we can just go to rhel9 + openstack 17 repo.
Some will just not move anytime soon:
busgateway - needs fedmsg, only in epel7 fedimg - needs fedmsg. will be replaced by some other solution github2fedmsg - needs fedmsg, only in epel7 mailman01 - needs fedmsg nuancer - needs fedmsg osbs - needs new container build system pdc - needs to be retired
Finally, some I am not sure about and would like input:
mbs - is this ready for rhel9? Should we move to Fedora instead? odcs - how about this?
So, as far as help goes, getting mirrormanager2 and pagure all in epel9 would be great... coming up with a fedmsg-irc replacement, and getting limnoria in epel9 seem to be the best ways right now. I can start on the easier reinstalls.
kevin
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
busgateway - needs fedmsg, only in epel7 fedimg - needs fedmsg. will be replaced by some other solution github2fedmsg - needs fedmsg, only in epel7 mailman01 - needs fedmsg nuancer - needs fedmsg osbs - needs new container build system pdc - needs to be retired
Finally, some I am not sure about and would like input:
mbs - is this ready for rhel9? Should we move to Fedora instead? odcs - how about this?
So, as far as help goes, getting mirrormanager2 and pagure all in epel9 would be great... coming up with a fedmsg-irc replacement, and getting limnoria in epel9 seem to be the best ways right now. I can start on the easier reinstalls.
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
The mailman stack is FTBFS on Fedora right now because of a single library (python-aiosmtpd) not working properly because of changes in the SSL module in Python 3.10. The whole stack can branch into RHEL 9 just fine.
busgateway - needs fedmsg, only in epel7 fedimg - needs fedmsg. will be replaced by some other solution github2fedmsg - needs fedmsg, only in epel7 mailman01 - needs fedmsg nuancer - needs fedmsg osbs - needs new container build system pdc - needs to be retired
Finally, some I am not sure about and would like input:
mbs - is this ready for rhel9? Should we move to Fedora instead? odcs - how about this?
So, as far as help goes, getting mirrormanager2 and pagure all in epel9 would be great... coming up with a fedmsg-irc replacement, and getting limnoria in epel9 seem to be the best ways right now. I can start on the easier reinstalls.
Getting Pagure and MM2 branched into EPEL 9 shouldn't be too bad, hopefully. I did it the last go around for EPEL 8 with not too much effort.
On Tue, Sep 27, 2022 at 3:20 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
The mailman stack is FTBFS on Fedora right now because of a single library (python-aiosmtpd) not working properly because of changes in the SSL module in Python 3.10. The whole stack can branch into RHEL 9 just fine.
And apparently that issue was fixed. Branching Mailman in EPEL9 is waiting on people adding infra-sig and epel-packagers-sig to dependencies of the stack.
See: https://bugzilla.redhat.com/show_bug.cgi?id=2030061
On Tue, Sep 27, 2022 at 04:04:41PM +0200, Neal Gompa wrote:
On Tue, Sep 27, 2022 at 3:20 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
Sure, we can.
But at this point I think we have already gotten rid of most of the things we really don't need. So, someone will need to make a compelling argument for dropping things (at least to me).
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
Well, until we figure out how to otherwise handle the use cases that mailing lists handle now?
For example: all the -sig groups in pagure have mailing lists that get all the bugs send to it. We would need to find another way to get bug content to those groups (and still keep it private). scm-commits is still important IMHO, because its a external record of changes. If someone messed with git history, that might be the only record of real changes. devel/test/a few other lists are still active. They would have to move to discourse or otherwise have something.
So, needs a concrete plan. I am not at all in favor of 'turn it off' without moving all the needs.
The mailman stack is FTBFS on Fedora right now because of a single library (python-aiosmtpd) not working properly because of changes in the SSL module in Python 3.10. The whole stack can branch into RHEL 9 just fine.
And apparently that issue was fixed. Branching Mailman in EPEL9 is
So, does that mean mailman3/hyperkitty/postorius can be fixed to not fail to build in Fedora now? That might be a bit less effort than them being retired and having to unretire them to fix them.
waiting on people adding infra-sig and epel-packagers-sig to dependencies of the stack.
Yeah, django has been a holdup, but I think it just needs someone to say 'I will maintain this in epel9'.
kevin
On Wed, Sep 28, 2022 at 8:35 PM Kevin Fenzi kevin@scrye.com wrote:
On Tue, Sep 27, 2022 at 04:04:41PM +0200, Neal Gompa wrote:
On Tue, Sep 27, 2022 at 3:20 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
Sure, we can.
But at this point I think we have already gotten rid of most of the things we really don't need. So, someone will need to make a compelling argument for dropping things (at least to me).
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
Well, until we figure out how to otherwise handle the use cases that mailing lists handle now?
For example: all the -sig groups in pagure have mailing lists that get all the bugs send to it. We would need to find another way to get bug content to those groups (and still keep it private). scm-commits is still important IMHO, because its a external record of changes. If someone messed with git history, that might be the only record of real changes. devel/test/a few other lists are still active. They would have to move to discourse or otherwise have something.
So, needs a concrete plan. I am not at all in favor of 'turn it off' without moving all the needs.
The mailman stack is FTBFS on Fedora right now because of a single library (python-aiosmtpd) not working properly because of changes in the SSL module in Python 3.10. The whole stack can branch into RHEL 9 just fine.
And apparently that issue was fixed. Branching Mailman in EPEL9 is
So, does that mean mailman3/hyperkitty/postorius can be fixed to not fail to build in Fedora now? That might be a bit less effort than them being retired and having to unretire them to fix them.
Yes, it should be possible. I'm not sure where in the chain it's FTBFS right now since all the logs were reaped, but since the core dependency that was breaking is not broken anymore, we can fix this.
waiting on people adding infra-sig and epel-packagers-sig to dependencies of the stack.
Yeah, django has been a holdup, but I think it just needs someone to say 'I will maintain this in epel9'.
Michel Salim was willing to do that, but some of Django's dependencies are stuck waiting for ownership to be fixed so they can be branched.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, 28 Sept 2022 at 14:54, Kevin Fenzi kevin@scrye.com wrote:
On Tue, Sep 27, 2022 at 04:04:41PM +0200, Neal Gompa wrote:
On Tue, Sep 27, 2022 at 3:20 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com
wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare
hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?'
'what happens if it doesn't have fedmsg', and 'do we need it?'
Sure, we can.
But at this point I think we have already gotten rid of most of the things we really don't need. So, someone will need to make a compelling argument for dropping things (at least to me).
mailman01 is one where maybe not having it on fedmsg wouldn't be
earth shattering
but it also has the bigger problem of all its libraries being FTBFS
in Fedora and
being retired from there. At which point we go with 'do we need to
run mailing lists?'
Well, until we figure out how to otherwise handle the use cases that mailing lists handle now?
For example: all the -sig groups in pagure have mailing lists that get all the bugs send to it. We would need to find another way to get bug content to those groups (and still keep it private). scm-commits is still important IMHO, because its a external record of changes. If someone messed with git history, that might be the only record of real changes. devel/test/a few other lists are still active. They would have to move to discourse or otherwise have something.
So, needs a concrete plan. I am not at all in favor of 'turn it off' without moving all the needs.
I am not in favour of it myself. I am however aware that it is a constant push from some corners.
My main concerns are that our mailman3 is in bad shape, and I don't know if it is the correct tool for the jobs needed for some lists. I agree scm-commits needs to exist but it may make more sense as just a list of people in ldap who get emails and a mailbox which can be downloaded.
Moving mailman to a newer version is going to be a major challenge for multiple reasons: * Major schema changes * Customizations we did for Fedora services * A move from Phoenix to IAD2 where I didn't calculate disk space correctly and some files were corrupted with no backups. I fixed as many as I could, but I am not sure I got all of them. * Other one-offs where our version of things do not match any versions of what mailman3 docs say but probably due to prerelease code. * Various tables will need to be cleaned so that we don't accidently unsubscribe everyone due to 'unimplemented' bounce features which store that a person should be someday unsubscribed or other actions.. when that code was finally implemented in a later version. * Other tables which contain large amounts of garbage data about spam emails which never need to be seen or subscriptions which never need to be fulfilled. However there is no easy way of cleaning them, and they seem to cause all kinds of other slowdown when emails go to affected lists. * Various slowdowns and non-responsive come when emails get sent to scm-commits and I am not sure why. It had neither a long list of subscribers or held messages. However the mailman rest api will go unresponsive when a lot of commits get to it.
Some of those have me very worried and looking for options to cut down work needed to make an update possible. However many of my options may not be useful and it is not my job to implement so I need to chill.
On Wed, Sep 28, 2022 at 11:35:09AM -0700, Kevin Fenzi wrote:
On Tue, Sep 27, 2022 at 04:04:41PM +0200, Neal Gompa wrote:
On Tue, Sep 27, 2022 at 3:20 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Sep 27, 2022 at 3:01 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
Sure, we can.
But at this point I think we have already gotten rid of most of the things we really don't need. So, someone will need to make a compelling argument for dropping things (at least to me).
mailman01 is one where maybe not having it on fedmsg wouldn't be earth shattering but it also has the bigger problem of all its libraries being FTBFS in Fedora and being retired from there. At which point we go with 'do we need to run mailing lists?'
Well, until we figure out how to otherwise handle the use cases that mailing lists handle now?
For example: all the -sig groups in pagure have mailing lists that get all the bugs send to it. We would need to find another way to get bug content to those groups (and still keep it private). scm-commits is still important IMHO, because its a external record of changes. If someone messed with git history, that might be the only record of real changes. devel/test/a few other lists are still active. They would have to move to discourse or otherwise have something.
So, needs a concrete plan. I am not at all in favor of 'turn it off' without moving all the needs.
I took it more as a: do we need to *run* mailing list? More than a: Do we need mailing list? Ie: should we look to host our lists for us?
That's a fair question and I can see pros and cons to it, but I'm definitely in the camp of "we need mailing lists" :)
Pierre
On Tue, 2022-09-27 at 09:00 -0400, Stephen Smoogen wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
FWIW, while I'm one of the holdouts on still needing PDC, I don't think anything I do with PDC relies on it publishing messages. Don't know if any other use cases of PDC do.
I do think we need to sort out the key use cases of PDC before getting rid of it, but leaving it on RHEL 8 seems like it'd be a reasonable choice. We really ought to be able to sort out getting off PDC before RHEL 8 kicks the bucket. I'm kinda kicking the tires on a plan to move critpath handling into Bodhi at the moment, for one thing.
On Tue, Sep 27, 2022 at 12:54:14PM -0700, Adam Williamson wrote:
On Tue, 2022-09-27 at 09:00 -0400, Stephen Smoogen wrote:
On Mon, 26 Sept 2022 at 17:56, Kevin Fenzi kevin@scrye.com wrote:
Here's my thoughts on rhel9 upgrades.
We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware).
Some will just not move anytime soon:
Is it possible to look at these as 'why does this need fedmsg?' 'what happens if it doesn't have fedmsg', and 'do we need it?'
FWIW, while I'm one of the holdouts on still needing PDC, I don't think anything I do with PDC relies on it publishing messages. Don't know if any other use cases of PDC do.
I do think we need to sort out the key use cases of PDC before getting rid of it, but leaving it on RHEL 8 seems like it'd be a reasonable
(It's rhel7 actually)
choice. We really ought to be able to sort out getting off PDC before RHEL 8 kicks the bucket. I'm kinda kicking the tires on a plan to move critpath handling into Bodhi at the moment, for one thing.
Yeah, we should definitely git rid of pdc... just need to move everything we use it for off. Might be good to sit down some folks and write up a detailed plan so we can poke at it.
kevin
On Wed, 2022-09-28 at 11:44 -0700, Kevin Fenzi wrote:
choice. We really ought to be able to sort out getting off PDC before RHEL 8 kicks the bucket. I'm kinda kicking the tires on a plan to move critpath handling into Bodhi at the moment, for one thing.
Yeah, we should definitely git rid of pdc... just need to move everything we use it for off. Might be good to sit down some folks and write up a detailed plan so we can poke at it.
ISTR there was already some effort to come up with a list of existing uses. At least I remember an email along those lines going out which I replied to with a detailed list of the things I'm involved in that use it.
badges (hopefully now ongoing?)
Yes it is, i hope that i will complete the list about what should be done in 1 or 2 days.
Some will just not move anytime soon:
busgateway - needs fedmsg, only in epel7
Just as an information, fedbadges project as well currently depends on fedmsg. But i think that it will not take too long to replace it with fedora-messaging. Replacing fedmsg with fedora-messaging on fedbadges project is one of the goals that i'm hoping to achieve as soon as i can.
On Mon, Sep 26, 2022 at 02:56:18PM -0700, Kevin Fenzi wrote:
Some of them need applications/packages built for rhel9 and we can't do them until that is sorted out:
badges (hopefully now ongoing?) notifs (ongoing) mm- (is mirrormanager2 ready to branch/build for rhel9?)
We were never able to build mirrormanager for rhel8 because some dependencies were missing in EPEL.
Missing dependencies in EPEL is probably the only thing stopping us using mirrormanager on rhel8 or rhel9 as far as I know. I am not aware of any other problems.
Adrian
On Thu, Sep 29, 2022 at 05:03:49PM +0200, Adrian Reber wrote:
On Mon, Sep 26, 2022 at 02:56:18PM -0700, Kevin Fenzi wrote:
Some of them need applications/packages built for rhel9 and we can't do them until that is sorted out:
badges (hopefully now ongoing?) notifs (ongoing) mm- (is mirrormanager2 ready to branch/build for rhel9?)
We were never able to build mirrormanager for rhel8 because some dependencies were missing in EPEL.
Missing dependencies in EPEL is probably the only thing stopping us using mirrormanager on rhel8 or rhel9 as far as I know. I am not aware of any other problems.
A quick and dirty rebuilding on epel9 gives me:
DEBUG util.py:443: No matching package to install: 'python3-IPy' DEBUG util.py:443: No matching package to install: 'python3-alembic' DEBUG util.py:443: No matching package to install: 'python3-fedmsg-core' DEBUG util.py:443: No matching package to install: 'python3-flask-admin' DEBUG util.py:443: No matching package to install: 'python3-flask-xml-rpc' DEBUG util.py:443: No matching package to install: 'python3-mock' DEBUG util.py:443: No matching package to install: 'python3-nose' DEBUG util.py:443: No matching package to install: 'python3-pyrpmmd' DEBUG util.py:443: No matching package to install: 'python3-webob'
and with some comments:
DEBUG util.py:443: No matching package to install: 'python3-IPy'
Should be pretty easy.
DEBUG util.py:443: No matching package to install: 'python3-alembic'
https://bugzilla.redhat.com/show_bug.cgi?id=2032641 coming in 9.1 I guess?
DEBUG util.py:443: No matching package to install: 'python3-fedmsg-core'
We should really move to fedora-messaging here.
DEBUG util.py:443: No matching package to install: 'python3-flask-admin' DEBUG util.py:443: No matching package to install: 'python3-flask-xml-rpc'
These should be doable to branch I would think.
DEBUG util.py:443: No matching package to install: 'python3-mock' DEBUG util.py:443: No matching package to install: 'python3-nose'
python maintainers don't want these in EPEL9. We should be able to change tests or just not run them in epel9
DEBUG util.py:443: No matching package to install: 'python3-pyrpmmd'
Should be doable.
DEBUG util.py:443: No matching package to install: 'python3-webob'
Also seems like it shouldn't be too hard.
kevin
infrastructure@lists.fedoraproject.org
-
Adam Williamson -
Adrian Reber -
Erol Keskin -
Kevin Fenzi -
Neal Gompa -
Pierre-Yves Chibon -
Stephen Smoogen