September 2022 - infrastructure - Fedora mailing-lists

planet
by Kevin Fenzi 19 Apr '23

19 Apr '23

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

13 44

About JS framework
by Pierre-Yves Chibon 17 Oct '22

17 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

RHEL9 migration
by Kevin Fenzi 30 Sep '22

30 Sep '22

Here's my thoughts on rhel9 upgrades. We have 188 RHEL7 or RHEL8 instances (counting both vm's and bare hardware). Some of them I think we can reinstall anytime: backup01.iad2.fedoraproject.org batcave13.iad2.fedoraproject.org cloud-noc-os01.rdu-cc.fedoraproject.org dl01.iad2.fedoraproject.org dl02.iad2.fedoraproject.org dl03.iad2.fedoraproject.org dl04.iad2.fedoraproject.org dl05.iad2.fedoraproject.org download-ib01.fedoraproject.org download-rdu-cc01.fedoraproject.org dedicationsolutions01.iad2.fedoraproject.org ibiblio01.iad2.fedoraproject.org ibiblio05.iad2.fedoraproject.org memcached01.iad2.fedoraproject.org noc01.iad2.fedoraproject.org noc02.fedoraproject.org ns01.iad2.fedoraproject.org ns02.iad2.fedoraproject.org ns02.fedoraproject.org ns13.rdu2.fedoraproject.org osuosl01.fedoraproject.org secondary01.iad2.fedoraproject.org smtp-mm-ib01.fedoraproject.org smtp-mm-osuosl01.fedoraproject.org smtp-mm-cc-rdu01.fedoraproject.org storinator01.rdu-cc.fedoraproject.org sundries01.iad2.fedoraproject.org sundries02.iad2.fedoraproject.org tang01.iad2.fedoraproject.org tang02.iad2.fedoraproject.org torrent02.fedoraproject.org unbound-cc-rdu01.fedoraproject.org unbound-cc-rdu01.fedoraproject.org Some of them we can do, but we will need an outage for them: bastion01.iad2.fedoraproject.org bastion02.iad2.fedoraproject.org bastion13.iad2.fedoraproject.org people02.fedoraproject.org db01.iad2.fedoraproject.org db03.iad2.fedoraproject.org db-datanommer.iad2.fedoraproject.org db-koji.iad2.fedoraproject.org db-openq.iad2.fedoraproject.org (we can see how long the upgrade takes in stg on the various db servers) Some of them need applications/packages built for rhel9 and we can't do them until that is sorted out: badges (hopefully now ongoing?) notifs (ongoing) mm- (is mirrormanager2 ready to branch/build for rhel9?) pagure (how about pagure?) pkgs (also need pagure) sign (needs the new sigul. I'll ping patrick about it again) value* - needs limnoria in epel9 and fedmsg-irc replacement somehow. The zodbot part perhaps could be moved to openshift. Some we need to carefully save local data and restore after reinstall: batcave01.iad2.fedoraproject.org log01.iad2.fedoraproject.org The vmhosts need some investigation to see if we can reinstall and keep the vm data around. Otherwise we need to migrate vm's off and back on. I tried this in a early 9.0 install and it didn't seem to work on the host I was testing with, but should try again. qvmhost bvmhost vmhost* Some, we need to talk about: db-fas01 - this used to have the fas db on it to be more secure (seperate from db01), now all it has is the ipsilon db. I guess we could rename it db-ipsilon01? or fold it into db01? ipa - I think, but want to confirm we can just reinstall one replica with rhel9, get it all synced and then do another and another until the entire cluster is rhel9. rabbitmq - currently we are using rhel8 + openstack 16 repo. I suppose we can just go to rhel9 + openstack 17 repo. Some will just not move anytime soon: busgateway - needs fedmsg, only in epel7 fedimg - needs fedmsg. will be replaced by some other solution github2fedmsg - needs fedmsg, only in epel7 mailman01 - needs fedmsg nuancer - needs fedmsg osbs - needs new container build system pdc - needs to be retired Finally, some I am not sure about and would like input: mbs - is this ready for rhel9? Should we move to Fedora instead? odcs - how about this? So, as far as help goes, getting mirrormanager2 and pagure all in epel9 would be great... coming up with a fedmsg-irc replacement, and getting limnoria in epel9 seem to be the best ways right now. I can start on the easier reinstalls. kevin

7 13

CPE Weekly Update for Week 39 of 2022
by Akashdeep Dhar 30 Sep '22

30 Sep '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. The report could be found at https://communityblog.fedoraproject.org/cpe-weekly-update-week-39-2022/. If you want to receive weekly reports by email in the future, please subscribe to either https://communityblog.fedoraproject.org/ or https://discussion.fedoraproject.org/c/news/commblog/61. We will stop sending them in the future. Regards, CPE Team

1 0

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 26 Sep '22

26 Sep '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-09-27 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

Planned Outage - networking work - 2022-09-26 21:00 UTC and 2022-09-27 21:00 UTC
by Kevin Fenzi 24 Sep '22

24 Sep '22

There will be an outage starting at 2022-09-26/2022-09-27 21:00 UTC, which will last approximately 4 hours on 2022-09-26 and 5 hours on 2022-09-27. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2022-09-26 21:00UTC' date -d '2022-09-27 21:00UTC' Reason for outage: On 2022-09-26 internet edge routers and switches will be upgraded. During this time period various Fedora services will be down as switches and routers are rebooted. On 2022-09-27 1gb internal switches will be upgraded. a 30min outage is likely during this time period. Affected Services: All services at the IAD2 datacenter Ticket Link: https://pagure.io/fedora-infrastructure/issue/10915 Please join #fedora-admin or #fedora-noc on irc.libera.chat or add comments to the ticket for this outage above.

1 0

CPE Weekly Update - Week 38 2022
by Michal Konecny 23 Sep '22

23 Sep '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. The report could be found at https://communityblog.fedoraproject.org/cpe-weekly-update---week-38-2022/. If you want to receive weekly reports by emails in the future, please subscribe to either https://communityblog.fedoraproject.org/ or https://discussion.fedoraproject.org/c/news/commblog/61. We will stop sending them in the future. Regards, CPE Team

1 0

Meeting Agenda Item: Introduction Erol Keskin
by Erol Keskin 23 Sep '22

23 Sep '22

Hi Dear Fedora Folks, My name is Erol Keskin. I'm 25 years old and i'm from Istanbul / Turkiye. Some of you folks might recognize me from projects such as anitya, fedora-messaging or noggin. I've been trying my best to contribute to fedora-infrastructure projects on a weekly basis for about 3 months. To this day, I've worked with languages such as Golang, Python, C#, C, NodeJS. I guess I could say that I have good knowledge about Docker, Podman, Kubernetes in containers generally. Besides all of that, I've been a long time user of Linux systems. I've been using them for about 10 years to this point. I should say that I've been using Fedora Linux for about 1 year till now, but in my opinion I can say that this is the best distribution that can be imagined. Because of my general workflow on Linux machines, i'm used to do Bash Scripting. To be honest, I do love working on fedora-infrastructure projects. I'm very interested in working on these projects, and I would love to be a part of this great team! I can get to use this on-edge reliable distribution thanks to many contributors behind it, i can't wait to be a one of them. I want to improve myself in topics Zuul and Ansible. I think this is a great place to work on those skills. I hope you all have a great day. GitHub Handle: https://github.com/erolkskn My IRC handle on Libera: erolkskn

4 3

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 19 Sep '22

19 Sep '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-09-20 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

CPE Weekly Update – Week 37 2022
by Lenka Segura 16 Sep '22

16 Sep '22

Hi everyone, You can find this week's CPE (Community Platform Engineering) weekly report here: https://communityblog.fedoraproject.org/cpe-weekly-update-week-37-2022/ If you want to keep receiving weekly reports by email, please subscribe to either https://communityblog.fedoraproject.org/ or https://discussion.fedoraproject.org/c/news/commblog/61. We will stop sending them in the future. Cheers, CPE Team

1 0

Fedora 37 Beta Freeze now in effect
by Kevin Fenzi 14 Sep '22

14 Sep '22

Greetings. We are now in the infrastructure freeze leading up to the Fedora 37 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 37 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2022-09-13 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list or a pull request for review. Thanks, Kevin

1 1

I made a new Fedora Apps page, do you want it?
by Jakub Kadlcik 13 Sep '22

13 Sep '22

Hello, I am reimplementing the current https://apps.fedoraproject.org page. It is not finished yet, but I wanted to share a demo with you: https://fedora-apps.netlify.app The upstream is here: https://github.com/frostyx/fedora-apps/ Do you have any interest to eventually switch to this implementation and sunset the original one? PS: Please don't judge me too harshly for the code quality, I am learning the language. I will refactor once the basic features are done (I know everybody is saying that :D) Jakub

3 4

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 12 Sep '22

12 Sep '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-09-13 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

FBR: DNS update
by Stephen Smoogen 12 Sep '22

12 Sep '22

Currently the ipv6for osuosl.org is not functioning and I would like to take them out of DNS. When looking at it, I found that the getfedora.org was in this state: ``` disable proxy09.fedoraproject.org disable proxy11.fedoraproject.org enable proxy11.fedoraproject.org disable proxy02.fedoraproject.org enable proxy02.fedoraproject.org disable proxy04.fedoraproject.org disable proxy06.fedoraproject.org enable proxy06.fedoraproject.org disable proxy13.fedoraproject.org enable proxy13.fedoraproject.org disable proxy03.fedoraproject.org enable proxy03.fedoraproject.org disable proxy12.fedoraproject.org enable proxy12.fedoraproject.org disable proxy01.iad2.fedoraproject.org disable proxy101.iad2.fedoraproject.org disable proxy101.iad2.fedoraproject.org ``` My change to the cmds files is to add proxy06.fedoraproject.org and proxy09 to the disable list and remove all others from getfedora.org. Then execute a change and cleanup. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren

3 2

CPE Weekly Update - Week 36 2022
by Michal Konecny 09 Sep '22

09 Sep '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. The report could be found at https://communityblog.fedoraproject.org/ cpe-weekly-update---week-36-2022/. If you want to receive weekly reports by emails in the future, please subscribe to either https://communityblog.fedoraproject.org/ or https://discussion.fedoraproject.org/c/news/commblog/61. We will stop sending them in the future. Regards, CPE Team

1 0

The state of mailing lists/mailman (not great)
by Stephen Smoogen 07 Sep '22

07 Sep '22

This is a general 'what I am seeing in the infrastructure' email taking various comments I made in irc into a more formal email I started looking at some email failures last week and realized that many of the mailing lists have vast amounts of 'held' email and 'held' subscriptions. In most cases these are SPAM messages and from many of the email addresses probably also spam accounts. In viewing this, I "felt" that lists with large numbers of these seemed slower to bring up various web pages so I started to clean up the loads. My rough estimate is that there are several hundred thousand outstanding subscriptions on many lists going back to 2015 and there are approximately million held messages in the database. Due to the age of our mailman3 implementation, I could not easily do this with any command line tool and found that the direct action given in various forums were not working either due to the schema we have in our DB not matching what is now the current version. I instead went through cleaning up various lists of old spam by hand. This takes a long time and led me to see that the general load average of the mailman was running around 10 to 12 over long periods of time. With only 4 CPU's, I thought this might be cause of my original issues so I increased the CPU count to 8. This lowered the long term load average but now led to reports where the mailman REST daemon does not respond in time due to timeouts. Looking at what is going on in the logs, and reading through the forums, my guess was these are issues due to the old django and early mailman we are using. At this point I started to look at what would be needed to update mailman to a newer version which could be supported on Fedora or RHEL9. The main issues are setting up the django "project" and then making sure that the needed packages are available. The upstream 'recommended' ways are to use either venv or containers but I found that the solutions are for a) docker, b) have a lot of security vulnerabilities in the quay listing and c) using it looks like Debian/Ubuntu packages. There is also a lot of customization needed to the point it might be best to start from scratch on that. I am thinking it might make better sense to work out a project and layout in staging, get a copy of the database from db01 and try experiments to make it work until we either burn it all down or not. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren

4 7

Freeze break request - eln signing
by Kevin Fenzi 06 Sep '22

06 Sep '22

I messed up and didn't update eln signing at branching. ;( This PR: https://pagure.io/fedora-infra/ansible/pull-request/1185 should enable that and also allow me to re-sign existing builds. +1s? kevin

1 0

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 05 Sep '22

05 Sep '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-09-06 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

Freeze Break Request: Increase mailman to 8 CPU
by Stephen Smoogen 02 Sep '22

02 Sep '22

I am trying to deal with some crashing issues with the mailman. It is currently at 100% CPU usage with 4 cpus and about 12-16 waiting services at all times. I would like to increase the number of CPUs to 8. There does not look to be memory problems at this time so no increase is needed there. Actions: update ansible update vm system via virsh reboot vm -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren

4 4

CPE Weekly Update - Week 35 2022
by Lenka Segura 02 Sep '22

02 Sep '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. The report could be found at https://communityblog.fedoraproject.org/cpe-weekly-update---week-35-2022/cp…. If you want to receive weekly reports by emails in the future, please subscribe to either https://communityblog.fedoraproject.org/ or https://discussion.fedoraproject.org/c/news/commblog/61. We will stop sending them in the future. Regards, CPE Team

1 0

Freeze break request: kernel builders
by Kevin Fenzi 02 Sep '22

02 Sep '22

So, last week I managed to break bkernel01 kernel builder (and we have been using just 02 since then). There were 2 issues: 1. I switched rawhide to use systemd-nspawn for isolation instead of chroot and this broke kernel builds due to ( https://github.com/rpm-software-management/mock/issues/140 ) 2. pesign in f36 had some issues with unlocking the smart cards. ( https://bugzilla.redhat.com/show_bug.cgi?id=2122777 ) The pesign maintainers fixed issue 2 this morning and I tested the workaround I have for issue 1 in mock on bkernel01. So, what I would like to do now is: * upgrade bkernel02 to the same pesign version as on 01 along with moving it to f36 (like 01 is). * Push a commit in ansible to adjust mock config for the workaround for issue 1 above. +1s or further questions? kevin

4 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2022