infrastructure May 2022

infrastructure@lists.fedoraproject.org

24 participants
26 discussions

planet
by Kevin Fenzi 19 Apr '23

19 Apr '23

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

13 44

About JS framework
by Pierre-Yves Chibon 17 Oct '22

17 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

old vm's
by Kevin Fenzi 08 Jun '22

08 Jun '22

Looking at our current hosts (ansible_distribution_major_version): 275 35 120 8 90 7 22 34 11 36 6 33 2 9 2 31 1 37 1 32 1 27 (The reason so many are f35 is because builders are f35) Anyhow: - 27 -> This is notifs-backend01. We are still trying to get the python3 FMN deployed and working in stg so we can repave this and move it to a supported version. - 31 -> These are resultsdb01 and resultsdb01.stg. We already have the openshift replacement in staging working, we just need to cut over prod. We should schedule a time/date to do that. - 32 -> This is odcs-backend-releng01. I am just going to reinstall it and hopefully odcs is happy on f36. :) I'll mail odcs folks to confirm. - 33 -> These are osbs-aarch64 prod and stg. I'm open to suggestion on what to do here. I'd personally say we should leave them and hasten our move to image builder and hopefully by then they have container building and aarch64? Other ideas welcome. - 34 -> These are still supported at least, but we need to move them soon. I can do most of these, will be contacting people about them. Finally after that we should try and start moving the 8's and 7's to 9's. ;) kevin

2 4

Upcoming MirrorManager changes (removal of Internet2 support)
by Adrian Reber 07 Jun '22

07 Jun '22

I will remove Internet2 support from MirrorManager in the next few weeks. The main reason is, that the source for the Internet2 routing tables does not exist any more: http://routes.net.internet2.edu/ It changed a couple of years ago and it seems Fedora was one of the only users of that data. They fixed it for us, but the latest ticket I opened to access this information was closed with "This service was decomd". I do not think that the removal of Internet2 support for our users will be noticed at all. Removing Internet2 will remove some complexity from the code and using ASN and GeoIP should still provide a reasonably close mirror for most of out users. I will start to update the different components to not expose and rely on Internet2 information any more in the next couple of weeks. Let me know if you see any major problems in removing Internet2 support from MirrorManager. Adrian

3 2

on rpms
by Kevin Fenzi 31 May '22

31 May '22

So, in the past we have always had a policy to package as rpms and get into fedora/epel applications we deploy and are upstream for. There were a number of good reasons for this: * We deployed everything on vm's using rpm. * Other users that wanted to reproduce our infrastructure could use the rpms. * It made us sure that the thing passed review and built on various Fedoras with the versions of things there it depended on. However, now a days we have a number of new apps that are deployed in openshift and aren't using rpms, but pip or s2i or other things. For these packaging them up as rpms becomes a burden with not too many gains. ;( So, I was thinking we should codify a new policy. (To avoid confusion for application authors and others). Something like: Applications in Fedora Infrastructure may be deployed via non rpm methods (as long as they obey licensing guidelines ( https://fedoraproject.org/wiki/Infrastructure_Licensing )). For those applications, creating and maintaining an rpm is optional. Thoughts? kevin

7 12

Announcing sqlalchemy-helpers
by Aurelien Bompard 31 May '22

31 May '22

Hey folks! A few months ago I started a library to share some boilerplate code in our applications when it comes to SQLAlchemy. Remember the thread about Flask and SQLAlchemy <https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr…>? It ended with us admitting that Flask-SQLAlchemy is pretty cool, but does not cover our non-Flask needs, can introduce issues with scripts that do multithreading manually, and ties us into Flask more than what we may feel comfortable with. But the problem remains the same: by doing SQLAlchemy integration manually over the years, we've copied code from one project to the next, improving it when we were working on it but not backporting those improvements to older projects, and having rather sparse (I should say "distributed" ;-) ) knowledge of what the SQLAlchemy & Alembic best practices are (everybody here knows about constraint naming conventions? Foreign key enforcement on SQLite?) In the spirit of avoiding code duplication and sharing maintenance, I've started the sqlalchemy-helpers library <https://github.com/fedora-infra/sqlalchemy-helpers>. The idea is to standardize on best practices with SQLAlchemy & Alembic usage without depending on Flask, but with an optional Flask integration. The README has a list of features and usage examples, so I'll let you go over there and skim read it. Basically what I've done is go over all our Flask apps using SQLAlchemy, look at what features they implemented, and bring them into sqlalchemy-helpers. As a result if you decide to move to it you will only gain features. The only notable exception is the pagination system that Flask-SQLAlchemy implements and that at least one project has copied. If it proves useful, we can always bring it to sqlalchemy-helpers, but let's start small for now. It targets SQLAlchemy 1.3+ & 2.0, Flask 2.0+ and Python 3.6+. I've used it personally for a few months already and I'd say it works quite well. I'm very interested in feedback. My next step is to improve user documentation because right now it's only the README and the APIdocs. I hope we can join forces and maintain this sort of boilerplate code in a central place. Cheers! Aurélien

1 0

[Fedocal] Reminder meeting : Websites & Apps Team Meeting
by t0xic0der＠fedoraproject.org 30 May '22

30 May '22

Dear all, You are kindly invited to the meeting: Websites & Apps Team Meeting on 2022-05-31 from 15:00:00 to 16:00:00 UTC At https://meet.jit.si/fedora-websites-apps-meeting The meeting will be about: Weekly team meeting for the Websites & Apps Team. This is part of the [Websites & Apps Community Revamp Objective](https://fedoraproject.org/wiki/Objectives/Websites_%26_Apps_Comm…. See [past discussion]( https://discussion.fedoraproject.org/t/planning-meeting-for-websites-apps-t…) to learn more about the Websites & Apps Team and how it came together. More information available at: [hackmd.io/Mxm2We3yTqKybLsdohadOA](https://hackmd.io/Mxm2We3yTqKybLsdohadOA?view) Source: https://calendar.fedoraproject.org//meeting/9990/

1 0

Renaming design-sw to design-suite
by Luya Tshimbalanga 28 May '22

28 May '22

Hello team, Will it be possible to rename the group "design-sw" to "design-suite" please? The goal is to keep consistancy with name. Thanks in advance. -- Luya Tshimbalanga Fedora Design Team Fedora Design Suite maintainer

3 2

CPE Weekly Update – Week 21 2022
by Lenka Segura 27 May '22

27 May '22

Hi everyone, This is a weekly report from the CPE (Community Platform Engineering) Team. If you have any questions or feedback, please respond to this report or contact us on #redhat-cpe channel on libera.chat (https://libera.chat/) Week: 23rd May - 27th May 2022 If you wish to read this in form of a blog post, check the post on Fedora community blog: https://communityblog.fedoraproject.org/cpe-weekly-update---week-21-2022/ # Highlights of the week ## Infrastructure & Release Engineering Goal of this Initiative ----------------------- Purpose of this team is to take care of day to day business regarding CentOS and Fedora Infrastructure and Fedora release engineering work. It’s responsible for services running in Fedora and CentOS infrastructure and preparing things for the new Fedora release (mirrors, mass branching, new namespaces etc.). The ARC (which is a subset of the team) investigates possible initiatives that CPE might take on. Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-05-25.pdf Update ------ * Infra and releng team was investigating Openshift in AWS for communishift and CentOS CI ### Fedora Infra * Ocp3 certs expired, had to renew/push new ones (~20min outage) * Finished up a mass update/reboot. Things up on 8.6 now. * Hit bug in new ansible-freeipa and filed it upstream * Got first RHEL9 staging virthost installed * Bunch of email issues to/from redhat.com, hopefully almost all solved now. * OCP4 clusters moved to 4.10. * Some compose machines moved to f36 already. ### CentOS Infra including CentOS CI * CentOS Stream storage migration spike (Netapp for nfs/iscsi) (blocked : no news) * Duffy/CI progress: * Site-to-site vpn tunnel between VPC (us-east-1) and CI infra * delegated/hosted (dynamic) pool.ci.centos.org zone on route53 * Dns forwarding (both directions : vpc ⇔ ci infra) * Ansible playbooks ready * Git.centos.org pagure upgrade/migration (blocked, waiting on internal - EXD) * Business as usual (mirrors, tags) ### Release Engineering * f32/33 archived. Old rc’s cleaned up ## CentOS Stream Goal of this Initiative ----------------------- This initiative is working on CentOS Stream/Emerging RHEL to make this new distribution a reality. The goal of this initiative is to prepare the ecosystem for the new CentOS Stream. Updates ------- * Working on first modules and software collections for CentOS Stream 9 * Changing the Fedora ELN Everything repository to only contain what's not in AppStream, BaseOS, and CRB, and renaming it to Extras * Getting CentOS Stream 8 workflow closer to 9: * Currently testing "import latest modules", expect to have modules imported early next week. * Updating is currently manual for both modules and normal RPMs, also working on an automated import script to run at regular intervals. Expect to be finished next week. ## CentOS Duffy CI Goal of this Initiative ----------------------- Duffy is a system within CentOS CI Infra which allows tenants to provision and access bare metal resources of multiple architectures for the purposes of CI testing. We need to add the ability to checkout VMs in CentOS CI in Duffy. We have OpenNebula hypervisor available, and have started developing playbooks which can be used to create VMs using the OpenNebula API, but due to the current state of how Duffy is deployed, we are blocked with new dev work to add the VM checkout functionality. Updates ------- * Skip beta and release 3.0.0 * Continue work on provisioning nodes for Duffy in EC2 * CLI for tenants (what was cicoclient, ongoing) ## Package Automation (Packit Service) Goal of this initiative ----------------------- Automate RPM packaging of infra apps/packages Updates ------- * Package proposal on fasjson created on bugzilla * Release of fedora-messaging 3.0.2 * Packit added to noggin-messaging (noggin dependency) * Packaging datanommer.models in progress (datagrepper dependency) * Business as usual, a lot of manual packaging ## Flask-oidc: oauth2client replacement Goal of this initiative ----------------------- Flask-oidc is a library used across the Fedora infrastructure and is the client for ipsilon for its authentication. flask-oidc uses oauth2client. This library is now deprecated and no longer maintained. This will need to be replaced with authlib. Updates: -------- * Test-auth app running, allows user to successfully login using Fedora acct details (https://app-flask-oidc-dev.apps.ocp.stg.fedoraproject.org/) * Implementing authlib function changes in codebase ( https://github.com/fedora-infra/test-auth/blob/authlib_dev/test_auth/flask_… ) * Trying to pull user info from login creds ## EPEL Goal of this initiative ----------------------- Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL). EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more. Updates ------- * epel9 available at the launch of rhel9 with 5764 packages, from 2678 source packages, built by 249 different packagers * epel9 update of python-jwt to fix CVE-2022-29217 * epel8 and epel9 update of epel-rpm-macros to require ansible-srpm-macros (provides macros for packaging ansible collections) * bdii installation from epel9 problem resolved for c9s, will be resolved in RHEL 9.1 * Documentation improvements: * added list of channels/repos epel9 and epel9-next are built against * updated quickstart guide for rhel9 GA and alma9 beta * removed mentions of CentOS Linux 8 Kindest regards, CPE Team

1 0

Upcoming Ipsilon update
by Aurelien Bompard 25 May '22

25 May '22

Hey folks! I have recently been given the powers to make Ipsilon releases, so I'm going to deploy the recent changes to production soonish (probably tomorrow early morning UTC). We've been working with a snapshot so it's not as big an update as you'd think when looking at the date of the last official release (2017). Which means things should not break (staging works fine). It should get us the *very much* requested feature of having a separate field for the OTP token, among other things. Also, the update is needed to update Bodhi later (something I still need to organize, btw) Cheers! Aurélien

1 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2022