CPE Weekly Update – Week 27 2022
by Michal Konecny
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
Week: 4th - 7th July 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update--week-27-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Fas2 is scaled down to 0 instances since last week. Farewell!
* Openshift3 is ready for retirement, will be taking it down later today
* Got toddlers working in staging openshift
* Email thread on infra-sig packages, please contribute
* Koji hubs reinstalled with f36, and z/vm s390x builders upgraded to f36
* Business as usual items
### CentOS Infra including CentOS CI
* Issue with mirrors and dnf being out of sync. PR merged today to fix
* Mirror requests
### Release Engineering
* A few rawhide compose issues, already fixed
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Figured out the root cause of missing module components in CentOS
Stream 8. Have a script to catch the cases, monitoring it.
## Flask-oidc: oauth2client replacement
Goal of this initiative
-----------------------
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need
to be replaced with authlib.
Updates:
--------
* Finally happy enough with our code, that we started to port it back
into the flask-oidc library itself.
* Working on a PR
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* EPEL9 is up to 6608 (+92) packages from 2956 (+45) source packages
* resolved "fails to install" bug for
[xfce4-sensors-plugin-devel](https://bodhi.fedoraproject.org/updates/FEDOR...
* filed various "fails to install" bugs for epel packages
* notable epel9 additions:
*
[chromium](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-002f30...
*
[python-paramiko](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022...
(unblocks multiple other packages)
Kindest regards,
CPE Team
1 year, 9 months
Bon-voyage to our openshift3 cluster
by Kevin Fenzi
Now that everything is moved off it as far as I can tell, I am going to
take down the openshift3 cluster tomorrow. If there's something you need
it for still, please let me know asap.
It's served long and well... we installed it back in Aug of 2017, almost
5 years ago now. It's run countless applications and gone through
countless upgrades. It helped us move from phx2 to iad2. It gave us a
nice way to deploy and manage applications.
The openshift 4 cluster has now taken over things and we look forward to
as many years or more with it. :)
kevin
1 year, 9 months
infra-sig packaging group packages
by Kevin Fenzi
Hello everyone.
As a bit of background, we setup a packaging group a long time back
called 'infra-sig'. This group was so we could collectively maintain
packages that are needed/used in Fedora Infrastructure and make sure
that packages we use aren't maintained by one team member who may move
on or not have cycles for it anymore.
These days it seems like its mostly me maintaining the packages, and not
too well since I am busy. :)
We also haven't curated the list of packages the sig maintains in a long
while. Currently we 'maintain' things we no longer use or want, and I am
sure there's packages we do use/maintain that aren't actually owned by
the infra-sig.
So, the purpose of this is several fold:
* Are you a packager already (ie, in the packager group) and would like
to help us out by maintaining infra-sig packages? Let me know if so!
* Are you someone who maintains something we use in fedora
infrastructure but no longer really have time to maintain the
packages? Or would prefer they are team maintained instead of just
you? Let me know!
* Finally, I'd like to drop some packages that the sig currently
maintains. There's 68 packages that the infra-sig is admin on currently.
There's 222 packages infra-sig has commit on.
I'm not sure how best to takle the list(s). I guess we could just go
through them one by one sometime? or everyone could propose a list we
should drop/add?
https://www.scrye.com/~kevin/fedora/infra-sig (admin)
and
https://www.scrye.com/~kevin/fedora/infra-sig2 (commit)
Some are obvious (pam_url), others less so (do we use python-gearbox?)
packager dashboard has a good view into open bugs, etc.
https://packager-dashboard.fedoraproject.org/dashboard?users=infra-sig
kevin
1 year, 9 months
Fedora OCP 3.11 -> 4.10 Data Migration Proposal
by Prakash Mishra
Hi all,
Over the last number of weeks we have been investigating a number of
outstanding issues related to the Openshift 3/4 clusters managed by
Fedora infrastructure. A large outstanding issue has been data
migration. We've been investigating various solutions, and have come
up with something which is relatively painless, and have verified it
works with a proof of concept. So here is our proposal:
## Proposal
- Using `Velero` https://velero.io/docs/main/migration-case/
- From 3.11, plan to migrate only PV/PVC data
- Velero will migrate multiple things, but the only thing we are
interested is the PV data. So there might be a little clean up in a
namespace once a migration has completed.
- Once completed we can run our Fedora ansible playbooks as normal,
the PVs/PVCs will be in place.
If it sounds like a reasonable course of action we'll put a SOP
together to describe how to carry out this work, and can either take
it on ourselves or allow Infra and Releng in CPE to carry it out.
Cheers,
Prakash Mishra & David Kirwan.
1 year, 9 months
Topic authorization on RabbitMQ
by Aurelien Bompard
Hey folks!
I have begun setting topic authorizations on our message bus: apps will no
longer be able to send messages to any topics, only to those they are
explicitly allowed to. I'll need your help to make sure I'm not forgetting
topics that your app wants to send to.
In RabbitMQ these authorizations are implemented as a set of regexps, so
it's not necessary to build an exhaustive list of the topics your app may
send to, thankfully. In the Ansible role I've implemented it as a variable
`sent_topics` that is a list of allowed regexps, usually matching the
application name right after the topic prefix. Example for batcave:
sent_topics:
- ^org\.fedoraproject\.{{ env_short }}\.ansible\..*
- ^org\.fedoraproject\.{{ env_short }}\.git\..*
- ^org\.fedoraproject\.{{ env_short }}\.infragit\..*
- ^org\.fedoraproject\.{{ env_short }}\.logger\.log\..*
Some questions you might ask:
- What happens if I try to send to a topic that is not allowed?
In this case fedora-messaging will raise an exception in the publish()
call
- What happens if I don't set the sent_topics list?
When the list is not set, all topics are allowed. Therefore if you don't
do anything, your app will technically keep working as before, but you will
make infra folks a bit sad because if your certificate gets compromised,
someone might send messages to the bus on any topic. If that happens, you
will feel bad. Take care of future you and set the variable now.
- What if I my app does not send any message?
Then set the sent_topic to a list containing a single element: ^$
- How do I test this?
At the moment, the sent_topics list is only taken into account on
staging. So what you can do is set it to a sensible value, run the
playbook(s) on staging, and check your applicaiton's logs for tracebacks
when a message should be sent.
- When do you plan to apply these restrictions on prod?
I don't know yet. When we are pretty confident that no topics have been
forgotten, we'll announce the prod activation here with a few days notice.
Please don't wait until then.
I've tried to set it for existing apps the best I could, but I may have
forgotten some topics you want to send to. Please verify your playbooks and
roles.
Then there's the issue of the accounts created in
roles/rabbitmq_cluster/tasks/apps.yml. My intent for this file was to
contain the account creations for applications that are not elsewhere in
ansible, such as CentOS applications, etc. As a result I can't examine
which topics these apps want to send to, because I don't even know which
apps use them. Please reach out to me if your application uses one of the
following rabbitmq accounts:
- coreos
- centos-ci
- osci-pipelines
- fedora-build-checks
- alt-src
- gitlab-centos
- koji-centos
- centos-koji
- cbs
- resultsdb-centos
- centos-stream-robosignatory
- distrobuildsync-eln
Thanks!
Aurélien
1 year, 9 months
CPE Weekly Update – Week 26 2022
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 27th of June to 1st of July 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update--week-26-2022
<https://communityblog.fedoraproject.org/?p=11183>
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-06-29.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Last app in openshift3 is old fas2/accounts. Once badges doesn’t use it
we can take down the openshift3 clusters.
* New mote in production: https://meetbot.fedoraproject.org
* Staging koji uplifted to f36.
* Business as usual (down to 60 tickets at one point!)
### CentOS Infra including CentOS CI
* Ongoing Duffy work (Fedora rawhide option available now)
* Granting SIGs “commit” rights to some projects on git.centos.org (still
using pagure-dist-git logic/filter) to let them review PR for their own
branches
* Finally back with redundancy/shared load to see mirror CDN (IAD2 node)
* Work in progress to let SIGs build against/for RHEL9
* Business as usual (mirrors, tags)
### Release Engineering
* Working on enabling IMA signing of rawhide packages
* New Fedora Media Writer release
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Fixed issue with changed host keys causing stall of packages into Stream
8.
* Continuing the work on getting CentOS Stream 8 closer to 9
## Package Automation (Packit Service)
Goal of this initiative
-----------------------
Automate RPM packaging of infra apps/packages
Updates
-------
* No major updates this week, work is winding down with a few PRs left to
be merged.
* Critical apps list in github is almost complete
## Flask-oidc: oauth2client replacement
Goal of this initiative
-----------------------
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need to
be replaced with authlib.
Updates:
--------
* Testing implementation for token expiry/refresh, final piece of work
required before submitting a PR upstream with changes.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* This week we have 6516 (+74) packages, from 2911 (+29) source packages
* Python Packaging Authority (PyPA) considering enabling EPEL by default in
the manylinux_2_28 image https://github.com/pypa/manylinux/issues/1338
* Working with RHEL maintainers to ship gtksourceview4-devel to enable more
packages
* Carl attended Open Source Summit North America and chatted with multiple
EPEL users and maintainers
Kindest regards,
CPE Team
1 year, 9 months
