CPE Weekly Update – Week 20 2022
by Michal Konecny
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
Week: 16th May - 20th May 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update---week-20-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-05-18.pdf
Update
------
### Fedora Infra
* Fedora Media Writer 5.0.1 is live/done for both windows and macos!
* Email from @redhat.com users to @fedoraproject.org users who use gmail
is broken due to SPF changes on rh side. [INC2210845
](https://redhat.service-now.com/surl.do?n=INC2210845)
* Switched to linux-system-roles.nbde_client (automatically unlocking
encrypted devices via network) role from our home grown incomplete one
after working thru a dracut bug in RHEL8.3+
* RHEL9 content synced and already switched epel9 to use it.
* Mass update/reboot in progress, outage later today to finish.
### CentOS Infra including CentOS CI
* CentOS Stream storage migration spike (Netapp for nfs/iscsi) (ongoing)
* Duffy fixes and tests (ec2 provisioning working)
* Git.centos.org pagure upgrade/migration (blocked, waiting on internal
Red Hat Team)
* 9 stream build targets on cbs/koji now consuming centos9s-buildroot
repositories straight from upstream kojihub for Stream 9 (no latency)
* Business as usual (mirrors, tags)
### Release Engineering
* Archiving older releases
* Updating the EOL release date will be part of the release schedule
### Any Other Bussiness
* All the zuul jobs are now migrated to [centralized
repository](https://pagure.io/fedora-infra/zuul)
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Changes to the c8s module migration code to automatically filter to
modules that are released.
* Migration of packages to new c8s infrastructure is moving along.
* Fixing the ELN Everything installer.
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to
provision and access bare metal resources of multiple architectures for
the purposes of CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We
have OpenNebula hypervisor available, and have started developing
playbooks which can be used to create VMs using the OpenNebula API, but
due to the current state of how Duffy is deployed, we are blocked with
new dev work to add the VM checkout functionality.
Updates
-------
* Deployment: apply database schema migrations
* Test per tenant quotas and provisioning on EC2
## Package Automation (Packit Service)
Goal of this initiative
-----------------------
Automate RPM packaging of infra apps/packages
Updates
-------
* Mostly business as usual, lots of dependencies
* A fair amount of manual packaging needs to happen for most of our
applications first
* Reducing a lot of dependency pinning
* Unfortunately packit [doesn’t support
monorepos](https://github.com/packit/packit/issues/1543) at the moment
so Bodhi and Datanommer will be blocked until they do. It’s on their
roadmap.
* scoady pto for ~2 weeks
## Flask-oidc: oauth2client replacement
Goal of this initiative
-----------------------
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need
to be replaced with authlib.
Updates:
--------
* Starting to
[implement](https://github.com/fedora-infra/test-auth/blob/authlib_dev/tes...
flask-oidc api using authlib.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* epel9 up to 2616 source packages (increase of 48 from last week).
* Completed transition of epel9 from building against c9s snapshot to
building against rhel9.
* Retired multiple epel8 packages that were added in rhel8.6.
* [Identified and
reported](https://bugzilla.redhat.com/show_bug.cgi?id=2084131) version
mismatch between c9s gcc and gcc-epel in epel9, working with maintainer
to provide a compatible version in epel9-next.
* [Ansible rebased to version 5 in
epel8](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-67f52f0700)
(major update) and [added to
epel9](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-73e215eb5c).
Kindest regards,
CPE Team
2 years
Fedora Contributor Tee Shirt Giveaway
by Vipul Siddharth
Hey all,
As we get ready to release Fedora Linux 36, another anticipated moment
has arrived: the Fedora contributor tee shirt giveaway! Sending a huge
“THANK YOU!!” to everyone who works to make Fedora the amazing
community it is. The Mindshare Committee is excited to be able to
share swag with our community to celebrate the latest release of
Fedora Linux.
Please visit the community blog post[0] to see how to claim yours!
[0] https://communityblog.fedoraproject.org/fedora-contributor-tee-shirt-give...
On behalf of the Mindshare committee,
--
Vipul Siddharth
He/His/Him
Mentored Project @ Mindshare | DEI advisor @ Fedora Council
2 years
Fedora Copr, new Power9 builders
by Pavel Raiskup
Hello all,
Fedora infra folks recently racked for a new Power9 machine for Fedora
Copr, and we now finished the installation. The new machine is hosting
~30 production builder VMs). This machine is going to replace the two
older Power8 boxes.
To fixup some startup issues we had to provide a new golden image, so
while on it—golden images for all our infrastructure locations were
regenerated (all architectures).
This is JFYI e-mail, should you have any issues please report them.
Happy building!
Pavel
2 years
Planned Outage - Updates / Reboots - 2022-05-18 20:00 UTC
by Mark O'Brien
Hi All,
There will be an outage starting at 2022-05-18 20:00 UTC
which will last approximately 4 hours.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2022-05-18 20:00 UTC'
Reason for outage:
We will be updating all our instances and rebooting hardware nodes into new
kernels. This will catch us up on security updates after the finishing of
the Fedora 36 final freeze.
Affected Services:
All services may experience slowdowns or short outages as servers are
rebooted and return to service. We will try and minimize disruptions, but
users/maintainers are advised to avoid activity in the outage window.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/10692
Please join #fedora-admin or #fedora-noc on irc.libera.chat
or add comments to the ticket for this outage above.
Mark
2 years
CPE Weekly Update – Week 19 2022
by Michal Konecny
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
Week: 09th May - 13th May 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update---week-19-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-05-11.pdf
Update
------
### Fedora Infra
* F34/F35 container builds failing due to 32bit arm (
https://bugzilla.redhat.com/show_bug.cgi?id=2077680 )
* git -core change broke koji. Downgraded git and upstream koji already
has a fix.
* Got a FMW macos build fully signed and notarized! Unfortunately, now
need to find out how to build it to be able to run on older macos. ;(
* Fedora 36 release went pretty smoothly, we are now out of Freeze
* Business as usual, misc tickets, etc.
### CentOS Infra including CentOS CI
* CentOS Stream storage migration spike (Netapp for nfs/iscsi)
* Duffy fixes and tests
* Investigating hardware issue on CI pool
* Investigating ci.centos.org decommission steps
* Git.centos.org pagure upgrade/migration (blocked, waiting on internal
Red Hat Team)
* Updated sshd host key signing (sha1 issue for el9 ssh clients)
* Bussiness as usual (mirrors, tags)
### Release Engineering
* F36 is out
* Firmware win binaries signed
* Bussiness as usual - stalled epel packages, package unretirements
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Finished the RPM import for c8s to Stream Koji
* Business as usual otherwise
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to
provision and access bare metal resources of multiple architectures for
the purposes of CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We
have OpenNebula hypervisor available, and have started developing
playbooks which can be used to create VMs using the OpenNebula API, but
due to the current state of how Duffy is deployed, we are blocked with
new dev work to add the VM checkout functionality.
Updates
-------
* More deployment tests
* Per tenant session lifetimes
* Some bug fixes
## Package Automation (Packit Service)
Goal of this initiative
-----------------------
Automate RPM packaging of infra apps/packages
Updates
-------
* The team is hitting lots of dependency and sub dependency issues,
working through them but its slow
* fasjson-client is our first package to be fully automated
* upstream release -> src.fp.o PR -> koji -> bodhi
* Thanks to Nils, Aurelien and Kevin for their help/advice
* fedora-messaging, datagrepper, fasjson currently being worked on (all
have deps issues)
* spec files will be staying downstream, packit has a way to facilitate this
## Flask-oidc: oauth2client replacement
Goal of this initiative
-----------------------
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need
to be replaced with authlib.
Updates:
--------
* Setup dev environment (Work In Progress!)
* Starting to implement flask-oidc api using authlib.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* epel9 up to 2568 source packages (increase of 113 from last week).
* Added rhel+epel-9 mock configs to mock-core-configs.
* Updated slurm in epel7 and epel8 to fix CVE-2022-29500 and CVE-2022-29501.
* Retired swtpm and libtpms from epel8 because they were added to RHEL8.6.
* Added python-texttable to epel9 to allow c8s maintainers (Johnny) to
run c9s as a workstation.
* Added missing devel packages for cogl, clutter, and clutter-gtk to
epel9 to unblock other epel9 requests.
Kindest regards,
CPE Team
2 years
attention: resultsdb changes
by Kevin Fenzi
Greetings everyone.
For a while now we have been wanting to make some changes to resultsdb
and these changes are now going to happen soon!
Basically we are moving it into openshift, adding password protection
for posting to it (before it was just a simple network check) and some
other minor changes like adding some new states. We also want to move it
to it's own url instead of leaving it under the taskotron virtual host
(since taskotron is no more. RIP).
We already have the new versions all setup in staging (along with the
old one still running).
old url: https://taskotron.stg.fedoraproject.org/resultsdb
new url: https://resultsdb.stg.fedoraproject.org/
So, what does this mean?
As far as I am aware, only 2 things post results to resultsdb.
1. resultsdb-ci-listener (listens for ci fedora-messages and posts
results). We control this application and it will be moving into
openshift as well.
2. openqa. Which AdamW can adjust when we move prod, but we have tested
and stg can post fine.
Many other things query/read/display links to results and will need to
be updated after we switch prod:
bodhi
pagure/distgit
greenwave
monitor_gating
waiverdb
If app owners could ready PR's for these changes that would be great,
otherwise we can make them before switching.
So, did we forget anything? Any issues or problems anyone can see?
We would ideally like to move into prod monday, but can delay if there's
issues or concerns.
Thanks!
kevin
2 years
