CPE Weekly Update – Week 33 2022
by Michal Konecny
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
Week: 15th August - 19th August 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update---week-33-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-17.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Mass update/reboot cycle this week (stg/nonoutage done, outage later
today)
* Freeze for f37 beta starts next week
### CentOS Infra including CentOS CI
* Discussion around the CentOS Stream infra hand over
* New tasks for the CI infra migration
* S3 bucket for the Stream CoreOS effort
* Some infra projects moved from gitea to gitlab
### Release Engineering
* Openh264 composes fo f36,37,38 send to cisco
* Package retirement issues after the branching, thanks to human error
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Face to face meeting in Boston.
* Penultimate parts of Module process sync. Between el8 and el9.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* EPEL9 is up to 7339 (+106) packages from 3278 (+71) source packages
* Found the bloaty package was uninstallable because of a libre2 soname
fix, rebuilding it fixed the issue.
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users
to create filters on messages sent to (currently) fedmsg and forward
these as notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Frontend
* Use CoreUI components
* Set up i18n
* Initial version of a “New Rule” page
* Authentication integration FE/BE (ongoing)
* Backend: SQLAlchemy integration (ongoing))
Kindest regards,
CPE Team
1 year, 8 months
Planned Outage - Updates / Reboots - 2022-08-17 20:00 UTC
by Mark O'Brien
Hi,
We are planning to get everything in order before the beta freeze next week
so require a small outage.
Planned Outage - Updates / Reboots - 2022-08-17 20:00 UTC
There will be an outage starting at 2022-08-17 20:00 UTC.
which will last approximately 4 hours.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2022-08-17 20:00 UTC'
Reason for outage:
We will be making sure all servers are up to date and rebooted into the
latest kernels.
Affected Services:
Many services will be affected for short times as various servers are
rebooted. Users and Maintainers are advised to wait until after the outage
to avoid any service problems.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/10854
Please join #fedora-admin or #fedora-noc on irc.libera.chat
or add comments to the ticket for this outage above.
Mark
1 year, 8 months
Debugging Datanommer performance issues
by Aurelien Bompard
Hey folks!
There's been a report of queries long enough to cause a timeout in
datagrepper:
https://github.com/fedora-infra/datagrepper/issues/467
I don't think those queries should take so much time, and I'd like to debug
this performance issue, possibly try a couple new indexes on the tables,
etc. However, I can't reproduce the issue on staging, probably because the
datanommer database there is much smaller.
So I was wondering what is the best course of action. I see the following
options:
1. Sync the prod DB to staging. This looks like an obvious first choice,
but the messages in the staging DB actually come from the staging
environment and are used by other contributors to check that their service
is behaving properly on staging. Also, the topic prefix of the messages is
different on staging and on prod, so syncing the DB with prod messages and
then adding staging ones may cause a mess.
2. Having a second datanommer DB in prod, and syncing them. The problem
here is of course the disk space required. I don't know if that's even
possible with the hardware we have.
3. Something else?
What do you think?
Thanks!
Aurélien
1 year, 8 months
congrats to our new sysadmin-mainers
by Kevin Fenzi
I'm happy to announce that We have approved several folks into the
sysadmin-main group:
nphilipp - Nils Philippsen
zlopez - Michal Konečný
ryanlerch - Ryan Lerch
This is the core group of trusted folks that high level access to most
everything in fedora infrastructure.
Nils is a long time contributor who has worked on a bunch of things,
including our ipa sudo/ssh permissions setup.
Michal has worked on many of our applications and is also serving to
help keep us on plan for planned work over the quarters.
Ryan has been around for a long time working on many things, including
our revamped status application, FMN, gitlab permissions and more.
They have both proved their dedication, trustworthiness, and ability.
Congrats to them all!
Use your powers for good! :)
kevin
1 year, 8 months
Self-introduction
by Mark E. Fuller
Hi all,
I'm Mark Fuller (FAS: fuller) a long-time (since 16/Verne) Fedora-user and packager.
I recently made the move from being an engineering researcher who managed the group IT on the side to being a full-time devops engineer.
As such, I'm looking to start getting involved with Fedora Infrastructure to apply what I'm doing and learning in my day job to contribute to the community and also to get more exposure to systems outside of work to develop a wider skill-set.
I'm open to suggestions as to where to start - I saw in the latest CPE announcement about the FMN replacement (but not links). Since I'm tinkering around the edges with RabbitMQ a bit, this is particularly interesting to me if it's an AMQP-based system.
Other than that, I'm competent in BASH and Python scripting and have some experience with Go as well. I primarily do Kubernetes deployments and monitoring and would also be particularly interested in working on CI/CD and authentication.
Thanks
-fuller
Mark E. Fuller, Ph.D.
+972 (0)53-872-6579
+49 (0)1577-1848188
fuller(a)fedoraproject.org
mark.e.fuller(a)gmail.com
@fuller:one.ems.host
@fuller:fedora.im[http://fedora.im]
https://www.stossrohr.net[https://www.stossrohr.net/]
PGP Fingerprint: 73F1 A30C BDF4 DB4B C75F FD0F D599 E76C FFCA BF60
1 year, 8 months
CPE Weekly Update – Week 32 2022
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 8th - 12th August 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/?p=11316
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-10.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Some great nest talks and discussions, check them out on replay!
* Debugging instability in 32bit arm builders again. ;(
* Rebalanced the s390x builders.
* Business as usual
### CentOS Infra including CentOS CI
* koji/cbs.centos.org upgrade complete
* Kick off for Stream infra handover to I&R
* Adapted ansible baseline role to include workaround for Rsync issue on
CentOS Stream 8
* Enabled RHEL9 Buildroots on cbs/koji for SIGs
### Release Engineering
* Mass branching yesterday (f37 split off rawhide, which is now f38)
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Git source moved from git.centos.org to gitlab, for c8s modules.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 is up to 7233 (+106) packages from 3207 (+27) source packages
* “State of EPEL” presentation at Nest conference, recording will be posted
to YouTube at a later date
* EPEL Survey has launched and is available through the end of August
https://tinyurl.com/epelsurvey2022
* epel-release has been improved with a recommends on
dnf-command(config-manager) to ensure crb enabling script can work out of
the box
* KDE Plasma updated from 5.23 to 5.24 (LTS release) in epel8-next and
epel9-next
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users to
create filters on messages sent to (currently) fedmsg and forward these as
notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Unit tests/coverage tests on frontend (Vue.js)
* Auth/OIDC work on both frontend and backend
* Initial backend connection via SQLAlchemy/fastAPI
* Basic functionality of connecting to FASJSON
* CI improvements and fixes
Kindest regards,
CPE Team
1 year, 8 months
CPE Weekly Update – Week 31 2022
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 1st - 5th August 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update-week-31-2022
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-03.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Unblocked osbuild in production, should be working now. (Uses script to
keep api ip updated in the firewalls)
* Ocp4 cluster api uses valid cert not (for webhooks/external oc)
* Disabled systemd-oomd in some places (koji hubs in particular)
* Barcamp at nest on saturday: https://hackmd.io/MiWdaZXCRDi4LEO1jmyLDg
* Some sysadmin-main additions: Nils, Michal, Ryan
### CentOS Infra including CentOS CI
* Duffy CI is now live (so hotfixes are also coming, thanks to Nils)
* Preparing CBS/koji upgrade to 1.29 (would unblock other RFEs on tracker)
### Release Engineering
* FTBFS bugs filed on failing to build packages
* Containers: rawhide fixed/updating, updated f35/f36
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Meetings about and started code, to move module source from git.centos to
gitlab.
* New ISOs for CentOS Linux 7 for installation that fixes libtimezonemap
(and other) issues.
* Rewrote the errata announcement scripts for CentOS Linux 7 to use new
endpoints after the decommissioning of the API search/rs/ on
access.redhat.com.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 is up to 7127 (+141) packages from 3180 (+97) source packages
* Prepared EPEL survey, it will be promoted on FedoraNest
* Provided a fix to nagios-plugins-check-updates to improve distro
compatibility
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users to
create filters on messages sent to (currently) fedmsg and forward these as
notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Frontend auth being developed
* Access token and refresh token
* Making pages require auth, if user is not authenticated, redirect to
login
* Backend auth still being developed (tests)
* Mockups for UI - bootstrap/HTML/CSS
* Agile ceremonies being planned
Kindest regards,
CPE Team
1 year, 8 months
nest infra&releng mini barcamp
by Kevin Fenzi
I put in for a 1/2 day mini hackfest for infra and releng at nest.
See:
https://pagure.io/flock/issue/393
My thought was to collect ideas for cool things we could work on or do,
vote on them before the event and do them in most popular order.
(bar camp style).
Please do chime in here or in the flock issue if you have ideas or would
rather do something else. :)
kevin
1 year, 8 months
