Hi all,
I am currently trying to open a UI / UX review of the FMN /
Notifications system and just chasing isome raw feedback on how you
all get notifications (email / IRC / or otherwise) when developing /
packaging / working on Fedora.
Please check out the following discussion thread to log your raw feedback:
https://discussion.fedoraproject.org/t/notifications-application-aka-fmn-ak…
cheers,
ryanerch
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/)
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update-week-of-february-…
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
take on.
Update
------
### Fedora Infra
* Got all armv7 builders moved to a stable install, hopefully they will
keep this until f36 (and 32bit arm) goes EOL
* In the middle of a mass update/reboot cycle. Also updating all servers to
latest bios and drac firmware and updating passwords. (outage later today!
- Wed February 16th)
* Got working ppc64le builder for FCOS. Had to reinstall a power9 with
Fedora 35 instead of RHEL8 as hypervisor.
* Cleaned up our dns repo (was up to 5GB). Everyone should check out a new
copy if you work with it.
* Bugzilla is forcing everyone to move to API tokens by end of month. All
applications MUST move to that or risk their accounts being locked!
* Fedimg is currently not working
### CentOS Infra including CentOS CI
* Cbs/koji sending notifications to fedora-messaging(
https://pagure.io/centos-infra/issue/650)
* Still hardware issues on main node in IAD2 (Work In Progress)
* Fixed www.centos.org deployment (different branch)
* Ansible-core now pushed to 8-stream so reverting for our own infra
ansible hosts
* Business as usual
### Release Engineering
* Branched (36) and rawhide (37) composes now on track and running fine.
### Any other bussiness
* The-new-hotness is being prepared for the Bugzilla API Authentication
change
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* cenpkg for EPEL 9 in testing (
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-fa2059dc76)
* the team is taking ownership of CVE checker, we'll follow up on future
plans
* business as usual
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to provision
and
access bare metal resources of multiple architectures for the purposes of
CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the current
state
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.
Updates
-------
* Finish provisioning and deprovisioning of nodes in background node pool
tasks
* Tie up loose ends (ongoing)
* Add some lines to the Q4 community blog post
## Image builder for Fedora IoT
Goal of this Initiative
-----------------------
Integration of Image builder as a service with Fedora infra to allow Fedora
IoT migrate their pipeline to Fedora infra.
Updates
-------
* 2 asks from the Image Builder team related to our koji. Detailed here:
https://pagure.io/fedora-infrastructure/issue/10561
* currently looking at pungi and how the fedora IoT team orchestrates their
builds
* 3 blockers on Image Builders side currently:
* koji service account issue above (in progress?)
* auth and API changes on their end (completed)
* they need to add multi-tenancy (in progress)
* Some small contributions upstream to plugins
## Bodhi
Goal of this Initiative
-----------------------
This initiative is to separate Bodhi into multiple sub packages, fix
integration and unit tests in CI, fix dependency management and automate
part of the release process.
Read ARC team findings in detail at:
https://fedora-arc.readthedocs.io/en/latest/bodhi/index.html
Updates
-------
* Move Bodhi from python-openid to OIDC (ongoing)
* Build beta / staging RPMs automatically in Koji (ongoing)
* Dependency management (ongoing)
* Fix the specfiles and RPMs since the split of packages (ongoing)
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 up to 1907 source packages (increase of 132 from last week)
* At current pace will surpass C9S in three weeks
* At current pace will surpass EPEL8 by June
* Fixing various fail-to-install bugs
* centpkg added to epel9 testing so maintainers can more easily dogfood c9s(
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-fa2059dc76)
* Presented State of EPEL to Fedora Council
Kindest regards,
CPE Team
Greetings all,
We will be doing a mass update/reboot next week. We haven't done
one in a couple of months and would like to get all up to date before the
upcoming beta freeze.
Rough plan:
* Monday (2022-02-14): *.stg.iad2.fedoraproject.org. update/reboot all
staging hosts. We need to watch for the openshift clusters and make sure
they come back up correctly.
* Tuesday (2022-02-15): any non outage causing hosts. This would include
proxies and their virthosts since we can take them out of dns. Also
might include some virthosts in iad2 as long as all the guests on them
have counterparts on other virthosts to take the load.
* Wed (2022-02-16): outage starting at 21UTC.
Everything else left that we haven't done yet we do here.
Outage ticket: https://pagure.io/fedora-infrastructure/issue/10536
This is mostly on Kevin and I, since it needs sysadmin-main, but our
SOP/doc on this is horribly out of date if someone(s) would like to work
on updating it. (
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/massupgrade/ )
Also there's talking to users during the outage and testing things after we
are done.
Thanks,
Mark
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/)
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update-week-of-february-…
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
take on.
Update
------
### Fedora Infra
* Got koji issues sorted and things back on track
* Worked with Fedora CoreOS folks on bringing up a ppc64le builder,
which exposed a virt issue we have. Will be installing a power9 box with
Fedora 35 to test if it fixes it.
* Got more ocp4 workers added, upgraded clusters a few times without
incident
* First live prod app on ocp4
([blockerbugs](https://qa.fedoraproject.org/blockerbugs/))
* Outage Friday caused by disk space issue on proxies. ;(
* 2 10TB iscsi volumes setup on fedora iad2 netapp for CentOS folks.
* Fedimg is broken apparently, caused ci issues. Still need to find a
fix to it. ( https://pagure.io/fedora-infrastructure/issue/10532 )
### CentOS Infra including CentOS CI
* Still storage/hardware issues to work on
* [Dell server](https://pagure.io/centos-infra/issue/649) for
iscsi/netapp usage
* CentOS [backup server](https://pagure.io/centos-infra/issue/618)
* https://debuginfod.centos.org should be
[live/announced](https://lists.centos.org/pipermail/centos-devel/2022-Februa…
(content for CentOS Stream 8 and 9 and SIGs packages)
* Bussiness as usual
### Release Engineering
* Branching of F36
* Rawhide nightlies are enabled again
* F37 builds are being signed with F37 key
* We have a Fedora 36 branched compose done already
* Started work on [automation of scm
requests](https://pagure.io/releng/issue/9274)
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* February planning done
* Adding centpkg to EPEL
* Continuing aligning c8s + c9s workflows with an optimum delivery
date now agreed
* Content Resolver upgrades continuing with maintainer page being added
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to
provision and
access bare metal resources of multiple architectures for the purposes of
CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the
current state
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.
Updates
-------
* Legacy API meta client merged
* Provisioning/Deprovisioning implemented (in review)
* Outstanding: integration tests & loose ends
## Image builder for Fedora IoT
Goal of this Initiative
-----------------------
Integration of Image builder as a service with Fedora infra to allow
Fedora IoT migrate their pipeline to Fedora infra.
Updates
-------
* local dev environment successfully deployed
* koji in containers, mocked oidc auth, builder in containers, and
osbuild-compose running as a service
* We can run a full koji build using the image-builder plugins to make a
successful compose
## Bodhi
Goal of this Initiative
-----------------------
This initiative is to separate Bodhi into multiple sub packages, fix
integration and unit tests in CI, fix dependency management and automate
part of the release process.
Read ARC team findings in detail at:
https://fedora-arc.readthedocs.io/en/latest/bodhi/index.html
Updates
-------
* Move Bodhi from python-openid to OIDC (ongoing)
* Automate staging process for RPM building in Koji (ongoing)
* Dependency management via Poetry and dependabot (ongoing)
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* EPEL9 up to 1775 source packages (increase of 127 from last week)
* [RDO considering shipping openstack client packages in
EPEL](https://meetings.opendev.org/meetings/rdo_meeting___2022_02_09/2022/r…
* EPEL talks at CentOS Dojo
* [State of EPEL on CentOS](https://youtu.be/JqnENGdvG88)
* [Bootstrapping new EPEL releases with
ebranch](https://youtu.be/VjPZmq_h2Rk)
* [EPEL Packaging Hackfest](https://youtu.be/vZ-CX5XJV2Q)
Kindest regards,
CPE Team
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/)
If you wish to read this in the form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/?p=10714
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
take on.
Update
--------
### Fedora Infra
* All fedora ansible hosts (except aws) now using
linux-system-roles/network for network config!
* Several upstream koji issues. One tagging issue fixed, one new one still
being worked on. ;(
* Got networking setup so CentOS iad2 master mirror could read Fedora
netapp. (to prep for adding space there for them)
* Invalid users disabled (ones with _ - . and one char)
### CentOS Infra including CentOS CI
* Centos 8 has been retired and removed from mirrors
* Big [storage reorganization](https://pagure.io/centos-infra/issue/618)
(multiple HDD issues, or pending - pfa mode-) , going from out of warranty
to still out of warranty node (risk mitigation)
* Business as usual
### Release Engineering
* Rawhide composes failing, been a long string of bugs
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* New version of [Content Resolver](https://tiny.distro.builders/) with an
integrated buildroot resolver is live! There are testing views ([ELN](
https://tiny.distro.builders/view--view-eln-test.html) and [Stream](
https://tiny.distro.builders/view--view-c9s-test.html)) using the new
resolver, the proper ones will be switched to that soon after some extra
validation. All following updates will be again showing up as they come,
this bigger one just had to come at once.
* Started the work on bringing CS8 and CS9 workflows together.
* Otherwise business as usual.
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to provision
and
access bare metal resources of multiple architectures for the purposes of
CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the current
state
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.
Updates
-------
* Still ongoing :)
* Legacy API
* Backend tasks integration
## Image builder for Fedora IoT
Goal of this Initiative
-----------------------
Integration of Image builder as a service with Fedora infra to allow Fedora
IoT migrate their pipeline to Fedora infra.
Updates
-------
* End to end running(ish) locally
* Configuring of the plugins this week
* Image builder team still blocking but not for long as they are landing
PRs now
* Spent some time as a team understanding the full e2e flow of osbuild and
how it will fit in the Fedora IoT architecture
## Bodhi
Goal of this Initiative
-----------------------
This initiative is to separate Bodhi into multiple sub packages, fix
integration and unit tests in CI, fix dependency management and automate
part of the release process.
Read ARC team findings in detail at:
https://fedora-arc.readthedocs.io/en/latest/bodhi/index.html
Updates
-------
* Bodhi has been split into multiple packages
* bodhi-ci has been cleaned up (split into multiple source files)
* Vagrant box now works out of the box (previously some systemd services
were failing after provisioning)
* CI tests now run using GitHub Actions
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 up to 1648 source packages (increase of 109 from last week)
* Three EPEL talks happening at CentOS Dojo, starting tomorrow
Kindest regards,
CPE Team