Plan / proposal: enable openQA update testing and potentially
gating on Rawhide updates
by Adam Williamson
Hi folks!
We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
openQA results).
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
pushed.
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
results).
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
10 months, 2 weeks
Small rant: installer environment size
by Adam Williamson
Hi folks! Today I woke up and found
https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me
down a bit of an "installer environment size" rabbit hole.
As of today, with that new dep in webkitgtk, Rawhide's network install
images are 703M in size. Here's a potted history of network install
image sizes:
Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M)
Fedora 13: 208M
Fedora 17: 162M (last "old UI")
Fedora 18: 294M (first "new UI")
Fedora 23: 415M
Fedora 28: 583M
Fedora 33: 686M
Fedora 37: 665M
Fedora Rawhide: 703M
The installer does not really do much more in Rawhide than it did in
FC8. Even after the UI rewrite in F18, we were only at 294M. Now the
image is well over 2x as big and does...basically the same.
Why does this matter? Well, the images being large is moderately
annoying in itself just in terms of transfer times and so on. But more
importantly, AIUI at least, the entire installer environment is loaded
into RAM at startup - it kinda has to be, we don't have anywhere else
to put it. The bigger it is, the more RAM you need to install Fedora.
The size of the installer environment (for which the size of the
network install image is more or less a perfect proxy) is one of the
two key factors in this, the other being how much RAM DNF uses during
package install.
So, I did a bit of poking about into *what* is taking up all that
space. There's a variety of answers, but there's two major culprits:
1. firmware
2. yelp (which pulls in webkitgtk and its deps)
I've been using du and baobab (the GNOME visual disk usage analyzer,
which is great) to examine the filesystems, but I ran a couple of test
builds to confirm these suspects, especially after the impact of
compression (it's hard to check the *compressed* size of things in the
installer environment directly).
I did a scratch build of lorax which does not pull in firmware
packages, and had openQA build a netinst using that lorax. It came out
at 489M - 214M smaller than current netinsts, a size we last managed in
Fedora 26. I did a scratch build of anaconda with its requirement of
yelp dropped (which would break help pages), and built a netinst with
that; it came out at 662M - 41M smaller than current images. I haven't
run a combined test yet, but it ought to come out around 448M, around
the size of Fedora 24.
Even then we'd still be about 50% larger than the Fedora 18 image, for
not really any added functionality.
I've moaned about the sheer amount and size of firmware blobs in other
forums before, but 214M compressed is *really* obnoxious. We must be
able to do something to clean this up (further than it's already
cleaned up - this is *after* we dropped low-hanging fruit like
enterprise switch 'firmwares' and garbage like that; most of the
remaining size seems to be huge amounts of probably-very-similar
firmware files for AMD graphics adapters and Intel wireless adapters).
I know some folks were trying to work on this (there was talk that we
could drop quite a lot of files that would only be loaded by older
kernels no longer in Fedora); any news on how far along that effort is?
Other obvious things that take up a lot of space:
1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my
system shows it compresses to around 11M, though, so that's probably
all it adds up to after compression (the image is an xz-compressed
squashfs)
2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
23M. We are, I think, basically stuck with this for mesa-dri-drivers ,
but does it have to be so *big*?
3. libicudata.so.71.1 - 30.4M, compresses to 7M. This is in the
webkitgtk dep chain but seems to still be pulled in without it, not
sure what else is requiring it.
4. /usr/share/locale - 112M in total (uncompressed, not sure how much
compressed) of translated strings from a ton of packages. No idea how
many of these are really *needed* in the installer environment. We can
maybe come up with a way to have lorax strip some, if we can come up
with a viable way to figure out which. Obviously-fairly-large ones are
from gnupg2 and libgweather4. I do recall we have some logic somewhere
to decide which languages have a certain level of translation in
anaconda; perhaps we could only include the strings for these
languages?
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 2 months
Fedora Workstation WG minutes, 2023-01-31
by Chris Murphy
==============================================
#fedora-meeting-2: Workstation WG (2023-01-31)
==============================================
Meeting started by brainycmurf at 17:12:05 UTC. The full logs are
available at
https://meetbot.fedoraproject.org/fedora-meeting-2/2023-01-31/workstation...
.
Meeting summary
---------------
* Present members: Michael, Owen, Kalev, Allan, Chris, Neal
(brainycmurf, 17:12:17)
* Guests: Jonathan, Justin (brainycmurf, 17:12:17)
* Regrets: Matthias, Jens, Tomas (brainycmurf, 17:12:17)
* Secretary: Owen (brainycmurf, 17:12:17)
* Fedora doesn't have kernel parameters for enabling IOMMU
(brainycmurf, 17:12:17)
* LINK: https://pagure.io/fedora-workstation/issue/346 (brainycmurf,
17:12:18)
* Reconsider scope of Fedora Flatpaks (brainycmurf, 17:12:40)
* LINK: https://pagure.io/fedora-workstation/issue/352 (brainycmurf,
17:12:42)
* Mouse acceleration options are unavailable (brainycmurf, 17:13:00)
* LINK: https://pagure.io/fedora-workstation/issue/354 (brainycmurf,
17:13:04)
* Announcements and status updates (brainycmurf, 17:13:09)
* LINK: https://pagure.io/fedora-workstation/issue/278 (brainycmurf,
17:13:13)
* The minutes from the previous meeting have been posted.
(brainycmurf, 17:13:15)
* LINK:
https://meetbot.fedoraproject.org/fedora-meeting-2/2023-01-30/workstation...
(brainycmurf, 17:13:17)
Meeting ended at 17:13:20 UTC.
Action Items
------------
Action Items, by person
-----------------------
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* brainycmurf (33)
* zodbot (7)
* Michael (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
1 year, 2 months
Proposed change to rendering of release schedule ics file
by Ben Cotton
Hi everyone,
I want to get feedback before I make a change to how Fedora Linux
schedules are generated. As reported in schedule#91[1], the current
ics files include very long tasks that aren't particularly useful. I
have a plan for how to address that, but first I want to check that it
won't impact how people currently use the ics files.
If you're using the Fedora Linux schedule ics files, please see the
details in the Pagure issue[2]. If this will affect how you or your
team use those files, comment in the issue before 13 February. Note
that this will not change how the html or json versions are produced.
[1] https://pagure.io/fedora-pgm/schedule/issue/91
[2] https://pagure.io/fedora-pgm/schedule/issue/91#comment-838492
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 2 months
Fedora Workstation WG minutes, 2023-01-24
by Chris Murphy
==============================================
#fedora-meeting-2: Workstation WG (2023-01-24)
==============================================
Meeting started by brainycmurf at 20:00:16 UTC. The full logs are
available at
https://meetbot.fedoraproject.org/fedora-meeting-2/2023-01-30/workstation...
.
Meeting summary
---------------
* Present members: Allan, Kalev, Michael, Tomas, Chris, Jens, Neal,
Matthias, Owen (brainycmurf, 20:00:28)
* Guests: Bastien, Sergio (brainycmurf, 20:00:28)
* Regrets: (brainycmurf, 20:00:28)
* Secretary: Allan (brainycmurf, 20:00:28)
* Switching default CPU governor to a more powerful one (brainycmurf,
20:00:28)
* LINK: https://pagure.io/fedora-workstation/issue/212 (brainycmurf,
20:00:28)
* LINK:
https://gitlab.freedesktop.org/hadess/power-profiles-daemon#operations-on...
(brainycmurf, 20:00:51)
* ACTION: we should close this ticket and ask the reporter to file a
bug against power-profiles-daemon. (brainycmurf, 20:01:09)
* ACTION: Owen to provide some followup questions on the
power-profile-daemon ticket. (brainycmurf, 20:01:17)
* Reconsider scope of Fedora Flatpaks (brainycmurf, 20:01:19)
* LINK: https://pagure.io/fedora-workstation/issue/352 (brainycmurf,
20:01:21)
* Mouse acceleration options are unavailable (brainycmurf, 20:01:25)
* LINK: https://pagure.io/fedora-workstation/issue/354 (brainycmurf,
20:01:27)
* Announcements and status updates (brainycmurf, 20:01:31)
* The Fedora Flatpak SIG had its first meeting this week
(brainycmurf, 20:01:33)
* LINK: https://etherpad.opensuse.org/p/fedora-flatpaks-sign
(brainycmurf, 20:01:42)
* GNOME 44 UI freeze is in ~3 weeks (brainycmurf, 20:01:44)
* Allan has published the results of the gnome-info-collect exercise
(brainycmurf, 20:01:46)
* LINK:
https://blogs.gnome.org/aday/2023/01/18/gnome-info-collect-what-we-learned/
(brainycmurf, 20:01:48)
* The minutes from the previous meeting have been posted.
(brainycmurf, 20:01:50)
* LINK:
https://meetbot.fedoraproject.org/fedora-meeting-2/2023-01-17/workstation...
(brainycmurf, 20:01:52)
Meeting ended at 20:04:16 UTC.
Action Items
------------
* we should close this ticket and ask the reporter to file a bug against
power-profiles-daemon.
* Owen to provide some followup questions on the power-profile-daemon
ticket.
Action Items, by person
-----------------------
* **UNASSIGNED**
* we should close this ticket and ask the reporter to file a bug
against power-profiles-daemon.
* Owen to provide some followup questions on the power-profile-daemon
ticket.
People Present (lines said)
---------------------------
* brainycmurf (41)
* zodbot (7)
* Michael (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
1 year, 2 months
Summary/Minutes from today's Fedora Flatpak Packaging SIG Meeting
(2023-01-23)
by Kalev Lember
=============================================
#fedora-meeting: Fedora Flatpak Packaging SIG
=============================================
Meeting started by kalev at 15:00:33 UTC. The full logs are available at
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-23/flatpak-sig.2...
.
Meeting summary
---------------
* First Meeting (kalev, 15:00:48)
* Introductions (kalev, 15:02:41)
* LINK: https://etherpad.opensuse.org/p/fedora-flatpaks-sig ?*
(travier, 15:03:05)
* Announcements (kalev, 15:09:26)
* yselkowitz has added ~20 new flatpaks over the last two weeks and
got flatseal packaged -- awesome work! (kalev, 15:10:16)
* Why do we need Fedora Flatpaks? (kalev, 15:16:22)
* openh264 (kalev, 15:31:44)
* Kalev has a plan how to do the openh264 flatpak runtime extension
using Cisco hosted openh264 rpms (kalev, 15:40:49)
* LINK: https://github.com/fedora-silverblue/issue-tracker/issues/288
(travier, 15:43:17)
* flatpak-sig Fedora group and access to flatpaks/ namespace in dist-git
(kalev, 15:45:43)
* ACTION: Kalev to set up flatpak-sig pagure group for flatpaks/
namespace (kalev, 15:49:21)
* KDE/Qt runtime? (kalev, 15:50:55)
* The KDE SIG is interested in taking the KDE/Qt runtime under KDE SIG
umbrella (kalev, 15:58:56)
Meeting ended at 16:00:58 UTC.
Action Items
------------
* Kalev to set up flatpak-sig pagure group for flatpaks/ namespace
Action Items, by person
-----------------------
* **UNASSIGNED**
* Kalev to set up flatpak-sig pagure group for flatpaks/ namespace
People Present (lines said)
---------------------------
* kalev (86)
* travier (55)
* tpopela[m] (18)
* zodbot (13)
* yselkowitz[m] (12)
* OwenTaylor[m] (10)
* rishi[m] (9)
* JanGrulich[m] (8)
* TheEvilSkeleton (7)
* AllanDay[m] (4)
* JanBeran[m] (1)
* adamw (1)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
1 year, 3 months
