FESCo wants to know what you use i686 packages for
by David Cantrell
Hi,
Our most recent FESCo meeting involved discussing the proposal to drop i686
builds of jdk8,11,17 from Fedora 37 onward. The topic quickly changed to the
larger question of "what do people use i686 packages for?"
Rather than guess, we wanted to ask the community what you use i686 packages
for in Fedora. There are no wrong answers here. We are seeking information.
Why? Since the removal of the i686 kernel in Fedora, we want to reduce the
number of i686 packages provided in the repo. As time marches on, the ability
to build a lot of things for i686 becomes unrealistic or even impossible.
Remember it goes beyond providing builds...providing support, bug fixes, and
security fixes for those packages too. Maybe some things using i686 packages
now can move to x86_64 packages. We do not know yet, but a goal is to figure
out what packages, if anything, can drop their i686 builds.
NOTE: Nothing is changing now. We are in an information gathering phase.
~~~~~~~~~~~~~~~~~~~~~~~~
If you use i686 packages for something now, please respond to this thread.
Thanks,
--
David Cantrell <dcantrell(a)redhat.com>
Red Hat, Inc. | Boston, MA | EST5EDT
2 weeks, 5 days
RISC-V -- are we ready for more, and what do we need to do it?
by Matthew Miller
Hi all! I just got back from Open Source Summit, several of the talks I
found interesting were on RISC-V -- a high-level one about the
organizational structure, and Drew Fustini's more technical talk.
In that, he noted that there's a Fedora build *, but it isn't an official
Fedora arch. As I understand it, the major infrastructure blocker is simply
that there isn't server-class hardware (let alone hardware that will build
fast enough that it isn't a frustrating bottleneck).
So, one question is: if we used, say, ARM or x86_64 Amazon cloud instances
as builders, could we build fast enough under QEMU emulation to work? We
have a nice early advantage, but if we don't keep moving, we'll lose that.
But beyond that: What other things might be limits? Are there key bits of
the distro which don't build yet? Is there a big enough risc-v team to
respond to arch-specific build failures? And, do we have enough people to do
QA around release time?
* see http://fedora.riscv.rocks/koji/
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3 weeks, 5 days
"rescue" boot entry files are not updated on OS upgrades
by Chris Murphy
Summary----------
Most all Fedora variants (except Cloud) have a GRUB menu entry
containing the word "rescue". This kernel+initramfs pair are never
updated for the life of a Fedora installation. And they quickly become
stale as a Fedora installation ages. This kernel's modules are
eventually deleted, and if selected at boot time, the typical user
experience is a dracut shell.
Basic background-------------
(skip this section if you know how it works)
During a new installation, a single kernel version is installed. e.g.
vmlinuz-5.17.0-0.rc4.96.fc36.x86_64 which is then duplicated as e.g.
vmlinuz-0-rescue-3a86878de5d649a983916543ece7bb7e.
Each of those (identical) kernels has an initramfs file:
initramfs-5.17.0-0.rc4.96.fc36.x86_64.img
initramfs-0-rescue-3a86878de5d649a983916543ece7bb7e.img
The sole difference is the first one is a smaller host-only initramfs,
the second one is a larger no host-only initramfs created with `dracut
-N`. The bigger one just contains a bunch of extra kernel modules and
dracut scripts, ostensibly to make it more likely to boot a system
with some change in hardware that the host-only initramfs doesn't
contain. The size of this rescue initramfs is around 100 MiB, with the
common day to day "host only" initramfs being around 33 MiB. [1]
As the system is updated, additional kernel versions are installed.
dnf.conf contains installonly_limit=3, which results in a maximum of
three kernel versions being installed at a time. Once a fourth kernel
is installed, the first kernel and its modules are removed from
/usr/lib/modules. The rescue kernel+initramfs pair are never updated
or upgraded, even during system upgrades.
Observations------------
This has been discussed by the Workstation working group [2] but since
this functionality is present in all of Fedora, we're moving the
discussion for greater visibility.
There's two separate complaints, if you will: (a) that the
kernel+initramfs pair are never update or upgraded for the life of the
installation; and (b) that even during one release cycle, the user
experience when booting the rescue entry, changes, i.e. when the
matching /usr/lib/modules for the rescue entry are present early on,
you do get a full runtime behavior, you will get to a graphical
environment. But then once the version matched /usr/lib/modules are
removed, you get a completely different behavior when booting the
rescue entry.
An important note from that ticket from Justin Forbes, the Fedora
kernel maintainer: " Remember, the only real purpose of the rescue
kernel is to get your system out of something completely unusable. It
isn't meant to be a full runtime."
Questions------------
* Considering the very narrow purpose of the entry, maybe the current
behavior is adequate?
* Does the rescue entry reliably get users to a dracut prompt, rather
than indefinite hang? I don't know whether it does.
* Is there any way to improve the situation without increasing the
risk that the rescue entry becomes totally non-functional?
* The chosen kernel version needs to be based on one that is known
to boot. Currently we know the kernel+initramfs pair work because it's
the same version used to boot the installation media when doing the
initial provisioning. We don't actually know an updated replacement
"no host-only" initramfs will work until it's tried. Is it possible to
automate this? And is it worth the risk, or even figuring out how to
assess the risk?
* At Flock 2021, Zbyszek proposed "Building Initrd Images from
RPMs" to reduce the complexity of building initramfs, maybe there's a
role for it here? More: https://www.youtube.com/watch?v=GATg_bqmASc
* What happens if we accept some scope creep, and go for many
improvements that make the extra work worth it?
* What about the unsigned nature of the initramfs? Should we be
creating initramfs's in Fedora infra and signing them?
* Stuff a graphical rescue environment into the initramfs? (This
might be ten leaps too far, but it's intended to encourage thinking
with a vivid imagination.)
[1] both values from a recent Fedora 36 Workstation installation
[2] https://pagure.io/fedora-workstation/issue/259
--
Chris Murphy
1 month
Planning to start unifying native and mingw packages
by Sandro Mani
Hi
Following recent discussions and to reduce the maintenance burden, I'm
planning to start merging native and mingw packages. Initially, I'll be
looking at these packages where I maintain both variants:
eigen3 mingw-eigen3
enchant2 mingw-enchant2
freeimage mingw-freeimage
gdal mingw-gdal
GeographicLib mingw-GeographicLib
geos mingw-geos
giflib mingw-giflib
gtkspell3 mingw-gtkspell3
gtkspellmm30 mingw-gtkspellmm30
jxrlib mingw-jxrlib
leptonica mingw-leptonica
libgeotiff mingw-libgeotiff
libimagequant mingw-libimagequant
libkml mingw-libkml
librttopo mingw-librttopo
libspatialite mingw-libspatialite
libwebp mingw-libwebp
openjpeg2 mingw-openjpeg2
OpenSceneGraph mingw-OpenSceneGraph
osgearth mingw-osgearth
podofo mingw-podofo
proj mingw-proj
python-pillow mingw-python-pillow
qtspell mingw-qtspell
shapelib mingw-shapelib
svg2svgt mingw-svg2svgt
tesseract mingw-tesseract
uriparser mingw-uriparser
I'm performing test builds here [1]. Once I've got them all building
there, if there are no objections, I plan to push to F37 and retire all
the corresponding mingw repos.
Sandro
[1] https://copr.fedorainfracloud.org/coprs/smani/mingw-unified-spec/builds/
1 month
Orphaned X11 packages
by Adam Jackson
The following packages, previously owned by xgl-maint, are now up for grabs:
xorg-x11-xfs
xorg-sgml-doctools
xorg-x11-drv-v4l
xorg-x11-xsm
xorg-x11-twm
xorg-x11-drv-sisusb
xorg-x11-xdm
xorg-x11-docs
Upstream development on all of these is pretty much nil, so if you're
serious about picking up any of these you may also wish to take on the
module upstream:
https://gitlab.freedesktop.org/xorg
- ajax
1 month, 1 week
F37 proposal: Add -fno-omit-frame-pointer to default compilation
flags (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Fedora will add -fno-omit-frame-pointer to the default C/C++
compilation flags, which will improve the effectiveness of profiling
and debugging tools.
== Owner ==
* Name: [[User:daandemeyer| Daan De Meyer]], [[User:Dcavalca| Davide
Cavalca]], [[ Andrii Nakryiko]]
* Email: daandemeyer(a)fb.com, dcavalca(a)fb.com, andriin(a)fb.com
== Detailed Description ==
Credits to Mirek Klimos, whose internal note on stacktrace unwinding
formed the basis for this change proposal (myreggg(a)gmail.com).
Any performance or efficiency work relies on accurate profiling data.
Sampling profilers probe the target program's call stack at regular
intervals and store the stack traces. If we collect enough of them, we
can closely approximate the real cost of a library or function with
minimal runtime overhead.
Stack trace capture what’s running on a thread. It should start with
clone - if the thread was created via clone syscall - or with _start -
if it’s the main thread of the process. The last function in the stack
trace is code that CPU is currently executing. If a stack starts with
[unknown] or any other symbol, it means it's not complete.
=== Unwinding ===
How does the profiler get the list of function names? There are two parts of it:
# Unwinding the stack - getting a list of virtual addresses pointing
to the executable code
# Symbolization - translating virtual addresses into human-readable
information, like function name, inlined functions at the address, or
file name and line number.
Unwinding is what we're interested in for the purpose of this
proposal. The important things are:
* Data on stack is split into frames, each frame belonging to one function.
* Right before each function call, the return address is put on the
stack. This is the instruction address in the caller to which we will
eventually return — and that's what we care about.
* One register, called the "frame pointer" or "base pointer" register
(RBP), is traditionally used to point to the beginning of the current
frame. Every function should back up RBP onto the stack and set it
properly at the very beginning.
The “frame pointer” part is achieved by adding push %rbp, mov
%rsp,%rbp to the beginning of every function and by adding pop %rbp
before returning. Using this knowledge, stack unwinding boils down to
traversing a linked list:
https://i.imgur.com/P6pFdPD.png
=== Where’s the catch? ===
The frame pointer register is not necessary to run a compiled binary.
It makes it easy to unwind the stack, and some debugging tools rely on
frame pointers, but the compiler knows how much data it put on the
stack, so it can generate code that doesn't need the RBP. Not using
the frame pointer register can make a program more efficient:
* We don’t need to back up the value of the register onto the stack,
which saves 3 instructions per function.
* We can treat the RBP as a general-purpose register and use it for
something else.
Whether the compiler sets frame pointer or not is controlled by the
-fomit-frame-pointer flag and the default is "omit", meaning we can’t
use this method of stack unwinding by default.
To make it possible to rely on the frame pointer being available,
we'll add -fno-omit-frame-pointer to the default C/C++ compilation
flags. This will instruct the compiler to make sure the frame pointer
is always available. This will in turn allow profiling tools to
provide accurate performance data which can drive performance
improvements in core libraries and executables.
== Feedback ==
=== Potential performance impact ===
* Meta builds all its libraries and executables with
-fno-omit-frame-pointer by default. Internal benchmarks did not show
significant impact on performance when omitting the frame pointer for
two of our most performance intensive applications.
* Firefox recently landed a change to preserve the frame pointer in
all jitted code
(https://bugzilla.mozilla.org/show_bug.cgi?id=1426134). No significant
decrease in performance was observed.
* Kernel 4.8 frame pointer benchmarks by Suse showed 5%-10%
regressions in some benchmarks
(https://lore.kernel.org/all/20170602104048.jkkzssljsompjdwy@suse.de/T/#u)
Should individual libraries or executables notice a significant
performance degradation caused by including the frame pointer
everywhere, these packages can opt-out on an individual basis as
described in https://docs.fedoraproject.org/en-US/packaging-guidelines/#_compiler_flags.
=== Alternatives to frame pointers ===
There are a few alternative ways to unwind stacks instead of using the
frame pointer:
* [https://dwarfstd.org DWARF] data - The compiler can emit extra
information that allows us to find the beginning of the frame without
the frame pointer, which means we can walk the stack exactly as
before. The problem is that we need to unwind the stack in kernel
space which isn't implemented in the kernel. Given that the kernel
implemented it's own format (ORC) instead of using DWARF, it's
unlikely that we'll see a DWARF unwinder in the kernel any time soon.
The perf tool allows you to use the DWARF data with
--call-graph=dwarf, but this means that it copies the full stack on
every event and unwinds in user space. This has very high overhead.
* [https://www.kernel.org/doc/html/v5.3/x86/orc-unwinder.html ORC]
(undwarf) - problems with unwinding in kernel led to creation of
another format with the same purpose as DWARF, just much simpler. This
can only be used to unwind kernel stack traces; it doesn't help us
with userspace stacks. More information on ORC can be found
[https://lwn.net/Articles/728339 here].
* [https://lwn.net/Articles/680985 LBR] - New Intel CPUs have a
feature that gives you source and target addresses for the last 16 (or
32, in newer CPUs) branches with no overhead. It can be configured to
record only function calls and to be used as a stack, which means it
can be used to get the stack trace. Sadly, you only get the last X
calls, and not the full stack trace, so the data can be very
incomplete. On top of that, many Fedora users might still be using
CPUs without LBR support which means we wouldn't be able to assume
working profilers on a Fedora system by default.
To summarize, if we want complete stacks with reasonably low overhead
(which we do, there's no other way to get accurate profiling data from
running services), frame pointers are currently the best option.
== Benefit to Fedora ==
Implementing this change will provide profiling tools with easy access
to stacktraces of installed libraries and executables which will lead
to more accurate profiling data in general. This in turn can be used
to implement optimizations to core libraries and executables which
will improve the overall performance of Fedora itself and the wider
Linux ecosystem.
Various debugging tools can also make use of the frame pointer to
access the current stacktrace, although tools like gdb can already do
this to some degree via embedded dwarf debugging info.
== Scope ==
* Proposal owners: Put up a PR to change the rpm macros to build
packages by default with -fno-omit-frame-pointer by default.
* Other developers: Review and merge the PR implementing the Change.
* Release engineering: [https://pagure.io/releng/issues #Releng issue
number]. A mass rebuild is required.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
This should not impact upgrades in any way.
== How To Test ==
# Build the package with the updated rpm macros
# Profile the binary with `perf record -g <binary>`
# Inspect the perf data with `perf report -g 'graph,0.5,caller'`
# When expanding hot functions in the perf report, perf should show
the full call graph of the hot function (at least for all functions
that are part of the binary compiled with -fno-omit-frame-pointer)
== User Experience ==
Fedora users will be more likely to have a streamlined experience when
trying to debug/profile system executables/libraries. Tools such as
perf will work out of the box instead of requiring to users to provide
extra options (e.g. --call-graph=dwarf/LBR) or requiring users to
recompile all relevant packages with -fno-omit-frame-pointer.
== Dependencies ==
The rpm macros for Fedora need to be adjusted to include
-fno-omit-frame-pointer in the default C/C++ compilation flags.
== Contingency Plan ==
* Contingency mechanism: The new version can be released without every
package being rebuilt with fno-omit-frame-pointer. Profiling will only
work perfectly once all packages have been rebuilt but there will be
no regression in behavior if not all packages have been rebuilt by the
time of the release. If the Change is found to introduce unacceptable
regressions, the PR implementing it can be reverted and affected
packages can be rebuilt.
* Contingency deadline: Final freeze
* Blocks release? No
== Documentation ==
* Original proposal for in-kernel DWARF unwinder (rejected):
https://lkml.org/lkml/2017/5/5/571
== Release Notes ==
Packages are now compiled with frame pointers included by default.
This will enable a variety of profiling and debugging tools to show
more information out of the box.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
2 months, 1 week
fedpkg clone fails with Permission denied (publickey).
by Richard Shaw
Long story short I lost my home directory where I do all of my packager
activities (separate from my main user) so I'm setting things up from
scratch.
I created new ssh keys and uploaded the public key to
admin.fedoraproject.org and pasted into pagure.io. It's been over an hour
and I'm still getting:
$ fedpkg clone hamlib
Cloning into 'hamlib'...
hobbes1069(a)pkgs.fedoraproject.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Could not execute clone: Failed to execute command.
I've also updated my API tokens, which is STILL not well documented. I
pasted them in the appropriate spot in "/etc/rpkg/fedpkg.conf" which isn't
real intuitive.
Thanks,
Richard
2 months, 1 week
Fedora SCM requests on the weekend
by Robert-André Mauchin
Hello,
I know a lot of you are working on Fedora during the week days, but for some of us, the
weekend is the only time we can spend time on it. The problem is, SCM requests are rarely
processed during that time, most of them get stuck from Friday afternoon to Monday afternoon
(CEST), so it really hampers my work. Would it be possible for a volunteer to agree to do it
on the weekend.
Best regards,
Robert-André
3 months
