F36 Change: Package information on ELF objects (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects
== Summary ==
All binaries (executables and shared libraries) are annotated with an
ELF note that identifies the rpm for which this file was built. This
allows binaries to be identified when they are distributed without any
of the rpm metadata. `systemd-coredump` uses this to log package
versions when reporting crashes.
== Owner ==
* Name: [[User:Zbyszek|Zbigniew Jędrzejewski-Szmek]]
* Email: zbyszek(a)in.waw.pl
* Name: Lennart Poettering
* Email: mzsrqben(a)0pointer.net
== Detailed Description ==
People mix binaries (programs and libraries) from different
distributions (for example using Fedora containers on Debian or vice
versa), and distribute binaries without packaging metadata (for
example by stripping everything except the binary from a container
image, also removing `/usr/lib/.build-id/*`), compile their own rpm
packages (for internal distribution and installation), and compile and
distribute their own binaries. Sometimes we need to introspect a
binary and figure out its provenance, for example when a program
crashes and we are looking at a core dump, but also when we have a
binary without the packaging metadata. When the need to introspect a
binary arises, we have some very good mechanisms to show the
provenance: when a file is installed through the package manager we
can directly list the providing package, but even without this we can
use build-ids embedded in the binary to uniquely identify the
originating build. But those mechanisms work best when we're in the
realm of a single distribution. In particular, build-ids can be easily
tied to a source rpm, but only when we have the source rpm is part of
the distribution and the build-id was registered in the appropriate
database which maps build-ids to real package names. When we move
outside of the realm of a single distribution, it can be hard to
figure out where a given binary originates from. If we know that a
binary is from a given distribution, we may be able to use some
distro-specific mechanism to figure out this information. But those
mechanisms will be different for different distributions and will
often require network access. With this change we aim to provide a
mechanism that is is very simple, provides a "human-readable" origin
information without further processing, is portable across distros,
and works without network access.
The directly motivating use case is display of core dumps. Right now
we have build-ids, but those are just opaque hexadecimal numbers that
are not meaningful to users. We would like to immediately list
versions of packages involved in the crash (including both the program
and any libraries it links to). It is not enough to query the rpm
database to do the equivalent of `rpm -qf …`: very often programs
crash after some packages have been upgraded and the binaries loaded
into memory are not the binaries that are currently present on disk,
or when through some mishap, the binaries on disk do not match the
installed rpms. A mechanism that works without rpm database lookup or
network access allows this information to be showed immediately in
`coredumpctl` listings and journal entries about the crash. This
includes crashes that happen in the initrd and sandboxed containers.
A second motivating use case is when users distribute their own
binaries and would like to collect crash information. Build-ids are a
solution that is technically possible, but easy to get wrong in
practice: users would need to immediately record the build-id after
the build and store the mapping to program names, versions, and build
number in some database. It's much easier to be able to record
something during the build in the build product itself.
A third motivating use case is the general mixing of Fedora binaries
with programs and libraries from different distributions, both with
our binaries being used as the base for foreign binaries, and the
other way around. Whilst most distributions provide some mechanism to
figure out the source build information, those mechanisms vary by
distribution and may not be easy to access from a "foreign" system.
Such mixing is expected with containers, flatpaks, snaps, Python
binary wheels, anaconda packages, and quite often when somebody
compiles a binary and puts it up on the web for other people to
download.
We propose a new mechanism which is designed to be very simple but
extensible: a small JSON document is embedded in an section in the ELF
binary. This document can be easily read by a human if necessary, but
it is also well-defined and can be processed programatically. For
example, `systemd-coredump` will immediately make use of this to
display package ''nevra'' information for crashes. The format is also
easy to generate, so it can be added to any build system, either using
the helpers that we provide or even reimplemented from scratch.
For the case where we mix binaries from different distros (the third
motivating use case above), this approach is the most useful when this
system is used by all distros and even non-distro builds. The more
widely it is used, the more useful it becomes. The specification was
developed in collaboration with Debian developers, and we hope that
Fedora and Debian will lead the way for this to become as widely used
as build-ids. But even if the information is only available from some
distros, it is still useful, except that fallback mechanisms need to
be implemented.
=== Existing system: `.note.gnu.build-id` ===
We already have build-ids: every ELF object has a `.note.gnu.build-id`
note, and given a core file, we can read the build-id and look it up
in the rpm database (`dnf repoquery --whatprovides debuginfo(build-id)
= …`) to map it to a package name.
Build-ids are unique and compact and very generic and work as expected
in general. But they have some downsides:
* build-ids are not very informative for users. Before the build-id is
converted back to the appropriate package, it's completely opaque.
* build-ids require a working rpm database or an internet connection
to map to the package name.
Three important cases:
* minimal containers: the rpm database is not installed in the
containers. The information about build-ids needs to be stored
externally, so package name information is not available immediately,
but only after offline processing. The new note doesn't depend on the
rpm db in any way.
* handling of a core from a container, where the container and host
have different distros
* self-built and external packages: unless a lot of care is taken to
keep access to the debuginfo packages, this information may be lost.
The new note is available even if the repository metadata gets lost.
Users can easily provide equivalent information in a format that makes
sense in their own environment. It should work even when rpms and debs
and other formats are mixed, e.g. during container image creation.
=== New system: `.note.package` ===
The new note is created and propagated similarly to
`.note.gnu.build-id`. The difference is that we inject the information
about package ''nevra'' from the build system.
The implementation is very simple: `%{build_ldflags}` are extended
with a command to insert a custom note as a separate section in an ELF
object. See [https://github.com/systemd/package-notes/blob/main/hello.spec
hello.spec] for an example. This is done in the default macros, so all
packages that use the prescribed link flags will be affected.
The note is a compact json string. This allows the format to be
trivially extensible (new fields can be added at will), easy to
process (json is extremely popular and parsers are widely available).
Using a single field rather than a set of separated notes is more
space-efficient. With multiple fields the padding and alignment
requirements cause unnecessary overhead.
The system was designed with cross-distro collaboration and is
flexible enough to identify binaries from different packaging formats
and build systems (rpms, debs, custom binaries).
See https://systemd.io/COREDUMP_PACKAGE_METADATA/ for detailed
description of the format.
One of the advantages of using an ELF note, as opposed to say a series
of extended attributes on the binary itself, is that the ELF note gets
automatically captured and copied into a core file by the kernel.
Extended attributes would have to be copied manually, which might not
even be possible because the binary on disk may have been removed by
the time the crash is analyzed.
The overhead is about 200 bytes for each ELF object.
We have about overall 33200 files in `/usr/s?bin/` and about 36600
`.so` files (F35, single architecture,
results from `dnf repoquery -l 2>/dev/null | rg '^/usr/s?bin/' | sort
-u | wc -l`,
`dnf repoquery -l 2>/dev/null | rg '^/usr/lib64/.*\.so$' |sort -u|wc -l`).
If we do this for the whole distro, we get 69800 × 200 = 13 MB.
For a typical installation, we can expect about 300–400 kB.
Thus the overhead of additionally used space is neglible (also see the
Feedback section for more discussion).
Precise measurements TBD once this is turned on and we have real
measurements for a larger number of builds.
=== Examples ===
<pre>
$ objdump -s -j .note.package build/libhello.so
build/libhello.so: file format elf64-x86-64
Contents of section .note.package:
02ec 04000000 63000000 7e1afeca 46444f00 ....c...~...FDO.
02fc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n
030c 616d6522 3a226865 6c6c6f22 2c227665 ame":"hello","ve
031c 7273696f 6e223a22 302d312e 66633335 rsion":"0-1.fc35
032c 2e783836 5f363422 2c226f73 43706522 .x86_64","osCpe"
033c 3a226370 653a2f6f 3a666564 6f726170 :"cpe:/o:fedorap
034c 726f6a65 63743a66 65646f72 613a3333 roject:fedora:33
035c 227d0000 "}..
</pre>
<pre>
$ readelf --notes build/hello | grep "description data" | sed -e
"s/\s*description data: //g" -e "s/ //g" | xxd -p -r | jq
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x10de
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x10af
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x119f
{
"type": "rpm",
"name": "hello",
"version": "0-1.fc35.x86_64",
"osCpe": "cpe:/o:fedoraproject:fedora:33"
}
</pre>
<pre>
$ coredumpctl info
PID: 44522 (fsverity)
...
Package: fsverity-utils/1.3-1
build-id: ac89bf7175b04d7eec7f6544a923f45be111f0be
Message: Process 44522 (fsverity) of user 1000 dumped core.
Found module
/home/bluca/git/fsverity-utils/libfsverity.so.0 with build-id:
fa40fdfb79aea84167c98ca8a89add9ac4f51069
Metadata for module
/home/bluca/git/fsverity-utils/libfsverity.so.0 owned by FDO found: {
"packageType" : "deb",
"package" : "fsverity-utils",
"packageVersion" : "1.3-1"
}
Found module linux-vdso.so.1 with build-id:
aba08e06103f725e26f1d7c178fb6b76a564a35d
Found module libpthread.so.0 with build-id:
e91114987a0147bd050addbd591eb8994b29f4b3
Found module libdl.so.2 with build-id:
d3583c742dd47aaa860c5ae0c0c5bdbcd2d54f61
Found module ld-linux-x86-64.so.2 with build-id:
f25dfd7b95be4ba386fd71080accae8c0732b711
Found module libcrypto.so.1.1 with build-id:
749142d5ee728a76e7cdc61fd79d2311a77405a2
Found module libc.so.6 with build-id:
18b9a9a8c523e5cfe5b5d946d605d09242f09798
Found module fsverity with build-id:
ac89bf7175b04d7eec7f6544a923f45be111f0be
Metadata for module fsverity owned by FDO found: {
"packageType" : "deb",
"package" : "fsverity-utils",
"packageVersion" : "1.3-1"
}
Stack trace of thread 44522:
#0 0x00007fe7c8af26f4 __GI___nanosleep (libc.so.6 + 0xc66f4)
#1 0x00007fe7c8af262a __sleep (libc.so.6 + 0xc662a)
#2 0x00005608481407dd main (fsverity + 0x27dd)
#3 0x00007fe7c8a5009b __libc_start_main (libc.so.6 + 0x2409b)
#4 0x000056084814094a _start (fsverity + 0x294a)
</pre>
== Feedback ==
See [https://github.com/systemd/systemd/issues/18433 systemd issue
#18433] for upstream discussion and implementation proposals.
=== Concerns about additional changes to files ===
<pre>
17:32:30 <Eighth_Doctor> I think zbyszek underestimates how much of a
problem it is to stamp every ELF binary with ''nevra'' data
17:32:44 <mhroncok> zbyszek: so, assuming python has ~100 ELF .so
files and I change one text file
17:33:22 <mhroncok> (ignore for the time being that the .so files
often changed because of toolchain updates and assume they are stable)
</pre>
I tested this with python3.10. So far there are 13 builds of that
package in F35:
`python3.10-3.10.0-1.fc35`,
`python3.10-3.10.0~a6-1.fc35`,
`python3.10-3.10.0~a6-2.fc35`,
`python3.10-3.10.0~a7-1.fc35`,
`python3.10-3.10.0~b1-1.fc35`,
`python3.10-3.10.0~b2-2.fc35`,
`python3.10-3.10.0~b2-3.fc35`,
`python3.10-3.10.0~b3-1.fc35`,
`python3.10-3.10.0~b4-1.fc35`,
`python3.10-3.10.0~b4-2.fc35`,
`python3.10-3.10.0~b4-3.fc35`,
`python3.10-3.10.0~rc1-1.fc35`,
`python3.10-3.10.0~rc2-1.fc35`.
I extracted the builds (for `.x86_64`) and made a list of all `.so`
files (1368 files), and calculated sha256 hashes for them. No two
files repeat, there are 1368 distinct hashes. So the files are
'''already''' different between builds and the additional proposed
metadata does will not make a significant difference.
Note that this range of Python versions encompasses periods when the
package is under development and undergoes significant changes (alpha
versions), and when it's only undergoing small changes (rc versions).
The fact that we get different files in each build is not surprising,
because files embed build-ids which differ between builds. But even if
we ignore those, binaries generally differ between builds. Even sizes
tend to vary between builds: there are 636 distinct `.so` file sizes,
i.e. on average any given size only repeats twice (presumably most
often for the same file). Running `diffoscope` on `.so` files from
different builds shows minor changes in the assembly which I did not
analyze futher.
If people have specific questions, for example about overhead in some
scenario, I'd be happy to answer them. Until now, the issues that were
raised were very vague, so it's impossible to answer them.
=== Why not just use the rpm database? ===
<pre>
17:34:33 <dcantrell> The main reason for this appears to be that we
need the RPM db locally to resolve build-ids to package names. But
since containers wipe /var/lib/rpm, we can't do that. So the solution
is to put the ''nevra'' in ELF metadata?
17:34:39 <dcantrell> That feels like the wrong approach.
</pre>
First, there are legitimate reasons to strip packaging metadata from
images. For example, for an initrd image from rpms, I get 117 MB of
files (without compression), and out of this `/var/lib/rpm` is 5.9 MB,
and `/var/lib/dnf` is 4.2 MB. This is an overhead of 9%. This is ''not
much'', but still too much to keep in the image unless necessary.
Similar ratios will happen for containers of similar size. Reducing
image size by one tenth is important. There is no `rpm` or `dnf` in
the image, to the package database is not even usable without external
tools.
As discussed on IRC
(https://meetbot.fedoraproject.org/teams/fesco/fesco.2021-05-11-17.01.log....),
the containers ''we'' build don't wipe this metadata, but custom
Dockerfiles do that.
Second, as described in Description section above, not everybody and
everything uses rpm. The Fedora motto is "we make an operating system
and we make it easy for you to do useful stuff with it" (and yes, this
is an actual quote from the official docs), and this stuff involves
reusing our binaries in containers and custom installations and
whatnot, not just straightforward installations with `dnf`. And in the
other direction, people will build their own binaries that are not
packaged as rpms. But it is still important to be able to figure out
the exact version of a binary, especially after it crashes.
=== Why do this in Fedora? ===
<pre>
17:36:49 <mhroncok> I don't understand how non-rpm distros and custom
built binaries are affected by our rpm-build environment :/
</pre>
The idea is that we inject this into our build system, and Debian
injects this into their build system, and so on… As mentioned, this is
a cross-distro effort. Also, people can use it in their custom build
systems if they build and distribute binaries internally. The scheme
would obviously be most useful if used comprehensively, but it's still
useful when available partially. We hope that Fedora can lead the way.
(This is similar to build-ids: when initially adopted, they were used
only by some distros, but were useful even then. Nowadays, with
comprehensive adoption, they are even more useful.)
https://hpc.guix.info/blog/2021/09/whats-in-a-package/ contains a nice
description of a pathological case of packaging hacks and binary
redistribution. When trying to unravel something like this,
information embedded directly in the binaries would be quite useful.
== Benefit to Fedora ==
A simple and reliable way to gather information about package versions
of programs is added.
It enhances, instead of replacing, the existing mechanisms.
It is particularly useful when reporting crash dumps, but can also be
used for image introspection and forensincs, license checks and
version scans on containers, etc.
If we adopt this in Fedora, Fedora leads the way on implementing the
standard. Fedora binaries used in any context can be easily
recognized. Fedora binaries provide a better basis to build things.
If other distros adopt this, we can introspect and report on those
binaries easily within the Fedora context. For example, when somebody
is using a container with some programs that originate in the Debian
ecosystem, we would be able to identify those programs without tools
like `apt` or `dpkg-query`. Core dump analaysis executed in the Fedora
host can easily provide useful information about programs from foreign
builds.
== Implementation in Other Distributions ==
=== Microsoft CBL-Mariner ===
[https://en.wikipedia.org/wiki/CBL-Mariner CBL-Mariner] is an
[https://github.com/microsoft/CBL-Mariner open source] Linux
distribution created by Microsoft, targeted at first-party and
container workloads on Azure. It is used both as a container runner
host and a base container image.
Mariner adopted the ELF stamping packaging metadata spec in
[https://github.com/microsoft/CBL-Mariner/blob/1.0/SPECS/mariner-rpm-macro...
version 1.0], initially to add OS metadata, and package-level metadata
will be added in a following release.
=== Debian ===
A package-level proof-of-concept is included in the
[https://github.com/systemd/package-notes/blob/main/dh_package_notes
package-notes] repository.
A [https://salsa.debian.org/bluca/debhelper/-/tree/notes_metadata
system-level proof-of-concept] that enables ELF stamping by default in
all builds implicitly will be proposed for adoption in the future.
== Scope ==
* Proposal owners:
** create a specification (First version DONE:
[https://systemd.io/COREDUMP_PACKAGE_METADATA
COREDUMP_PACKAGE_METADATA]. We might need to make some adjustments
based on the deployment in Fedora, but no big changes are expected.)
** write a script to generate the package note (First version DONE:
[https://github.com/systemd/package-notes/blob/main/generate-package-notes.py
generate-package-notes.py])
** provide a patch for `redhat-rpm-config` to insert appropriate
compilation options
** extend systemd's coredumpctl to extract and display this
information (DONE: [https://github.com/systemd/systemd/pull/19135 PR
#19135], available in systemd-249)
** submit pull request to Packaging Guidelines
* Other developers:
** possibly add support in abrt?
* Release engineering: There should be no impact.
* Policies and guidelines:
The new flags should be mentioned in Packaging Guidelines.
* Trademark approval: N/A (not needed for this Change)
N/A
* Alignment with Objectives:
It might be relevant for Minimization. Even though it increases the
image size a tiny bit, it makes minimized images work a bit better.
== Upgrade/compatibility impact ==
No impact.
== How To Test ==
<pre>
$ bash -c 'kill -SEGV $$'
$ coredumpctl
TIME PID UID GID SIG COREFILE EXE
SIZE PACKAGE
Mon 2021-03-01 14:37:22 CET 855151 1000 1000 SIGSEGV present
/usr/bin/bash 51.7K bash-5.1.0-2.fc34.x86_64
</pre>
== User Experience ==
`coredumpctl` should display information about package versions.
`readelf --notes` or similar tools can be used on `.so` files and
compiled programs
to extract the JSON blurb that describes the originating package.
== Dependencies ==
None.
== Contingency Plan ==
* Contingency mechanism: Remove the new compilation flags. Rebuild any
packages that were build with the new flags.
* Contingency deadline: Beta freeze.
* Blocks release? No.
== Documentation ==
* https://systemd.io/COREDUMP_PACKAGE_METADATA/
* https://github.com/systemd/package-notes
See also [[Changes/DebuginfodByDefault]].
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
2 years, 2 months
upgrading RH 9 system->Fedora with iso files and apt only
by Didier Casse
I have the yarrow's iso files on my HD in a RH9 system. Let's say I want
to upgrade selected packages using an "apt-get install" pointing to my
iso-mounted files, how do I do it?
i.e I mount the iso into some /mnt/yarrow1, /mnt/yarrow 2 etc..
Then what is the complete procedure to make my apt look into my own HD to
upgrade packages. Can anybody redirect me to the correct
resource or some literature hanging on the web? Thanks.
Assume also that I do not wish to burn CDs! I do not want to use
apt-cdrom. Thanks.
With kind regards,
Didier.
---
PhD student
Singapore Synchrotron Light Source (SSLS)
5 Research Link,
Singapore 117603
Email: slsbdfc at nus dot edu dot sg \or\
didierbe at sps dot nus dot edu dot sg
Website: http://ssls.nus.edu.sg
2 years, 3 months
Self Introduction: Jan Baudisch
by Jan Baudisch
Hi,
I am a computer science student from Germany and have used Fedora for
at least 2 years by now.
Using COPR I gained some experience in buidling packages but recently I
wanted to try building packages conforming to the guidelines and
getting them into Fedora. My interest is mostly in Rust and that is why
I would like to contribute Rust packages. I already got started here:
https://copr.fedorainfracloud.org/coprs/janbaudisch/rust/packages
For now my main interest would be to get the Rocket framework packaged,
as many projects rely on it and on the way I discovered some key Rust
crates not packaged yet.
I feel like some of them are ready for submission and will file the
first review requests soon.
Looking forward to working on this!
Jan
2 years, 3 months
Orphaning luit
by Peter Hutterer
I've orphaned luit. The only user of it was xterm and it recently dropped
support for luit so there are no users left. Whether there are *any* users
left is unclear too :)
The luit package we shipped is still the freedesktop.org one which has been
unmaintained for about a decade now. Upstream now points to
Thomas Dickey's fork at http://invisible-island.net/luit/ so if anyone
wants to take this, I advise switching to that version as part of the first
steps.
Cheers,
Peter
2 years, 3 months
F36 Change: Make Authselect Mandatory (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Make_Authselect_Mandatory
== Summary ==
This change wants to make authselect required to configure
authentication and identity sources and forcefully update
non-authselect configuration to the sssd authselect profile to
eliminate any existing non-authselect setups.
Even though it will still be possible to manually modify the
configuration, users that require special configuration should create
and use custom authselect profile.
''Authselect is available in Fedora since Fedora 27 and enabled by
default on new installations since Fedora 28. Authconfig compatibility
tool was removed from Fedora 35 as a
[[Changes/RemoveAuthselectCompatPackage|system wide change page]]. It
is now well accepted by the community as well as the package
maintainers. The package maintainers have repeatedly requested to make
authselect mandatory for the users which lead to creation of
[https://bugzilla.redhat.com/show_bug.cgi?id=2000936 this bugzilla].''
== Owner ==
* Name: [[User:pbrezina|Pavel Březina]]
* Email: pbrezina(a)redhat.com
== Detailed Description ==
The following components must be updated to make authselect mandatory:
* authselect
* pam
* glibc
* packages that use it: systemd, ecryptfs, nss-mdns and fingerprint.
Required changes:
# Remove user-nsswitch.conf functionality from authselect
# Move ownership of /etc/nsswitch.conf and /etc/pam.d/{system-auth,
password-auth, smartcard-auth, fingerprint-auth, postlogin} to
authselect from glibc and pam
# Require authselect in pam
# Remove non-authselect support from systemd, ecryptfs, nss-mdns and fingerprint
# Select default profile when authselect is installed
# Select default profile when authselect is upgraded
=== Remove user-nsswitch.conf functionality ===
File /etc/authselect/user-nsswitch.conf was introduced in authselect
to allow partial user modifications of nsswitch.conf without the need
to create a custom authselect profile. The main driver was to enable
modules that are not included in authselect such as systemd-resolved
and nss-mdns.
This however made the situation more confusing to users and it is not
desirable any more if authselect is mandatory.
'''Authselect will drop user-nsswitch.conf functionality and instead
add more nsswitch modules to existing profiles and be more open about
future inclusion requests.'''
=== Own /etc/nsswitch.conf and /etc/pam.d/{system-auth, password-auth,
smartcard-auth, fingerprint-auth, postlogin} instead of glibc and pam
===
File /etc/nsswitch.conf is currently owned by glibc. It will be now
owned by authselect and removed from glibc.
PAM configuration generated by authselect is currently owned by pam.
It will be now owned by authselect and removed from pam.
''Note: that config-util and other will still be owned by pam since
these files are not generated by authselect.''
'''All files that are generated by authselect are now owned by authselect.'''
=== Require authselect in pam ===
The pam package will require authselect. This will tie pam and
authselect together and it will be impossible to uninstall authselect
without uninstalling pam which fundamentally makes authselect a hard
dependency on each system.
'''This step will make it impossible to uninstall authselect, making
it always available to RPM packages.'''
=== Remove non-authselect support from systemd, ecryptfs, nss-mdns and
fingerprint ===
'''Non-authselect configuration support will be dropped in these packages.'''
=== Select default profile when authselect is installed ===
If authselect configuration is not detected and this is a new
installation of authselect it will automatically select the
distribution default authselect profile by calling authselect select
--force with distribution specific parameters.
If existing authselect configuration is detected (perhaps from
previous installation), it will be updated (current behavior).
This makes sure that if authselect is installed (which is always) a
configuration is created.
Select default profile when authselect is upgraded
If authselect is upgraded from an older version and non-authselect
configuration is detected, it will forcefully overwrite it with
distribution defaults by calling authselect select --force with
distribution specific parameters.
This is a one time event so if someone does not want to use
authselect, it remains possible. However, non-authselect
configurations will not be supported by RPM packages mentioned above.
If authselect is upgraded on a system that already is configured by
it, the update process remains the same as it is now.
'''This step will forcefully update existing installations to
authselect configuration. It is a one time event and opt-out is still
possible but no longer supported.
'''
== Benefit to Fedora ==
'''Making authselect mandatory will provide better user and
maintainers experience and significantly reduce risk of breaking
system configuration.'''
The use of authselect-generated configuration is currently optional.
This means that users can uninstall authselect or just opt-out from it
by modifying the PAM and nsswitch.conf manually.
Additionally users that upgrade from pre-authselect systems (Fedora 27
and older) must opt-in to authselect manually, which is not the case
most of the time. Also sometimes an upgrade of a particular package
that did not handle authselect configuration correctly automatically
opted-out its users. Therefore we can split users into four groups:
* Those who are using authselect
* Those who were using authselect but are not anymore due to various
errors in packages
* Those who are not using authselect because they are not aware of it
and upgraded from older Fedora versions thus it was not automatically
enabled
* Those who are not using authselect knowingly
''Fedora is now split into two worlds: those who use authselect and
those who don’t. This however keeps two problems existing. Authselect
was designed to solve them both. But it will be solved only if the use
of authselect is a mandatory requirement.''
=== 1. It is difficult to deliver updates to configurations ===
FIles /etc/nsswitch.conf and /etc/pam.d/* are distributed as
%config(noreplace) which means that they are configuration files and
are only installed if they are not yet present. If they are present
then they are never overwritten with package updates, instead an
*.rpmnew file is created and the update responsibility is left
completely to the user.
It is done this way to prevent overwriting user changes
configurations. But at the same time it means that even configurations
that are not modified by the users can not be changed so we can not
deliver fixes and changes efficiently.
It is only possible through difficult scriptlets. As an example, we
can show this bugzilla where a change in Gnome required an update to
PAM otherwise the user could not authenticate. Delivering the change
was easy with authselect, but difficult for non-authselect systems.
Authselect already knows how the resulting configuration should look
and does not risk overriding user configuration. Making it mandatory
will help distribute important updates to nsswitch and PAM
configuration.
=== 2. Package maintainers wants to enable/disable nsswitch or PAM
modules automatically upon installation/uninstallation ===
Packages that install nsswitch modules (e.g. systemd, nss-mdns) or PAM
modules (e.g. ecryptfs, fprintd) want to enable the module
automatically upon RPM installation to provide a better user
experience. So they use various more or less complex and fragile
scriptlets in %post action of the RPM.
This is usually easier for the nsswitch module, when simple sed works
for most configurations. But it is not really possible for PAM
(because there is high risk of introducing security vulnerabilities or
breaking authentication completely) so the packages used authconfig
and now authselect.
Packages need to detect the current authselect state and either call
authselect when available or fallback to non-authselect scriptlets.
The end results tends to be the same for nsswitch modules but
different for PAM modules where it is either enabled with authselect
or nothing happens.
'''If authselect is made mandatory, packages can only support
authselect configuration which makes the change simple,
straightforward, easier to test and safe.'''
== Scope ==
* Proposal owners:
** authselect maintainers:
*** implement the changes described in "Detailed Description" in all
affected packages
*** open pull requests with the changes against affected packages
* Other developers:
** maintainers of affected packages must accept submitted merge request
* Release engineering: [https://pagure.io/releng/issue/10337 #10337]
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
The system will be forcefully configured with authselect, selecting
the default distribution profile (sssd), if non-authselect
configuration is detected. It will will override users configuration.
A backup will be created, users can run 'authselect backup-restore
$id' to restore their configuration when desired.
== How To Test ==
* Authselect must keep working as expected.
* Ownership of the following files is transferred to authselect-libs package
** /etc/pam.d/fingerprint-auth
** /etc/pam.d/password-auth
** /etc/pam.d/postlogin
** /etc/pam.d/smartcard-auth
** /etc/pam.d/system-auth
** /etc/nsswitch.conf
* pam requires authselect
* updating from previous system with non-authselect configuration must
result in applied authselect configuration with sssd profile
* updating from previous system with authselect configuration must
remain intact (besides applying udated profiles)
* installing/removing affected packages (see Detailed Description)
must not break authselect configuration
== User Experience ==
PAM and nsswitch.conf configuration will be automatically updated when changed.
''Note: To request inclusion of a new nsswitch or PAM module in
default authselect profiles, either open an issue on authselect GitHub
project or open a bug against authselect component in Fedora
bugzilla.''
''Note: Users will still be able to create custom profiles to include
modules that are not supported by default authselect profiles.
However, such profiles will have limited update support. It is
possible to create a custom profile with some files as symlinks to
default profiles, those files will receive updates automatically.
However, modified files of the custom profiles still need to be
updated manually and the users are responsible for the update.
''
''Note: Users will still be able to opt-out from authselect, but such
configuration will no longer be updated on upgrades or touched by
package installation.''
== Dependencies ==
This change does not break any packages that depends on authselect.
== Contingency Plan ==
* Contingency mechanism: Revert changes in affected packages.
* Contingency deadline: Beta
* Blocks release? No
== Documentation ==
There is no upstream documentation.
== Release Notes ==
Configuring system authentication and identity sources with authselect
is now mandatory. Systems with non-authselect configuration will be
automatically forcefully upgraded to authselect configuration. Users
that require special configuration are encouraged to create and use
custom authselect profile.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
2 years, 4 months
Onboarding package
by Vít Ondruch
Hi,
Recently, there have been a lot of discussions on this list as well as
we have internally about onboarding. During our internal brainstorming,
we were initially discussing that it could be useful to have some
package one can experiment with without being too much worried about the
result.
However, discussing this back and forth, we figured that it might also
be good idea to actually have something such as "onboarding" package,
where new coming package maintainer could gradually gain experience with
the packaging workflows. So the simplest tasks could be:
1) Add changelog entry into onboarding package and open PR with the
change. This would not require too many privileges. Alternatively this
could include change to "CONTRIBUTORS" file. I suspect that also some
current Fedora contributors might be interested to send such PR ;)
2) Second step could be something similar, but that would require the
packager to be already sponsored and they could go through the whole
process themeselves just with some light guidance if needed.
This could be extended in the future. E.g. next step could be:
3) Submit module update.
Apart from gaining experience, this could also help with the common
question "where should I start". And of course our sponsoring guidelines
could be refreshed suggesting/requesting to take these steps at some point.
Thoughts?
Vít
2 years, 4 months
Bodhi critpath package updates now gated on openQA results
by Adam Williamson
Hey folks!
Just wanted to flag up that, now the new Bodhi version has been
deployed to production, critpath updates are gated on openQA test
results. If any openQA test for your critpath update failed, the gating
status will be marked as 'failed' and you will not be able to push it
stable.
Waivers can be issued for failed tests where appropriate:
https://docs.fedoraproject.org/en-US/ci/gating/#_waive
But in almost all cases a failure indicates either a genuine bug or an
opportunity to improve the test, so I'd prefer to avoid use of waivers
where possible. I am trying to keep an eye on all failed tests, but if
you have a blocked update and you don't understand the failure and I
haven't yet commented on it, please do poke me and I'll take a look.
If a failure looks like some kind of transient issue, several folks
have the power to rerun tests: myself, lruzicka, kparal, tflink,
abokovoy (abbra / ab), pwhalen, and sumantrom. You can ask one of us to
do it. There have been plans in the past to implement some sort of
rerun request system in Bodhi but no-one's quite had the roundtuits to
work it out yet; sorry about that.
Thanks everyone, please be patient with any kinks while we see how this
goes :)
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
2 years, 5 months
protobuf 3.18.1 update coming to rawhide
by Adrian Reber
The protobuf maintainers prepared an update to protobuf 3.18.1 in
rawhide. protobuf comes, as always, with a new SO name and requires
a rebuild of all dependencies. The list of dependencies grows with each
rebuild and we have now reached 58 protobuf dependencies according to
repoquery.
This time the number of rebuild failures is unusually high with 13
broken dependencies.
Because of missing dependencies we had to disable the Java bindings
which breaks the build of:
1. osmpbf
Problem: package protobuf-java-3.14.0-6.fc35.noarch conflicts with protobuf-compiler > 3.14.0 provided by protobuf-compiler-3.18.1-1.fc36.x86_64
There are two openssl error:
2. community-mysql
Cannot find appropriate system libraries for WITH_SSL=system.
Make sure you have specified a supported SSL version.
Valid options are :
system (use the OS openssl library),
yes (synonym for system),
</path/to/custom/openssl/installation>
4. mumble
error: 'CRYPTO_mem_ctrl' was not declared in this scope; did you mean 'CRYPTO_memcmp'?
A python 3.10 dependency problem break:
4. opencv
nothing provides (python3.10dist(pyflakes) < 2.5 with python3.10dist(pyflakes) >= 2.4) needed by python3-flake8-4.0.1-1.fc36.noarch
which breaks:
6. gazebo
package opencv-core-4.5.4-1.fc36.x86_64 requires libprotobuf.so.25()(64bit), but none of the providers can be installed
which breaks:
7. fawkes
package gazebo-10.1.0-21.fc36.x86_64 requires libprotobuf.so.25()(64bit), but none of the providers can be installed
There are also couple of seemingly protobuf unrelated compiler errors:
8. et
catch.hpp:10827:58: error: call to non-'constexpr' function 'long int sysconf(int)'
catch.hpp:10887:45: error: size of array 'altStackMem' is not an integral constant-expression
9. qgis
sip: Py_ssize_t is undefined
10. bear
type_traits.hpp:362:46: error: incomplete type 'nlohmann::detail::is_constructible<nlohmann::basic_json<>, std::filesystem::__cxx11::path>' used in nested name specifier
11. opentrep
action_dispatch.hpp:135:29: error: no match for call to '(const OPENTREP::PorParserHelper::storeAltLangCodeHist) (std::vector<char32_t, std::allocator<char32_t> >&,
And two more dependency errors:
12. mir
Problem: package wlcs-devel-1.3.0-2.fc35.x86_64 requires wlcs(x86-64) = 1.3.0-2.fc35, but none of the providers can be installed
- cannot install the best candidate for the job
- nothing provides libgtest.so.1.10.0()(64bit) needed by wlcs-1.3.0-2.fc35.x86_64
- nothing provides libgmock.so.1.10.0()(64bit) needed by wlcs-1.3.0-2.fc35.x86_64
13. postgres-decoderbufs
Problem: package postgresql-server-devel-13.4-3.fc36.x86_64 requires postgresql-private-devel, but none of the providers can be installed
- package postgresql-private-devel-13.4-3.fc36.i686 conflicts with libpq-devel provided by libpq-devel-13.4-2.fc36.x86_64
- package postgresql-private-devel-13.4-3.fc36.x86_64 conflicts with libpq-devel provided by libpq-devel-13.4-2.fc36.x86_64
- cannot install the best candidate for the job
Besides these 13 errors I will rebuild everything else in a side tag
starting in one week with the rebuilds.
Current rebuild results are available at:
https://copr.fedorainfracloud.org/coprs/adrian/protobuf-3-18/
Adrian
2 years, 5 months
