1:34 a.m.
Hi,
heads-up: systemd 250-rc1 has been built in rawhide.
It's a fairly big release (140 paragraphs in the NEWS file entry) [1],
but there are also some packaging changes that are relevant for Fedora.
There's a bunch of new configuration settings
(RestrictNetworkInterfaces=, StartupAllowedCPUs=, StartupAllowedMemoryNodes=,
[Condition|Assert][Memory|CPU|IO]Pressure=, ExecSearchPath=, RuntimeRandomizedExtraSec=,
TTYRows=/TTYColumns=, [Service] ExitType=, [Automount] ExtraOptions=),
new units (systemd-boot-update.service, factory-reset.target),
support for TPM2-backed credential storage, file system integrity,
and a lot of new features in systemd-networkd, systemd-analyze,
systemd-dissect, systemd-homed, systemd-coredump, sd-boot, and various
other smaller features and enhancements.
Most tools now provide json output for easy postprocessing.
The credential storage stuff could be relevant in the light of the recent
discussion about logins in emergency mode.
There are some dependency changes in packaging in Fedora.
We changed most of the crypto code to use openssl instead of gcrypt.
We also changed various libraries to be dlopened when actually used,
instead of being dynamically linked. The goal is to reduce the
required dependencies. This means that various libraries that were
previously always pulled in, are not listed as Recommends by the various
subpackages:
$ rpm -q --recommends systemd | grep '^lib'
libbpf.so.0()(64bit)
libbpf.so.0(LIBBPF_0.4.0)(64bit)
libcryptsetup.so.12()(64bit)
libcryptsetup.so.12(CRYPTSETUP_2.4)(64bit)
libdw.so.1()(64bit)
libdw.so.1(ELFUTILS_0.186)(64bit)
libelf.so.1()(64bit)
libelf.so.1(ELFUTILS_1.7)(64bit)
libidn2.so.0()(64bit)
libidn2.so.0(IDN2_0.0.0)(64bit)
libpcre2-8.so.0()(64bit)
libpwquality.so.1()(64bit)
libpwquality.so.1(LIBPWQUALITY_1.0)(64bit)
libqrencode.so.4()(64bit)
libxkbcommon(x86-64)
Recommends are normally installed, but if you're using 'dnf --setopt
install_weak_deps=False'
or similar, please make sure that you install those libraries too if appropriate.
libgcrypt is pulled in by libsystemd.so, which means it is pretty much
everywhere. The plan is to switch it to being dlopened, because it is only
used for one minor feature irrelevant to almost all users.
Also, I moved more stuff into the -udev subpackage:
systemd-integritysetup, systemd-veritysetup, systemd-binfmt,
systemd-sysctl, systemd-coredump. IIUC, none of those are useful
in containers, so they fit better in -udev which is supposed to be installed
on real hardware or VMs.
systemd-resolved has also seen work, but it mostly has been behind-the-scenes
refactoring. All of the crypto code had been ported from libgcrypt to openssl,
but hopefully that change will be completely invisible. There have been some
bugfixes too, but not enough. I hope to clear some of the outstanding issues
before v250-final.
If there are regressions, please file a bug or let me know otherwise.
[1] https://raw.githubusercontent.com/systemd/systemd/v250-rc1/NEWS
Zbyszek
3:40 a.m.
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
We also changed various libraries to be dlopened when actually used,
instead of being dynamically linked. The goal is to reduce the
required dependencies.
What happens if these libraries are not installed an cannot be dl-opened?
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
6:34 a.m.
Am 10.12.21 um 10:40 schrieb Vitaly Zaitsev via devel:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
> We also changed various libraries to be dlopened when actually used,
> instead of being dynamically linked. The goal is to reduce the
> required dependencies.
dl-opening instead of dynamic linkage doesn't remove
these deps.
It only hides away the deps and moves them from "packaging" to run-time.
What happens if these libraries are not installed an cannot be
dl-opened?
Programs will fail at run-time.
Openly said, I feel this is a massive regression and design flaw.
Ralf
7:39 a.m.
On 10/12/2021 13:34, Ralf Corsépius wrote:
Programs will fail at run-time.
Openly said, I feel this is a massive regression and design flaw.
+1. And crash on init-system level will fail the whole system.
All library dependencies should be tracked by linker.
dlopen() should be used only for loading plugins or some restricted
libraries, like openh264 or ffmpeg-libs.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
8 a.m.
On Fr, 10.12.21 10:40, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
> We also changed various libraries to be dlopened when actually used,
> instead of being dynamically linked. The goal is to reduce the
> required dependencies.
What happens if these libraries are not installed an cannot be
dl-opened?
We handle that gracefully. i.e. the specific feature is disabled, but
everything else stil works as it should.
here's an example: systemd-cryptenroll allows you to enroll additional
keys into LUKS2 volumes for systemd-cryptsetup to honour. Besides
pkcs11/fido2/tpm2 stuff it also supports "recovery keys", which are
pretty much the same as a regular password, but are computer generated
rather than user chosen, and thus have guaranteed high entropy. Since
users have to read the generated key off screen and write it
down/store it away somewhere, we will normally show you a QR code in
the terminal you can quickly scan it off screen with. This is
implemented via dlopen()ing libqrencode. If you install that lib,
you'll get the QR code + text string displayed. If not, you'll just
get the text string.
And the other cases where we use dlopen() are pretty much the same:
the specific feature is skipped or made unavailable, but the basic
behaviour remains the same. When we really really need something, then
we'll use regular share library deps as always, not dlopen(). We use
dlopen() only for the stuff that isn't a "core" feature,
i.e. essential for the program to do its thing.
It's an excercise in minimizing our footprint (for containers,
embedded devices) , but still providing a rich feature set for those
who want it, without requiring recompilation, and whith automatic
upgrading to the full feature set, simply by dropping in a few more of
the optional deps.
Lennart
--
Lennart Poettering, Berlin
8:42 p.m.
On Fri, Dec 10, 2021 at 03:00:29PM +0100, Lennart Poettering wrote:
It's an excercise in minimizing our footprint (for containers,
embedded devices) , but still providing a rich feature set for those
who want it, without requiring recompilation, and whith automatic
upgrading to the full feature set, simply by dropping in a few more of
the optional deps.
This seems like a big step forward in flexiblity in using the same binaries
for a lot of different situations. Cool!
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8:13 a.m.
On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel wrote:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
>We also changed various libraries to be dlopened when actually used,
>instead of being dynamically linked. The goal is to reduce the
>required dependencies.
What happens if these libraries are not installed an cannot be dl-opened?
The functionality that requires those libraries will not be available.
An easy example is libqrencode: some systemd tools (journalctl,
homectl, cryptenroll) will print recovery keys and such as qrcode
on the terminal for easy transfer to a cellphone, in addition to the usual
text output. If the library is not present, the qrcode is not printed.
Another example is systemd-coredump: it uses libdw and libelf to analyze
a core file and generate a traceback. If those libs cannot be loaded,
the crash is reported without it.
Other examples: libpwquality is used to warn about too-easy passwords,
without it the warning does not happen, libpcre2 is used for 'journalctl --grep',
and without it the --grep option just returns an error, etc.
Systemd allows almost such dependencies to be turned on/off at compile time.
In Fedora we compile with everything on, which means that systemd would
pull in a lot of dependencies. In particular, most systemd binaries are
link to libsystemd-shared.so, which in turn is linked to most of those
libraries. This means that pulling in even a single binary pulls in
many many library dependencies, and even if we were to split systemd
into multiple subpackages, we wouldn't avoid pulling in the dependencies.
Using dlopen allows us to make the requirements more flexible: for example,
in the initrd you certainly don't need qrcodes or pwquality checks.
Thus for the initrd we just don't install those libraries.
We have been doing this for a while. I think the first was libxkbcommon
(which is used by localed to check keyboard mapping configurations),
back in 2016 (https://github.com/systemd/systemd/commit/5de344704d).
When this done correctly, people mostly don't notice.
Some other replies in this thread assumed that the programs will crash.
I have no idea why you would assume that. We're not complete idiots.
Zbyszek
10:15 a.m.
Am 10.12.21 um 15:13 schrieb Zbigniew Jędrzejewski-Szmek:
Some other replies in this thread assumed that the programs will
crash.
I have no idea why you would assume that.
If properly implemented, such programs
typically won't crash, they will
"just not work" oder "error out".
Functionally, dlopening libraries, instead of dynamically linking really
does not provide any advantages.
We're not complete idiots.With all due respect, tt's not the
first time systemd communicates the
attitude of blindly imitating MS-Windows.
Whether you like it or not,
this appears to be one of these cases.
Ralf
3:54 p.m.
On 12/10/21 08:15, Ralf Corsépius wrote:
Am 10.12.21 um 15:13 schrieb Zbigniew Jędrzejewski-Szmek:
> Some other replies in this thread assumed that the programs will crash.
> I have no idea why you would assume that.
If properly implemented, such programs typically won't crash, they will
"just not work" oder "error out".
Functionally, dlopening libraries, instead of dynamically linking really
does not provide any advantages.
Maybe you need to read the other messages in the thread again. It's the
difference between crashing or gracefully reducing functionality while
avoiding hard dependencies.
> We're not complete idiots.With all due respect, tt's not
the first
> time systemd communicates the
attitude of blindly imitating MS-Windows. Whether you like it or not,
this appears to be one of these cases.
There's no need to be rude and you aren't being respectful. This has
nothing to do with Windows. I don't even understand what that
comparison is supposed to mean in this context.
12:58 p.m.
On 12/10/21 09:13, Zbigniew Jędrzejewski-Szmek wrote:
On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel
wrote:
What happens if these libraries are not installed an cannot be dl-opened?
The functionality that requires those libraries will not be available.
This is quite clever, +1.
It occurred to me that a downside might be in mysterious functionality
variances in case of latent bugs. Say, something happens to libpcre2
(file gets damaged, or ABI incompatibility creeps in, etc) so that
suddenly 'journalctl --grep' stops working. How would one find out about
the cause? From reading the journalctl code, it seems to me that
pcre2-dlopen.c:dlopen_pcre2() is called when --grep is used, and it logs
the failure, which I think addresses my concern.
So the apps using this technique would have to be written carefully, to
only dlopen() (and therefore also print possible error messages) when
the optional functionality is called for, right? If the dlopen()s were
just shotgunned at the startup, it could result in a lot of chatter.
If this technique became widely used, as it probably should, maybe there
ought to be a common way of finding out what is and isn't available. For
instance, the apps could have a special common option, like 'journalctl
--capabilities' returning
...
libpcre2 loaded, --grep enabled
libfancy NOT loaded, --FANCY disabled
..
Or, there could be a system-wide status matrix of executables vs
libraries, listing the outcome of recent dlopen()s, so that a query like
'capabilites journalctl' would return the status above (except that such
general utility would not in general be able to print what app
functionality is tied to the particular library)
4:40 a.m.
On Fri, Dec 10, 2021 at 01:58:28PM -0500, przemek klosowski via devel wrote:
On 12/10/21 09:13, Zbigniew Jędrzejewski-Szmek wrote:
>On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel wrote:
> What happens if these libraries are not installed an cannot be dl-opened?
>The functionality that requires those libraries will not be available.
This is quite clever, +1.
It occurred to me that a downside might be in mysterious functionality
variances in case of latent bugs. Say, something happens to libpcre2 (file
gets damaged, or ABI incompatibility creeps in, etc) so that suddenly
'journalctl --grep' stops working. How would one find out about the cause?
From reading the journalctl code, it seems to me that
pcre2-dlopen.c:dlopen_pcre2() is called when --grep is used, and it logs the
failure, which I think addresses my concern.
So the apps using this technique would have to be written carefully, to only
dlopen() (and therefore also print possible error messages) when the
optional functionality is called for, right? If the dlopen()s were just
shotgunned at the startup, it could result in a lot of chatter.
In general, we do this because we think the advantages outweigh the issues,
but this does mean that we're happy with the implementation.
Right now the function signatures are copied into our code, and there's
boilerplate code, and in general the whole system is a bit clunky.
In practice it works fine because the libraries we link to are stable
system libraries that just don't break API or ABI compatibility. We
also import just a handful of symbols. But this is not a general solution
— it'd be horrible for any library whose API evolves or many functions need
to be used.
Ideally, we would have support in the linker to do this automatically.
We would specify that certain libraries should be linked weakly, and the
linker or compiler would generate stubs that'd be resolved to either
NULL or the actual symbol at runtime. I.e. just like calling statically
or dynamically linked libraries requires no modifications in the code,
and is entirely handled by linker commandline options.
If this technique became widely used, as it probably should, maybe
there
ought to be a common way of finding out what is and isn't available. For
instance, the apps could have a special common option, like 'journalctl
--capabilities' returning
...
libpcre2 loaded, --grep enabled
libfancy NOT loaded, --FANCY disabled
That's a reasonable request. Right now it'd need to be implemented manually,
i.e. we'd need to remember that e.g. journalctl --grep dlopens libpcre2, and add
call the same load function in the function implementing --capabilities. This
is certainly doable, but I hope we could make this all more automatic…
Maybe we should just add this --version output. It already prints +SUPPORTED
-UNSUPPORTED.
Zbyszek
5:16 a.m.
Zbigniew Jędrzejewski-Szmek wrote:
This means that various libraries that were
previously always pulled in, are not listed as Recommends by the various
subpackages:
[...]
Recommends are normally installed, but if you're using 'dnf
--setopt install_weak_deps=False'
or similar, please make sure that you install those libraries too if appropriate.
Was "not" supposed to be "now"? Otherwise these statements don't
make
sense together.
Björn Persson
7:52 a.m.
On Fri, Dec 10, 2021 at 12:16:37PM +0100, Björn Persson wrote:
Zbigniew Jędrzejewski-Szmek wrote:
> This means that various libraries that were
> previously always pulled in, are not listed as Recommends by the various
> subpackages:
[...]
> Recommends are normally installed, but if you're using 'dnf --setopt
install_weak_deps=False'
> or similar, please make sure that you install those libraries too if appropriate.
Was "not" supposed to be "now"? Otherwise these statements don't
make
sense together.
Yes, "now". Sorry for the typo.
Zbyszek
11:03 a.m.
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Also, I moved more stuff into the -udev subpackage:
systemd-integritysetup, systemd-veritysetup, systemd-binfmt,
systemd-sysctl, systemd-coredump. IIUC, none of those are useful
in containers, so they fit better in -udev which is supposed to be installed
on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package
for extras? This change affects libguestfs which only wants udev, not
systemd or anything else.
(Ideally, package udev as a separate project of course.)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
4:10 a.m.
On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek
wrote:
> Also, I moved more stuff into the -udev subpackage:
> systemd-integritysetup, systemd-veritysetup, systemd-binfmt,
> systemd-sysctl, systemd-coredump. IIUC, none of those are useful
> in containers, so they fit better in -udev which is supposed to be installed
> on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package
for extras? This change affects libguestfs which only wants udev, not
systemd or anything else.
Hmm, I don't think this changes anything for libguestfs. The
systemd-udev rpm requires systemd, so if have systemd-udev, this is a
noop.
Zbyszek
6:13 a.m.
On Sat, Dec 11, 2021 at 5:43 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
> On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > Also, I moved more stuff into the -udev subpackage:
> > systemd-integritysetup, systemd-veritysetup, systemd-binfmt,
> > systemd-sysctl, systemd-coredump. IIUC, none of those are useful
> > in containers, so they fit better in -udev which is supposed to be installed
> > on real hardware or VMs.
>
> Could we keep udev to being for udev and introduce a different package
> for extras? This change affects libguestfs which only wants udev, not
> systemd or anything else.
Hmm, I don't think this changes anything for libguestfs. The
systemd-udev rpm requires systemd, so if have systemd-udev, this is a
noop.
Please put systemd-coredump back in the main systemd package. Unlike
the others, this is actually required functionality for backtrace
stuff, especially if you're running OS containers (nspawn, ubi-init,
etc.).
--
真実はいつも一つ！/ Always, there's only one truth!
7:44 a.m.
On Wed, Dec 15, 2021 at 07:13:49AM -0500, Neal Gompa wrote:
On Sat, Dec 11, 2021 at 5:43 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
> > On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > > Also, I moved more stuff into the -udev subpackage:
> > > systemd-integritysetup, systemd-veritysetup, systemd-binfmt,
> > > systemd-sysctl, systemd-coredump. IIUC, none of those are useful
> > > in containers, so they fit better in -udev which is supposed to be
installed
> > > on real hardware or VMs.
> >
> > Could we keep udev to being for udev and introduce a different package
> > for extras? This change affects libguestfs which only wants udev, not
> > systemd or anything else.
>
> Hmm, I don't think this changes anything for libguestfs. The
> systemd-udev rpm requires systemd, so if have systemd-udev, this is a
> noop.
>
Please put systemd-coredump back in the main systemd package. Unlike
the others, this is actually required functionality for backtrace
stuff, especially if you're running OS containers (nspawn, ubi-init,
etc.).
The handler for coredumps (/proc/sys/kernel/core_pattern) is not namespaced,
so coredumps reported in a container are handled on the host (*).
Are you sure that your coredump processing happens in the container?
How does the pattern look like and how is the whole thing set up?
(*) In principle systemd-coredump on the host could feed the core file
back to some handler invoked in the container. We discussed this a few
times in the past and it would be quite useful for some types of containers.
But I'm not aware of this being implemented anywhere.
Zbyszek
9:59 a.m.
Hi Zbyszek,
On 10. 12. 21 8:34, Zbigniew Jędrzejewski-Szmek wrote:
Hi,
heads-up: systemd 250-rc1 has been built in rawhide.
If there are regressions, please file a bug or let me know otherwise.
just a minor thing about systemd cryptsetup plugins, but please take a
look at: https://bugzilla.redhat.com/show_bug.cgi?id=2031873
Regards
O.
922
days inactive
927
days old
17 comments
11 participants
participants (11)
-
Björn Persson -
Lennart Poettering -
Matthew Miller -
Neal Gompa -
Ondrej Kozina -
przemek klosowski -
Ralf Corsépius -
Richard W.M. Jones -
Samuel Sieb -
Vitaly Zaitsev -
Zbigniew Jędrzejewski-Szmek