4:37 a.m.
I'm not pretending these will solve everything, but they should make
attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib.
It's extremely convoluted and almost no one understands how it really
works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about
porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do
the initial injection (original:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
These are just my thoughts on a Saturday morning. Feedback welcome of
course.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
4:43 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
W dniu 30.03.2024 o 10:37, Richard W.M. Jones pisze:
(1) We should routinely delete autoconf-generated cruft from
upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
There were also projects where configure script was generated long time
ago and then edited by hand. Usually because no one in project knew m4.
4:45 a.m.
Hey Richard,
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
It sounds like a good plan to put certain dependencies on a critical path. Perhaps
anything that is used by packages included in the various editions of Fedora that
allow for remote access (even if disabled by default) could fall under that path?
We could also try to ensure that packages do not contain any binary blobs and instead
require generation scripts for those that we can run ourselves.
Regards,
Simon
5:42 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30 2024 at 09:37:44 AM +00:00:00, Richard W.M. Jones
<rjones(a)redhat.com> wrote:
In the xz case this wouldn't have been enough, it turns out we
would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
I agree that running autoreconf on our packages makes sense to start
doing. Still, to avoid this backdoored m4 file, we would have needed to
stop using release tarballs altogether and switch to using git tags
directly instead. That would at least force the malicious attacker to
commit their code to version control, making it slightly harder to hide
the attack.
Michael
7:25 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Once upon a time, Michael Catanzaro <mcatanzaro(a)redhat.com> said:
I agree that running autoreconf on our packages makes sense to start
doing. Still, to avoid this backdoored m4 file, we would have needed
to stop using release tarballs altogether and switch to using git
tags directly instead. That would at least force the malicious
attacker to commit their code to version control, making it slightly
harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra
level of author attestation)... but where both are available, it'd be
nice to have a way to denote in the spec file that there are two URLs
for the source. In this case, if both URLs were listed and something
could be run to automatically fetch and compare them, the attack code
would have been flagged.
Just because something is public in a git tag doesn't mean somebody else
reviewed it and caught an attack (after all, in this case, part of it
was committed to git and at least one other maintainer didn't notice
anything odd).
--
Chris Adams <linux(a)cmadams.net>
10:23 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
Once upon a time, Michael Catanzaro <mcatanzaro(a)redhat.com>
said:
> I agree that running autoreconf on our packages makes sense to start
> doing. Still, to avoid this backdoored m4 file, we would have needed
> to stop using release tarballs altogether and switch to using git
> tags directly instead. That would at least force the malicious
> attacker to commit their code to version control, making it slightly
> harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra
level of author attestation)... but where both are available, it'd be
nice to have a way to denote in the spec file that there are two URLs
for the source. In this case, if both URLs were listed and something
could be run to automatically fetch and compare them, the attack code
would have been flagged.
Tarball production should be reproducible. Then the maintainer can
both make a signature locally and make it public, and users can download
the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried
to improve the compression method (i.e. .tar would be still identical,
but .tar.gz would be different), and after a huge outcry this was reverted.
They still haven't officially said that it's stable, but let's hope that
it never changes, at least not without a suitable advance warning.
Zbyszek
10:41 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 03:23:55PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
> Once upon a time, Michael Catanzaro <mcatanzaro(a)redhat.com> said:
> > I agree that running autoreconf on our packages makes sense to start
> > doing. Still, to avoid this backdoored m4 file, we would have needed
> > to stop using release tarballs altogether and switch to using git
> > tags directly instead. That would at least force the malicious
> > attacker to commit their code to version control, making it slightly
> > harder to hide the attack.
>
> Using a signed tarball is ideally better than a git tag (it's an extra
> level of author attestation)... but where both are available, it'd be
> nice to have a way to denote in the spec file that there are two URLs
> for the source. In this case, if both URLs were listed and something
> could be run to automatically fetch and compare them, the attack code
> would have been flagged.
Tarball production should be reproducible. Then the maintainer can
both make a signature locally and make it public, and users can download
the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried
to improve the compression method (i.e. .tar would be still identical,
but .tar.gz would be different), and after a huge outcry this was reverted.
They still haven't officially said that it's stable, but let's hope that
it never changes, at least not without a suitable advance warning.
I used the following to check all 193 github tarballs provided for systemd
(those are autogenerated):
git clone --bare https://github.com/systemd/systemd
dir=systemd.git
tags=$(git --git-dir=$dir tag|rg '^v\d+(\.\d+)?$'|sed 's/^v//'|sort -g)
for v in $tags; do
echo $v
if [[ v =~ . ]]; then
wget https://github.com/systemd/systemd-stable/archive/v$v/systemd-$v.tar.gz
git --git-dir=$dir archive v$v --prefix=systemd-stable-$v/ | gzip
>systemd-$v.local.tar.gz
else
wget https://github.com/systemd/systemd/archive/v$v/systemd-$v.tar.gz
git --git-dir=$dir archive v$v --prefix=systemd-$v/ | gzip
>systemd-$v.local.tar.gz
fi
cmp systemd-$v.local.tar.gz systemd-$v.tar.gz || break
rm systemd-$v.local.tar.gz systemd-$v.tar.gz
echo
done
Fortunately, they all match ;)
Zbyszek
10:35 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, 2024-03-30 at 15:23 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
> Once upon a time, Michael Catanzaro <mcatanzaro(a)redhat.com> said:
> > I agree that running autoreconf on our packages makes sense to start
> > doing. Still, to avoid this backdoored m4 file, we would have needed
> > to stop using release tarballs altogether and switch to using git
> > tags directly instead. That would at least force the malicious
> > attacker to commit their code to version control, making it slightly
> > harder to hide the attack.
>
> Using a signed tarball is ideally better than a git tag (it's an extra
> level of author attestation)... but where both are available, it'd be
> nice to have a way to denote in the spec file that there are two URLs
> for the source. In this case, if both URLs were listed and something
> could be run to automatically fetch and compare them, the attack code
> would have been flagged.
Tarball production should be reproducible. Then the maintainer can
both make a signature locally and make it public, and users can download
the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried
to improve the compression method (i.e. .tar would be still identical,
but .tar.gz would be different), and after a huge outcry this was reverted.
They still haven't officially said that it's stable, but let's hope that
it never changes, at least not without a suitable advance warning.
I do not know if this changed in the last couple of years, but tarballs
in github are not stable (or weren't up to a couple years ago), they
are cached for a period of time, so they may look stable in busy
projects where you have regular downloads that keep the cache alive,
but they are *regenerated* from the tag for seldom downloaded tarballs.
And when that happens then hashes change.
Simo.
--
Simo Sorce
Distinguished Engineer
RHEL Crypto Team
Red Hat, Inc
10:46 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 30. 03. 24 v 1:25 odp. Chris Adams napsal(a):
Using a signed tarball is ideally better than a git tag (it's an
extra
level of author attestation).
In this case signed tarball would not help at all. And git-tag would prevent this attack.
--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
10:48 a.m.
On Sat, Mar 30, 2024 at 11:47 AM Miroslav Suchý <msuchy(a)redhat.com> wrote:
Dne 30. 03. 24 v 1:25 odp. Chris Adams napsal(a):
Using a signed tarball is ideally better than a git tag (it's an extra
level of author attestation).
In this case signed tarball would not help at all. And git-tag would prevent this
attack.
Only because that person didn't think to check it in and tag it. They
very well could have since they had direct commit access.
--
真実はいつも一つ！/ Always, there's only one truth!
6:22 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Richard W.M. Jones wrote on Sat, Mar 30, 2024 at 09:37:44AM +0000:
In the xz case this wouldn't have been enough, it turns out we
would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
I'm not 100% sure about the theory, but it looks like `autoreconf -fi`
looks at the 'serial' in the first line of the script.
The one bundled in the xz sources was marked very high:
# build-to-host.m4 serial 30
But the one in the system (as of f39) is only 'serial 1'.
Artificially lowering the serial back to 0 in the file and running
`autoreconf -fi` again properly reinstall the one from the system over
it, but anything higher will keep it...
So if we want to go this route, we should remove the full m4 dir, but
unfortunately I've seen quite a few projects that depend on non-standard
m4 scripts so we'll need to fix these as we go... (At which point I'd
suggest it's probably faster to convert it all to meson or another new
shiny, and saner, build system, but getting upstreams to agree will be
fun)
(2) We should discourage gnulib as much as possible.
[...]
In the xz case it was a gnulib-derived script which was modified to do
the initial injection (original:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(Honestly I did compare the backdoored script and the real one this
morning and I would be hard pressed to say if either is backdoored just
looking at either version... Admitedly it was 3AM when I looked at it,
but I don't think it's just a late hour problem)
(3) We should have a "security path", like "critical
path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
Before making each of these safer we should make sshd not link with so
many things in the first place.
On oss-security, Solar Designer made a lot of good points about it
(around here: https://www.openwall.com/lists/oss-security/2024/03/29/27
, but the full thread is interesting)
But that's specific to sshd, and this problem could happen with any
network-facing service: I guess sshd is a juciy target because it runs
as root (whereas backdooring e.g. nginx would run as www user and not be
as interesting), but in practice that could be bad enough.
(hm, well, your point about 'enabled by default' strikes home though -
we definitely could pay more attention to these first)
--
Dominique Martinet | Asmadeus
7:06 a.m.
Dominique Martinet wrote:
I'm not 100% sure about the theory, but it looks like `autoreconf
-fi`
looks at the 'serial' in the first line of the script.
The one bundled in the xz sources was marked very high:
# build-to-host.m4 serial 30
But the one in the system (as of f39) is only 'serial 1'.
Artificially lowering the serial back to 0 in the file and running
`autoreconf -fi` again properly reinstall the one from the system over
it, but anything higher will keep it...
So if we want to go this route, we should remove the full m4 dir, but
unfortunately I've seen quite a few projects that depend on non-standard
m4 scripts so we'll need to fix these as we go...
Well, it all depends on whether those m4 scripts are really source code or
whether they are autoimported from somewhere like gnulib. True source code
needs to be retained, anything that can be autoimported should be
autoimported at build time. (And upstreams should stop using imported
copylibs to begin with, but that is a different story.)
(At which point I'd suggest it's probably faster to convert
it all to
meson or another new shiny, and saner, build system, but getting upstreams
to agree will be fun)
CMake! :-)
> (2) We should discourage gnulib as much as possible.
> [...]
> In the xz case it was a gnulib-derived script which was modified to do
> the initial injection (original:
>
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(Honestly I did compare the backdoored script and the real one this
morning and I would be hard pressed to say if either is backdoored just
looking at either version... Admitedly it was 3AM when I looked at it,
but I don't think it's just a late hour problem)
That is exactly the problem with autotools code, almost nobody understands
what the heck it does, almost everybody just copies and pastes somebody
else's snippet hoping it does not do bad things. And gnulib is a
particularly ugly piece of the puzzle.
Before making each of these safer we should make sshd not link with
so
many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma.
Fedora does because of this innocuous-looking patch:
https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...
which is what ultimately allowed this to happen. This drags in libsystemd
for sd_notify, and libsystemd is linked to way too much stuff including
liblzma. Either we need a split libsdnotify that contains only sd_notify, or
we should just stop using sd_notify at all. It increases the attack surface
of daemons a lot just to allow the service to be "Type=notify" rather than
one of the other available approaches. Arch Linux is also systemd-based
nowadays, but still does not link OpenSSH against libsystemd.
Kevin Kofler
7:23 a.m.
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Dominique Martinet wrote:
> I'm not 100% sure about the theory, but it looks like `autoreconf -fi`
> looks at the 'serial' in the first line of the script.
> The one bundled in the xz sources was marked very high:
> # build-to-host.m4 serial 30
> But the one in the system (as of f39) is only 'serial 1'.
>
> Artificially lowering the serial back to 0 in the file and running
> `autoreconf -fi` again properly reinstall the one from the system over
> it, but anything higher will keep it...
>
> So if we want to go this route, we should remove the full m4 dir, but
> unfortunately I've seen quite a few projects that depend on non-standard
> m4 scripts so we'll need to fix these as we go...
Well, it all depends on whether those m4 scripts are really source code or
whether they are autoimported from somewhere like gnulib. True source code
needs to be retained, anything that can be autoimported should be
autoimported at build time. (And upstreams should stop using imported
copylibs to begin with, but that is a different story.)
> (At which point I'd suggest it's probably faster to convert it all to
> meson or another new shiny, and saner, build system, but getting upstreams
> to agree will be fun)
CMake! :-)
Funnily enough, xz can also be built with CMake. :)
I don't know whether the CMake scripts are complete enough to switch
to yet, but we should consider doing it.
>> (2) We should discourage gnulib as much as possible.
>> [...]
>> In the xz case it was a gnulib-derived script which was modified to do
>> the initial injection (original:
>>
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
>
> (Honestly I did compare the backdoored script and the real one this
> morning and I would be hard pressed to say if either is backdoored just
> looking at either version... Admitedly it was 3AM when I looked at it,
> but I don't think it's just a late hour problem)
That is exactly the problem with autotools code, almost nobody understands
what the heck it does, almost everybody just copies and pastes somebody
else's snippet hoping it does not do bad things. And gnulib is a
particularly ugly piece of the puzzle.
> Before making each of these safer we should make sshd not link with so
> many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma.
Fedora does because of this innocuous-looking patch:
https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...
which is what ultimately allowed this to happen. This drags in libsystemd
for sd_notify, and libsystemd is linked to way too much stuff including
liblzma. Either we need a split libsdnotify that contains only sd_notify, or
we should just stop using sd_notify at all. It increases the attack surface
of daemons a lot just to allow the service to be "Type=notify" rather than
one of the other available approaches. Arch Linux is also systemd-based
nowadays, but still does not link OpenSSH against libsystemd.
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library
instead of being part of libsystemd. I'm sure systemd upstream will
consider it now. :)
--
真実はいつも一つ！/ Always, there's only one truth!
9:44 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
> > (At which point I'd suggest it's probably faster to convert it all to
> > meson or another new shiny, and saner, build system, but getting upstreams
> > to agree will be fun)
>
> CMake! :-)
Meson outclasses CMake in functionality, clarity, and brevity.
I doesn't make sense to consider switching to CMake at this point.
> This drags in libsystemd
> for sd_notify, and libsystemd is linked to way too much stuff including
> liblzma. Either we need a split libsdnotify that contains only sd_notify, or
> we should just stop using sd_notify at all. It increases the attack surface
> of daemons a lot just to allow the service to be "Type=notify" rather
than
> one of the other available approaches. Arch Linux is also systemd-based
> nowadays, but still does not link OpenSSH against libsystemd.
>
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library
instead of being part of libsystemd. I'm sure systemd upstream will
consider it now. :)
I don't think a separate library would be particularly useful.
sd_notify() as used by sshd just writes a static string to a unix
socket. This can be implemented in ~10 lines of C, including error handling.
If that is the _only_ thing needed from libsystemd, then open-coding it
is a good option. I guess it'd make sense for systemd to provide a header
file that provides a reference documentation as an inline function.
One thing which is happening, mostly for unrelated reasons, it that
systemd code is being changed to use dlopen() for various libraries which
are usually not needed at runtime. The primary motivation for this is to
omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap,
and libgcrypt + libgpg-error. A pull request to convert that that last
pair to dlopen is under review.
Zbyszek
11:09 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 02:44:32PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
> On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel
> <devel(a)lists.fedoraproject.org> wrote:
> > > (At which point I'd suggest it's probably faster to convert it all
to
> > > meson or another new shiny, and saner, build system, but getting
upstreams
> > > to agree will be fun)
> >
> > CMake! :-)
Meson outclasses CMake in functionality, clarity, and brevity.
I doesn't make sense to consider switching to CMake at this point.
Which is the better build system doesn't much matter here, as they all
let you run shell scripts so source level backdoors could be inserted
in all of them.
The big difference they all have over autoconf is that they don't
usually distribute a giant obfuscated (basically binary) shell script.
We can fix autoconf now by running autoreconf and we don't need to
boil the ocean to do so.
> > This drags in libsystemd
> > for sd_notify, and libsystemd is linked to way too much stuff including
> > liblzma. Either we need a split libsdnotify that contains only sd_notify, or
> > we should just stop using sd_notify at all. It increases the attack surface
> > of daemons a lot just to allow the service to be "Type=notify" rather
than
> > one of the other available approaches. Arch Linux is also systemd-based
> > nowadays, but still does not link OpenSSH against libsystemd.
> >
>
> This is related to philosophical differences between Arch and Fedora.
>
> That said, it probably makes sense for sd_notify to be its own library
> instead of being part of libsystemd. I'm sure systemd upstream will
> consider it now. :)
I don't think a separate library would be particularly useful.
sd_notify() as used by sshd just writes a static string to a unix
socket. This can be implemented in ~10 lines of C, including error handling.
If that is the _only_ thing needed from libsystemd, then open-coding it
is a good option. I guess it'd make sense for systemd to provide a header
file that provides a reference documentation as an inline function.
I think systemd should do this, just so we have one implementation,
and not loads of buggy ones.
One thing which is happening, mostly for unrelated reasons, it that
systemd code is being changed to use dlopen() for various libraries which
are usually not needed at runtime. The primary motivation for this is to
omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap,
and libgcrypt + libgpg-error. A pull request to convert that that last
pair to dlopen is under review.
Also a good change, thanks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
11:12 a.m.
On Sat, Mar 30, 2024 at 12:10 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Sat, Mar 30, 2024 at 02:44:32PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
> > On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel
> > <devel(a)lists.fedoraproject.org> wrote:
> > > > (At which point I'd suggest it's probably faster to convert
it all to
> > > > meson or another new shiny, and saner, build system, but getting
upstreams
> > > > to agree will be fun)
> > >
> > > CMake! :-)
>
> Meson outclasses CMake in functionality, clarity, and brevity.
> I doesn't make sense to consider switching to CMake at this point.
Which is the better build system doesn't much matter here, as they all
let you run shell scripts so source level backdoors could be inserted
in all of them.
The big difference they all have over autoconf is that they don't
usually distribute a giant obfuscated (basically binary) shell script.
We can fix autoconf now by running autoreconf and we don't need to
boil the ocean to do so.
> > > This drags in libsystemd
> > > for sd_notify, and libsystemd is linked to way too much stuff including
> > > liblzma. Either we need a split libsdnotify that contains only sd_notify,
or
> > > we should just stop using sd_notify at all. It increases the attack
surface
> > > of daemons a lot just to allow the service to be "Type=notify"
rather than
> > > one of the other available approaches. Arch Linux is also systemd-based
> > > nowadays, but still does not link OpenSSH against libsystemd.
> > >
> >
> > This is related to philosophical differences between Arch and Fedora.
> >
> > That said, it probably makes sense for sd_notify to be its own library
> > instead of being part of libsystemd. I'm sure systemd upstream will
> > consider it now. :)
>
> I don't think a separate library would be particularly useful.
> sd_notify() as used by sshd just writes a static string to a unix
> socket. This can be implemented in ~10 lines of C, including error handling.
>
> If that is the _only_ thing needed from libsystemd, then open-coding it
> is a good option. I guess it'd make sense for systemd to provide a header
> file that provides a reference documentation as an inline function.
I think systemd should do this, just so we have one implementation,
and not loads of buggy ones.
> One thing which is happening, mostly for unrelated reasons, it that
> systemd code is being changed to use dlopen() for various libraries which
> are usually not needed at runtime. The primary motivation for this is to
> omit such libraries from the initrd. But this also helps with overlinking.
>
> In systemd git main, libsystemd is only linked to libc, libcap,
> and libgcrypt + libgpg-error. A pull request to convert that that last
> pair to dlopen is under review.
Also a good change, thanks.
Note that dlopen() doesn't fix the problem of the giant libsystemd in
the first place. It just obfuscates the true dependency graph of
libsystemd.
Long ago (I think like ~10 years ago), libsystemd was actually several
separate smaller libraries. Perhaps we could consider asking upstream
to switch back to that model?
--
真実はいつも一つ！/ Always, there's only one truth!
12:58 p.m.
Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant
libsystemd in
the first place. It just obfuscates the true dependency graph of
libsystemd.
At least it (hopefully) means liblzma will not be opened if you do not use
an API that needs liblzma. But it makes it even harder to tell whether
liblzma will end up being loaded or not.
Long ago (I think like ~10 years ago), libsystemd was actually
several
separate smaller libraries. Perhaps we could consider asking upstream
to switch back to that model?
+1
Kevin Kofler
4:43 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 06:58:05PM +0100, Kevin Kofler via devel wrote:
Neal Gompa wrote:
> Note that dlopen() doesn't fix the problem of the giant libsystemd in
> the first place. It just obfuscates the true dependency graph of
> libsystemd.
At least it (hopefully) means liblzma will not be opened if you do not use
an API that needs liblzma. But it makes it even harder to tell whether
liblzma will end up being loaded or not.
> Long ago (I think like ~10 years ago), libsystemd was actually several
> separate smaller libraries. Perhaps we could consider asking upstream
> to switch back to that model?
+1
This comes up occasionally, and the main reason to not do this is that
systemd code has a lot of helpers that are used across a lot of the
codebase and we end up with recursive calls between many of the different
"components". So if tried to split things, we'd either need to remove
a lot of polish, or copy code, or have the shared code duplicated.
Some of those split-out libraries would probably end up embedding
almost all of some of the other ones. libsystemd now consists of 12
parts (sd-bus, sd-daemon, sd-device, sd-event, sd-hwdb, sd-id128,
sd-journal, sd-login, sd-netlink, sd-network, sd-path, sd-resolve)
and it'd be a lot of work to untangle, and the overall footprint would
likely grow.
Zbyszek
4 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-03-30 09:12, Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant
libsystemd in
the first place. It just obfuscates the true dependency graph of
libsystemd.
This isn't my area of expertise, but I am curious:
Why doesn't dlopen() solve the problem? As best I understand it,
liblzma was able to steal one (or more) of the symbols from
libcrypto.so.3 because it ran constructors at a point in time when the
GOT was still writable. After loading shared objects is complete, that
table is made read-only. If dlopen() is used after the program starts,
then even if the library is loaded, it can't steal symbols in the table
any more.
Or do I misunderstand this entirely?
4:38 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 02:00:52AM -0700, Gordon Messmer wrote:
On 2024-03-30 09:12, Neal Gompa wrote:
> Note that dlopen() doesn't fix the problem of the giant libsystemd in
> the first place. It just obfuscates the true dependency graph of
> libsystemd.
This isn't my area of expertise, but I am curious:
Why doesn't dlopen() solve the problem? As best I understand it, liblzma
was able to steal one (or more) of the symbols from libcrypto.so.3 because
it ran constructors at a point in time when the GOT was still writable.
After loading shared objects is complete, that table is made read-only. If
dlopen() is used after the program starts, then even if the library is
loaded, it can't steal symbols in the table any more.
Or do I misunderstand this entirely?
You understood correctly ;)
As you wrote, dlopen() is done after RELRO has taken effect.
Also, dlopen() is only called lazily when the required library is to
be used for the first time… So in this particular case, the program
could call sd_notify(), but no dlopen() call would be done.
Zbyszek
4:57 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
* Gordon Messmer:
Why doesn't dlopen() solve the problem? As best I understand
it,
liblzma was able to steal one (or more) of the symbols from
libcrypto.so.3 because it ran constructors at a point in time when the
GOT was still writable. After loading shared objects is complete,
that table is made read-only. If dlopen() is used after the program
starts, then even if the library is loaded, it can't steal symbols in
the table any more.
The rsa_pkcs1_ossl_meth variable that contains a function pointer to the
actual implementation resides in read-write memory:
static RSA_METHOD rsa_pkcs1_ossl_meth = {
"OpenSSL PKCS#1 RSA",
rsa_ossl_public_encrypt,
rsa_ossl_public_decrypt, /* signature verification */
rsa_ossl_private_encrypt, /* signing */
rsa_ossl_private_decrypt,
rsa_ossl_mod_exp,
BN_mod_exp_mont, /* XXX probably we should not use Montgomery
* if e == 3 */
rsa_ossl_init,
rsa_ossl_finish,
RSA_FLAG_FIPS_METHOD, /* flags */
NULL,
0, /* rsa_sign */
0, /* rsa_verify */
NULL, /* rsa_keygen */
NULL /* rsa_multi_prime_keygen */
};
The missing “const” keyword is probably an oversight, but I don't think
it matters because the variable is not used directly. All access
happens through deefault_RSA_meth:
static const RSA_METHOD *default_RSA_meth = &rsa_pkcs1_ossl_meth;
void RSA_set_default_method(const RSA_METHOD *meth)
{
default_RSA_meth = meth;
}
const RSA_METHOD *RSA_get_default_method(void)
{
return default_RSA_meth;
}
const RSA_METHOD *RSA_PKCS1_OpenSSL(void)
{
return &rsa_pkcs1_ossl_meth;
}
And that's clearly required to be read-write. RSA_set_default_method is
a documented interface, too. So I don't think GOT patching was required
to achieve the intended effect.
Furthermore, it's possible to undo the effect of RELRO with mprotect.
Using dlopen, liblzma would not have been loaded at run time, but it's
not clear if it had mattered because liblzma would still have been
loaded by RPM, both at build and install time.
Thanks,
Florian
11:31 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 30.03.24 um 15:44 schrieb Zbigniew Jędrzejewski-Szmek:
Meson outclasses CMake in functionality, clarity, and brevity.
I doesn't make sense to consider switching to CMake at this point.
While I do agree on clarity and brevity, I don't on functionality.
Meson doesn't allow you do create your own functions. While one should
try to avoid them in build systems, there are cases where you need them.
I work on a project where they are needed, but it also wouldn't make
sense to upstream it because it's too project specific, and creating a
loop out of every call like Meson's FAQ recommends
(https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functi...)
would create one heck of spagetthi code.
So yeah, there are edge cases where I would recommend CMake over Meson
purely without looking at the rest of the ecosystem.
Another is ofc that a quite big chunk of the C and C++ ecosystem uses
CMake and integration between build systems kinda sucks (but the CMake
developers try to work on making it better by working together with
other projects).
But I don't think this is the right place to talk about which build
system to switch to FOR OTHER PROJECTS.
(But if we are at it, ever looked at Zig as a buildsystem for a C or C++
project? I know some companies which do this at this point.)
Sincerely
Kilian Hanich
1:06 p.m.
Kilian Hanich via devel wrote:
Meson doesn't allow you do create your own functions. While one
should
try to avoid them in build systems, there are cases where you need them.
I work on a project where they are needed, but it also wouldn't make
sense to upstream it because it's too project specific, and creating a
loop out of every call like Meson's FAQ recommends
(https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functi...)
would create one heck of spagetthi code.
Yes, that is exactly what makes Meson all hardcoded and not extensible. If
you need to do anything that the Meson developers did not think of, you end
up having to work around the build system (using external commands, or a
Meson file generator that preprocesses macros, or some other ugly hack)
instead of with the build system or just having to switch to a better build
system (such as CMake ;-) ).
Using loops instead of functions or macros also misses one important point:
the function or macro can be shared between projects and even eventually
upstreamed to the build system (both of which happen regularly in the CMake
world).
By the way, CMake allows defining both macros and actual functions, and
macros are what you want to use in most cases. The main reason functions
were introduced is to allow recursion (which is a two-edged sword because it
makes the language Turing-complete with all its implications).
Kevin Kofler
12:56 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
Meson outclasses CMake in functionality,
LOL, how so? Everything in Meson is hardcoded, you have very little
flexibility (but still enough to plant a backdoor if that is what you want,
because you can just invoke a shell script or any other external command).
CMake is completely extensible with a complete programming language (it is
Turing-complete if and only if you use recursive functions; if you avoid
FUNCTION or at least recursive ones, then it is guaranteed to always
terminate and still usually expressive enough, but Turing-completeness
through recursion is there if you really need it).
clarity, and brevity.
That is very much in the eye of the beholder, and it also depends on whether
you use modern CMake or legacy constructs that are often overly verbose.
Meson does not have so much legacy stuff simply because 1. it is much newer
(which also means it is less mature) and 2. it cares less about backwards
compatibility (which is not a good thing, backwards compatibility is
essential if you want to avoid upstreams bundling old copies of build
tools).
I doesn't make sense to consider switching to CMake at this
point.
CMake being what the whole Qt and KDE community uses nowadays (even Qt
itself switched to it with Qt 6), it very much makes sense to switch to
CMake now more than ever.
The main reason Meson is a thing at all (and not just one person's little-
known project as it initially was) is that GNOME did not want to use the
same build tool KDE was already using (even though they had the exact same
issues with autotools).
I don't think a separate library would be particularly useful.
sd_notify() as used by sshd just writes a static string to a unix
socket. This can be implemented in ~10 lines of C, including error
handling.
Copy&paste is exactly what we do not want here. (It can easily be done
wrong, either accidentally or deliberately, introducing security issues, and
any security issue discovered in the copied&pasted code has to be fixed in
all the copies.) If daemons are likely to need only one function
(sd_notify), then a library containing only that one function is exactly
what is needed, no matter how small it is.
If that is the _only_ thing needed from libsystemd, then open-coding
it
is a good option. I guess it'd make sense for systemd to provide a header
file that provides a reference documentation as an inline function.
That is an option which is better than copy&paste, but it still means that
all users will need to be rebuilt if a bug in that header file is fixed (as
for a static library, but unlike a shared library).
One thing which is happening, mostly for unrelated reasons, it that
systemd code is being changed to use dlopen() for various libraries which
are usually not needed at runtime. The primary motivation for this is to
omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap,
and libgcrypt + libgpg-error. A pull request to convert that that last
pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working),
but it also makes things even less transparent: How do we know whether some
random sd_foobarify() function or some random foobard linked against
libsystemd will (always or ever (and when?)) end up dlopening liblzma or
not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies
it introduces.
Kevin Kofler
5:15 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 06:56:27PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
> Meson outclasses CMake in functionality,
LOL, how so? Everything in Meson is hardcoded, you have very little
flexibility (but still enough to plant a backdoor if that is what you want,
because you can just invoke a shell script or any other external command).
CMake is completely extensible with a complete programming language (it is
Turing-complete if and only if you use recursive functions; if you avoid
FUNCTION or at least recursive ones, then it is guaranteed to always
terminate and still usually expressive enough, but Turing-completeness
through recursion is there if you really need it).
Pretty much every build system is Turing-complete, and pretty much every
build system allows scripts to be called. Upstreams can do crazy stuff with
any build system too.
I think Meson hits the sweet spot where simple projects can be expressed
in very few very readable lines. And common configuration tests that
need to be done are still very readable.
The handling of dependencies and options is very declarative.
(Meson is also evolving, in early versions it was less so.)
It'd be interesting to see some real statistics, e.g. over all Fedora
packages, but in my experience, CMake projects tend to have many more
files with lots of lines.
> clarity, and brevity.
That is very much in the eye of the beholder, and it also depends on whether
you use modern CMake or legacy constructs that are often overly verbose.
True ;) In _my_ eye, it's prettier. I guess we'll not achieve consensus.
One particular issue I have with CMake as a downstream maintainer is
it's often very hard to override linking or compilation options
or when the project is using one of the cmake find scripts that gets
something wrong… It's possible that those projects are "doing cmake wrong",
but it seems that CMake makes it easy to do things wrong.
> I doesn't make sense to consider switching to CMake at this
point.
CMake being what the whole Qt and KDE community uses nowadays (even Qt
itself switched to it with Qt 6), it very much makes sense to switch to
CMake now more than ever.
Right, but if one doesn't work on Qt, then this argument doesn't do much.
[snip]
> One thing which is happening, mostly for unrelated reasons, it
that
> systemd code is being changed to use dlopen() for various libraries which
> are usually not needed at runtime. The primary motivation for this is to
> omit such libraries from the initrd. But this also helps with overlinking.
>
> In systemd git main, libsystemd is only linked to libc, libcap,
> and libgcrypt + libgpg-error. A pull request to convert that that last
> pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working),
but it also makes things even less transparent: How do we know whether some
random sd_foobarify() function or some random foobard linked against
libsystemd will (always or ever (and when?)) end up dlopening liblzma or
not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies
it introduces.
This is true. It'd be great if the compilers and linkers supported a
"weak dependency" model, where we'd use the normal call syntax and things
would be the same as with a normal shared library link, but if that library
is not found, the program would still load and run. Then we could
still autogenerate those dependencies on optional libraries, but using
Recommends instead of Requires.
Right now the dependency handling needs to be handled manually, which
is error-prone and annoying.
Zbyszek
8:52 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 10:15:47PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
One particular issue I have with CMake as a downstream maintainer is
it's often very hard to override linking or compilation options
or when the project is using one of the cmake find scripts that gets
something wrong… It's possible that those projects are "doing cmake
wrong",
but it seems that CMake makes it easy to do things wrong.
I just had to interact with CMake, so let's use that as example:
I'm rebuilding rpm.rpm with some local patches using 'fedpkg local' in F40.
The build fails with:
Directory not found:
/home/zbyszek/rpmbuild/BUILDROOT/rpm-4.19.1.1-2.fc41.x86_64/usr/lib64/python3.12/site-packages/rpm
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of
python, and it found /usr/bin/python3.13t that I have installed, and subsequently
it got all the paths wrong. How do I override this?
('cmake -LAH' doesn't yield anything useful.)
(When compiling a python module, any other alternative would be better.
The build uses %python3, i.e. /usr/bin/python3.12, so there's no need
to detect anything, the location of python is known and all this binary
can be called to print all paths. Otherwise, either call
'python3-config' or 'pkgconf python', don't do you own
reimplementation.)
Zbyszek
11:44 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
That's why you should never build packages outside of mock.
Kevin Kofler
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Sat, Mar 30, 2024 at 10:15:47PM +0000, Zbigniew
Jędrzejewski-Szmek wrote:
> One particular issue I have with CMake as a downstream maintainer is
> it's often very hard to override linking or compilation options
> or when the project is using one of the cmake find scripts that gets
> something wrong… It's possible that those projects are "doing
> cmake wrong",
> but it seems that CMake makes it easy to do things wrong.
I just had to interact with CMake, so let's use that as example:
I'm rebuilding rpm.rpm with some local patches using 'fedpkg local'
in F40.
The build fails with:
Directory not found:
/home/zbyszek/rpmbuild/BUILDROOT/rpm-4.19.1.1-2.fc41.x86_64/usr/lib64/python3.12/site-packages/rpm
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special
detection of
python, and it found /usr/bin/python3.13t that I have installed, and
subsequently
it got all the paths wrong. How do I override this?
('cmake -LAH' doesn't yield anything useful.)
(When compiling a python module, any other alternative would be
better.
The build uses %python3, i.e. /usr/bin/python3.12, so there's no need
to detect anything, the location of python is known and all this
binary
can be called to print all paths. Otherwise, either call
'python3-config' or 'pkgconf python', don't do you own
reimplementation.)
Zbyszek
5:03 p.m.
I wrote:
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew
Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> Hmm, why? Oh, rpm uses cmake, and cmake has it's own special
> detection of python, and it found /usr/bin/python3.13t that I have
> installed, and subsequently it got all the paths wrong.
That's why you should never build packages outside of mock.
PS: Autotools also loves to autodetect random libraries that happen to be
installed on the system. It is in no way specific to CMake.
> How do I override this?
> ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for
the variables it uses. (First, try to figure out whether RPM is using a
system-installed FindPython or its own custom version, so you look at the
correct version.) But the safest (for all build systems) is to always build
in a mock chroot with only the expected BuildRequires installed, as I have
written.
Kevin Kofler
4:36 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
I wrote:
> On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
>> Hmm, why? Oh, rpm uses cmake, and cmake has it's own special
>> detection of python, and it found /usr/bin/python3.13t that I have
>> installed, and subsequently it got all the paths wrong.
>
> That's why you should never build packages outside of mock.
That's another way of saying "it's broken" ;]
Mock is great, but I'm doing local development and a local build
against my envirnoment is what I need.
PS: Autotools also loves to autodetect random libraries that happen
to be
installed on the system. It is in no way specific to CMake.
>> How do I override this?
>> ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for
the variables it uses. (First, try to figure out whether RPM is using a
system-installed FindPython or its own custom version, so you look at the
correct version.)
Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't
want to integrate with the Linux userspace in the normal way.
Zbyszek
4:52 a.m.
On Mon, Apr 8, 2024 at 5:37 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
> I wrote:
> > On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek
> > <zbyszek(a)in.waw.pl> wrote:
> >> Hmm, why? Oh, rpm uses cmake, and cmake has it's own special
> >> detection of python, and it found /usr/bin/python3.13t that I have
> >> installed, and subsequently it got all the paths wrong.
> >
> > That's why you should never build packages outside of mock.
That's another way of saying "it's broken" ;]
Mock is great, but I'm doing local development and a local build
against my envirnoment is what I need.
> PS: Autotools also loves to autodetect random libraries that happen to be
> installed on the system. It is in no way specific to CMake.
>
> >> How do I override this?
> >> ('cmake -LAH' doesn't yield anything useful.)
>
> Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for
> the variables it uses. (First, try to figure out whether RPM is using a
> system-installed FindPython or its own custom version, so you look at the
> correct version.)
Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't
want to integrate with the Linux userspace in the normal way.
You know that pkgconfig *also* uses variables too, right? It's
perfectly "normal" to do it this way. Arguably, the fact that every
variable is known and saved in the CMakeLists and CMakeCache data for
you to read is a boon, because if you need to know a variable to
override, you can find it easily in the logs.
And this your statement on normalcy is silly since we don't *have* a
normal way. The GNU Build System (Autotools), Meson, and CMake all do
things differently, and all three have significant share in our
packages. Projects that use plain Makefiles are even *more*
non-standard, as they do whatever they want too.
--
真実はいつも一つ！/ Always, there's only one truth!
5:21 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 08, 2024 at 05:52:07AM -0400, Neal Gompa wrote:
On Mon, Apr 8, 2024 at 5:37 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
> > I wrote:
> > > On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek
> > > <zbyszek(a)in.waw.pl> wrote:
> > >> Hmm, why? Oh, rpm uses cmake, and cmake has it's own special
> > >> detection of python, and it found /usr/bin/python3.13t that I have
> > >> installed, and subsequently it got all the paths wrong.
> > >
> > > That's why you should never build packages outside of mock.
>
> That's another way of saying "it's broken" ;]
> Mock is great, but I'm doing local development and a local build
> against my envirnoment is what I need.
>
> > PS: Autotools also loves to autodetect random libraries that happen to be
> > installed on the system. It is in no way specific to CMake.
> >
> > >> How do I override this?
> > >> ('cmake -LAH' doesn't yield anything useful.)
> >
> > Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for
> > the variables it uses. (First, try to figure out whether RPM is using a
> > system-installed FindPython or its own custom version, so you look at the
> > correct version.)
>
> Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't
> want to integrate with the Linux userspace in the normal way.
>
You know that pkgconfig *also* uses variables too, right? It's
perfectly "normal" to do it this way. Arguably, the fact that every
variable is known and saved in the CMakeLists and CMakeCache data for
you to read is a boon, because if you need to know a variable to
override, you can find it easily in the logs.
So… this is what I'm talking about: there is no obvious way to
figure out what to set. Looking at the logs and trying to figure out
some variables from that is not very attractive.
Of course I know pgkconfig uses variables, I don't know what you
are trying to say.
I think CMake hits the "sour spot" here in a few different ways:
- because of the historical mindset, instead of using standard
mechanisms that everybody else uses, it implements custom detection,
- this autodetection very often does things wrong,
- because of the approach with Find* scripts, each package is different
and each detection script has a custom interface, so overriding things
is a lot of work,
- the complicated language that is partially variable-based, partially
macro-based is not easy to follow.
autotools has './configure --help',
meson has all options in `meson_options.txt' and also 'meson configure'
will print all options,
but with CMake the options are buried in Find* and not easy to figure out.
And this your statement on normalcy is silly since we don't
*have* a
normal way. The GNU Build System (Autotools), Meson, and CMake all do
things differently, and all three have significant share in our
packages. Projects that use plain Makefiles are even *more*
non-standard, as they do whatever they want too.
Yeah, I know we have a bazillion build systems. And all have
to work and the packagers need to use and understand all of them.
Nevertheless, for me, CMake and autotools are outdated technologies
that shouldn't be used in new projects.
Zbyszek
2:57 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
So… this is what I'm talking about: there is no obvious way to
figure out what to set. Looking at the logs and trying to figure out
some variables from that is not very attractive.
The comments at the top of the relevant Find*.cmake module are the best
source for which variables you are supposed to set directly.
But there is also cmake-gui that can show you all the available options in a
pretty Qt GUI.
Nevertheless, for me, CMake and autotools are outdated technologies
that shouldn't be used in new projects.
And for me, Meson is just a poor Not Invented Here imitation of CMake, with
fewer features and in a slower programming language. :-)
And the kind of automagic you complain about is something all 3 major build
systems do (and plenty of obscure ones, too). Maybe not for the specific
case of the Python executable, but there are plenty of other cases where
autotools and Meson also do automagic, which is why building outside of a
chroot is such a bad idea.
Kevin Kofler
5:27 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sa, 30.03.24 18:56, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:
> In systemd git main, libsystemd is only linked to libc, libcap,
> and libgcrypt + libgpg-error. A pull request to convert that that last
> pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working),
but it also makes things even less transparent: How do we know whether some
random sd_foobarify() function or some random foobard linked against
libsystemd will (always or ever (and when?)) end up dlopening liblzma or
not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies
it introduces.
Well, this is certainly a valid point, but the solution is not to make
all deps hard ones. Instead, in order to just make these deps visible
I see multiple better alternatives:
1. teach Linux/ELF something inspired by MacOS's weak deps. i.e. allow
ELF programs declare that some symbols shall be backed by some lib,
but make it a weak dep that is only resolved on demand, and
gracefully is set to NULL if the library cannot be found. If we had
that, we'd stop using dlopen() for systemd's deps, since all we do
is basically reimplement this concept manually.
2. without Linux/ELF supporting the above we could still teach systemd
and other tools in similar situations to declare the dlopen deps
they have in some ELF note or section, so that it can be processed
by initrd generators, automatic dep generators in dpkg/rpm,
ldd-like tools and everything else that wants this info. This would
require some C macro magic, but could be added in probably just a
few lines of codes added to relevant projects.
Lennart
--
Lennart Poettering, Berlin
6:55 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
* Lennart Poettering:
On Sa, 30.03.24 18:56, Fedora Development ML
(devel(a)lists.fedoraproject.org) wrote:
> > In systemd git main, libsystemd is only linked to libc, libcap,
> > and libgcrypt + libgpg-error. A pull request to convert that that last
> > pair to dlopen is under review.
>
> That helps somewhat (it would have prevented this backdoor from working),
> but it also makes things even less transparent: How do we know whether some
> random sd_foobarify() function or some random foobard linked against
> libsystemd will (always or ever (and when?)) end up dlopening liblzma or
> not?
>
> Distribution packagers tend to dislike dlopen due to the hidden dependencies
> it introduces.
Well, this is certainly a valid point, but the solution is not to make
all deps hard ones. Instead, in order to just make these deps visible
I see multiple better alternatives:
1. teach Linux/ELF something inspired by MacOS's weak deps. i.e. allow
ELF programs declare that some symbols shall be backed by some lib,
but make it a weak dep that is only resolved on demand, and
gracefully is set to NULL if the library cannot be found. If we had
that, we'd stop using dlopen() for systemd's deps, since all we do
is basically reimplement this concept manually.
How exactly is this implemented? Is the object loaded on the first
function call? I'd be worried that this made initialization even more
complicated than it is today.
Loading the object unconditionally (prior to the function call) doesn't
make a difference for objects which are always installed on the system.
They would always be loaded.
Thanks,
Florian
10:50 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Kevin Kofler via devel wrote:
Dominique Martinet wrote:
> Before making each of these safer we should make sshd not link with so
> many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma.
Fedora does because of this innocuous-looking patch:
https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...
which is what ultimately allowed this to happen. This drags in libsystemd
for sd_notify, and libsystemd is linked to way too much stuff including
liblzma. Either we need a split libsdnotify that contains only sd_notify, or
we should just stop using sd_notify at all.
Upstream openssh-portable has a proposed patch which simply
implements the sdnotify protocol directly. That would
provide the benefits with none of the over-linking risk.
https://bugzilla.mindrot.org/show_bug.cgi?id=2641#c13
It could use some review from distro folks familiar with
sshd systemd integration.
(The wider point about splitting the sdnotify functionality
is still quite useful, to avoid everyone re-implementing the
same thing and possibly adding bugs in _that_ process.)
--
Todd
1:52 p.m.
On Sat, Mar 30, 2024 at 1:07 PM Kevin Kofler via devel <
devel(a)lists.fedoraproject.org> wrote:
> Before making each of these safer we should make sshd not link
with so
> many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma.
Fedora does because of this innocuous-looking patch:
https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...
which is what ultimately allowed this to happen. This drags in libsystemd
for sd_notify, and libsystemd is linked to way too much stuff including
liblzma. Either we need a split libsdnotify that contains only sd_notify,
or
we should just stop using sd_notify at all. It increases the attack
surface
of daemons a lot just to allow the service to be "Type=notify" rather than
one of the other available approaches. Arch Linux is also systemd-based
nowadays, but still does not link OpenSSH against libsystemd.
We have an upstream-adjusted version of this patch, see
https://bugzilla.mindrot.org/show_bug.cgi?id=2641
I'm OK to bring the updated version of this script to Fedora as soon as it
is finalized.
--
Dmitry Belyavskiy
6:34 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-03-30 11:52, Dmitry Belyavskiy wrote:
We have an upstream-adjusted version of this patch, see
https://bugzilla.mindrot.org/show_bug.cgi?id=2641
I'm OK to bring the updated version of this script to Fedora as soon
as it is finalized.
I proposed https://src.fedoraproject.org/rpms/openssh/pull-request/72 ,
which uses the implementation from
https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
2:10 a.m.
On Wednesday, 3 April 2024 01:34:00 CEST Gordon Messmer wrote:
On 2024-03-30 11:52, Dmitry Belyavskiy wrote:
> We have an upstream-adjusted version of this patch, see
> https://bugzilla.mindrot.org/show_bug.cgi?id=2641
> I'm OK to bring the updated version of this script to Fedora as soon
> as it is finalized.
I proposed https://src.fedoraproject.org/rpms/openssh/pull-request/72 ,
which uses the implementation from
https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
Thanks for the contribution, but please use https://bugzilla.mindrot.org/
show_bug.cgi?id=2641#c13 instead.
6:47 a.m.
I wrote:
That is exactly the problem with autotools code, almost nobody
understands
what the heck it does, almost everybody just copies and pastes somebody
else's snippet hoping it does not do bad things. And gnulib is a
particularly ugly piece of the puzzle.
PS: Here is a pretty good post summarizing the issues with autotools, both
generally and in the context of the xz vulnerability:
https://felipec.wordpress.com/2024/04/04/xz-backdoor-and-autotools-insanity/
Kevin Kofler
7:20 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 08:22:06PM +0900, Dominique Martinet wrote:
> the initial injection (original:
>
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(Honestly I did compare the backdoored script and the real one this
morning and I would be hard pressed to say if either is backdoored just
looking at either version... Admitedly it was 3AM when I looked at it,
but I don't think it's just a late hour problem)
Right! Definitely not a 3am problem :-/
> (3) We should have a "security path", like
"critical path".
...
Before making each of these safer we should make sshd not link with
so
many things in the first place.
On oss-security, Solar Designer made a lot of good points about it
(around here: https://www.openwall.com/lists/oss-security/2024/03/29/27
, but the full thread is interesting)
Agreed.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
6:53 a.m.
Richard W.M. Jones wrote:
(1) We should routinely delete autoconf-generated cruft from
upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
I have always said that. We do not allow other prebuilt binary blobs and
require rebuilding everything from source. I do not see how the obfuscated
autogenerated shell scripts from the autotools are any different. Sadly, I
was ignored. Now you folks (Fedora) see where this leads. Told You So.
In the xz case this wouldn't have been enough, it turns out we
would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
Looks like autoreconf does not work as advertised. With -f, it is supposed
to regenerate all files that it knows how to regenerate. It clearly does not
do what it is supposed to and ought to be fixed.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib.
It's extremely convoluted and almost no one understands how it really
works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about
porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do
the initial injection (original:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
There too, I agree completely. Also because gnulib is a "copylib", which is
a very broken concept. A true library would not be subject to this kind of
"take the file from the copylib and inject bad code into it" attack.
(3) We should have a "security path", like "critical
path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
I think the issue is that there is just too much stuff in critpath these
days. Whole desktop environments and all their transitive dependencies
probably ought to not be in there. If at all, I would put the display
manager in there, maybe the window manager, and no further.
Kevin Kofler
7:15 a.m.
On Sat, Mar 30, 2024 at 7:54 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Richard W.M. Jones wrote:
> (1) We should routinely delete autoconf-generated cruft from upstream
> projects and regenerate it in %prep. It is easier to study the real
> source rather than dig through the convoluted, generated shell script
> in an upstream './configure' looking for back doors.
>
> For most projects, just running "autoreconf -fiv" is enough.
>
> Yes, there are some projects that depend on a specific or old version
> of autoconf. We should fix those. But that doesn't need to delay us
> from using autoreconf on many projects today.
I have always said that. We do not allow other prebuilt binary blobs and
require rebuilding everything from source. I do not see how the obfuscated
autogenerated shell scripts from the autotools are any different. Sadly, I
was ignored. Now you folks (Fedora) see where this leads. Told You So.
This is not helpful in the slightest and the tone is not appreciated at all.
That said, this is being tracked by the Packaging Committee:
https://pagure.io/packaging-committee/issue/1350
> In the xz case this wouldn't have been enough, it turns out
we would
> also have to delete m4/build-to-host.m4, which then autoreconf
> regenerates. I don't fully understand why that is.
Looks like autoreconf does not work as advertised. With -f, it is supposed
to regenerate all files that it knows how to regenerate. It clearly does not
do what it is supposed to and ought to be fixed.
> (2) We should discourage gnulib as much as possible.
>
> In libvirt we took the decision a few years ago to remove gnulib.
> It's extremely convoluted and almost no one understands how it really
> works. It's written in obscure m4 macros and shell script.
>
> It's also not necessary for Linux since gnulib is mainly about
> porting to non-Linux platforms. There are better ways to do this.
>
> In the xz case it was a gnulib-derived script which was modified to do
> the initial injection (original:
>
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
There too, I agree completely. Also because gnulib is a "copylib", which is
a very broken concept. A true library would not be subject to this kind of
"take the file from the copylib and inject bad code into it" attack.
Yes, we should scrutinize things like this. Though I will note that we
didn't actually suffer an attack through this venue with library code,
just the build scripts. Generally, people do not pay attention to
build scripts, and that was how this slipped by for so long. But even
so, Autotools is particularly difficult to understand and I don't
think we would have ordinarily caught it anyway.
> (3) We should have a "security path", like
"critical path".
>
> sshd is linked to a lot of libraries:
>
> /lib64/libaudit.so.1 audit-libs
> /lib64/libc.so.6 glibc
> /lib64/libcap-ng.so.0 libcap-ng
> /lib64/libcap.so.2 libcap
> /lib64/libcom_err.so.2 libcom_err
> /lib64/libcrypt.so.2 libxcrypt
> /lib64/libcrypto.so.3 openssl-libs
> /lib64/libeconf.so.0 libeconf
> /lib64/libgcc_s.so.1 libgcc
> /lib64/libgssapi_krb5.so.2 krb5-libs
> /lib64/libk5crypto.so.3 krb5-libs
> /lib64/libkeyutils.so.1 keyutils-libs
> /lib64/libkrb5.so.3 krb5-libs
> /lib64/libkrb5support.so.0 krb5-libs
> /lib64/liblz4.so.1 lz4-libs
> /lib64/liblzma.so.5 xz-libs
> /lib64/libm.so.6 glibc
> /lib64/libpam.so.0 pam-libs
> /lib64/libpcre2-8.so.0 pcre2
> /lib64/libresolv.so.2 glibc
> /lib64/libselinux.so.1 libselinux
> /lib64/libsystemd.so.0 systemd-libs
> /lib64/libz.so.1 zlib / zlib-ng
> /lib64/libzstd.so.1 zstd
>
> Should we have a higher level of attention to these packages? We
> already have "critical path", but that's a broad category now. These
> seem like they are "security path" packages, an intentionally small
> subset associated with very secure services which are enabled by
> default.
I think the issue is that there is just too much stuff in critpath these
days. Whole desktop environments and all their transitive dependencies
probably ought to not be in there. If at all, I would put the display
manager in there, maybe the window manager, and no further.
I don't know if the security path package list would help any, since
we still have no one to do security work.
That said, I agree that pretty much every display manager and
compositor for every Fedora variant should be critpath'd.
--
真実はいつも一つ！/ Always, there's only one truth!
7:52 a.m.
Neal Gompa wrote:
This is not helpful in the slightest and the tone is not appreciated
at
all.
Well, I have been arguing against this exception (exempting prebuilt
autotools output) from the "no prebuilt blobs" rule for years, and it
saddens me that something like this had to happen for Fedora to finally
realize that that exception has always been a bad idea.
That said, this is being tracked by the Packaging Committee:
https://pagure.io/packaging-committee/issue/1350
Finally, thanks! (But you filed that only now after this incident, see
above. Still, thanks that you are bringing this up now!)
Yes, we should scrutinize things like this. Though I will note that
we
didn't actually suffer an attack through this venue with library code,
just the build scripts. Generally, people do not pay attention to
build scripts, and that was how this slipped by for so long. But even
so, Autotools is particularly difficult to understand and I don't
think we would have ordinarily caught it anyway.
I definitely agree there, build scripts are indeed an attractive target for
backdoor authors, and autotools is indeed a big part of the problem.
The whole architecture of autotools was designed for a situation that is
mostly obsolete these days: people running some proprietary Unix with some
buggy implementation of a Bourne-like shell and no centralized build tools
who want to just untar a tarball and build it with only what they already
have installed (the buggy shell). Hence all this concept of shipping
prebuilt obfuscated shell blobs full of workarounds for bugs in ancient
shell implementations. Nowadays, people are either running GNU/Linux, where
centralized build tools such as CMake or Meson are readily installable from
the repository (and where most builds are done by distributions, for whom it
is just a matter of adding, e.g., "BuildRequires: cmake"), or Microsoft
Windows, where an environment that can run autotools scripts (e.g., MSYS2)
is NOT part of the system and actually as hard or harder to install than
something like CMake. So, nowadays, pregenerating shell scripts is a
completely outdated and unhelpful way of working.
That said, I agree that pretty much every display manager and
compositor for every Fedora variant should be critpath'd.
Well, where we disagree is that I actually want to see LESS stuff in
critpath, not more. It cannot be scrutinized well enough because there is
just too much stuff in it. E.g., at times, we had MySQL/MariaDB in critpath
because Akonadi required it. (Nowadays, Akonadi actually recommends using
SQLite instead.) That just does not make sense.
Kevin Kofler
8:09 a.m.
On Sat, Mar 30, 2024 at 8:53 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> This is not helpful in the slightest and the tone is not appreciated at
> all.
Well, I have been arguing against this exception (exempting prebuilt
autotools output) from the "no prebuilt blobs" rule for years, and it
saddens me that something like this had to happen for Fedora to finally
realize that that exception has always been a bad idea.
> That said, this is being tracked by the Packaging Committee:
> https://pagure.io/packaging-committee/issue/1350
Finally, thanks! (But you filed that only now after this incident, see
above. Still, thanks that you are bringing this up now!)
> Yes, we should scrutinize things like this. Though I will note that we
> didn't actually suffer an attack through this venue with library code,
> just the build scripts. Generally, people do not pay attention to
> build scripts, and that was how this slipped by for so long. But even
> so, Autotools is particularly difficult to understand and I don't
> think we would have ordinarily caught it anyway.
I definitely agree there, build scripts are indeed an attractive target for
backdoor authors, and autotools is indeed a big part of the problem.
The whole architecture of autotools was designed for a situation that is
mostly obsolete these days: people running some proprietary Unix with some
buggy implementation of a Bourne-like shell and no centralized build tools
who want to just untar a tarball and build it with only what they already
have installed (the buggy shell). Hence all this concept of shipping
prebuilt obfuscated shell blobs full of workarounds for bugs in ancient
shell implementations. Nowadays, people are either running GNU/Linux, where
centralized build tools such as CMake or Meson are readily installable from
the repository (and where most builds are done by distributions, for whom it
is just a matter of adding, e.g., "BuildRequires: cmake"), or Microsoft
Windows, where an environment that can run autotools scripts (e.g., MSYS2)
is NOT part of the system and actually as hard or harder to install than
something like CMake. So, nowadays, pregenerating shell scripts is a
completely outdated and unhelpful way of working.
And in CMake's favor, there's a huge ecosystem of helpers and
integrations that make it easier for people to understand what CMake
is doing as it's being developed, built, and shipped. That makes it
much more attractive than Autotools simply because the knowledge and
the tooling is there for developers and users to dig into it.
> That said, I agree that pretty much every display manager and
> compositor for every Fedora variant should be critpath'd.
Well, where we disagree is that I actually want to see LESS stuff in
critpath, not more. It cannot be scrutinized well enough because there is
just too much stuff in it. E.g., at times, we had MySQL/MariaDB in critpath
because Akonadi required it. (Nowadays, Akonadi actually recommends using
SQLite instead.) That just does not make sense.
I didn't say every single one we package, just the ones shipped in
deliverables. I think it's worth making sure those are on the
critpath.
--
真実はいつも一つ！/ Always, there's only one truth!
1:13 p.m.
Neal Gompa wrote:
And in CMake's favor, there's a huge ecosystem of helpers
and
integrations that make it easier for people to understand what CMake
is doing as it's being developed, built, and shipped. That makes it
much more attractive than Autotools simply because the knowledge and
the tooling is there for developers and users to dig into it.
Well, to be fair, I have also seen (more than once!) arcane stuff being done
in CMake, with almost a whole new build system being built on top of CMake
(tons of custom macros implementing things such as bundled libraries, before
CMake had native support for that), which was not particularly easy to
understand either.
If you use CMake the way it was intended, a CMakeLists.txt is a lot more
readable than even a configure.ac and Makefile.am, let alone the generated
blobs autoconf spits out based on those. But there is potential for abuse,
too.
That said, I do not believe completely banning custom functions and macros
as Meson does is a workable solution.
Kevin Kofler
8:38 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Kevin Kofler wrote:
> This is not helpful in the slightest and the tone is not
appreciated at
> all.
Well, I have been arguing against this exception (exempting prebuilt
autotools output) from the "no prebuilt blobs" rule for years, and it
saddens me that something like this had to happen for Fedora to finally
realize that that exception has always been a bad idea.
[…]
CMIIW, but it would not have made any difference as the
source code had been shipped as part of the tar ball and
auto(re)conf would have happily integrated it into the next
build. I suspect that a modification to CMakeLists.txt and
its includes would not have been detected either; even a
daring, but obvious change in the 30000+ lines of source
itself might have gone unnoticed.
A major factor seems to have been the discrepancy between
"the source code" at GitHub & Co. that was probably
scrutinized by many eyes and the shipped, but different
artifact. So one step (as a inter-distribution effort)
could be to continuously automatically compare shipped
artifacts with their "make dist" equivalents and publishing
the results.
Tim
9:55 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 09:09:35AM -0400, Neal Gompa wrote:
And in CMake's favor, there's a huge ecosystem of helpers
and
integrations that make it easier for people to understand what CMake
is doing as it's being developed, built, and shipped.
That is actually a weakness:
On Sat, Mar 30, 2024 at 01:38:45PM +0000, Tim Landscheidt wrote:
Kevin Kofler wrote:
> Well, I have been arguing against this exception (exempting prebuilt
> autotools output) from the "no prebuilt blobs" rule for years, and it
> saddens me that something like this had to happen for Fedora to finally
> realize that that exception has always been a bad idea.
CMIIW, but it would not have made any difference as the
source code had been shipped as part of the tar ball and
auto(re)conf would have happily integrated it into the next
build. I suspect that a modification to CMakeLists.txt and
its includes would not have been detected either; even a
daring, but obvious change in the 30000+ lines of source
itself might have gone unnoticed.
CMake for many years fought against pkgconf and pushed people towards
copying those scripts into sources. It is still very common for projects
using CMake to come with a whole directory of badly written detection
scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it
easy to smuggle something through there.
Zbyszek
10:02 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew
Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> wrote:
CMake for many years fought against pkgconf and pushed people
towards
copying those scripts into sources. It is still very common for
projects
using CMake to come with a whole directory of badly written detection
scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it
easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to
switch to Meson for whatever reason, then CMake is a reasonable
alternative.
I agree that CMake is not as good as Meson, and that CMake find modules
are inferior to pkg-config. The CMake developers are working on
replacing find modules with CPS [1] which is intended to be a
replacement for pkg-config that will work better on non-Linux
platforms, where pkg-config is not always adequate. It looks like that
work has maybe stalled? but if successful that would fix the problem
with find modules.
[1] https://cps-org.github.io/cps/overview.html
10:36 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew
Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> CMake for many years fought against pkgconf and pushed people towards
> copying those scripts into sources. It is still very common for projects
> using CMake to come with a whole directory of badly written detection
> scripts that each replace a single-line pkgconf invocation.
>
> And of course nobody has time to look into those scripts, making it
> easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to
switch to Meson for whatever reason, then CMake is a reasonable alternative.
I agree that CMake is not as good as Meson, and that CMake find modules are
inferior to pkg-config.
But then we shouldn't describe them as equivalent alternatives ;)
If we say "switch to a modern build systemd, e.g. cmake or meson",
people will randomly choose on or the other and since the whole CMake
ecosystem is built around find modules, we'll end with a bazillion of
those.
Instead we should say: "Use meson. If you can't for some reason, consider
CMake, but come talk to us first."
The CMake developers are working on replacing find
modules with CPS [1] which is intended to be a replacement for pkg-config
that will work better on non-Linux platforms, where pkg-config is not always
adequate. It looks like that work has maybe stalled? but if successful that
would fix the problem with find modules.
[1] https://cps-org.github.io/cps/overview.html
Ack. But from our point of view, this wouldn't be great. We already have
pkgconf files for almost everything and CPS files for nothing…
In fact, we should probably make the effort to add pkgconf files for the
few libraries that don't have it to make it completely standard and
expected.
Zbyszek
10:53 a.m.
On Sat, Mar 30, 2024 at 11:36 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
> On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
> > CMake for many years fought against pkgconf and pushed people towards
> > copying those scripts into sources. It is still very common for projects
> > using CMake to come with a whole directory of badly written detection
> > scripts that each replace a single-line pkgconf invocation.
> >
> > And of course nobody has time to look into those scripts, making it
> > easy to smuggle something through there.
>
> It's still better than Autotools, though. If a project doesn't want to
> switch to Meson for whatever reason, then CMake is a reasonable alternative.
>
> I agree that CMake is not as good as Meson, and that CMake find modules are
> inferior to pkg-config.
But then we shouldn't describe them as equivalent alternatives ;)
If we say "switch to a modern build systemd, e.g. cmake or meson",
people will randomly choose on or the other and since the whole CMake
ecosystem is built around find modules, we'll end with a bazillion of
those.
Instead we should say: "Use meson. If you can't for some reason, consider
CMake, but come talk to us first."
Meson's own module instability and lack of extensibility make it
unsuitable for a wide range of projects, especially complex ones. The
lack of stability in Meson itself is so bad that Meson upgrades break
GNOME, libvirt, and others. And the lack of extensibility is an
anti-feature. It means that Meson cannot scale to the infinite world
of project needs because everything has to be bent around it or hacks
need to be written in the projects to work around its weaknesses.
No way would I personally recommend it. I'm not going to go as far as
to recommend one explicitly over the other from a distribution
perspective, but personally I would never choose Meson anymore.
--
真実はいつも一つ！/ Always, there's only one truth!
11:15 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 11:53:07AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 11:36 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
> > On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek
> > <zbyszek(a)in.waw.pl> wrote:
> > > CMake for many years fought against pkgconf and pushed people towards
> > > copying those scripts into sources. It is still very common for projects
> > > using CMake to come with a whole directory of badly written detection
> > > scripts that each replace a single-line pkgconf invocation.
> > >
> > > And of course nobody has time to look into those scripts, making it
> > > easy to smuggle something through there.
> >
> > It's still better than Autotools, though. If a project doesn't want to
> > switch to Meson for whatever reason, then CMake is a reasonable alternative.
> >
> > I agree that CMake is not as good as Meson, and that CMake find modules are
> > inferior to pkg-config.
>
> But then we shouldn't describe them as equivalent alternatives ;)
> If we say "switch to a modern build systemd, e.g. cmake or meson",
> people will randomly choose on or the other and since the whole CMake
> ecosystem is built around find modules, we'll end with a bazillion of
> those.
>
> Instead we should say: "Use meson. If you can't for some reason, consider
> CMake, but come talk to us first."
>
Meson's own module instability and lack of extensibility make it
unsuitable for a wide range of projects, especially complex ones. The
lack of stability in Meson itself is so bad that Meson upgrades break
GNOME, libvirt, and others. And the lack of extensibility is an
anti-feature. It means that Meson cannot scale to the infinite world
of project needs because everything has to be bent around it or hacks
need to be written in the projects to work around its weaknesses.
No way would I personally recommend it. I'm not going to go as far as
to recommend one explicitly over the other from a distribution
perspective, but personally I would never choose Meson anymore.
Also we haven't found the meson lead developer to be very open to
fixing an obvious bug that affects Fedora/RISC-V, even after providing
and carefully testing a patch for it. It wasn't encouraging.
Rich.
--
真実はいつも一つ！/ Always, there's only one truth!
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
1:28 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
In fact, we should probably make the effort to add pkgconf files for
the
few libraries that don't have it to make it completely standard and
expected.
That is a particularly bad idea. Downstream .pc files are a nuisance. If
someone develops upstream software on Fedora, they will end up relying on
those .pc files, thinking they are a supported way to link that library,
then only after releasing the code, finding out that those .pc files do not
exist upstream or in any other distribution (or worse, other distributions
may have their own, incompatible, downstream .pc file for the same library).
I have already been in that situation as a developer. It just sucked.
If a library does not support pkg-config upstream, you MUST NOT use pkg-
config to find it. It is not portable. So shipping such a downstream .pc
file advertises a false interface to developers.
Kevin Kofler
4:07 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 07:28:43PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
> In fact, we should probably make the effort to add pkgconf files for the
> few libraries that don't have it to make it completely standard and
> expected.
That is a particularly bad idea. Downstream .pc files are a nuisance.
I agree, I never suggested doing this downstream. This kind of thing
needs to happen upstream so that other upstreams can start depending on
it.
A pkgconf file is very simple to add, it's the kind of a thing that
can be provided as a pull request by an external contributor, in particular
a downstream maintainer.
Zbyszek
1:24 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
CMake for many years fought against pkgconf and pushed people
towards
copying those scripts into sources. It is still very common for projects
using CMake to come with a whole directory of badly written detection
scripts that each replace a single-line pkgconf invocation.
Find*.cmake scripts for many common libraries are actually shipped with
CMake itself. More and more libraries even ship their own *Config.cmake
which makes a Find*.cmake script entirely unnecessary.
For less common libraries, you are probably better off simply using the
CMake pkg-config integration, which exists. Assuming of course that that
library ships a .pc file to begin with, otherwise, pkg-config is not going
to help you. (And when I write "pkg-config", I mean either the original pkg-
config or pkgconf, the build system should not need to care about the
difference.)
The reason CMake upstream recommends against using pkg-config directly, and
instead to use it at most as a source of hints for CMake's find_* commands
(e.g., find_library) in a Find*.cmake script, is that using pkg-config on
Windows is often frowned upon. So typically, the way it works is that
someone first writes a Find*.cmake that finds the library in the standard
paths using find_library, then extends it to invoke pkg-config using the
CMake pkg-config integration under "if (UNIX)" and to add the result as a
path hint to the already written find_library call.
And of course nobody has time to look into those scripts, making it
easy to smuggle something through there.
You are right that bundled Find*.cmake scripts are a problem.
Kevin Kofler
10:35 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Tim Landscheidt writes:
A major factor seems to have been the discrepancy between
"the source code" at GitHub & Co. that was probably
scrutinized by many eyes and the shipped, but different
artifact. So one step (as a inter-distribution effort)
could be to continuously automatically compare shipped
artifacts with their "make dist" equivalents and publishing
the results.
There are several versions of autoconf, automake, and libtool in wide use.
There are also many libraries that ship autotools macros for inclusion in
code that builds against those libraries. Those macros don't change very
often, but they do change.
You will need to match the version for everything in the build environment
in order to meaningfully compare autotools-generated shellcode.
11:35 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, 2024-03-30 at 12:53 +0100, Kevin Kofler via devel wrote:
I think the issue is that there is just too much stuff in critpath
these
days. Whole desktop environments and all their transitive dependencies
probably ought to not be in there. If at all, I would put the display
manager in there, maybe the window manager, and no further.
I wrote a mail about this a while ago. The problem is really that the
"critical path" concept has changed somewhat over time, and gotten a
bit overloaded.
The original idea of critical path was to require special testing
attention for it. Back in Ye Earlie Days, critpath packages had all
kinds of special rules around them, including requiring +2 or +3 (it's
a long time ago, I forget) from "proven testers" (remember those?)
*Most* of that has now gone. The only significant of critpath for
manual testing in the current update policy is that critpath packages
have a longer minimum wait in updates-testing (14 days vs. 7 days, at
least after a certain point in the release cycle). They do not have
higher karma requirements (at least, not by policy; Bodhi doesn't
actually implement the policy correctly ATM, but I'm fixing that). The
karma minima defined in the updates policy are currently identical at
all points in the cycle for critpath and non-critpath updates. The
"proven testers" concept was put on ice long ago.
The primary 'meaning' of critpath these days is that it triggers openQA
testing, and critpath updates are gated on openQA testing. I set things
up this way really just because it was convenient, and as is the way of
things, now it's kinda baked in.
We probably want to look at separating out the concepts a bit. It's
certainly technically possible, it just requires some work. The
'releng.py' script that "generates the critical path" is really just a
comps-informed depsolver that spits out JSON. It could generate all
kinds of groups besides "critical path" groups. We'd just have to wire
them up to *mean*...whatever we want them to mean.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
9:57 a.m.
On Sat, Mar 30, 2024 at 9:38 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
I'm not pretending these will solve everything, but they should make
attacks a little harder in future.
Thanks for starting the discussion.
A well resourced supply chain attack is probably
not preventable (no matter how many eyes are
looking). That does not mean we should not try
to minimize the likelihood of future such attacks.
(3) We should have a "security path", like "critical
path".
.....
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
Obligatory xkcd:
https://xkcd.com/2347/
What I do think we should start with is look at the
list of dependencies in the list of whatever we
can agree are security critical packages (running
as root and opening network ports is always a
good start) and dependencies which are not
supported by a large-ish organization (even if
only informal), with a set of experienced
developers, and sufficiently funded to continue
support of the package, and has a good security
reporting and response process in place.
xz would not seem to meet that vague hand
waving criteria, and so it should either be
replaced with something else (if possible) or get
it (or in this case, likely its new team) resourced
to its level of importance in the ecosystem.
I expect, with a critical eye, other such projects
will be identified.
The response to Heartbleed was (among other
things), resourcing OpenSSL. If a decision is
made that xz needs to stay as part of the critical
chain, it needs to be resourced too (although, as
others have suggested, removing xz from that
chain may be a better choice).
1:39 p.m.
Gary Buhrmaster wrote:
What I do think we should start with is look at the
list of dependencies in the list of whatever we
can agree are security critical packages (running
as root and opening network ports is always a
good start) and dependencies which are not
supported by a large-ish organization (even if
only informal), with a set of experienced
developers, and sufficiently funded to continue
support of the package, and has a good security
reporting and response process in place.
What if, as in the case of SELinux, said "large-ish organization" is exactly
the kind of organization one would expect to plant a backdoor like this?
Also, a "large-ish organization" can be secretly contacted by the
intelligence agencies of the country it resides in and tasked to implement
secret backdoors for them. It has happened with large proprietary software
providers, so why could it not happen with a large organization developing
Free Software?
Projects done by a "large-ish organization" are NOT immune to this kind of
attack. It would just be executed differently, not as a hostile takeover by
one "motivated new maintainer" as for an individual hobbyist project like
xz.
Kevin Kofler
10:58 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 30. 03. 24 v 10:37 dop. Richard W.M. Jones napsal(a):
I'm not pretending these will solve everything, but they should
make
attacks a little harder in future.
4) Fetch build artifacts before executing tests
https://github.com/rpm-software-management/mock/issues/1352
(3) We should have a "security path", like "critical
path".
Generally good idea. But several packages that JiaT75 GH-starred were:
* doxygen - when you infect this, you have open path to 700 Fedora packages, including
gcc.
* squashfs-tools - when you infect this, you have open path to all images (just example,
not sure if our toolchain use
this or -ng version).
So the security patch should be much wider.
--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
11:20 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
I like the idea of the security path as well, where all packages in that
path have upstream subject to higher security standards (that means
helping them to achieve it as well), and greater defense downstream in
any way possible.
Two things that came to mind I shared in another channel:
* no binary blobs in the upstream, or no blob referred to in the source
built, or referred to in the build tools
* diffoscope should show no difference except file stats between the
tar.gz being pulled by the spec, and the source brought with a git clone.
Both things could be automated with tools.
On 3/30/24 08:58, Miroslav Suchý wrote:
Dne 30. 03. 24 v 10:37 dop. Richard W.M. Jones napsal(a):
> I'm not pretending these will solve everything, but they should make
> attacks a little harder in future.
4) Fetch build artifacts before executing tests
https://github.com/rpm-software-management/mock/issues/1352
> (3) We should have a "security path", like "critical path".
Generally good idea. But several packages that JiaT75 GH-starred were:
* doxygen - when you infect this, you have open path to 700 Fedora
packages, including gcc.
* squashfs-tools - when you infect this, you have open path to all
images (just example, not sure if our toolchain use this or -ng version).
So the security patch should be much wider.
1:54 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 09:20:28AM -0700, Carlos Rodriguez-Fernandez wrote:
I like the idea of the security path as well, where all packages in
that
path have upstream subject to higher security standards (that means helping
them to achieve it as well), and greater defense downstream in any way
possible.
Two things that came to mind I shared in another channel:
* no binary blobs in the upstream, or no blob referred to in the source
built, or referred to in the build tools
I think there's some useful points here, but this would need to be
qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of
text, mojibake, and actual binary protocol samples. For example, dhcp
input packets, dns packets. There are also other ~binary test files,
for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to
in the build tools", and would not be allowed by the above definition.
But if we dropped those, we'd lose very valuable testing of the codebase.
* diffoscope should show no difference except file stats between the
tar.gz
being pulled by the spec, and the source brought with a git clone.
Here, OTOH, I'd just say that the tarball generated from a git clone
should be bit-identical to the tar.gz pulled in by the spec.
Both things could be automated with tools.
Zbyszek
2 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
I think there's some useful points here, but this would need to
be
qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of
text, mojibake, and actual binary protocol samples. For example, dhcp
input packets, dns packets. There are also other ~binary test files,
for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to
in the build tools", and would not be allowed by the above definition.
But if we dropped those, we'd lose very valuable testing of the codebase.
On the other hand, "test files" are exactly how the payload of this backdoor
was disguised! So a policy that deletes all binary test files or even all
test files altogether would have prevented this backdoor.
Kevin Kofler
4:14 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 08:00:29PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
> I think there's some useful points here, but this would need to be
> qualified and/or made more flexible to be applied.
>
> For example, systemd repo has fuzzer case files, which are a mix of
> text, mojibake, and actual binary protocol samples. For example, dhcp
> input packets, dns packets. There are also other ~binary test files,
> for example corrupted journal files.
>
> The tests are defined via meson.build, so those files are "referred to
> in the build tools", and would not be allowed by the above definition.
> But if we dropped those, we'd lose very valuable testing of the codebase.
On the other hand, "test files" are exactly how the payload of this backdoor
was disguised! So a policy that deletes all binary test files or even all
test files altogether would have prevented this backdoor.
Well, if we made a rule that "binary" files are not allowed, the
attacker would e.g. commit some minified bash that generates whatever output
is needed. The real problem was the complex and unreadable build system
that made it easy to embed _multiple_ obfuscated components for the attack.
To trust code, it needs to be reviewed. And if the code is reviewed,
and the build system is sane, it's usually fairly easy to say "oh,
this is a sample and the only way it's used is as input to a fuzzer
binary", or "this sample is used as input to a unit test", etc.
A policy against binary files would hinder this particular implementation
of the attack, but the attacker had full control of the repo, so they
would be able to arrange the attack around such a policy without too
much trouble.
Zbyszek
4:07 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
hi Zbyszek,
how did you review the corrupted journal files committed in systemd? Can you know for
certain that they do not contain any backdoor or anything illegal or unlicensed?
6:45 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 09:07:21AM -0000, François Rigault wrote:
hi Zbyszek,
how did you review the corrupted journal files committed in systemd? Can you know for
certain that they do not contain any backdoor or anything illegal or unlicensed?
The licensing and legal side is easy: those files are produced by a
program that we wrote (journald), so copyright and patents don't
apply. In principle there could be some privileged information in
those files, but it was disclosed by the person who submitted a pull
request with those files, so at this point distributing this
information wouldn't make further difference. And also the person
submitting them accept the license which allows redistribution.
If there's a backdoor: those files are read by a program which is
supposed to be resilient against broken input. We execute this program
under multiple sanitizers over this input file. So we're doing pretty
strong testing that the input is parsed correctly (or refused).
I wrote a bit more abou this in other part of the thread [1].
I think that while it's theoretically possible to do something
malicious with bad fuzzer samples, it'd be very very do pull off.
[1]
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...,
Zbyszek
3:46 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
To echo
To trust code, it needs to be reviewed.
If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in systemd were not reviewed
neither by co-maintainers nor by downstream package maintainers.
I understand that the build system used by systemd makes it much less probable that some
binary blob used in a test obfuscates something that could be used for other purposes
outside the test; still, wouldn't you agree it would be a good practice to make sure
everyone is able to review everything in the source code repository?
6:11 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 01, 2024 at 08:46:39AM -0000, François Rigault wrote:
To echo
> To trust code, it needs to be reviewed.
> If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in
systemd were not reviewed neither by co-maintainers nor by
downstream package maintainers.
Yes, some of those blobs are treated as opaque.
I understand that the build system used by systemd makes it much
less probable that some binary blob used in a test obfuscates
something that could be used for other purposes outside the test;
still, wouldn't you agree it would be a good practice to make sure
everyone is able to review everything in the source code repository?
It's a trade-off. We can include a useful test case (e.g. a journal
file that causes journalctl to busyloop or crash), to verify that the
issue was fixed and that we don't regress, or we can reject the file
and forego the test.
With a reasonable build system, it's fairly easy to figure out how
the file is used, and I think it's entirely reasonable to review _that_.
OTOH, figuring out what effect that file would have if (hypothetically)
used as input to a different tool or whether it might embed some code
which might be extracted somehow is hard. But I really think that the
risk is low. Also, consider that systemd has 2500 .c and .h files with
875k lines… It's not like you can review that in a weekend.
Zbyszek
7:02 a.m.
On Mon, 1 Apr 2024 at 04:47, François Rigault <rigault.francois(a)gmail.com>
wrote:
To echo
> To trust code, it needs to be reviewed.
> If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in systemd
were not reviewed neither by co-maintainers nor by downstream package
maintainers.
Are we talking about the blobs which contained the malware? Those blobs
were not in systemd, they were in the compression library. You are going to
see weird blobs in any compression library because compression needs to
test how it does against different data types.. especially binary data
because you need to see if you improved or worsened the compression with a
change. The places you are going to see binary data which may not make any
'sense' are: compression, anything graphics related, sound and general
archiving tools. You will find actually 'binaries' in various parts of any
compilation set because you may be linking or 'embedding' it into code that
will run something else.
But that is just where you are just sticking plain binary data. You can
uuencode, base64 or many other things and put it into regular code in
places where you might need to run some stuff. They went after ssh most
likely because the main target was internet based attacks. However they
could have gone a slightly different path and used dnf/apt (rpm/dpkg) as
the target application and placed additional scripts to open systems up
somewhere. This could be done with a nice bit of C code which usually does
one thing but for what would look like legitimate reasons does something
else after a certain date or code sequence. Anyone who has tried to read
through an obfuscated C contest entry can see where this is going.
I understand that the build system used by systemd makes it much
less
probable that some binary blob used in a test obfuscates something that
could be used for other purposes outside the test; still, wouldn't you
agree it would be a good practice to make sure everyone is able to review
everything in the source code repository?
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
9:23 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Those blobs were not in systemd,
that was not my point, nevertheless putting it this way: nobody knows.
For the example about compression methods you could generate your binary using a piece of
code, that can be reviewed (maybe using a fixed seed as inspired by
https://git.rootprojects.org/root/xz/commit/6e636819e8f070330d835fce46289...
btw!). If you want to test systemd against a broken journal then can't you commit a
valid journal (that can be reviewed) and some code that generates a corrupted one?
The obfuscated C code is a different problem - at least it can be reviewed/audited and the
maintainer can ask to simplify it.
My point is that everything should get reviewed before merge. I would hope that, as a
lesson learnt from this attack, no unreviewed "corrupted binary" exist anymore
in any project, since really, nobody knows what they actually contain.
11:17 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 01, 2024 at 02:23:19PM -0000, François Rigault wrote:
> Those blobs were not in systemd,
that was not my point, nevertheless putting it this way: nobody knows.
For the example about compression methods you could generate your binary using a piece of
code, that can be reviewed (maybe using a fixed seed as inspired by
https://git.rootprojects.org/root/xz/commit/6e636819e8f070330d835fce46289...
btw!). If you want to test systemd against a broken journal then can't you commit a
valid journal (that can be reviewed) and some code that generates a corrupted one?
The obfuscated C code is a different problem - at least it can be reviewed/audited and
the maintainer can ask to simplify it.
My point is that everything should get reviewed before merge. I would hope that, as a
lesson learnt from this attack, no unreviewed "corrupted binary" exist anymore
in any project, since really, nobody knows what they actually contain.
Or (if production of the binary is expensive or can be fiddly), commit
the binary but include a script to generate it that can be run by others
to check that the included binary is legit.
Call it "Reproducible Tests" to go along with reproducible builds.
Cryptography has the same concept now, learning from the Dual EC DBRG
backdoor: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number
So "nothing-up-my-sleeve scripts" could be another moniker.
--
Scott Schmit
8:48 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 30. 03. 24 v 22:14 Zbigniew Jędrzejewski-Szmek napsal(a):
On Sat, Mar 30, 2024 at 08:00:29PM +0100, Kevin Kofler via devel
wrote:
> Zbigniew Jędrzejewski-Szmek wrote:
>> I think there's some useful points here, but this would need to be
>> qualified and/or made more flexible to be applied.
>>
>> For example, systemd repo has fuzzer case files, which are a mix of
>> text, mojibake, and actual binary protocol samples. For example, dhcp
>> input packets, dns packets. There are also other ~binary test files,
>> for example corrupted journal files.
>>
>> The tests are defined via meson.build, so those files are "referred to
>> in the build tools", and would not be allowed by the above definition.
>> But if we dropped those, we'd lose very valuable testing of the codebase.
> On the other hand, "test files" are exactly how the payload of this
backdoor
> was disguised! So a policy that deletes all binary test files or even all
> test files altogether would have prevented this backdoor.
Well, if we made a rule that "binary" files are not allowed
I think that we should really not talk about binary files and talk e.g.
about pre-generated files instead. This chapter in guidelines is already
trying to cover this:
https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-pac...
But maybe we should rewrite the "No inclusion of pre-built binaries or
libraries" chapter above to be more generic.
Vít
, the
attacker would e.g. commit some minified bash that generates whatever output
is needed. The real problem was the complex and unreadable build system
that made it easy to embed _multiple_ obfuscated components for the attack.
To trust code, it needs to be reviewed. And if the code is reviewed,
and the build system is sane, it's usually fairly easy to say "oh,
this is a sample and the only way it's used is as input to a fuzzer
binary", or "this sample is used as input to a unit test", etc.
A policy against binary files would hinder this particular implementation
of the attack, but the attacker had full control of the repo, so they
would be able to arrange the attack around such a policy without too
much trouble.
Zbyszek
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
2:01 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> said:
Here, OTOH, I'd just say that the tarball generated from a git
clone
should be bit-identical to the tar.gz pulled in by the spec.
That might change with zlib-ng... expecting compressed data to match is
probably not a reasonable expectation, as there are several that can
generate slightly different output at the same level but with other
different options (such as multiple threads).
I think it's probably more reasonable to say the compressed file should
pass a test by the compressor (e.g. gzip -t) and the uncompressed data
should match.
--
Chris Adams <linux(a)cmadams.net>
4:26 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
It was unreasonable before zlib-ng too [0], but zlib-ng does make it a slightly bigger
issue.
[0] https://github.com/rpm-software-management/createrepo_c/pull/336
2:11 p.m.
Miroslav Suchý wrote:
4) Fetch build artifacts before executing tests
https://github.com/rpm-software-management/mock/issues/1352
Or better: Do not execute tests to begin with! rm -rf test in %prep and
NEVER run tests during builds. Even when the tests are all legitimate, all
it does is slow down the build (e.g., compare glibc build times without and
with tests) and every so often break it because the test, not the software,
is broken. And a claimed "test file" is what allowed the payload to be snuck
in here.
Unit tests are something for upstream developers. They should NEVER be run
in a distribution build.
Kevin Kofler
2:32 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 30.03.24 um 20:11 schrieb Kevin Kofler via devel:
Or better: Do not execute tests to begin with! rm -rf test in %prep
and
NEVER run tests during builds. Even when the tests are all legitimate, all
it does is slow down the build (e.g., compare glibc build times without and
with tests) and every so often break it because the test, not the software,
is broken. And a claimed "test file" is what allowed the payload to be snuck
in here.
Unit tests are something for upstream developers. They should NEVER be run
in a distribution build.
As a developer I need to say: If your build also runs your tests
implicitly even if you just want to make a normal build of your
software, your build setup is broken. That should just not happen.
But "rm -rf test" or something like that may not be possible. Quite some
projects (or just straight up languages) these days have their tests in
the same file as the code they test.
Kilian Hanich
2:51 p.m.
Dear Kevin,
On Sat, Mar 30, 2024 at 8:12 PM Kevin Kofler via devel <
devel(a)lists.fedoraproject.org> wrote:
Miroslav Suchý wrote:
> 4) Fetch build artifacts before executing tests
>
> https://github.com/rpm-software-management/mock/issues/1352
Or better: Do not execute tests to begin with! rm -rf test in %prep and
NEVER run tests during builds. Even when the tests are all legitimate, all
it does is slow down the build (e.g., compare glibc build times without
and
with tests) and every so often break it because the test, not the
software,
is broken. And a claimed "test file" is what allowed the payload to be
snuck
in here.
It's a terrible idea. Sorry.
Unit tests are something for upstream developers. They should NEVER
be run
in a distribution build.
The first thesis is completely wrong. Having, say, a 30+ downstream patches
and declining to run upstream tests is the most effective way to break a
gazillion use-cases.
But the fuzzing tests look quite dangerous to me here and now. No one can
review a corpse of binary files :(
--
Dmitry Belyavskiy
4:27 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 08:51:03PM +0100, Dmitry Belyavskiy wrote:
Dear Kevin,
On Sat, Mar 30, 2024 at 8:12 PM Kevin Kofler via devel <
devel(a)lists.fedoraproject.org> wrote:
> Miroslav Suchý wrote:
> > 4) Fetch build artifacts before executing tests
> >
> > https://github.com/rpm-software-management/mock/issues/1352
>
> Or better: Do not execute tests to begin with! rm -rf test in %prep and
> NEVER run tests during builds. Even when the tests are all legitimate, all
> it does is slow down the build (e.g., compare glibc build times without
> and
> with tests) and every so often break it because the test, not the
> software,
> is broken. And a claimed "test file" is what allowed the payload to be
> snuck
> in here.
>
It's a terrible idea. Sorry.
Yes, it's a terrible idea. Also it doesn't solve anything. Because if
you can inject code into the tarball to inject code into the tests,
then you can also inject code into the output binaries, but you can
also inject it into configuration tests.
Our official policy is to run tests during build, because that helps
to catch many silly mistakes and miscompilations. If we stop doing that,
we'll close one avanue of attack, leaving a few ~equivalent avanues of attack
open which share the characteristic that if one is accessible to the attacker,
most likely all are.
> Unit tests are something for upstream developers. They should
NEVER be run
> in a distribution build.
>
The first thesis is completely wrong. Having, say, a 30+ downstream patches
and declining to run upstream tests is the most effective way to break a
gazillion use-cases.
But the fuzzing tests look quite dangerous to me here and now. No one can
review a corpse of binary files :(
Meh, the primary purpose of such tests is to stress the code under
sanitizers. Those test cases are not added randomly, they are added by
upstream developers when they solve an issue. To construct an attack,
somebody would need to construct a sample that does not trigger any failures
with sanitizers, but actually causes a misbehaviour allowing taking control
of execution. Then this control would need to be used to modify the build
artifacts (because presumably doing something in the sandboxed build
environment is not particularly interesting), and _then_ the attacker
would need to convince the maintainers to commit the sample. This is all
theoretically possible, but pretty hard to pull off.
Again, I think that the important aspect is reviewability, not some
simple rule about the format of files.
Zbyszek
4:30 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Once upon a time, Kevin Kofler <kevin.kofler(a)chello.at> said:
Unit tests are something for upstream developers. They should NEVER
be run
in a distribution build.
Upstream developers aren't testing in every Fedora release (which
whatever the current compiler+library+app combo is), so unit tests
should always be run when available in the Fedora environment.
--
Chris Adams <linux(a)cmadams.net>
4:45 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 04:30:53PM -0500, Chris Adams wrote:
Once upon a time, Kevin Kofler <kevin.kofler(a)chello.at> said:
> Unit tests are something for upstream developers. They should NEVER be run
> in a distribution build.
Upstream developers aren't testing in every Fedora release (which
whatever the current compiler+library+app combo is), so unit tests
should always be run when available in the Fedora environment.
Don't forget architectures. One of the reasons I use to justify getting my
company's open source projects in distributions like Fedora and Debian
is to surface issues when compiling against various compiler+library+app
combo as you noted but as importantly, architecture specific issues.
Preserving the build artifacts so they can't be tampered with during the
test run is probably worthwhile though.
Best regards,
--
_o) Michel Lind (né Salim)
_( ) identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2
4:11 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel wrote:
Unit tests are something for upstream developers. They should NEVER
be run
in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see
unit test failures. The developer ran the tests, but not on S390. Or, with a
different timezone database than current in Fedora. Or etc.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
7:01 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 1, 2024 at 17:11:46 -0400, Matthew Miller via devel
wrote:
On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel wrote:
> Unit tests are something for upstream developers. They should NEVER be run
> in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see
unit test failures. The developer ran the tests, but not on S390. Or, with a
different timezone database than current in Fedora. Or etc.
IMHO, there's no good way to *programmatically* protect ourselves
from a malicious upstream on which we depend. If their goal is to
compromise us, they will work around whatever programmatic/technical
measures we happen to have in place at the time they decide to launch
their attack.
Any potential defense against this sort of thing will have to be
*social*, and/or *process* based. Packagers should get to know (as
best as possible) their upstream maintainers and developers -- by
reaching out over upstream's dev fora, by meeting up at events and
conferences, etc. Packagers should hopefully be familiar with the
human *and* technical situation of upstream, and have a chance to
notice when things go "weird".
Just another $0.02 from the peanut gallery...
Cheers,
--Gabriel
8:47 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Once upon a time, Gabriel Somlo <gsomlo(a)gmail.com> said:
IMHO, there's no good way to *programmatically* protect
ourselves
from a malicious upstream on which we depend. If their goal is to
compromise us, they will work around whatever programmatic/technical
measures we happen to have in place at the time they decide to launch
their attack.
Yeah. This was clearly an attack targeted at Fedora and Debian; trying
to fix the specific point of entry is a losing battle, as at the end of
the day, Fedora will still be taking code from upstreams and
distributing it to systems far and wide. The particular use of test
and autoconf files to try to hide the attack may be novel, but there are
other ways it could have been done. If there's easy and minimal-impact
ways to help (which I haven't really seen suggested), that's good to
look at, but putting lots of effort into how tests are run or wholesale
changes to configuration seem to not be all that useful.
However, it's a good trigger to review Fedora's security approach in
general (like 2FA use).
--
Chris Adams <linux(a)cmadams.net>
11:25 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Chris,
The specific points of entry were evading the strength of open source:
many skilled eyes. Therefore, there is value in programmatically
enforcing that everything used as an input in a build must have been
exposed to *normal opensource workflows*. It is a very simple principle,
yet very powerful: let's flag everything that can evade review:
basically, tar.gz must be equal in content to what it is browseable on
the git repository, and also blobs must be flagged for additional
analysis, because released tar.gz and blobs are not normally reviewed on
a opensource workflow.
Those problems can be codified in a tool (which it is already done in a
proof of concept to show that it is simple to do and possible), or the
approach can be changed all together to something like source-git?
On 4/1/24 18:47, Chris Adams wrote:
Yeah. This was clearly an attack targeted at Fedora and Debian;
trying
to fix the specific point of entry is a losing battle, as at the end of
the day, Fedora will still be taking code from upstreams and
distributing it to systems far and wide. The particular use of test
and autoconf files to try to hide the attack may be novel, but there are
other ways it could have been done. If there's easy and minimal-impact
ways to help (which I haven't really seen suggested), that's good to
look at, but putting lots of effort into how tests are run or wholesale
changes to configuration seem to not be all that useful.
However, it's a good trigger to review Fedora's security approach in
general (like 2FA use).
2 p.m.
Chris Adams wrote:
However, it's a good trigger to review Fedora's security
approach in
general (like 2FA use).
Using such an issue that made it through upstream 2FA and would also have
made it through any 2FA enforcement in Fedora as an excuse to force 2FA on
us is just pure nonsense.
Kevin Kofler
2:53 p.m.
I personally would very much agree with enforcing the use of 2fa on the
Fedora Account System. Maybe take that opportunity to make it a bit more
user friendly? (Such as the fkinit prompt requiring the 2fa code being
added at the end of your password -- to be clear I think the 2fa code
should be separate)
On Tue, Apr 2, 2024 at 3:06 PM Kevin Kofler via devel <
devel(a)lists.fedoraproject.org> wrote:
Chris Adams wrote:
> However, it's a good trigger to review Fedora's security approach in
> general (like 2FA use).
Using such an issue that made it through upstream 2FA and would also have
made it through any 2FA enforcement in Fedora as an excuse to force 2FA on
us is just pure nonsense.
Kevin Kofler
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
3:38 p.m.
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette <farchord(a)gmail.com> wrote:
I personally would very much agree with enforcing the use of 2fa on the Fedora Account
System. Maybe take that opportunity to make it a bit more user friendly? (Such as the
fkinit prompt requiring the 2fa code being added at the end of your password -- to be
clear I think the 2fa code should be separate)
https://pagure.io/fedora-packager/pull-request/179
6:40 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette
<farchord(a)gmail.com> wrote:
>
> I personally would very much agree with enforcing the use of 2fa on the Fedora
Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as
the fkinit prompt requiring the 2fa code being added at the end of your password -- to be
clear I think the 2fa code should be separate)
https://pagure.io/fedora-packager/pull-request/179
I agree that fixing the mismatch in prompts might be nice, but why does
having 2fa seperate make things any better? I mean, it's one more return
you get to hit. ;)
And... I am not sure about moving the handling of passwords to a bash
script from a kinit prompt.
kevin
6:27 a.m.
On Tue, Apr 2, 2024 at 7:41 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
> On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette <farchord(a)gmail.com> wrote:
> >
> > I personally would very much agree with enforcing the use of 2fa on the Fedora
Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as
the fkinit prompt requiring the 2fa code being added at the end of your password -- to be
clear I think the 2fa code should be separate)
>
> https://pagure.io/fedora-packager/pull-request/179
I agree that fixing the mismatch in prompts might be nice, but why does
having 2fa seperate make things any better? I mean, it's one more return
you get to hit. ;)
And... I am not sure about moving the handling of passwords to a bash
script from a kinit prompt.
The kinit is already being run inside a bash script, so if bash is
compromised with a keylogger, you've already lost the game... I'm not
sure how this is worse.
Yeah, it's an extra keystroke, but I think there's value in helping
the user provide the input in the proper format. Right now it's
confusing (particularly since the kinit prompt gives bad information
that we have to warn about).
12:01 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Wed, Apr 03, 2024 at 07:27:12AM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 7:41 PM Kevin Fenzi <kevin(a)scrye.com>
wrote:
>
> On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
> > On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette <farchord(a)gmail.com>
wrote:
> > >
> > > I personally would very much agree with enforcing the use of 2fa on the
Fedora Account System. Maybe take that opportunity to make it a bit more user friendly?
(Such as the fkinit prompt requiring the 2fa code being added at the end of your password
-- to be clear I think the 2fa code should be separate)
> >
> > https://pagure.io/fedora-packager/pull-request/179
>
> I agree that fixing the mismatch in prompts might be nice, but why does
> having 2fa seperate make things any better? I mean, it's one more return
> you get to hit. ;)
>
> And... I am not sure about moving the handling of passwords to a bash
> script from a kinit prompt.
>
The kinit is already being run inside a bash script, so if bash is
compromised with a keylogger, you've already lost the game... I'm not
sure how this is worse.
Well, I meant more that now $PASSWORD has your password where before
kinit was the only thing you input your password into. :)
So, if someone does say a 'sh -x fkinit' to look at something, their
password will show up, but it's probibly fine.
Yeah, it's an extra keystroke, but I think there's value in
helping
the user provide the input in the proper format. Right now it's
confusing (particularly since the kinit prompt gives bad information
that we have to warn about).
Sure.
kevin
11:29 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Matthew Miller,
Unit tests, even though in theory developer should mock dependencies to
isolate their code to the maximum, in reality, it is not that clear cut.
Therefore, those unit tests do serve to some extent as a validation that
their code works with the system libraries and platforms present in the
targeted configuration. I think it is valuable to run them during the
rpm builds, and contribute upstream when they break.
On 4/1/24 14:11, Matthew Miller wrote:
On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel
wrote:
> Unit tests are something for upstream developers. They should NEVER be run
> in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see
unit test failures. The developer ran the tests, but not on S390. Or, with a
different timezone database than current in Fedora. Or etc.
1:59 p.m.
Matthew Miller wrote:
I sometimes see unit test failures. The developer ran the tests, but
not
on S390.
Why would I want a test failure on such an exotic architecture to fail my
build? The only reason Fedora supports that architecture at all is pressure
from IBM. Basically nobody uses it.
Kevin Kofler
5:50 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 08:59:32PM +0200, Kevin Kofler via devel wrote:
Matthew Miller wrote:
> I sometimes see unit test failures. The developer ran the tests, but not
> on S390.
Why would I want a test failure on such an exotic architecture to fail my
build?
The architecture is weird enough (big endian!) that it may show your
code has various incorrect assumptions. We found one the other day
actually:
https://sourceware.org/pipermail/binutils/2024-January/132096.html
Rich.
The only reason Fedora supports that architecture at all is pressure
from IBM. Basically nobody uses it.
Kevin Kofler
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
12:26 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Hi,
It was sheer luck that the exploit was discovered and major distros haven't yet
included it in their stable releases. It's quite possible and plausible it could have
reached RHEL, Debian, Ubuntu, SLES and other distros and it's almost reached Fedora
40.
I don't know how to talk to RedHat/IBM/FSF/Ubuntu and all the big players behind Open
Source/Linux but I want to raise a very important issue.
There's near zero accountability for the tens of thousands of packages included in
Linux distros, often by maintainers who have no resources, qualifications or even know any
programming languages to spot the "bad" code and raise an alarm. Upstream
packages are pushed into Linux distros without considerationand that's it.
That's all completely unacceptable on multiple levels. Security is a joke as a result
of this considering the infamous "Jia Tan" who was almost the sole maintainer of
XZ for over two years.
I propose this issue to be tackled in a centralized way by the collaboration of major
distros.
There must be a website or a central authority which includes known to be
good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever
hashes of the source tarballs. In addition, the source tarballs (not their compressed
versions because people may use different compressors and compression settings) and their
hashes must be digitally signed or have the appropriate PGP signatures from the trusted
parties.
Some parties must be assigned trust to be able to push new packages to this repository.
Each push must be verified by at least two independent parties, let's say RedHat and
Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties
must be people whose whereabouts are known to confirm who they physically are. No
nicknames allowed.
This website must also have/allow a revocation mechanism for situations like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing they are safe to
use.
If that's the wrong place to come up with this proposal, please forward it to the
people who are responsible for making such decisions. I'm not willing to dig through
the dirt to understand how the Fedora project works, who is responsible for what, and what
are the appropriate communication channels. If you care, you'll simply forward my
message. Thanks a lot.
Best regards,
Artem
12:51 p.m.
On Sat, 30 Mar 2024 at 13:26, Artem S. Tashkinov via devel <
devel(a)lists.fedoraproject.org> wrote:
I propose this issue to be tackled in a centralized way by the
collaboration of major distros.
There must be a website or a central authority which includes known to be
good/safe/verified/vetted open source packages along with e.g.
SHA256/384/512/whatever hashes of the source tarballs. In addition, the
source tarballs (not their compressed versions because people may use
different compressors and compression settings) and their hashes must be
digitally signed or have the appropriate PGP signatures from the trusted
parties.
Some parties must be assigned trust to be able to push new packages to
this repository. Each push must be verified by at least two independent
parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't matter.
The representatives of these parties must be people whose whereabouts are
known to confirm who they physically are. No nicknames allowed.
This website must also have/allow a revocation mechanism for situations
like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing
they are safe to use.
If that's the wrong place to come up with this proposal, please forward it
to the people who are responsible for making such decisions. I'm not
willing to dig through the dirt to understand how the Fedora project works,
who is responsible for what, and what are the appropriate communication
channels. If you care, you'll simply forward my message. Thanks a lot.
There is no one who makes such decisions for any of the distros. Most of
the distributions make decisions by consensus of hundreds of individuals
who read a list and come to the conclusion that they are 'going to dig
through the dirt' to make something happen or not. For changes like what
you propose, you need groups of people to work for years to get all the
agreements in place, get the various tooling adapted, and work out all the
personalities involved. It will usually start with an email like this, and
then various disagreements about how it will never work, and then some
group of people to actually try to get something like it to work somewhere.
At which point, the next round of 'well did you think about..' problems
arrive and either the people are able to fix them or the idea gets shelved
until later.
Best regards,
Artem
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
2:28 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
I'm not sure my proposal has been understood at all.
This website/authority is a sort of advisory board where each member's participation
is 100% voluntary and distros are free to **ignore** it altogether.
What this website will contain is just a nice list of vetted open source packages,
versions and their hashes, signed by at least two independent parties (people or orgs,
doesn't matter), that's it. Who's going to populate this website, is up to
people to decide.
This is just fundamentally not how Free Software works.
Fundamentally I don't understand your comment at all. The proposal of mine is not
there to limit anyone's freedom, it's to provide guarantees that certain packages
have been vetted (checked and verified to be malware free), and you are safe to use it.
Actually it's a huge stinking problem for a **ton** of open source users who want to
install certain packages that their distros don't have. It's especially relevant
for Fedora given it's a basically a precursor of RedHat and it cannot contain a ton of
packages related to software patents.
As a result of it, BTW, your users blindly trust RPMFusion. A seemingly absolutely shady
website with no official ties to RedHat, which guarantees neither that the packages it
builds are malware free, nor that there are any actual people responsible for them. If
there are RPMFusion maintainers here, I'm not here to insult you, I'm just
relaying the status quo. RPMFusion does not look legit. I stopped using it over a decade
ago because I simply cannot understand why I should trust it. If RedHat denies anything
patent related, there's zero legal obligations for RedHat if someone starts spreading
malware via it. That sucks.
Back to the topic.
Then you have to painstakingly scour the web for distros already using this package and
check whether their have the same version with a hash. Then you download the package and
verify the hash and pray to God the distro has at least given a cursory look to this
package, so it's actually safe to install.
I guess I'm not coming from @fedora.org or @redhat.com, so my proposal is
"anti-freedom".
Sorry for wasting your time. You have not even provided the very basic counter-arguments
why my proposal makes no sense.
RedHat absolutely can start this initiative. You have all the means and resources, and
I'm not talking about something super complex or expensive. For all I know, it could
be the most basic website running on top of SQLite which costing the company $50 a month
to run.
And of course, without this website, distros will continue to valiantly include upstream
packages and get royally screwed and screw their poor users because a ton of your
maintainers have neither the time/resources, nor qualifications to check whether the code
you happily push to users is malware free.
I guess we'll have to have a few more accidents like this before someone will come up
with a similar solution only not coming from me personally, because I'm a no one and
just rending the air.
Sorry for intervening,
Artem
3:11 p.m.
On Sat, 30 Mar 2024 at 15:29, Artem S. Tashkinov via devel <
devel(a)lists.fedoraproject.org> wrote:
I'm not sure my proposal has been understood at all.
Probably not.. proposals like this need to be thought about, reviewed and
thought about. Some people who like to say NO to various things will of
course voice their NO as soon as they see it.
Back to the topic.
Then you have to painstakingly scour the web for distros already using
this package and check whether their have the same version with a hash.
Then you download the package and verify the hash and pray to God the
distro has at least given a cursory look to this package, so it's actually
safe to install.
I guess I'm not coming from @fedora.org or @redhat.com, so my proposal is
"anti-freedom".
No. You are taking one comment from a person who is known to yell at
everyone and doesn't work for Red Hat as 'being an official comment'. Most
everyone on this list does not speak for Fedora at all. This is the part of
the proposal that I was trying to bring up earlier that needs work:
1. There is NO one who speaks for most distributions. There is a group of
many people with many different opinions who speak for a part of a
distribution. They work together as best they can, but they are all going
to have differing opinions and any proposal has to understand that changing
400 people's minds (for the number of active devs in Fedora) takes time.
2. Red Hat sponsors Fedora but doesn't control what Fedora does. There are
many developers who are paid by Red Hat but only work on Fedora in their
spare time so Red Hat may 'influence' but can not command what they do.
Debian and Arch are even further along the anarchy meter for who gets to
decide what happens where.
3. Getting distros to work together is hard. People choose their distros
like they choose their sports teams. When they see another distro doing
something, their first reaction is to do the opposite. This is why it takes
a long time to get changes done and it takes a lot of people time to make
it happen.
Sorry for wasting your time. You have not even provided the very
basic
counter-arguments why my proposal makes no sense.
It isn't a waste of time, but I see this response with a lot of good
proposals which get any criticism. You are going to get criticism, you are
going to get people yelling about stupid things in any proposal you make.
People don't change their minds quickly and there is a lot of meat-space
circuitry to try and make sure it isn't easy. To get a proposal through,
people need to be able to understand that most of the time the first thing
they are going to get is NO. Some of it is because people have a lot going
on in life and they need space and time to see something for what
its worth. Other times they have seen a lot of empty proposals and are
trying to see if the person making it is going to actually do something
about it or not.
RedHat absolutely can start this initiative. You have all the means
and
resources, and I'm not talking about something super complex or expensive.
For all I know, it could be the most basic website running on top of SQLite
which costing the company $50 a month to run.
And of course, without this website, distros will continue to valiantly
include upstream packages and get royally screwed and screw their poor
users because a ton of your maintainers have neither the time/resources,
nor qualifications to check whether the code you happily push to users is
malware free.
I guess we'll have to have a few more accidents like this before someone
will come up with a similar solution only not coming from me personally,
because I'm a no one and just rending the air.
Sorry for intervening,
Artem
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
2:43 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
This approach
let's delete autoconf-generated cruft from upstream projects and
regenerate it in %prep
To me sounds woefully inappropriate for the task at hand. You remove a single attack
vector while completely overlooking that many of your maintainers don't have the
qualifications to vet the included code even if it's autoconf-cruft free.
AFAIK, the bad code discovered in XZ was on its way to the GIT repo, could have been
included in a slightly different way and if not for the wonderful discovery by the
Microsoft software engineer of all people, we'd be dealing with a whole much more
terrible outcome.
The source code must be vetted. In a perfect world, considering the entire source code is
available, an API calls map for each release could be built/extracted, so that whenever a
new version is published by the upstream, a new map for the new release could easily show
at least all the new API calls that the project now uses to easily discover whether new
features correspond to what the project maintainer published in their changelog. Probably
another "anti-freedom" proposal.
And of course, would be great to employ all the methods of automated software
verification, like static analysis, sandboxing, fuzzying, etc. I'm just dreaming.
Let's pretend this incident is entirely about autoconf stuff, and not about a software
project being hijacked by hostile actors. And of course, you're absolutely sure this
hasn't already happened in other widely used projects and will never happen again.
Again, sorry for intervening. I just freaked out when I realized my Linux box might have
been compromised after almost believing in the persistent myth of "multiple
eyes" (five?) scanning open source software for malware. It's not been the case,
the entire proposal that we are discussing here is not talking about it either. I'm
confused.
Regards,
Artem
3:05 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Hi Artem,
I disagree that the idea is not appropriate. Ensuring that the tar.gz
you are getting is exactly what it is in the git repo reduces a lot of
risk. So, it makes a lot of sense to get rid of the practice of
distributing tar.gz with pregenerated build scripts not present in the
git repo that has not been reviewed, without an audit trail implicit in
the git history, and without a clear non-repudiateable authorship.
Ignoring them and regenerating them is a step forward.
I bet the JT user would not have been able to put the attack in the git
source code without Lasse seeing it, or other developers, or tools
eventually seeing it, sooner than if it had been obfuscated in the
"release" downloads (as it was).
Regarding the multiple eyes, that is not the goal, but the goal is
multiple *skilled* eyes.
At the end of the day, this is not about building a bullet-proof system
since that's unrealistic, but a system that makes it very hard by
setting up tools and processes that mitigate the risk significantly.
Regards,
Carlos.
On 3/30/24 12:43, Artem S. Tashkinov via devel wrote:
And of course, would be great to employ all the methods
1:49 p.m.
Artem S. Tashkinov via devel wrote:
There must be a website or a central authority which includes known
to be
good/safe/verified/vetted open source packages along with e.g.
SHA256/384/512/whatever hashes of the source tarballs. In addition, the
source tarballs (not their compressed versions because people may use
different compressors and compression settings) and their hashes must be
digitally signed or have the appropriate PGP signatures from the trusted
parties.
Some parties must be assigned trust to be able to push new packages to
this repository. Each push must be verified by at least two independent
parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't
matter. The representatives of these parties must be people whose
whereabouts are known to confirm who they physically are. No nicknames
allowed.
This is just fundamentally not how Free Software works.
Kevin Kofler
4:07 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
It's not how free software works, but there are some interesting projects working on
(distributed, not centrally managed) code review systems that are kind of similar in
spirit to what OP describes.
https://github.com/crev-dev/crev
https://github.com/crev-dev/cargo-crev
https://mozilla.github.io/cargo-vet/
That is, individuals and organizations can publish the results of their code audits
publicly in a standardized format, tied to a package artifact, and a downstream consumer
could denote which individuals and organizations they trust to perform said audits.
It's technically possible and not an entirely ridiculous idea, it's just
economically challenging. How do you find enough people willing and able to audit a
package (including e.g. autotools build scripts), in multiple, to the level of scrutiny
that would be required to catch something like this.
5:38 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 09:07:19PM -0000, Daniel Alley wrote:
It's not how free software works, but there are some interesting
projects working on (distributed, not centrally managed) code review systems that are kind
of similar in spirit to what OP describes.
https://github.com/crev-dev/crev
https://github.com/crev-dev/cargo-crev
https://mozilla.github.io/cargo-vet/
That is, individuals and organizations can publish the results of their code audits
publicly in a standardized format, tied to a package artifact, and a downstream consumer
could denote which individuals and organizations they trust to perform said audits.
It's technically possible and not an entirely ridiculous idea, it's just
economically challenging. How do you find enough people willing and able to audit a
package (including e.g. autotools build scripts), in multiple, to the level of scrutiny
that would be required to catch something like this.
I think such cross-checks already happen in practice. When distro
packages are updated, maintainers will do _some_ review. We can't
force them to full reviews every time, and it wouldn't even make
sense, but we can be certain that if anything malicious is found,
notifications too the other distros will go out immediately.
Or to put in a different way: if distribution publishes a package,
we implicitly know that they did the review of that package in that
version to the level they consider acceptable. We if asked them to
make an additional signature, it wouldn't add much.
As a maintainer, the level of review I do for updates varies a lot:
for some upstreams I'll do a patch-by-patch review and maybe use
diffoscope to see what changed in the internals, for some upstreams
I'll scan the NEWS file, and for other upstreams I'll not even do
that, if I know what is hapenning in the project and I trust the
quality and integrity. If there was a "task force" auditing packages,
I think it should apply some variant of that rule: focus on packages
which are a) important, b) have few maintainers, c) have lots of
ugly internals, d) raise suspicion for whatever other reason.
This would probably yield better results then trying to apply the
same level of scrutiny accross the board.
--
That said, one useful variant of the auditing proposal would be to
check that different distributions actually have the same source tarballs,
for the same versions of the same packages. This is something that
could scripted and occasionally executed. It'd be very suspicious
if it turned out that the "upstream tarball" differs in Fedora or Arch
or Debian or Suse or any other distro.
Zbyszek
8:53 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 30. 03. 24 v 18:26 Artem S. Tashkinov via devel napsal(a):
Hi,
It was sheer luck that the exploit was discovered and major distros haven't yet
included it in their stable releases. It's quite possible and plausible it could have
reached RHEL, Debian, Ubuntu, SLES and other distros and it's almost reached Fedora
40.
I don't know how to talk to RedHat/IBM/FSF/Ubuntu and all the big players behind Open
Source/Linux but I want to raise a very important issue.
There's near zero accountability for the tens of thousands of packages included in
Linux distros, often by maintainers who have no resources, qualifications or even know any
programming languages to spot the "bad" code and raise an alarm. Upstream
packages are pushed into Linux distros without considerationand that's it.
That's all completely unacceptable on multiple levels. Security is a joke as a result
of this considering the infamous "Jia Tan" who was almost the sole maintainer of
XZ for over two years.
I propose this issue to be tackled in a centralized way by the collaboration of major
distros.
If I was JT, I would applaud this proposal. This would give me an
opportunity to infiltrate such powerful body and either
1) close my eyes above some of the reviewed content from the right
parties or
2) have some nice proposal such as "do you think your code is correct
and won't you include rather this specially crafted piece of code?"
But since I am not JT, I prefer the current decentralized approach.
Vít
There must be a website or a central authority which includes known to be
good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever
hashes of the source tarballs. In addition, the source tarballs (not their compressed
versions because people may use different compressors and compression settings) and their
hashes must be digitally signed or have the appropriate PGP signatures from the trusted
parties.
Some parties must be assigned trust to be able to push new packages to this repository.
Each push must be verified by at least two independent parties, let's say RedHat and
Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties
must be people whose whereabouts are known to confirm who they physically are. No
nicknames allowed.
This website must also have/allow a revocation mechanism for situations like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing they are safe
to use.
If that's the wrong place to come up with this proposal, please forward it to the
people who are responsible for making such decisions. I'm not willing to dig through
the dirt to understand how the Fedora project works, who is responsible for what, and what
are the appropriate communication channels. If you care, you'll simply forward my
message. Thanks a lot.
Best regards,
Artem
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
3:18 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-03-30 02:37, Richard W.M. Jones wrote:
(3) We should have a "security path", like "critical
path".
sshd is linked to a lot of libraries:
I really don't want to start a systemd thread, but... the xz, lz4, and
zstd libraries are pulled in by libsystemd, merely so that sshd can call
"sd_notify"
(https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...),
which raises a couple of questions.
The first one that comes to mind is: Is the increased attack surface
incurred by linking to these additional libraries worth the value
provided by calling "sd_notify", or should that patch be dropped to
improve sshd security?
The second is: Is libsystemd too large? I could very easily be
misreading it, but it looks like at least some of
src/libsystemd/sd-journal is used by journald, including the compression
bits. Do those really belong in libsystemd? If they need to be shared
components, could the journald components be split out to reduce the
size of libsystemd? (That is, to avoid linking to the compression libs?)
Moving on to a broader topic:
The write up describing the back door indicates that the malicious xz
library "changes the value of RSA_public_decrypt@....plt to point to its
own code." So the back door has pointed one of the symbols that should
point to a page mapped to OpenSSL's libcrypto.so.3 to a page mapped to
liblzma.so.5, instead.
Would it be possible to audit the value of a process's symbols at
runtime to look for this kind of shenanigans? Could this type of
auditing be added to functional tests or rpminspect?
1:59 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-03-30 13:18, Gordon Messmer wrote:
The write up describing the back door indicates that the malicious xz
library "changes the value of RSA_public_decrypt@....plt to point to
its own code." So the back door has pointed one of the symbols that
should point to a page mapped to OpenSSL's libcrypto.so.3 to a page
mapped to liblzma.so.5, instead.
Would it be possible to audit the value of a process's symbols at
runtime to look for this kind of shenanigans? Could this type of
auditing be added to functional tests or rpminspect?
As a proof of concept, I extended GEF a tiny bit:
https://github.com/gordonmessmer/gef
Now gdb can print the GOT with the paths providing the memory section
containing a function. For example, on a Debian 12 system with liblzma 5.6:
---
got RSA
GOT protection: Full RelRO | GOT functions: 463
[0x555957c780f8] RSA_set0_key(a)OPENSSL_3.0.0 → 0x7f7bce676c90 :
/usr/lib/x86_64-linux-gnu/libcrypto.so.3
[0x555957c78218] RSA_public_decrypt(a)OPENSSL_3.0.0 → 0x7f7bce948510 :
/usr/lib/x86_64-linux-gnu/liblzma.so.5
[0x555957c782a8] EVP_PKEY_set1_RSA(a)OPENSSL_3.0.0 → 0x7f7bce618f30 :
/usr/lib/x86_64-linux-gnu/libcrypto.so.3
...
---
If the full table were collected and logged in functional testing, and
compared from build to build, this seems like it could detect this class
of attack. RSA_public_decrypt has clearly moved from libcrypto.so.3 to
liblzma.so.5.
Is this worth pursuing?
2:45 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-04-01 23:59, Gordon Messmer wrote:
Now gdb can print the GOT with the paths providing the memory section
containing a function. For example, on a Debian 12 system with
liblzma 5.6:
Purely as trivia, and as I haven't seen it discussed elsewhere, the
malware steals a different set of symbols on Fedora, where
RSA_public_decrypt doesn't seem to appear in the GOT at all. On Fedora 40:
gef➤ got RSA
GOT protection: Full RelRO | GOT functions: 503
[0x556ac0b94ff8] RSA_set0_key(a)OPENSSL_3.0.0 → 0x7f4e95dafce0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b951c0] RSA_bits(a)OPENSSL_3.0.0 → 0x7f4e95daf0a0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b951e0] EVP_PKEY_set1_RSA(a)OPENSSL_3.0.0 → 0x7f4e960e23b0 :
/usr/lib64/liblzma.so.5.6.1
[0x556ac0b95310] RSA_set0_crt_params(a)OPENSSL_3.0.0 → 0x7f4e95dafea0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b953c8] RSA_size(a)OPENSSL_3.0.0 → 0x7f4e95daf0b0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95518] RSA_new(a)OPENSSL_3.0.0 → 0x7f4e95db3330 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95778] RSA_get0_crt_params(a)OPENSSL_3.0.0 → 0x7f4e95dae490 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95870] RSA_free(a)OPENSSL_3.0.0 → 0x7f4e95db2f00 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95b90] RSA_get0_key(a)OPENSSL_3.0.0 → 0x7f4e960e1ac0 :
/usr/lib64/liblzma.so.5.6.1
[0x556ac0b95c00] RSA_get0_factors(a)OPENSSL_3.0.0 → 0x7f4e95dae470 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95c88] EVP_PKEY_get1_RSA(a)OPENSSL_3.0.0 → 0x7f4e95d59710 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95da0] RSA_get_ex_data(a)OPENSSL_3.0.0 → 0x7f4e95db3440 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95e50] RSA_set0_factors(a)OPENSSL_3.0.0 → 0x7f4e95dafdc0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95f00] RSA_blinding_on(a)OPENSSL_3.0.0 → 0x7f4e95db17f0 :
/usr/lib64/libcrypto.so.3.2.1
10:32 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 12:45:18AM -0700, Gordon Messmer wrote:
On 2024-04-01 23:59, Gordon Messmer wrote:
>Now gdb can print the GOT with the paths providing the memory
>section containing a function. For example, on a Debian 12 system
>with liblzma 5.6:
Purely as trivia, and as I haven't seen it discussed elsewhere, the
malware steals a different set of symbols on Fedora, where
RSA_public_decrypt doesn't seem to appear in the GOT at all. On
Fedora 40:
gef➤ got RSA
GOT protection: Full RelRO | GOT functions: 503
[0x556ac0b94ff8] RSA_set0_key(a)OPENSSL_3.0.0 → 0x7f4e95dafce0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b951c0] RSA_bits(a)OPENSSL_3.0.0 → 0x7f4e95daf0a0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b951e0] EVP_PKEY_set1_RSA(a)OPENSSL_3.0.0 → 0x7f4e960e23b0 :
/usr/lib64/liblzma.so.5.6.1
[0x556ac0b95310] RSA_set0_crt_params(a)OPENSSL_3.0.0 → 0x7f4e95dafea0
: /usr/lib64/libcrypto.so.3.2.1
[0x556ac0b953c8] RSA_size(a)OPENSSL_3.0.0 → 0x7f4e95daf0b0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95518] RSA_new(a)OPENSSL_3.0.0 → 0x7f4e95db3330 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95778] RSA_get0_crt_params(a)OPENSSL_3.0.0 → 0x7f4e95dae490
: /usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95870] RSA_free(a)OPENSSL_3.0.0 → 0x7f4e95db2f00 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95b90] RSA_get0_key(a)OPENSSL_3.0.0 → 0x7f4e960e1ac0 :
/usr/lib64/liblzma.so.5.6.1
[0x556ac0b95c00] RSA_get0_factors(a)OPENSSL_3.0.0 → 0x7f4e95dae470 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95c88] EVP_PKEY_get1_RSA(a)OPENSSL_3.0.0 → 0x7f4e95d59710 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95da0] RSA_get_ex_data(a)OPENSSL_3.0.0 → 0x7f4e95db3440 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95e50] RSA_set0_factors(a)OPENSSL_3.0.0 → 0x7f4e95dafdc0 :
/usr/lib64/libcrypto.so.3.2.1
[0x556ac0b95f00] RSA_blinding_on(a)OPENSSL_3.0.0 → 0x7f4e95db17f0 :
/usr/lib64/libcrypto.so.3.2.1
Since no one else replied yet, this is a nice bit of analysis.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
8:10 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 04:32:24PM +0100, Richard W.M. Jones wrote:
On Tue, Apr 02, 2024 at 12:45:18AM -0700, Gordon Messmer wrote:
> On 2024-04-01 23:59, Gordon Messmer wrote:
> >Now gdb can print the GOT with the paths providing the memory
> >section containing a function. For example, on a Debian 12 system
> >with liblzma 5.6:
Since no one else replied yet, this is a nice bit of analysis.
+1. I put the tool on my TODO list of things to look into.
Zbyszek
4:46 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-04-04 06:10, Zbigniew Jędrzejewski-Szmek wrote:
+1. I put the tool on my TODO list of things to look into.
When you get that time, I've also opened the following PR that includes
a proof-of-concept test
https://src.fedoraproject.org/rpms/openssh/pull-request/73
It's sloppy at the moment. The GEF extension should probably be
packaged and not bundled in the test. If this is an interesting
approach it should probably be a shared test instead, and used in
various other packages that commonly listed on network ports. But I
think this at least illustrates what I'm suggesting.
4:14 p.m.
Gordon Messmer wrote:
Purely as trivia, and as I haven't seen it discussed elsewhere,
the
malware steals a different set of symbols on Fedora, where
RSA_public_decrypt doesn't seem to appear in the GOT at all.
This proves again that this is a very targeted attack that carefully
analyzed the individual targeted distributions, the distributions whose
packaging tools the build script attempts to detect were not just picked
because they are known to link OpenSSH to liblzma, but also individually
tested and targeted.
Kevin Kofler
1 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-04-01 23:59, Gordon Messmer wrote:
On 2024-03-30 13:18, Gordon Messmer wrote:
> The write up describing the back door indicates that the malicious xz
> library "changes the value of RSA_public_decrypt@....plt to point to
> its own code." So the back door has pointed one of the symbols that
> should point to a page mapped to OpenSSL's libcrypto.so.3 to a page
> mapped to liblzma.so.5, instead.
>
> Would it be possible to audit the value of a process's symbols at
> runtime to look for this kind of shenanigans? Could this type of
> auditing be added to functional tests or rpminspect?
As a proof of concept, I extended GEF a tiny bit:
https://github.com/gordonmessmer/gef
I spent a little more time extending GEF further, as a new "got-audit"
command. The command will report an error if two or more libraries
appear to export conflicting symbols. It will also report an error if a
symbol in the GOT points to a shared object that doesn't appear to
export that symbol. For all symbols in the GOT, it reports a mapping
between the symbol and the path where that symbol is mapped.
I'll work on a functional test for the openssh package. I think the
naive approach is to simply record the known-good output of the audit in
a file in the test's directory, run the "got-audit" command, and compare
the two files. Any difference is an error.
I haven't started on that yet, but the "port-forward" test seems fairly
small and simple, so I'll try writing something similar, unless anyone
has suggestions otherwise.
5:42 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sa, 30.03.24 13:18, Gordon Messmer (gordon.messmer(a)gmail.com) wrote:
On 2024-03-30 02:37, Richard W.M. Jones wrote:
> (3) We should have a "security path", like "critical path".
>
> sshd is linked to a lot of libraries:
I really don't want to start a systemd thread, but... the xz, lz4, and zstd
libraries are pulled in by libsystemd, merely so that sshd can call
"sd_notify"
(https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-s...),
which raises a couple of questions.
The first one that comes to mind is: Is the increased attack surface
incurred by linking to these additional libraries worth the value provided
by calling "sd_notify", or should that patch be dropped to improve sshd
security?
As discussed elsewhere: these deps are now dlopen() in systemd git
main. So they are basically gone, unless you actually use one of the
API calls that need the code.
The second is: Is libsystemd too large? I could very easily be
misreading
it, but it looks like at least some of src/libsystemd/sd-journal is used by
journald, including the compression bits. Do those really belong in
libsystemd?
There are various programs that use sd-journal to read journal
files. It's after all *the* place to you get your logs from these
days. And journal files can be compressed. We only use one compressor
when writing, but we have changed the compression library
(i.e. nowadays use zstd), and need to provide read
compatibility. Hence the other compression libs.
If they need to be shared components, could the journald
components be split out to reduce the size of libsystemd? (That is, to
avoid linking to the compression libs?)
This creates many other problems. For an explanation see the various
comments here: https://github.com/systemd/systemd/issues/32028
Also, I don't think we should get hung up too much on the libsystemd
thing. I know people like to hit on systemd, but the attacker could
have used various other vehicle libs for their goal, too. I mean, sshd
uses PAM, and that pull in variety of things through its modules, and
that's just very hard to properly review even when one just focusses
on the default PAM config on Fedora.
Lennart
--
Lennart Poettering, Berlin
2:18 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 2024-04-02 03:42, Lennart Poettering wrote:
Also, I don't think we should get hung up too much on the
libsystemd
thing. I know people like to hit on systemd,
I know, and one of the problems that results from having just a torrent
of undeserved criticism is that it naturally predisposes the maintainers
to reflexively disregard all criticism. To be clear: I *like* systemd.
One of the things I like about it is that while the project is fairly
large, the individual functions/services are modular and mostly
independent of each other. But that isn't really true of libsystemd. I
understand that there are some dependencies between different components
of libsystemd, but sd-daemon doesn't seem to have any dependencies on
sd-journal, for example.
but the attacker could
have used various other vehicle libs for their goal, too. I mean, sshd
uses PAM, and that pull in variety of things through its modules
PAM modules aren't a problem for the same reason that the compression
libraries aren't a problem now that they're dlopen()ed.
3:58 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should
make
attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some
of the grandiose ideas in later posts than Richard's list...I think
we're in danger of building castles in the sky while not cleaning up
the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world
off autotools or banning binaries in repos or centralized source depots
authenticated by a committee of Top People, can we remember:
1. We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
2. Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the
packager FAS group. Any one of those people - or someone who
compromises any one of those 140 accounts - can grant any other person
on earth Fedora packager status. Our policy on how they should do this
is
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
. The words "trust" and "identity" do not appear in it. There is,
AFAIK, no policy or procedure by which inactive sponsors have this
power removed. There is no mandatory 2FA policy for sponsors.
3. We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder,
*we are a project that allows 1,601 minimally-vetted people to deliver
arbitrary code executed as root on hundreds of thousands of systems*,
and this mechanism allows any one of those people to cause the package
they have complete control over to be automatically pulled in as a
dependency on virtually every single one of those systems.
4. Our main auth system was written years ago by someone who no longer
contributes and nobody is really actively maintaining it[0].
These are just the ones that leap to my tired mind at this moment. I'm
sure we can think of many more things we should probably look at before
we start pontificating (or, worse, lecturing) about how things should
be done upstream of us.
[0]: https://pagure.io/ipsilon/commits/master
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
4:34 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 31. 03. 24 v 10:58 dop. Adam Williamson napsal(a):
1. Westill don't have compulsory 2FA for Fedora packagers. We
*still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
Fair enough.
Let's do something about it: https://pagure.io/fesco/issue/3186
--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
5:29 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 31/03/2024 10:58, Adam Williamson wrote:
1. We *still don't have compulsory 2FA for Fedora packagers*. We
*still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
This finally motivated me to enable 2fa for my Fedora acount...
An important note, the docs [0] should probably mention `fkinit -u
<username>`. It's much easier than using kinit.
[0]: https://docs.fedoraproject.org/en-US/fedora-accounts/user/#pkinit
--
Arthur Bols
fas/irc: principis
6:03 a.m.
Adam Williamson wrote:
1. We *still don't have compulsory 2FA for Fedora packagers*. We
*still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for
contributors for a while, starting with "important" projects, then getting
stricter and stricter. It has done absolutely nothing to stop this attack.
How could it, when the backdoor was apparently introduced by the authorized
maintainer? (Or if not, the attacker must have had access to their 2FA
secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
US! And especially DO NOT abuse this incident as an excuse to force 2FA down
our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2. Our process for vetting packagers is, let's face it, from a
security
perspective almost *comically* patchy. There are 140 sponsors in the
packager FAS group. Any one of those people - or someone who
compromises any one of those 140 accounts - can grant any other person
on earth Fedora packager status. Our policy on how they should do this
is
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
. The words "trust" and "identity" do not appear in it. There is,
AFAIK, no policy or procedure by which inactive sponsors have this
power removed. There is no mandatory 2FA policy for sponsors.
We already have a manpower problem, how is removing sponsors going to
improve the situation?
3. We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder,
*we are a project that allows 1,601 minimally-vetted people to deliver
arbitrary code executed as root on hundreds of thousands of systems*,
and this mechanism allows any one of those people to cause the package
they have complete control over to be automatically pulled in as a
dependency on virtually every single one of those systems.
This would get noticed pretty quickly, when that package comes up in update
transactions for no reason. I believe this has never happened so far. It is
just too obvious.
Kevin Kofler
6:35 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA
for
contributors for a while, starting with "important" projects, then getting
stricter and stricter. It has done absolutely nothing to stop this attack.
How could it, when the backdoor was apparently introduced by the authorized
maintainer? (Or if not, the attacker must have had access to their 2FA
secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
US! And especially DO NOT abuse this incident as an excuse to force 2FA down
our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve *this* issue, but that wasn't
Adam's point. What Adam is saying is that we're in danger of focusing
too much on a specific issue while we should spent our time and energy
on the general security aspect of Fedora. 2FA isn't nonsense, it
strengthens security by a lot. A compromised (proven)packager account
can do a lot of harm and can take a while to be noticed. If this would
happen to us, Fedora's reputation would tank immediately. Mint is still
regarded as a insecure distro (in my circles) for things that happened
before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they
used to be. Yelling about it isn't going to solve anything. Meanwhile,
enabling 2FA helps A LOT even if used incorrectly (e.g. storing it in
the same keepassxc database).
--
Arthur Bols
fas/irc: principis
6:42 a.m.
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols <arthur(a)bols.dev> wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for
contributors for a while, starting with "important" projects, then getting
stricter and stricter. It has done absolutely nothing to stop this attack.
How could it, when the backdoor was apparently introduced by the authorized
maintainer? (Or if not, the attacker must have had access to their 2FA
secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
US! And especially DO NOT abuse this incident as an excuse to force 2FA down
our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve this issue, but that wasn't Adam's
point. What Adam is saying is that we're in danger of focusing too much on a specific
issue while we should spent our time and energy on the general security aspect of Fedora.
2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager
account can do a lot of harm and can take a while to be noticed. If this would happen to
us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure
distro (in my circles) for things that happened before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they used to be. Yelling
about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT even if
used incorrectly (e.g. storing it in the same keepassxc database).
At this point, I'm used to MFA for stuff (and I use a password manager
that handles 2FA OTPs too), but the Fedora implementation of MFA is
uniquely bad because we have to do a lot in the terminal, and our MFA
implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks
2. You need to concatenate password and OTP for getting a krb5 session ticket
3. The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently
unpleasant that I really don't want to use it.
--
真実はいつも一つ！/ Always, there's only one truth!
7:02 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 31/03/2024 13:42, Neal Gompa wrote:
At this point, I'm used to MFA for stuff (and I use a password
manager
that handles 2FA OTPs too), but the Fedora implementation of MFA is
uniquely bad because we have to do a lot in the terminal, and our MFA
implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks
2. You need to concatenate password and OTP for getting a krb5 session ticket
3. The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently
unpleasant that I really don't want to use it.
Thank you, these are valid
points that I was mostly unaware of.
1 and 2 (I use a mooltipass, so it's just one extra click) aren't a
problem for me and hopefully I'll never need the 3rd, but I can see how
this could have a big impact on other users.
--
Arthur Bols
fas/irc: principis
10:56 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 07:42:24AM -0400, Neal Gompa wrote:
At this point, I'm used to MFA for stuff (and I use a password manager
that handles 2FA OTPs too), but the Fedora implementation of MFA is
uniquely bad because we have to do a lot in the terminal, and our MFA
implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks
To clarify, goa cannot get a new token for you, but once you have gotten
one with your otp, it will renew it for you until it's renewal time is
over.
2. You need to concatenate password and OTP for getting a krb5
session ticket
Yep. I think this is being worked on...
3. The recovery mechanism involves GPG signed emails
Yep. Or... you can enroll multiple otps. You only need one to be
working. You can enroll more and keep backup ones.
The experience using 2FA for Fedora accounts is sufficiently
unpleasant that I really don't want to use it.
:(
kevin
11:17 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 07:42 -0400, Neal Gompa wrote:
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols <arthur(a)bols.dev>
wrote:
>
> On 31/03/2024 13:03, Kevin Kofler via devel wrote:
>
> This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for
> contributors for a while, starting with "important" projects, then
getting
> stricter and stricter. It has done absolutely nothing to stop this attack.
> How could it, when the backdoor was apparently introduced by the authorized
> maintainer? (Or if not, the attacker must have had access to their 2FA
> secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
> US! And especially DO NOT abuse this incident as an excuse to force 2FA down
> our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
> repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
>
>
> 2FA for Fedora packagers doesn't solve this issue, but that wasn't
Adam's point. What Adam is saying is that we're in danger of focusing too much on
a specific issue while we should spent our time and energy on the general security aspect
of Fedora. 2FA isn't nonsense, it strengthens security by a lot. A compromised
(proven)packager account can do a lot of harm and can take a while to be noticed. If this
would happen to us, Fedora's reputation would tank immediately. Mint is still regarded
as a insecure distro (in my circles) for things that happened before I even entered the
linux scene...
>
> Like it or not, this is 2024 and passwords are not as secure as they used to be.
Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT
even if used incorrectly (e.g. storing it in the same keepassxc database).
>
At this point, I'm used to MFA for stuff (and I use a password manager
that handles 2FA OTPs too), but the Fedora implementation of MFA is
uniquely bad because we have to do a lot in the terminal, and our MFA
implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks
2. You need to concatenate password and OTP for getting a krb5 session ticket
3. The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently
unpleasant that I really don't want to use it.
Copying this comment here from the FESCo ticket at Kevin's request -
please follow up here, not there:
I think the points above and others made later about areas where the
2FA experience on Fedora are bad are absolutely correct and justified.
I also think there is a solid argument that we should do this anyway.
I think the vulnerability associated with having packager (and
provenpackager? Do we require 2FA for provenpackager yet? I wasn't
actually sure, when I wrote my list post) accounts without 2FA is
sufficiently horrendous that we just cannot accept it. Yes, our
implementation of 2FA is suboptimal and would frustrate people. Is that
worse than the consequences of letting ourselves be compromised?
Imagine how brutal the response would be if it came to light that
Fedora had been compromised through exploitation of single-factor
authentication. It would not be kind. It would be a huge and lasting
stain on our reputation. People would say, justifiably so, that it was
absolutely unacceptable for us to be allowing single-factor
authentication for contributors to a general-purpose operating system
in 2024. It is.
We now have incontrovertible evidence that extremely sophisticated
attackers are willing to mount extremely sophisticated attacks on the
supply chain of which we are a significant component. We are busy
Monday morning quarterbacking about extremely complex ways to try and
counteract an attack of that sophistication. Meanwhile we are still
leaving this huge vulnerability to much less sophisticated attacks,
which has been known to be one for literal decades at this point, open.
We know that much less sophisticated attackers exploit single-factor
authentication for purposes as trivial as stealing cryptocurrency, and
it happens frequently. I don't think we can pretend we can't connect
the dots here.
On a practical level, if we just made 2FA compulsory, it would provide
people with a much stronger motivation to contribute and make the
experience of it better. So long as we let people not use it, that
motivation does not exist. I suspect if we just did it, we'd get a more
sophisticated experience with Kerberos and recovery tokens and all the
rest of it much faster than we will if we don't.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:31 p.m.
Adam Williamson wrote:
Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to
be).
People would say, justifiably so, that it was absolutely unacceptable
for
us to be allowing single-factor authentication for contributors to a
general-purpose operating system in 2024. It is.
This is nonsense propaganda. Most 2FA implementations cannot even guarantee
that the second factor is not stored right next to the first factor. Open
standards that do not depend on commercial hardware or telecommunication
operators, such as TOTP, cannot guarantee it by design. Any 2FA app that
works on my PinePhone is also going to work directly on my computer, so you
have no way to enforce that I use a different device for the second factor.
2FA is pointless security theater that just makes it a pain to contribute,
when we are all this time talking about lowering, not rising, the barrier to
entry.
Kevin Kofler
2:19 p.m.
Hi Kevin,
On Sun, Mar 31, 2024, at 7:31 PM, Kevin Kofler via devel wrote:
Adam Williamson wrote:
> Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to
be).
> People would say, justifiably so, that it was absolutely unacceptable for
> us to be allowing single-factor authentication for contributors to a
> general-purpose operating system in 2024. It is.
This is nonsense propaganda. Most 2FA implementations cannot even guarantee
that the second factor is not stored right next to the first factor. Open
standards that do not depend on commercial hardware or telecommunication
operators, such as TOTP, cannot guarantee it by design. Any 2FA app that
works on my PinePhone is also going to work directly on my computer, so you
have no way to enforce that I use a different device for the second factor.
2FA is pointless security theater that just makes it a pain to contribute,
when we are all this time talking about lowering, not rising, the barrier to
entry.
I don't quite agree with you. Two factor authentication whether an actual second
factor device or not does prevent credential stuffing which is a common attack
method that is easy to perform. It is when people take databases of previously leaked
passwords and try them on other accounts that belong to the same person. Since two
factors are generally unique per login situation they can't be stuffed in the same
way.
Of course there are many things two factor does not protect against.
Regards,
Simon
8:09 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 31.03.24 um 21:19 schrieb Simon de Vlieger:
I don't quite agree with you. Two factor authentication whether
an actual second
factor device or not does prevent credential stuffing which is a common attack
method that is easy to perform. It is when people take databases of previously leaked
passwords and try them on other accounts that belong to the same person. Since two
factors are generally unique per login situation they can't be stuffed in the same
way.
Of course there are many things two factor does not protect against.
2FA in a lot of cases is just access to a different account (e.g. email
or even SMS) and these normally aren't unique. Sure, there are other
ways like FIDO2, but these are not necessarily used (or liked, quite
frankly I know a lot of people who would loose them on a monthly basis,
but still are quite smart about other stuff).
This can also lead to a pretty interesting "circle" of 2FA where for
example email a is the 2FA address for email b and email b is the 2FA
address for email a. If it's the only option it can also lead to a
chicken and egg problem for young people who want to create e.g. their
first email account. But this paragraph is besides the point.
So, sure, 2FA would prevent people from just trying out leaked
passwords. But an attack like this would not be a "spray and pray"
attack, but it would be a targeted one. This means that the acceptable
effort from the attacker would be quite a bit higher.
2FA would prevent script kiddies and "spray and pray"-style attacks from
being successful. But more? Doubtful.
Regards
Kilian Hanich
7:09 p.m.
On Mon, Apr 1, 2024 at 1:10 AM Kilian Hanich via devel
<devel(a)lists.fedoraproject.org> wrote:
2FA in a lot of cases is just access to a different account (e.g.
email
or even SMS) and these normally aren't unique. Sure, there are other
ways like FIDO2, but these are not necessarily used (or liked, quite
frankly I know a lot of people who would loose them on a monthly basis,
but still are quite smart about other stuff).
Given that FIDO2 credentials can be stored
on your mobile device (and exchanged with
other devices), if those people are losing their
mobile devices every month they likely have
other issues (including a very expensive
mobile device replacement budget) for which
there is likely no viable solution.
FAS' use of TOTP 2FA is not a great solution
compared to FIDO2, and there are well known
attacks against TOTP 2FA, but even TOTP
2FA can reduce the doorknob rattling exploits.
As TOTP 2FA generators exist for most
mobile devices one will tend to have a
TOTP 2FA generator with one most of the
time.
To the Fedora leadership:
What is the best way to formally propose
that 2FA is required for packagers after
some date (I suppose we could have
different dates for PPs vs others if we
wanted to do that in order to get started
sooner). Do we need a formal Change
Proposal to be voted on by someone?
It does not really seem like a FESCo
issue to me, but more of a policy issue
that might need to go to the Council?
I have no doubt that such a proposal
will be controversial with some, and
all those issues should get a (re-)airing
in front of those making the decision.
7:49 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
What is the best way to formally propose
that 2FA is required for packagers after
some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 /
Please don't discuss there, discuss here; FESCo will vote in that
ticket or a meeting when they feel it appropriate.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
11:04 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Thu, Apr 11, 2024 at 05:49:27PM -0700, Adam Williamson wrote:
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
>
> What is the best way to formally propose
> that 2FA is required for packagers after
> some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 /
Please don't discuss there, discuss here; FESCo will vote in that
ticket or a meeting when they feel it appropriate.
I was wanting to circle back and add some more info to this thread too.
So, right now as far as I know, IPA doesn't have a way to easily say
'require a otp to be enrolled if you want to be added to this group'.
We do have a script that can check current members of a group(s) for otp
and nag them. This is what we do for sysadmin groups, although we
haven't done it in a while.
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or
whatever, right now that would require some work on some scripting,
which I guess would remove people without otp? But then there would
still be a window when the user was added and before the script removed
them. Or some way for sponsors to check otp status before sponsoring
someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager
members without otp and perhaps email them a note about how to set
things up, etc.
kevin
2:45 p.m.
What about simply blocking access to the git repos/koji/bodhi for those
without 2fa?
On Fri, Apr 12, 2024 at 12:05 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
On Thu, Apr 11, 2024 at 05:49:27PM -0700, Adam Williamson wrote:
> On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
> >
> > What is the best way to formally propose
> > that 2FA is required for packagers after
> > some date
>
> There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 /
> Please don't discuss there, discuss here; FESCo will vote in that
> ticket or a meeting when they feel it appropriate.
I was wanting to circle back and add some more info to this thread too.
So, right now as far as I know, IPA doesn't have a way to easily say
'require a otp to be enrolled if you want to be added to this group'.
We do have a script that can check current members of a group(s) for otp
and nag them. This is what we do for sysadmin groups, although we
haven't done it in a while.
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or
whatever, right now that would require some work on some scripting,
which I guess would remove people without otp? But then there would
still be a window when the user was added and before the script removed
them. Or some way for sponsors to check otp status before sponsoring
someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager
members without otp and perhaps email them a note about how to set
things up, etc.
kevin
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
11:13 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, Apr 12, 2024 at 03:45:40PM -0400, Steve Cossette wrote:
What about simply blocking access to the git repos/koji/bodhi for
those
without 2fa?
Well, git I suppose could be a hook that checks the status of the user,
but koji and bodhi don't really have any place to hook that in directly.
They would have to add something in their permissions models to check
for specific actions.
Denying access to koji and bodhi entirely for people without 2fa is...
way too wide. bodhi updates would never get karma from users who didn't
bother to set it up, people just doing scratch builds would be affected,
etc.
So, sure, it's possible, but would be a lot of new code needing written.
kevin
12:53 p.m.
On Fri, Apr 12, 2024 at 4:05 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or
whatever, right now that would require some work on some scripting,
which I guess would remove people without otp? But then there would
still be a window when the user was added and before the script removed
them. Or some way for sponsors to check otp status before sponsoring
someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager
members without otp and perhaps email them a note about how to set
things up, etc.
Finding the (proven)package
members without 2FA might be a
useful thing to know in order to
influence any decision or the
implementation time frame (is it
20% or 80% of (P)Ps?).
That said, I would rather see any
such follow up directed email
happen after a FESCo decision
about 2FA is made in order to
avoid possible multiple emails
(one sent soonish saying that
you *could* add 2FA, should you
want to, and another, should
the decision be made to require
2FA, to say that you will be
required to add 2FA after some
date; one email is better).
That does presume that FESCo
will make a decision in the near
term such that any email text
can be appropriately crafted.
While there will always be some
window/edge cases, I think we
should start with the presumption
that most (proven)packagers will
wish to do the right thing, and
will use 2FA if that is the stated
requirement. After the fact
cleanup/removals as the community
now does for inactive packages
may not be ideal but is arguable
sufficient as a first step.
1:18 p.m.
On Thu, Apr 11, 2024 at 6:50 PM Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
>
> What is the best way to formally propose
> that 2FA is required for packagers after
> some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 /
Please don't discuss there, discuss here; FESCo will vote in that
ticket or a meeting when they feel it appropriate.
If this is going to end up required in one way or another to a
larger group, we should also commit to getting FAS FIDO2 enabled and GOA
fixed. The current FAS MFA is less than ideal, though you do get used to it.
--
Jonathan Steffan
9:52 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
I was hesitant to have MFA for a while. Imagine losing a phone with tons
of tokens. What a hassle to recover from that. I found it less than
ideal for practical reasons.
However, I decided instead to buy two Yubikey (primary and backup), and
I add the QRs to both of them with the Yubico App. I also screenshot my
QRs, tar them, encrypt them with openssl and gpg, and upload them to two
cloud locations also protected by MFA, and remove them from my computer.
I repeat that when I add a new QR. I also have a txt together with the
encrypted tars documenting the commands used to encrypt/decrypt so I
remember the parameters to use. The reason I do that is to be able to
load them into a new Yubikey in case I lose one.
There are alternative to Yubikeys if they are too expensive for some. I
do find them a good investment in general, though. I found having
Yubikeys (at least two), or other similar devices cheaper than phones,
to be the most practical way to do MFA. You can even use those same
Yubikeys to unlock hard drives (luks), and go passwordless for some
applications.
On 4/11/24 17:09, Gary Buhrmaster wrote:
On Mon, Apr 1, 2024 at 1:10 AM Kilian Hanich via devel
<devel(a)lists.fedoraproject.org> wrote:
> 2FA in a lot of cases is just access to a different account (e.g. email
> or even SMS) and these normally aren't unique. Sure, there are other
> ways like FIDO2, but these are not necessarily used (or liked, quite
> frankly I know a lot of people who would loose them on a monthly basis,
> but still are quite smart about other stuff).
Given that FIDO2 credentials can be stored
on your mobile device (and exchanged with
other devices), if those people are losing their
mobile devices every month they likely have
other issues (including a very expensive
mobile device replacement budget) for which
there is likely no viable solution.
FAS' use of TOTP 2FA is not a great solution
compared to FIDO2, and there are well known
attacks against TOTP 2FA, but even TOTP
2FA can reduce the doorknob rattling exploits.
As TOTP 2FA generators exist for most
mobile devices one will tend to have a
TOTP 2FA generator with one most of the
time.
To the Fedora leadership:
What is the best way to formally propose
that 2FA is required for packagers after
some date (I suppose we could have
different dates for PPs vs others if we
wanted to do that in order to get started
sooner). Do we need a formal Change
Proposal to be voted on by someone?
It does not really seem like a FESCo
issue to me, but more of a policy issue
that might need to go to the Council?
I have no doubt that such a proposal
will be controversial with some, and
all those issues should get a (re-)airing
in front of those making the decision.
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
11:47 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with
tons
of tokens. What a hassle to recover from that. I found it less than
ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup
codes that you're meant to print out and keep in a safe place, for
recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just
install an MFA app on that and enrol it too, lock it in a cabinet, then
if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup
mechanism. FreeOTP lets you back up to a file (which you should, of
course, keep somewhere safe and ideally encrypted). Google
Authenticator can backup To The Cloud.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:09 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
> I was hesitant to have MFA for a while. Imagine losing a phone with tons
> of tokens. What a hassle to recover from that. I found it less than
> ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup
codes that you're meant to print out and keep in a safe place, for
recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just
install an MFA app on that and enrol it too, lock it in a cabinet, then
if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup
mechanism. FreeOTP lets you back up to a file (which you should, of
course, keep somewhere safe and ideally encrypted). Google
Authenticator can backup To The Cloud.
yeah, I'll put in a plug for the one I use:
https://github.com/beemdevelopment/Aegis
It's open source, available on f-droid and play store, can to encrypted
backups, pretty active upstream, highly rated in reviews.
kevin
12:10 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Yes that works too. By the time I was setting up MFA everywhere, and
doing the code printing, I recall not all systems giving me that option,
and finding the paper thing not very good as recovery mechanism for me,
so I went with Yubikeys and my own backup-in-the-cloud mechanism. I was
just sharing an option just in case it could serve someone that is
hesitating by giving some ideas :).
On 4/12/24 09:47, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez
wrote:
> I was hesitant to have MFA for a while. Imagine losing a phone with tons
> of tokens. What a hassle to recover from that. I found it less than
> ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup
codes that you're meant to print out and keep in a safe place, for
recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just
install an MFA app on that and enrol it too, lock it in a cabinet, then
if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup
mechanism. FreeOTP lets you back up to a file (which you should, of
course, keep somewhere safe and ideally encrypted). Google
Authenticator can backup To The Cloud.
12:34 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, 2024-04-12 at 10:10 -0700, Carlos Rodriguez-Fernandez wrote:
Yes that works too. By the time I was setting up MFA everywhere, and
doing the code printing, I recall not all systems giving me that option,
Yeah, FreeOTP resisted doing backups for a long time on the basis that
it wasn't "secure" enough, which may be what you're remembering. (you
could still kinda back it up from a rooted phone, but it was a bit
weird.) These days it has a native backup option, though.
and finding the paper thing not very good as recovery mechanism for
me,
so I went with Yubikeys and my own backup-in-the-cloud mechanism. I was
just sharing an option just in case it could serve someone that is
hesitating by giving some ideas :).
For sure, whatever works for you is always the best way :)
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
4:38 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
> I was hesitant to have MFA for a while. Imagine losing a phone with tons
> of tokens. What a hassle to recover from that. I found it less than
> ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup
codes that you're meant to print out and keep in a safe place, for
recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just
install an MFA app on that and enrol it too, lock it in a cabinet, then
if you ever lose your primary phone, use it to recover.
So the problem with github is they don't allow you to have 2FA on a
backup device (or rather, it *is* possible, but the process is
ludicrous[1]). If you have your phone as second FA and lose it then
you have to immediately fall back to the piece of paper.
[1] https://github.com/orgs/community/discussions/78027
I really hope we can avoid that mistake.
Rich.
Also, these days, most authenticator apps support some kind of
backup
mechanism. FreeOTP lets you back up to a file (which you should, of
course, keep somewhere safe and ideally encrypted). Google
Authenticator can backup To The Cloud.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
4:50 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Once upon a time, Richard W.M. Jones <rjones(a)redhat.com> said:
So the problem with github is they don't allow you to have 2FA on
a
backup device (or rather, it *is* possible, but the process is
ludicrous[1]). If you have your phone as second FA and lose it then
you have to immediately fall back to the piece of paper.
I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all
just store one. It's trivial to scan the TOTP code into multiple
devices (depending on the software used, you can sometimes "export" a
TOTP code from one device to another by showing a QR code on the first
device), so that's hardly a "ludicrous" method.
--
Chris Adams <linux(a)cmadams.net>
3:44 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Fri, Apr 12, 2024 at 04:50:13PM -0500, Chris Adams wrote:
Once upon a time, Richard W.M. Jones <rjones(a)redhat.com> said:
> So the problem with github is they don't allow you to have 2FA on a
> backup device (or rather, it *is* possible, but the process is
> ludicrous[1]). If you have your phone as second FA and lose it then
> you have to immediately fall back to the piece of paper.
I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all
just store one. It's trivial to scan the TOTP code into multiple
devices (depending on the software used, you can sometimes "export" a
TOTP code from one device to another by showing a QR code on the first
device), so that's hardly a "ludicrous" method.
I sometimes think how hard it would be to explain all of this to my
mother. I don't understand why 2FA needs to be so obscure and clumsy
to use.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
9:39 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 4/13/24 01:44, Richard W.M. Jones wrote:
On Fri, Apr 12, 2024 at 04:50:13PM -0500, Chris Adams wrote:
> Once upon a time, Richard W.M. Jones <rjones(a)redhat.com> said:
>> So the problem with github is they don't allow you to have 2FA on a
>> backup device (or rather, it *is* possible, but the process is
>> ludicrous[1]). If you have your phone as second FA and lose it then
>> you have to immediately fall back to the piece of paper.
>
> I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all
> just store one. It's trivial to scan the TOTP code into multiple
> devices (depending on the software used, you can sometimes "export" a
> TOTP code from one device to another by showing a QR code on the first
> device), so that's hardly a "ludicrous" method.
I sometimes think how hard it would be to explain all of this to my
mother. I don't understand why 2FA needs to be so obscure and clumsy
to use.
Rich.
You got a really good point there. All MFA implementations the industry
has come up with are less than ideal in one way or the other.
10:38 a.m.
On Sat, Apr 13, 2024 at 8:44 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
I sometimes think how hard it would be to explain all of this to my
mother. I don't understand why 2FA needs to be so obscure and clumsy
to use.
FIDO2 (Apple branded[0] as "passkeys") is
not that hard to use, or explain. The problem
is that (a) passkeys are not yet universally
supported (and, in this case specifically, by
FAS[1]), and (b) unlike Apple (macOS, iOS,
etc.), Microsoft (Windows), and Android,
where the passkey is integrated into the
OS inside a protected enclave, there is no
trusted integrated support in Linux without
an external FIDO2 key[2][3] or using the
"scan a QR code" workaround with a
mobile device which does support use of
passkeys.
Unless your mother is using Linux (and
while Mrs. Roberts has been using Linux
for a long time, most moms don't), this is
likely a time limited issue as more and
more sites support passkeys and from
the consumer point of view it all mostly
just works.
I would like to imagine that FAS' current
2FA will eventually also be reasonably
easy with FIDO2/passkeys, which is why
I occasionally ask about the FIDO2
support status.
[0] I don't remember if there was any
official assignment of the branding, but
I heard that Apple was the org that
suggested the name.
[1] As I understand it, if/when some of the
FAS IdP moves to keycloak, FIDO2 2FA
*could* be supported. However, there is
no current schedule for that move that I
am aware of, and unless Fedora uses the
RHBK runtime, building keycloak from
source for Fedora can be a real PITA
(at least last I looked at it, maybe it has
gotten easier).
[2] As I understand it, the issue is the
lack of the required trusted environment
in generic Linux. There are software
implementations that do not have the
hardware enclave protections,
[3] External FIDO2 keys are also not free,
although I did see a $10 Adafruit FIDO2
key, which is the cheapest I have seen.
4:34 p.m.
Gary Buhrmaster wrote:
[2] As I understand it, the issue is the
lack of the required trusted environment
in generic Linux. There are software
implementations that do not have the
hardware enclave protections,
And to be honest, I do not see the problem there. I will use whatever will
let me pass the Fedora security theater checks without investing in extra
hardware. (This also means that if I am offered the choice, I will pick TOTP
over FIDO2 any day, because TOTP does not require me to emulate a fake
hardware crypto device like FIDO2 does.)
And in my view, the fact that, in those implementations, there is no
Treacherous Computing hardware preventing me from doing what I want with my
own private key (e.g., just copying the same key to all my devices, as I can
also do with TOTP) is actually a feature, even if it goes against the
"security" model.
Kevin Kofler
6:37 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 17.04.24 um 23:34 schrieb Kevin Kofler via devel:
And in my view, the fact that, in those implementations, there is no
Treacherous Computing hardware preventing me from doing what I want with my
own private key (e.g., just copying the same key to all my devices, as I can
also do with TOTP) is actually a feature, even if it goes against the
"security" model.
The fact that you can share the keys is actually part of the design and
wanted. Apple for exmaple has (or wants to) implement easy sharing of
passkeys via AirDrop.
Regards
Kilian Hanich
9:21 a.m.
Kilian Hanich via devel wrote:
The fact that you can share the keys is actually part of the design
and
wanted. Apple for exmaple has (or wants to) implement easy sharing of
passkeys via AirDrop.
So the Apple Cloud can see your private key, but you cannot? Sounds like
GREAT "security", LOL…
Kevin Kofler
4:53 p.m.
The idea is rather to scan the same QR twice, for two yubikeys, and then
screenshot it and save it securely in case you lose one yubikey and need to
load it into a new one.
On Fri, Apr 12, 2024 at 2:39 PM Richard W.M. Jones <rjones(a)redhat.com>
wrote:
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
> On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
> > I was hesitant to have MFA for a while. Imagine losing a phone with
tons
> > of tokens. What a hassle to recover from that. I found it less than
> > ideal for practical reasons.
>
> This is one reason most systems provide a sheet of one-time backup
> codes that you're meant to print out and keep in a safe place, for
> recovery from exactly that scenario.
>
> Alternatively, if you have an old phone or tablet lying around, just
> install an MFA app on that and enrol it too, lock it in a cabinet, then
> if you ever lose your primary phone, use it to recover.
So the problem with github is they don't allow you to have 2FA on a
backup device (or rather, it *is* possible, but the process is
ludicrous[1]). If you have your phone as second FA and lose it then
you have to immediately fall back to the piece of paper.
[1] https://github.com/orgs/community/discussions/78027
I really hope we can avoid that mistake.
Rich.
> Also, these days, most authenticator apps support some kind of backup
> mechanism. FreeOTP lets you back up to a file (which you should, of
> course, keep somewhere safe and ideally encrypted). Google
> Authenticator can backup To The Cloud.
> --
> Adam Williamson (he/him/his)
> Fedora QA
> Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
> https://www.happyassassin.net
>
>
>
> --
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
6:06 p.m.
Adam Williamson wrote:
Also, these days, most authenticator apps support some kind of
backup
mechanism. FreeOTP lets you back up to a file (which you should, of
course, keep somewhere safe and ideally encrypted). Google
Authenticator can backup To The Cloud.
If you use Keysmith, you can just SFTP your
~/.config/org.kde.keysmith/Keysmith.conf from/to all your GNU/Linux
computers including the PinePhone or equivalent, and they will all be able
to generate the same TOTP keys with the same master key.
Kevin Kofler
3:12 p.m.
On Sun, Mar 31, 2024 at 5:35 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Adam Williamson wrote:
> Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to
be).
Whenever 2FA comes up, the requirement
for provenpackages to have it enabled has
always come out very high on the list of
first steps.
I still recommend that. PP is a privilege,
not a right. With privileges often comes
additional responsibilities, and 2FA should
absolutely be one of them.
12:55 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 31 Mar 2024, Neal Gompa wrote:
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols <arthur(a)bols.dev>
wrote:
>
> On 31/03/2024 13:03, Kevin Kofler via devel wrote:
>
> This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for
> contributors for a while, starting with "important" projects, then getting
> stricter and stricter. It has done absolutely nothing to stop this attack.
> How could it, when the backdoor was apparently introduced by the authorized
> maintainer? (Or if not, the attacker must have had access to their 2FA
> secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
> US! And especially DO NOT abuse this incident as an excuse to force 2FA down
> our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
> repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
>
>
> 2FA for Fedora packagers doesn't solve this issue, but that wasn't Adam's
point. What Adam is saying is that we're in danger of focusing too much on a specific
issue while we should spent our time and energy on the general security aspect of Fedora.
2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager
account can do a lot of harm and can take a while to be noticed. If this would happen to
us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure
distro (in my circles) for things that happened before I even entered the linux scene...
>
> Like it or not, this is 2024 and passwords are not as secure as they used to be.
Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT
even if used incorrectly (e.g. storing it in the same keepassxc database).
>
At this point, I'm used to MFA for stuff (and I use a password manager
that handles 2FA OTPs too), but the Fedora implementation of MFA is
uniquely bad because we have to do a lot in the terminal, and our MFA
implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks
2. You need to concatenate password and OTP for getting a krb5 session ticket
3. The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently
unpleasant that I really don't want to use it.
We need to fix these problems anyway. For the first two I am working on
a potential solution as a part of FreeIPA passwordless authentication
support. As you know, FreeIPA supports more than just OTP method that
Fedora Project is currently using. The way how that support is
implemented through Kerberos makes it uniform for OTP, RADIUS, passkeys,
and external IdP pre-authentication methods. Since internal bits of SSSD
already implement support for all these methods in a proper way, we
might reuse those to improve Fedora user experience.
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
11:12 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 13:35 +0200, Arthur Bols wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
> This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for
> contributors for a while, starting with "important" projects, then
getting
> stricter and stricter. It has done absolutely nothing to stop this attack.
> How could it, when the backdoor was apparently introduced by the authorized
> maintainer? (Or if not, the attacker must have had access to their 2FA
> secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON
> US! And especially DO NOT abuse this incident as an excuse to force 2FA down
> our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being
> repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve *this* issue, but that wasn't
Adam's point. What Adam is saying is that we're in danger of focusing
too much on a specific issue while we should spent our time and energy
on the general security aspect of Fedora. 2FA isn't nonsense, it
strengthens security by a lot.
Thanks Arthur, yes, that was *exactly* the point.
I would argue there's a danger of getting too tied up in very specific
technical details of this attack. Yes, it's reasonable to think of ways
to mitigate those specific mechanisms, at least at the appropriate
levels (arguably, most of this is really directly an issue upstream of
us). But it has been - for me - persuasively argued that the specific
technical details were decided based on the wider goals of the attack.
I buy the scenario where the starting point was "how can we poison the
major distributions?" and everything from there derived from that
starting point. xz was picked as the target because of all the specific
technical stuff about systemd and ssh links which people are keen to
dive deep into the details of, but *if that vector hadn't existed the
attacker would just have chosen the next best one*. The specific form
of the attack was then customized to the specific properties of xz,
very cunningly - the whole thing about hiding the payload in binary
test files. But if the attacker had chosen to attack a different
project with different properties, they would have customized their
attack to *that* environment with equal cunning, and it would probably
look quite different.
Worrying *only* about binary blobs in source repos or the specific
details of how systemd opens compression libraries feels a bit narrow,
to me - and especially so when we do it down at the distribution level
where it's not necessarily primarily our responsibility, and I would
argue is definitely not the *lowest* hanging fruit if we take a broader
view of "what should we as a project be doing to raise the difficulty
of supply chain attacks?"
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
11:39 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 09:12:30AM -0700, Adam Williamson wrote:
Thanks Arthur, yes, that was *exactly* the point.
I would argue there's a danger of getting too tied up in very specific
technical details of this attack. Yes, it's reasonable to think of ways
to mitigate those specific mechanisms, at least at the appropriate
levels (arguably, most of this is really directly an issue upstream of
us). But it has been - for me - persuasively argued that the specific
technical details were decided based on the wider goals of the attack.
I buy the scenario where the starting point was "how can we poison the
major distributions?" and everything from there derived from that
starting point. xz was picked as the target because of all the specific
technical stuff about systemd and ssh links which people are keen to
dive deep into the details of, but *if that vector hadn't existed the
attacker would just have chosen the next best one*. The specific form
of the attack was then customized to the specific properties of xz,
very cunningly - the whole thing about hiding the payload in binary
test files. But if the attacker had chosen to attack a different
project with different properties, they would have customized their
attack to *that* environment with equal cunning, and it would probably
look quite different.
I think even stepping further back from the technical details is worth
considering. I think xz was picked because it had one maintainer who had
personal issues and low time/desire to continue work, and they were a
good target to be bullied into adding another maintainer. Are there
other critical packages we ship in similar situations?
Worrying *only* about binary blobs in source repos or the specific
details of how systemd opens compression libraries feels a bit narrow,
to me - and especially so when we do it down at the distribution level
where it's not necessarily primarily our responsibility, and I would
argue is definitely not the *lowest* hanging fruit if we take a broader
view of "what should we as a project be doing to raise the difficulty
of supply chain attacks?"
Agreed. I think theres lots of places we should improve, and focusing on
our own backyard first might be wise. But we can also work on the other
parts too!
There are lots of things we _could_ do... we should discuss and consider
what ones make sense in what order. We should just do something because
we can. ;)
Also, a reminder... nothing is ever 'secure'. security is a process. We
consider likelt threats, we try and mitigate them, and we keep doing it.
Things change and we should too.
kevin
11:58 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 09:39:43AM -0700, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 09:12:30AM -0700, Adam Williamson wrote:
> Thanks Arthur, yes, that was *exactly* the point.
>
> I would argue there's a danger of getting too tied up in very specific
> technical details of this attack. Yes, it's reasonable to think of ways
> to mitigate those specific mechanisms, at least at the appropriate
> levels (arguably, most of this is really directly an issue upstream of
> us). But it has been - for me - persuasively argued that the specific
> technical details were decided based on the wider goals of the attack.
>
> I buy the scenario where the starting point was "how can we poison the
> major distributions?" and everything from there derived from that
> starting point. xz was picked as the target because of all the specific
> technical stuff about systemd and ssh links which people are keen to
> dive deep into the details of, but *if that vector hadn't existed the
> attacker would just have chosen the next best one*. The specific form
> of the attack was then customized to the specific properties of xz,
> very cunningly - the whole thing about hiding the payload in binary
> test files. But if the attacker had chosen to attack a different
> project with different properties, they would have customized their
> attack to *that* environment with equal cunning, and it would probably
> look quite different.
I think even stepping further back from the technical details is worth
considering. I think xz was picked because it had one maintainer who had
personal issues and low time/desire to continue work, and they were a
good target to be bullied into adding another maintainer. Are there
other critical packages we ship in similar situations?
I can imagine the attacker had some sort of scoring system for projects:
- No upstream patch review process (+10)
- Complicated codebase (+20)
- Small group of maintainers (+5)
- Single, vulnerable maintainer (+50)
- Linked to sshd (+1000)
There's something to be said for us doing a similar sort of analysis.
> Worrying *only* about binary blobs in source repos or the
specific
> details of how systemd opens compression libraries feels a bit narrow,
> to me - and especially so when we do it down at the distribution level
> where it's not necessarily primarily our responsibility, and I would
> argue is definitely not the *lowest* hanging fruit if we take a broader
> view of "what should we as a project be doing to raise the difficulty
> of supply chain attacks?"
Agreed. I think theres lots of places we should improve, and focusing on
our own backyard first might be wise. But we can also work on the other
parts too!
There are lots of things we _could_ do... we should discuss and consider
what ones make sense in what order. We should just do something because
we can. ;)
Also, a reminder... nothing is ever 'secure'. security is a process. We
consider likelt threats, we try and mitigate them, and we keep doing it.
Things change and we should too.
kevin
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
1:12 p.m.
Adam Williamson wrote:
I would argue there's a danger of getting too tied up in very
specific
technical details of this attack.
But the fact is:
What WOULD have stopped this attack: (one or more of:)
* Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
* Deleting ALL files automatically generated or imported by autotools in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would
NOT have done the right thing here. Delete the files, THEN run autoreconf.)
* NOT patching OpenSSH downstream to link it against libsystemd against
upstream's recommendation.
What WOULD have greatly reduced the impact of this attack:
* NOT enabling updates-testing by default for Branched releases.
What WOULD NOT have stopped this attack: (any or all of:)
* 2FA. GitHub already enforces 2FA. It did NOT stop this attack.
* Any stricter vetting of Fedora contributions. The attack was performed
upstream, NOT in Fedora.
* More distrust of new Fedora contributors. The offending upgrade was
imported by a TRUSTED Fedora contributor. The untrusted new person operated
upstream, NOT in Fedora.
Kevin Kofler
3:09 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
But the fact is:
What WOULD have stopped this attack: (one or more of:)
* Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
While it’s technically correct that deleting tests would have
disrupted
this specific attack, a policy of deleting and and never running
upstream test code would have prevented me from finding and helping
upstreams fix dozens and dozens of bugs due to accidentally faulty
assumptions that turned out to be violated on different architectures,
in different system environments, or with various allegedly-compatible
dependency versions. There are even GCC bugs (miscompilations, not only
failures to compile) that were discovered and fixed only because
packages I maintain were running upstream unit and integration tests.
Frankly, “testing the packages we ship, as built in our distribution, is
actually bad” seems like a pretty strange and extreme conclusion to draw
from all of this.
4:02 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 04:09:36PM -0400, Ben Beasley wrote:
On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
> But the fact is:
>
> What WOULD have stopped this attack: (one or more of:)
> * Deleting ALL unit tests in %prep (and then of course not trying to run
> them later).
While it’s technically correct that deleting tests would have disrupted this
specific attack, a policy of deleting and and never running upstream test
code would have prevented me from finding and helping upstreams fix dozens
and dozens of bugs due to accidentally faulty assumptions that turned out to
be violated on different architectures, in different system environments, or
with various allegedly-compatible dependency versions. There are even GCC
bugs (miscompilations, not only failures to compile) that were discovered
and fixed only because packages I maintain were running upstream unit and
integration tests. Frankly, “testing the packages we ship, as built in our
distribution, is actually bad” seems like a pretty strange and extreme
conclusion to draw from all of this.
Deleting the tests makes no sense to me either, but it seems like a
mechanism that ensures the test code can't change the build outputs (or
a mechanism to detect that it's happened and abort the build) would
allow upstream tests to be run without compromising the integrity of the
build itself.
7:06 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
Deleting the tests makes no sense to me either, but it seems like a
mechanism that ensures the test code can't change the build outputs (or
a mechanism to detect that it's happened and abort the build) would
allow upstream tests to be run without compromising the integrity of the
build itself.
Just to be clear here that wouldn't have been enough: it's not the test
step that's modifying the binaries, the actual build step is modified in
the right conditions to use data that looks like it belongs to a test
(I've read the actual files aren't actually used in any test and just
look like test data, I didn't check, it wouldn't be hard to make a test
that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have
been prevented, just not running tests isn't enough.
In theory it'd be possible to build twice:
- one normal build with test data, and run tests at the end
- a second build without test data (and confirm we've got an identical
binary, builds are reproducible right?!)
But while we might be able to afford the computing cost, I'm not sure
it's worth it -- this attack vector happened to use test data, but there
are plenty of other ways of doing this, and even just identifying /
removing test data in the first place is hard work for packagers (I
guess we could try to detect binary files but there is no end to that
either, and many builds would just break if we were to automatically
remove tests...)
(Anyway, I also think tests bring more benefits than risks in our case)
--
Dominique Martinet | Asmadeus
11:31 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 01, 2024 at 09:06:16AM +0900, Dominique Martinet wrote:
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
> Deleting the tests makes no sense to me either, but it seems like a
> mechanism that ensures the test code can't change the build outputs (or
> a mechanism to detect that it's happened and abort the build) would
> allow upstream tests to be run without compromising the integrity of the
> build itself.
Just to be clear here that wouldn't have been enough: it's not the test
step that's modifying the binaries, the actual build step is modified in
the right conditions to use data that looks like it belongs to a test
(I've read the actual files aren't actually used in any test and just
look like test data, I didn't check, it wouldn't be hard to make a test
that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have
been prevented, just not running tests isn't enough.
Ugh! I'd missed that detail.
In theory it'd be possible to build twice:
- one normal build with test data, and run tests at the end
- a second build without test data (and confirm we've got an identical
binary, builds are reproducible right?!)
FWIW, I don't think you'd need to build twice:
One approach:
1. do the build
2. do the install
3. generate the RPMs
4. quarantine the RPMs so they're safe from modification
- I believe this could be done via SELinux policy
- there are probably other mechanisms
5. run the tests
- for SELinux, this might be via an `rpmbuild-test` binary that
doesn't have rights to touch the output RPMs
6a. if the tests fail, destroy the RPMs and fail out, reproducing the
result today
6b. if the tests pass, move/copy the RPMs to the result location and
exit cleanly, reproducing the result today
A variation of this would order things as normal but capture hashes of
all the files before the tests start and check the hashes after they're
done. As long as the hashes can't be changed in between, you can trust
that.
Another approach (perhaps more doable in mock, not so easy otherwise):
1. do the build as user1
2. do the tests as user2
3. if the tests pass, resume as user1 as today
Given that the build pollution came from loading data out of the tests,
the above might still be useful, but it's not sufficient to prevent a
recurrence. (Though if you can identify the test files, you could still
segregate them via similar means -- make them unreadable until you're
ready to run the tests.)
But while we might be able to afford the computing cost, I'm not
sure
it's worth it -- this attack vector happened to use test data, but there
are plenty of other ways of doing this, and even just identifying /
removing test data in the first place is hard work for packagers (I
guess we could try to detect binary files but there is no end to that
either, and many builds would just break if we were to automatically
remove tests...)
Another idea I've seen floated
<https://chaos.social/@ollibaba/112189619904512612> is to modify the
build tools (compilers, etc) to generate SBOM files that log every input
file used to produce a given output. Then you have a record to audit to
see if stuff is pulled from weird places. That said, that wouldn't help
either, because all the machinations were via shell commands that
aren't, strictly speaking, build tools. (Except if the linker shows a
file as input that was never generated by a compiler…)
But we could adapt that idea -- couldn't we use strace(1)/ptrace(2) to
detect open calls to determine the inputs and outputs? And we could
protect the resulting records from build/test tampering similarly to the
above.
And if the upstream makes it difficult to tell what's test vs not…I
think at this point we have this incident as precedent to help explain
why we'd prefer less convoluted builds, or at least some explanations.
I dunno how workable that is in practice, though. Though a part of me
says that it all boils down to making the build process more transparent
than we've previously put up with.
Anyway, just putting that out there in case anyone thinks these are good
ideas (or brainstorming toward better ones).
(Anyway, I also think tests bring more benefits than risks in our
case)
And it also occurs to me that test failures could just as easily help
find a malicious change if some new feature we're adding to Fedora
breaks the tests unexpectedly and exposes it (or just something unique
about how we've configured things).
--
Scott Schmit
12:09 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
+1
Am 01.04.24 um 06:31 schrieb Scott Schmit:
One approach:
1. do the build
2. do the install
3. generate the RPMs
4. quarantine the RPMs so they're safe from modification
- I believe this could be done via SELinux policy
- there are probably other mechanisms
5. run the tests
- for SELinux, this might be via an `rpmbuild-test` binary that
doesn't have rights to touch the output RPMs
6a. if the tests fail, destroy the RPMs and fail out, reproducing the
result today
6b. if the tests pass, move/copy the RPMs to the result location and
exit cleanly, reproducing the result today
Boils down to separate source and test code/phase
source code:
(hopefully not obfuscated to the point where no review is possible)
no binaries allowed, best possible review
needed to build
build phase: source to binary
test code:
binaries allowed
only needed to test
test phase: binary unmodified
Allowing a test file to modify the binary makes it a source file. ?
Christoph
6:24 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 01, 2024 at 09:06:16AM +0900, Dominique Martinet wrote:
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
> Deleting the tests makes no sense to me either, but it seems like a
> mechanism that ensures the test code can't change the build outputs (or
> a mechanism to detect that it's happened and abort the build) would
> allow upstream tests to be run without compromising the integrity of the
> build itself.
Just to be clear here that wouldn't have been enough: it's not the test
step that's modifying the binaries, the actual build step is modified in
the right conditions to use data that looks like it belongs to a test
(I've read the actual files aren't actually used in any test and just
look like test data, I didn't check, it wouldn't be hard to make a test
that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have
been prevented, just not running tests isn't enough.
Yep.
And since we're talking about xz, note that a second malicious issue has
beend found: [1] is a revert of [2] which sabotages CMakeLists.txt
to always disable Landlock sandbox.
Clearly, the only reasonable solution is to delete all the CMake cruft ;)
[1]
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f...
[2]
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644ef...
Zbyszek
8:16 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 31.03.24 um 23:02 schrieb Scott Schmit:
On Sun, Mar 31, 2024 at 04:09:36PM -0400, Ben Beasley wrote:
> On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
>> But the fact is:
>>
>> What WOULD have stopped this attack: (one or more of:)
>> * Deleting ALL unit tests in %prep (and then of course not trying to run
>> them later).
> While it’s technically correct that deleting tests would have disrupted this
> specific attack, a policy of deleting and and never running upstream test
> code would have prevented me from finding and helping upstreams fix dozens
> and dozens of bugs due to accidentally faulty assumptions that turned out to
> be violated on different architectures, in different system environments, or
> with various allegedly-compatible dependency versions. There are even GCC
> bugs (miscompilations, not only failures to compile) that were discovered
> and fixed only because packages I maintain were running upstream unit and
> integration tests. Frankly, “testing the packages we ship, as built in our
> distribution, is actually bad” seems like a pretty strange and extreme
> conclusion to draw from all of this.
Deleting the tests makes no sense to me either, but it seems like a
mechanism that ensures the test code can't change the build outputs (or
a mechanism to detect that it's happened and abort the build) would
allow upstream tests to be run without compromising the integrity of the
build itself.
It would prevent the attack from being hidden in the tests, but not in
any other part of the build.
Also, I have seen build setups which encode the status of tests in the
eventual binary and as such info page or integrated bug report
generators. Often because some distros sometimes turned them off or
ships software even with failed tests and thus pushing that headache to
upstream.
Regards
Kilian Hanich
11:10 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Dne 01. 04. 24 v 3:16 dop. Kilian Hanich via devel napsal(a):
Also, I have seen build setups which encode the status of tests in
the
eventual binary and as such info page or integrated bug report
generators. Often because some distros sometimes turned them off or
ships software even with failed tests and thus pushing that headache to
upstream.
Can you point me to such case, please? Just being curious.
--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
10:27 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 20:12 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
> I would argue there's a danger of getting too tied up in very specific
> technical details of this attack.
But the fact is:
What WOULD have stopped this attack: (one or more of:)
* Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
This comes with the huge cost that we no longer have any idea if many
things actually work in Fedora. There are much better options here,
such as ensuring the package build process *isolates* the test data
from the compile process without just *removing* it. but again, this is
getting far too tied up in the details of this *one* attack. it's the
"everyone has to take their shoes off at the airport forever" mistake:
just because the *last* attack involved a binary test file, does not
mean the *next* one will. This is also known as preparing for the last
war, in military strategy.
* Deleting ALL files automatically generated or imported by autotools
in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would
NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have
regenerated the malicious file. We have already established that.
* NOT patching OpenSSH downstream to link it against libsystemd
against
upstream's recommendation.
Again, this is far too tied to the specific attack. I'm not interested,
as I said, in Monday morning quarterbacking this specific attack. I'm
interested in the question of what are the most immediate and effective
ways Fedora can lower its likelihood to be directly targeted in a
supply chain attack *in future*. Not the last one.
What WOULD have greatly reduced the impact of this attack:
* NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the
compromise was discovered so quickly. Otherwise, we were busy trying to
get this update pushed stable as fast as possible, because rwmj wanted
that. If the compromise had happened to be caught a few days later, the
update would have been pushed stable anyway. Our gating and manual
testing processes are not designed to catch security issues, and
certainly do not reliably do that job. They did not in this case. So it
seems wrong, to me, to suggest we should rely on them to do it in
future. Thus I disagree that any possible security benefit gained by
doing this would outweigh the substantial disadvantage that we wouldn't
get testing of stuff promptly any more.
What WOULD NOT have stopped this attack: (any or all of:)
* 2FA. GitHub already enforces 2FA. It did NOT stop this attack.
Again. I'm not interested in what, with hindsight, might have stopped
the attack that already happened. We know for sure that weak
authentication processes absolutely are a giant flashing ATTACK HERE
neon sign for *future* attackers.
* Any stricter vetting of Fedora contributions. The attack was
performed
upstream, NOT in Fedora.
See above.
* More distrust of new Fedora contributors. The offending upgrade was
imported by a TRUSTED Fedora contributor. The untrusted new person operated
upstream, NOT in Fedora.
See above again.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:13 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Adam,
Is there a way already to achieve test isolation during the rpm build?
Beside doing something ad-hoc with git init or some file system feature,
I was wondering if there is something already available and standardized.
Regards,
Carlos R.F.
On 3/31/24 20:27, Adam Williamson wrote:
On Sun, 2024-03-31 at 20:12 +0200, Kevin Kofler via devel wrote:
> Adam Williamson wrote:
>> I would argue there's a danger of getting too tied up in very specific
>> technical details of this attack.
>
> But the fact is:
>
> What WOULD have stopped this attack: (one or more of:)
> * Deleting ALL unit tests in %prep (and then of course not trying to run
> them later).
This comes with the huge cost that we no longer have any idea if many
things actually work in Fedora. There are much better options here,
such as ensuring the package build process *isolates* the test data
from the compile process without just *removing* it. but again, this is
getting far too tied up in the details of this *one* attack. it's the
"everyone has to take their shoes off at the airport forever" mistake:
just because the *last* attack involved a binary test file, does not
mean the *next* one will. This is also known as preparing for the last
war, in military strategy.
> * Deleting ALL files automatically generated or imported by autotools in
> %prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it
would
> NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have
regenerated the malicious file. We have already established that.
> * NOT patching OpenSSH downstream to link it against libsystemd against
> upstream's recommendation.
Again, this is far too tied to the specific attack. I'm not interested,
as I said, in Monday morning quarterbacking this specific attack. I'm
interested in the question of what are the most immediate and effective
ways Fedora can lower its likelihood to be directly targeted in a
supply chain attack *in future*. Not the last one.
> What WOULD have greatly reduced the impact of this attack:
> * NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the
compromise was discovered so quickly. Otherwise, we were busy trying to
get this update pushed stable as fast as possible, because rwmj wanted
that. If the compromise had happened to be caught a few days later, the
update would have been pushed stable anyway. Our gating and manual
testing processes are not designed to catch security issues, and
certainly do not reliably do that job. They did not in this case. So it
seems wrong, to me, to suggest we should rely on them to do it in
future. Thus I disagree that any possible security benefit gained by
doing this would outweigh the substantial disadvantage that we wouldn't
get testing of stuff promptly any more.
>
> What WOULD NOT have stopped this attack: (any or all of:)
> * 2FA. GitHub already enforces 2FA. It did NOT stop this attack.
Again. I'm not interested in what, with hindsight, might have stopped
the attack that already happened. We know for sure that weak
authentication processes absolutely are a giant flashing ATTACK HERE
neon sign for *future* attackers.
> * Any stricter vetting of Fedora contributions. The attack was performed
> upstream, NOT in Fedora.
See above.
> * More distrust of new Fedora contributors. The offending upgrade was
> imported by a TRUSTED Fedora contributor. The untrusted new person operated
> upstream, NOT in Fedora.
See above again.
1:20 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because
there is no one universal Standard Test System (not even within a
single language ecosystem, let alone across all of them). Currently
you'd have to design something unique, if you wanted to implement this
for your package.
I suppose one approach would be to split the sources-required-for-build
and the test suite into Source0 and Source1 (respectively) and only
extract Source0 during %prep. Don't extract Source1 until %check (i.e.
after %build and %install are already done). I'm just spitballing,
though, haven't checked if this is really practical.
Of course, another approach is to really do what Kevin suggests and not
ship the test suite in the package source at all, but instead run the
tests via Fedora CI, and configure the package to be gated on that CI
check (so it wouldn't go to stable without the tests passing). But
that's rather a different approach (and would still require 'custom'
work to cut the tests out of the source, or at least delete them before
running the build).
And I still think at this point we are falling into the trap of
thinking too specifically about an attack vector which just *happens*
to be the one chosen in *this specific instance*. It's still
worthwhile, on some level, for someone to think about that kind of
hardening, of course. I am just not convinced it is the most useful
thing Fedora could be doing right now in the general area of "supply
chain hardening".
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
6:38 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
> Adam,
>
> Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because
there is no one universal Standard Test System (not even within a
single language ecosystem, let alone across all of them). Currently
you'd have to design something unique, if you wanted to implement this
for your package.
If we wanted to pursue that, I'd suggest the following:
remount $RPM_BUILD_ROOT read-only for the %check phase
(or maybe overmount it with a writable overlayfs that is thrown
away after %check finishes, and warn if any modifications were made.)
%check is executed after %install, so everything should be in place
before %check, and %check may be skipped, so no modifications to
installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and
'bind --read-only' that might be useful here. But mock uses
systemd-nspawn in a way that does register the container with machined.
So maybe it'd be more reasonable to just execute a mount command directly
from mock.
This is independent of the test system and does not require splitting
of the test sources.
Zbyszek
6:46 a.m.
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
> On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
> > Adam,
> >
> > Is there a way already to achieve test isolation during the rpm build?
>
> Nothing systematic that I'm aware of, no. It would be tricky because
> there is no one universal Standard Test System (not even within a
> single language ecosystem, let alone across all of them). Currently
> you'd have to design something unique, if you wanted to implement this
> for your package.
If we wanted to pursue that, I'd suggest the following:
remount $RPM_BUILD_ROOT read-only for the %check phase
(or maybe overmount it with a writable overlayfs that is thrown
away after %check finishes, and warn if any modifications were made.)
%check is executed after %install, so everything should be in place
before %check, and %check may be skipped, so no modifications to
installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and
'bind --read-only' that might be useful here. But mock uses
systemd-nspawn in a way that does register the container with machined.
So maybe it'd be more reasonable to just execute a mount command directly
from mock.
This is independent of the test system and does not require splitting
of the test sources.
Another thing to consider is making RPM unshare each build phase like
it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can
also help with this. I believe moss[2]' boulder tool does this.
[1]: https://github.com/rpm-software-management/rpm/pull/2666
[2]: https://github.com/serpent-os/moss/tree/main/boulder
--
真実はいつも一つ！/ Always, there's only one truth!
10 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
I created a discussion issue for this idea:
https://github.com/rpm-software-management/rpm/discussions/3009
I think it worth pursuing further.
On 4/1/24 04:46, Neal Gompa wrote:
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
>> On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
>>> Adam,
>>>
>>> Is there a way already to achieve test isolation during the rpm build?
>>
>> Nothing systematic that I'm aware of, no. It would be tricky because
>> there is no one universal Standard Test System (not even within a
>> single language ecosystem, let alone across all of them). Currently
>> you'd have to design something unique, if you wanted to implement this
>> for your package.
>
> If we wanted to pursue that, I'd suggest the following:
> remount $RPM_BUILD_ROOT read-only for the %check phase
> (or maybe overmount it with a writable overlayfs that is thrown
> away after %check finishes, and warn if any modifications were made.)
> %check is executed after %install, so everything should be in place
> before %check, and %check may be skipped, so no modifications to
> installed files should be done in %check.
>
> Considering possible implemention details, machinectl has 'bind' and
> 'bind --read-only' that might be useful here. But mock uses
> systemd-nspawn in a way that does register the container with machined.
> So maybe it'd be more reasonable to just execute a mount command directly
> from mock.
>
> This is independent of the test system and does not require splitting
> of the test sources.
>
Another thing to consider is making RPM unshare each build phase like
it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can
also help with this. I believe moss[2]' boulder tool does this.
[1]: https://github.com/rpm-software-management/rpm/pull/2666
[2]: https://github.com/serpent-os/moss/tree/main/boulder
--
真実はいつも一つ！/ Always, there's only one truth!
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
3:56 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On 4/1/24 14:46, Neal Gompa wrote:
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
>> On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
>>> Adam,
>>>
>>> Is there a way already to achieve test isolation during the rpm build?
>>
>> Nothing systematic that I'm aware of, no. It would be tricky because
>> there is no one universal Standard Test System (not even within a
>> single language ecosystem, let alone across all of them). Currently
>> you'd have to design something unique, if you wanted to implement this
>> for your package.
>
> If we wanted to pursue that, I'd suggest the following:
> remount $RPM_BUILD_ROOT read-only for the %check phase
> (or maybe overmount it with a writable overlayfs that is thrown
> away after %check finishes, and warn if any modifications were made.)
> %check is executed after %install, so everything should be in place
> before %check, and %check may be skipped, so no modifications to
> installed files should be done in %check.
>
> Considering possible implemention details, machinectl has 'bind' and
> 'bind --read-only' that might be useful here. But mock uses
> systemd-nspawn in a way that does register the container with machined.
> So maybe it'd be more reasonable to just execute a mount command directly
> from mock.
>
> This is independent of the test system and does not require splitting
> of the test sources.
>
Another thing to consider is making RPM unshare each build phase like
it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can
also help with this. I believe moss[2]' boulder tool does this.
[1]: https://github.com/rpm-software-management/rpm/pull/2666
[2]: https://github.com/serpent-os/moss/tree/main/boulder
Sure, this is an area we've been planning to look into even before this
incident. For example
https://github.com/rpm-software-management/rpm/issues/2989 and as a
result of this thread,
https://github.com/rpm-software-management/rpm/issues/3010. And builds
with a read-only source:
https://github.com/rpm-software-management/rpm/issues/2985
None of this will fix a compromised usptream of course.
- Panu -
7:58 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Test isolation is still assuming the attack comes in the test phase. The
attack can come in the `make`, or in the `make install` too. That's why
the idea of other techniques being discussed are still valid, but
perhaps not abstracted out enough for a wider defense.
However, the test isolation concept is a good defense. Once the build
artifacts is generated, the integrity of the build artifacts must be
ensured during any post build phase. Some tests do expect to be executed
right after the build, when the build artifacts are still in specific
directories within the source root, so that's why I was thinking of
something more like at the file system level (e.g., a snapshot that it
is then restored, or something like that).
On 3/31/24 23:20, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez
wrote:
> Adam,
>
> Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because
there is no one universal Standard Test System (not even within a
single language ecosystem, let alone across all of them). Currently
you'd have to design something unique, if you wanted to implement this
for your package.
I suppose one approach would be to split the sources-required-for-build
and the test suite into Source0 and Source1 (respectively) and only
extract Source0 during %prep. Don't extract Source1 until %check (i.e.
after %build and %install are already done). I'm just spitballing,
though, haven't checked if this is really practical.
Of course, another approach is to really do what Kevin suggests and not
ship the test suite in the package source at all, but instead run the
tests via Fedora CI, and configure the package to be gated on that CI
check (so it wouldn't go to stable without the tests passing). But
that's rather a different approach (and would still require 'custom'
work to cut the tests out of the source, or at least delete them before
running the build).
And I still think at this point we are falling into the trap of
thinking too specifically about an attack vector which just *happens*
to be the one chosen in *this specific instance*. It's still
worthwhile, on some level, for someone to think about that kind of
hardening, of course. I am just not convinced it is the most useful
thing Fedora could be doing right now in the general area of "supply
chain hardening".
11:29 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, 2024-04-01 at 05:58 -0700, Carlos Rodriguez-Fernandez wrote:
Test isolation is still assuming the attack comes in the test phase.
As I initially suggested it, it does not. My suggestion was that we
ensure the test code is not available to the prep / build / install
phases *at all*, and is only made available to the test phase.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:01 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Understood. However, at least for those unit tests run in the %check, it
is going to be almost unfeasible, because of the variability of the way
things are done in the different programming ecosystems. In Java, unit
tests are nicely separated in a different folder (i.e., `src/test`), but
in golang, it is mingled with the source code in `_test.go` files. In C,
it depends on the programmers convention.
On 4/1/24 09:29, Adam Williamson wrote:
On Mon, 2024-04-01 at 05:58 -0700, Carlos Rodriguez-Fernandez
wrote:
> Test isolation is still assuming the attack comes in the test phase.
As I initially suggested it, it does not. My suggestion was that we
ensure the test code is not available to the prep / build / install
phases *at all*, and is only made available to the test phase.
1:38 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 20:27 -0700, Adam Williamson wrote:
> What WOULD have greatly reduced the impact of this attack:
> * NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the
compromise was discovered so quickly. Otherwise, we were busy trying to
get this update pushed stable as fast as possible, because rwmj wanted
that. If the compromise had happened to be caught a few days later, the
update would have been pushed stable anyway. Our gating and manual
testing processes are not designed to catch security issues, and
certainly do not reliably do that job. They did not in this case. So it
seems wrong, to me, to suggest we should rely on them to do it in
future. Thus I disagree that any possible security benefit gained by
doing this would outweigh the substantial disadvantage that we wouldn't
get testing of stuff promptly any more.
Having thought this over a bit, I think I should moderate my position
here and make it a bit more nuanced.
It occurs to me - maybe you don't agree, but this is how it looks to me
- that, ironically, you and I usually argue the exact *opposite* side
of this case, no? I argue in *favor* of somewhat-arbitrary delays to
packages appearing in 'stable', and you argue *against* them. :D
So, I don't think I can in full conscience justify being so
categorical. So let me put more emphasis on the "difference between
security and quality" aspect here.
Let's look at the *intent* of the existing policies, here. Practically
speaking, what our policies *achieve* is that for stable releases,
updates-testing acts to *guard Fedora users' systems*. That is its
function there. That's why it is not enabled by default, and we ask
only people who want to contribute to testing to enable it, and
understand that they are "taking a bullet for quality" by doing so. The
idea is that by preventing updates reaching 'ordinary' users' systems
before our testers have at least had an opportunity to try the packages
out, we will catch at least some quality issues and prevent ordinary
users from being affected by them.
The process is specifically designed to achieve this. We have automated
*quality* checks on packages in updates-testing (but almost nothing in
the way of automated *security* checks). We have a whole process geared
towards getting testers to test out the packages and report their
findings - in *quality* terms. This is, to me, a job it's reasonable to
ask a fairly broad range of folks to help with. It works reasonably
well. We have done a reasonable job of implementing a process that lets
a person without really specific training install a test update,
experience a problem in it, and communicate that they encountered that
problem in such a way that we are able to prevent the package reaching
a wider swathe of users. It's not *perfect*, but it does provide some
demonstrable value. Similarly on the automated testing side, both
Fedora CI and openQA provide useful gating of some packages. It is by
no means comprehensive, but it *is* useful. We catch bugs and prevent
them reaching regular users. We don't catch *all* bugs, but we can
reasonably claim to be purposefully and intentionally catching a
reasonable number with this system.
OK, that's for stable releases. What about development releases? It's
key to realize that for development releases, the process looks very
similar, but just that one change - updates-testing defaulting to off -
*changes its entire purpose*, and this is by intent. For development
releases, the purpose of updates-testing is **NOT** to protect
'ordinary users'. It is to protect **the Fedora development process**.
We don't really intend for there to *be* "ordinary users" of Fedora
development releases. We make it fairly clear that, by using a
development release, you are taking yourself out of the "regular user"
category and putting yourself in the "tester / developer" category. We
do not feel such an obligation to provide users of development releases
with quality-tested code. On the contrary, to a degree, they are
supposed to be *helping* with the process of quality testing for the
ultimate goal of making the *final* release good for "ordinary users".
So if you're a user of a development release, we do not really feel an
obligation to provide the same "safety net" for you that updates-
testing provides for "regular users" of stable releases.
Instead, the point of updates-testing in development releases is to let
us prevent breakages in *the process of building the distribution*. We
expect *all* users of development releases to help with this by
flagging up when they see breakage, so we can prevent broken packages
appearing in the buildroot or the compose process. *That* is why we
have updates-testing for development releases.
So...to return to the initial question, ultimately my argument is that
it doesn't make much sense to try and use the existing updates-testing
process as a "security gate". Not even for production releases, really,
but *especially* not for development releases. That's *not what it's
designed to do*. There is very little meaningful automated *security*
testing going on for updates-testing. Nor do we have any kind of
process to try and get informed *manual* security checks done on
packages in updates-testing. Given that, I think the benefit of trying
to use our existing updates-testing process to improve security would
be very small.
Now, if we were to actually *intentionally design a process* to do some
meaningful security checks on 'candidate' packages, that would be an
entirely different story. That might be something worth actually doing,
and if we were to set out to do that, the question of whether it should
be kinda merged into the existing 'updates-testing' process, or should
use some different kinda gating approach, would be a reasonable one. My
vote might be very different in that case. But I just don't think we
would get much security value from *just* enabling updates-testing on
Branched, given the background above. Sorry this got much longer and
less clear. :D
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
5:15 p.m.
Adam Williamson wrote:
It occurs to me - maybe you don't agree, but this is how it looks
to me
- that, ironically, you and I usually argue the exact *opposite* side
of this case, no? I argue in *favor* of somewhat-arbitrary delays to
packages appearing in 'stable', and you argue *against* them. :D
I have never argued against updates-testing existing or that all packages
should skip updates-testing. "Please pick up this new upstream version, it
has some great new features" as was done here is exactly the kind of changes
that SHOULD go through updates-testing. But if the maintainer has something
urgent to push out, such as an important regression fix or a critical
security fix (e.g., a fix for a backdoor like this one), they should be
allowed to decide to skip testing and not be treated as being too
incompetent for that (while at the same time allowing any other person, with
no other credentials than a FAS account, to +1 the package, allowing it to
be autopushed to stable – everyone except the one person most qualified to
make that decision). THAT is what I have been arguing for all this time, and
I do not see how this contradicts my position here in any way.
Kevin Kofler
5:25 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Wed, 2024-04-03 at 00:15 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
> It occurs to me - maybe you don't agree, but this is how it looks to me
> - that, ironically, you and I usually argue the exact *opposite* side
> of this case, no? I argue in *favor* of somewhat-arbitrary delays to
> packages appearing in 'stable', and you argue *against* them. :D
I have never argued against updates-testing existing or that all packages
should skip updates-testing. "Please pick up this new upstream version, it
has some great new features" as was done here is exactly the kind of changes
that SHOULD go through updates-testing. But if the maintainer has something
urgent to push out, such as an important regression fix or a critical
security fix (e.g., a fix for a backdoor like this one), they should be
allowed to decide to skip testing and not be treated as being too
incompetent for that (while at the same time allowing any other person, with
no other credentials than a FAS account, to +1 the package, allowing it to
be autopushed to stable – everyone except the one person most qualified to
make that decision). THAT is what I have been arguing for all this time, and
I do not see how this contradicts my position here in any way.
Fair enough. I think the rest of my post stands, though, as it was more
about my argument than yours.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
4:37 p.m.
Adam Williamson wrote:
> * Deleting ALL files automatically generated or imported by
autotools in
> %prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it
> would NOT have done the right thing here. Delete the files, THEN run
> autoreconf.)
No. This would not have avoided the attack, because it would not have
regenerated the malicious file. We have already established that.
Just running autoreconf would not. As I wrote: "DO NOT trust autoreconf, it
would NOT have done the right thing here." Deleting the file with an
explicit rm -f in %prep, and THEN running autoreconf would have regenerated
(reimported, actually, this comes from gnulib and is copied unchanged, but
in any case it would NOT have contained the malicious additions) the file.
That said, autoreconf needs fixing too, because -f is supposed to regenerate
all files that can be regenerated, which is not happening. But if you
explicitly delete the files before running autoreconf, then it has to
regenerate them no matter what.
Kevin Kofler
6:28 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, 2024-04-01 at 23:37 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
> > * Deleting ALL files automatically generated or imported by autotools in
> > %prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it
> > would NOT have done the right thing here. Delete the files, THEN run
> > autoreconf.)
>
> No. This would not have avoided the attack, because it would not have
> regenerated the malicious file. We have already established that.
Just running autoreconf would not. As I wrote: "DO NOT trust autoreconf, it
would NOT have done the right thing here." Deleting the file with an
explicit rm -f in %prep, and THEN running autoreconf would have regenerated
(reimported, actually, this comes from gnulib and is copied unchanged, but
in any case it would NOT have contained the malicious additions) the file.
That said, autoreconf needs fixing too, because -f is supposed to regenerate
all files that can be regenerated, which is not happening. But if you
explicitly delete the files before running autoreconf, then it has to
regenerate them no matter what.
Sure, but as others posted upthread, this still doesn't help much. To
do this you have to know what m4s are 'standard' and will actually be
regenerated, and which are custom and you can't wipe them. And then an
attacker could just slip in an extra custom one instead of modifying a
'standard' one.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
6:55 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sunday, 31 March 2024 20:12:38 CEST Kevin Kofler via devel wrote:
* Deleting ALL unit tests in %prep (and then of course not trying to
run
them later).
I strongly oppose this suggestion. While it would have prevented this
particular backdoor as a side-effect, the primary effect of going without unit
tests would be an outsize hole in Fedora's QA.
As maintainer of the mold package, I regularly come across failures in %check,
mostly in the builds for secondary architectures. This allows me to help
upstream - a one-person project with limited resources - fix these bugs before
Fedora ships the package.
Moreover, %check has already caught bugs that only manifest themselves with
Fedora's compile flags. Here's an example:
https://github.com/rui314/mold/pull/877
You used the term 'pointless security theatre' in another email within this
thread. *This* would be a prime example.
Best,
Christoph
7:10 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Christoph Erhardt writes:
I strongly oppose this suggestion. While it would have prevented
this
particular backdoor as a side-effect, the primary effect of going without
unit
tests would be an outsize hole in Fedora's QA.
There have been several suggestions here for ways that this specific attempt
from succeeding.
Any one of them will be very useful as long as it is guaranteed that all
backdoor/supply chain attacks in the future are attempted in the exact same
technical way.
11:06 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 13:03 +0200, Kevin Kofler via devel wrote:
> 2. Our process for vetting packagers is, let's face it, from a security
> perspective almost *comically* patchy. There are 140 sponsors in the
> packager FAS group. Any one of those people - or someone who
> compromises any one of those 140 accounts - can grant any other person
> on earth Fedora packager status. Our policy on how they should do this
> is
>
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
> . The words "trust" and "identity" do not appear in it. There
is,
> AFAIK, no policy or procedure by which inactive sponsors have this
> power removed. There is no mandatory 2FA policy for sponsors.
We already have a manpower problem, how is removing sponsors going to
improve the situation?
Removing inactive sponsors makes it harder for an attacker to
compromise an inactive sponsor's account and use it for evil.
However, looking into it it looks like I made a mistake here: we do
have an inactive *packager* policy, as well as an inactive *proven
packager* policy. I had misremembered that we did not have the former,
only the latter. So at *least* a sponsor who becomes completely
inactive in the project will eventually get removed as a packager
entirely (and hence also as a sponsor). I don't think that's entirely
sufficient, but it's better than I thought.
> 3. We have no mechanism to flag when J. Random Packager adds
> "Supplements: glibc" to their random leaf node package. As a reminder,
> *we are a project that allows 1,601 minimally-vetted people to deliver
> arbitrary code executed as root on hundreds of thousands of systems*,
> and this mechanism allows any one of those people to cause the package
> they have complete control over to be automatically pulled in as a
> dependency on virtually every single one of those systems.
This would get noticed pretty quickly, when that package comes up in update
transactions for no reason. I believe this has never happened so far. It is
just too obvious.
"Would get noticed pretty quickly" is not really acceptable. It should
not be possible to do it at all.
This was just an example really, too. I am not a very good red teamer.
I'm not imaginative enough. There are probably all sorts of more subtle
ways a malicious person with packager privileges could leverage them to
attack more people than they could attack just through 'normal'
maintenance of a leaf package. Just for another example, we have had
two significant cases recently of "package X incorrectly provides
capability Y".
1. sdubby was providing grubby with a higher version than the real
grubby package, which seems to have caused it to get installed in
preference to grubby on fresh installs and upgrades when it should not
have been.
2. some relatively obscure package somehow started including bundled
copies of dozens of core system libraries, which made the auto-provides
generator generate provides for those libraries. dnf then started
preferring it over the 'real' providers of some of those libraries, I
think because it had a shorter dep chain.
Neither of these was, AFAICT, flagged as a potential security issue by
anyone, but it seems reasonable to me that they should have been. I
know about them because we happened to catch them as quality issues,
but it is not hard to imagine that a malicious packager may be
interested in using the same effect intentionally for evil, and they
would obviously be careful to try and set things up such that this
would *not* cause anything easily noticeable as a quality defect.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
7:28 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 01:58:21AM -0700, Adam Williamson wrote:
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
> I'm not pretending these will solve everything, but they should make
> attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some
of the grandiose ideas in later posts than Richard's list...I think
we're in danger of building castles in the sky while not cleaning up
the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world
off autotools or banning binaries in repos or centralized source depots
authenticated by a committee of Top People, can we remember:
1. We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
2. Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the
packager FAS group. Any one of those people - or someone who
compromises any one of those 140 accounts - can grant any other person
on earth Fedora packager status. Our policy on how they should do this
is
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
. The words "trust" and "identity" do not appear in it. There is,
AFAIK, no policy or procedure by which inactive sponsors have this
power removed. There is no mandatory 2FA policy for sponsors.
3. We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder,
*we are a project that allows 1,601 minimally-vetted people to deliver
arbitrary code executed as root on hundreds of thousands of systems*,
and this mechanism allows any one of those people to cause the package
they have complete control over to be automatically pulled in as a
dependency on virtually every single one of those systems.
This is as much a distro design problem, as a Fedora process
problem. The typical Linux distro model is that everything is
installed in the same namespace, and we only avoid interference
(whether accidental or intentional) by careful packaging design
and review.
This is somewhere where the image based Linux distro model has
a potential benefit, with a comparatively slim distro base, and
then applications as self contained separated entities, whether
server apps in podman containers, or GUI apps in flatpaks.
No easy anwere here though, as the traditional Linux model isn't
going away any time in the forseeable future.
4. Our main auth system was written years ago by someone who no
longer
contributes and nobody is really actively maintaining it[0].
5. We review initial new packages in a moderate level of detail,
but after that it is largely a free for all for every rebase.
It is up to the maintainer's discretion what to do for new
releases, and any oversight is patchy at best. The threat here
was in relation to an update to an existing package, and where
we have no formal process that was even /interested/ in addressing
that threat, let alone capable of stopping it.
Now, in this particular case, it would have been challenging
to detect it the problem even with "new package" review process,
so I'm sceptical we can easily build a "updated package" process
that would have blocked it and yet remain practical for maintainers.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:15 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, 2024-03-31 at 13:28 +0100, Daniel P. Berrangé wrote:
>
> 3. We have no mechanism to flag when J. Random Packager adds
> "Supplements: glibc" to their random leaf node package. As a reminder,
> *we are a project that allows 1,601 minimally-vetted people to deliver
> arbitrary code executed as root on hundreds of thousands of systems*,
> and this mechanism allows any one of those people to cause the package
> they have complete control over to be automatically pulled in as a
> dependency on virtually every single one of those systems.
This is as much a distro design problem, as a Fedora process
problem. The typical Linux distro model is that everything is
installed in the same namespace, and we only avoid interference
(whether accidental or intentional) by careful packaging design
and review.
This is somewhere where the image based Linux distro model has
a potential benefit, with a comparatively slim distro base, and
then applications as self contained separated entities, whether
server apps in podman containers, or GUI apps in flatpaks.
No easy anwere here though, as the traditional Linux model isn't
going away any time in the forseeable future.
Well, definitely no easy answers, but I would argue this is the kind of
thing we as a distributor *should* be worrying about and addressing,
maybe with more urgency than things that are not primarily the
responsibility of the distributor, in the supply chain.
Maybe this needs to go on the growing pile of reasons why the
traditional Linux model *does* need to go away. Maybe Fedora, with its
foundation of First, should be kind of at the forefront of making that
happen.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:54 p.m.
Adam Williamson wrote:
Maybe this needs to go on the growing pile of reasons why the
traditional Linux model *does* need to go away. Maybe Fedora, with its
foundation of First, should be kind of at the forefront of making that
happen.
Switching to a container-based model is just going to introduce more
different library versions (in the worst case, one per container) with a
higher probability that one of them is compromised.
Kevin Kofler
5:56 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
> Maybe this needs to go on the growing pile of reasons why the
> traditional Linux model *does* need to go away. Maybe Fedora, with its
> foundation of First, should be kind of at the forefront of making that
> happen.
Switching to a container-based model is just going to introduce more
different library versions (in the worst case, one per container) with a
higher probability that one of them is compromised.
Our traditional distro model is not perfect — far from it — and we
certainly try to improve it. But I agree with Kevin that in _this
particular case_, the other models have smaller chances of catching
the issue.
Here the upstream was compromised, so 2FA, upstream signatures, and any
other checks don't help at all.
But in our "traditional model" we have one version of the dependency
used for everbody, so there is a strong incentive to review and
improve this one particular version. The packaging process is also
very open: it is absolutely routine for people to change packagaging
for packages owned by other maintainers.
The newfangled models are much more about picking particular versions
of dependencies and duplicating them in multiple projects. This makes
some things easier, and makes things more independent, but I think
it'd make the xz bug less likely to be caught. If sshd was packaged as
a container or a flatpak, and I saw that it takes .8 instead of .1
seconds to log in, I certainly wouldn't spend the time to figure out
why. I'd assume that the authors did something strange and move on to
my own things.
We talk a lot about the "new ways", but software must still come from
somewhere, and the dependencies need to be maintained… Changing the
delivery format is not going to magically makes this unnecessary.
Zbyszek
11:21 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, 2024-04-01 at 10:56 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel
wrote:
> Adam Williamson wrote:
> > Maybe this needs to go on the growing pile of reasons why the
> > traditional Linux model *does* need to go away. Maybe Fedora, with its
> > foundation of First, should be kind of at the forefront of making that
> > happen.
>
> Switching to a container-based model is just going to introduce more
> different library versions (in the worst case, one per container) with a
> higher probability that one of them is compromised.
Our traditional distro model is not perfect — far from it — and we
certainly try to improve it. But I agree with Kevin that in _this
particular case_, the other models have smaller chances of catching
the issue.
Here the upstream was compromised, so 2FA, upstream signatures, and any
other checks don't help at all.
Yes, to be clear, my "this" was not "the specific technical details of
this attack". It was more:
i) the factors I listed in my email about just how many people are
trusted to build 'Fedora', when 'Fedora' is essentially a collection of
arbitrary scripts executed as root
ii) the fact that this attack reinforces the painful truth that
sophisticated attackers *are* extremely interested in attacking the
supply chain of which we form a significant component
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
11:27 a.m.
On Mon, Apr 1, 2024 at 12:22 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Mon, 2024-04-01 at 10:56 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel wrote:
> > Adam Williamson wrote:
> > > Maybe this needs to go on the growing pile of reasons why the
> > > traditional Linux model *does* need to go away. Maybe Fedora, with its
> > > foundation of First, should be kind of at the forefront of making that
> > > happen.
> >
> > Switching to a container-based model is just going to introduce more
> > different library versions (in the worst case, one per container) with a
> > higher probability that one of them is compromised.
>
> Our traditional distro model is not perfect — far from it — and we
> certainly try to improve it. But I agree with Kevin that in _this
> particular case_, the other models have smaller chances of catching
> the issue.
>
> Here the upstream was compromised, so 2FA, upstream signatures, and any
> other checks don't help at all.
Yes, to be clear, my "this" was not "the specific technical details of
this attack". It was more:
i) the factors I listed in my email about just how many people are
trusted to build 'Fedora', when 'Fedora' is essentially a collection of
arbitrary scripts executed as root
ii) the fact that this attack reinforces the painful truth that
sophisticated attackers *are* extremely interested in attacking the
supply chain of which we form a significant component
Can we please reframe it for what it actually is? This is an attack on
open source communities. "Supply chain" implies a lot of things that
simply don't exist in open source development. Almost the entirety of
the sophistication of the attack was social engineering, not technical
engineering. There *are* technical things to improve, for sure, but
let's not try to make it sound like it's a wholly technical thing that
can be solved with technical solutions exclusively. There are people
and community problems that need addressing too.
--
真実はいつも一つ！/ Always, there's only one truth!
11:41 a.m.
New subject: What we mean when we talk about "supply chains" [was Re: Three
steps we could take to make supply chain attacks a bit harder]
On Mon, 2024-04-01 at 12:27 -0400, Neal Gompa wrote:
>
> ii) the fact that this attack reinforces the painful truth that
> sophisticated attackers *are* extremely interested in attacking the
> supply chain of which we form a significant component
Can we please reframe it for what it actually is? This is an attack on
open source communities. "Supply chain" implies a lot of things that
simply don't exist in open source development. Almost the entirety of
the sophistication of the attack was social engineering, not technical
engineering. There *are* technical things to improve, for sure, but
let's not try to make it sound like it's a wholly technical thing that
can be solved with technical solutions exclusively. There are people
and community problems that need addressing too.
This feels like a derail, so splitting it into a separate subthread.
Honestly, I don't see how the first part of your paragraph relates to
the second. I agree with a lot of the second part, but not the first.
I think we *are* part of a supply chain, regardless of any handwaving
about The Open Source Model. If you are part of producing stuff that
people use to do Real Life Stuff, you are part of a supply chain. You
might want to disclaim various responsibilities for various reasons,
but you still are. If you don't want to be part of a supply chain, stop
supplying stuff. I get the argument that there's a difference between
putting a plank over a stream with a CROSS AT YOUR OWN RISK sign and
charging people to cross your toll bridge, but there are *also* some
similarities.
I agree that the social engineering aspects of this attack were very
significant (though I disagree that was "almost the entirety of the
sophistication" - the technical elements were also pretty
sophisticated). But I don't see why that leads to bikeshedding about
whether this is a "supply chain" attack or not. Why is a "social
engineering attack" not a supply chain attack, but a "technical attack"
is?
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net
12:07 p.m.
New subject: What we mean when we talk about "supply chains" [was Re: Three
steps we could take to make supply chain attacks a bit harder]
On Mon, Apr 1, 2024 at 4:42 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
I think we *are* part of a supply chain, regardless of any
handwaving
about The Open Source Model.
And, more importantly, the industry has agreed
to use the term supply chain. Is the term
perhaps overloaded, or perhaps too
ill-defined/imprecise? Sure. But if one wants
to use a different term one would need to work
across the industry to change the term, and
that is not going to happen.
5:25 p.m.
New subject: What we mean when we talk about "supply chains" [was Re: Three steps we could take to make supply chain attacks a bit harder]
Gary Buhrmaster wrote:
And, more importantly, the industry has agreed
to use the term supply chain. Is the term
perhaps overloaded, or perhaps too
ill-defined/imprecise? Sure. But if one wants
to use a different term one would need to work
across the industry to change the term, and
that is not going to happen.
Well, one could argue that Free Software is a community, not an industry, so
"the industry" cannot agree on anything, and "supply chain" as an
industrial
term obviously does not apply. And also that it is called "Free Software"
and not "Open Source". :-)
Kevin Kofler
11:10 a.m.
New subject: What we mean when we talk about "supply chains" [was Re: Three steps we
could take to make supply chain attacks a bit harder]
Hello,
I have been deleting most of these emails, but I feel like this is a bit
myopic.
On Tuesday, April 2, 2024 6:25:56 PM EDT Kevin Kofler via devel wrote:
Gary Buhrmaster wrote:
> And, more importantly, the industry has agreed
> to use the term supply chain. Is the term
> perhaps overloaded, or perhaps too
> ill-defined/imprecise? Sure. But if one wants
> to use a different term one would need to work
> across the industry to change the term, and
> that is not going to happen.
Well, one could argue that Free Software is a community, not an industry,
so "the industry" cannot agree on anything, and "supply chain" as
an
industrial term obviously does not apply.
But it does. The term "supply chain" refers to the process of acquiring,
organizing, and distributing the necessary resources and components to build
and maintain the distribution. This includes acquiring source code from
various upstream projects, integrating them into the distribution, and then
packaging and distributing the final product to end-users. The supply chain
involves a variety of tasks such as software development, testing, quality
assurance, responding to bug reports, documentation, and support. IOW, it is
a process.
It is the community that carries out the process.
-Steve
8:50 a.m.
On Sun, Mar 31, 2024 at 8:58 AM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
1. We *still don't have compulsory 2FA for Fedora packagers*. We
*still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
What is the status of the FIDO2 implementation in the
authentication processes? Whenever 2FA comes
up I agree with the principal, but ask for an update
on FIDO2. Last I knew, it was still a under-resourced
WIP.
10:10 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Going in that same route, if 2FA becomes mandatory, then we have a
stronger defense of the GPG public key in the user profile. The attacker
would need not only the credentials, but the 2FA device to change the
public GPG.
That then makes one step further possible: enforce commit signing on the
downstream project. That makes authorship of changes non repudiateable
(that is, important changes like `sources`, all patches, specs, etc...),
and also allows the build to check that all commits between the last
release and the current build are signed by the committers of the
project. This can be done by having the build importing the public GPGs
of the committers of the downstream project from pagure/FAS and
checking, again, the commits since the last release. The downstream
project *source code* would be effectively signed, so the signed rpm
then would carry even a higher assurance: "this artifact was built by
Fedora Infrastructure", but also "all the downstream sources used to
make it are assuredly coming from the maintainers."
It is definitely an additional hassle for the maintainers, but there is
value in it, especially in this day and age.
On 3/31/24 01:58, Adam Williamson wrote:
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
> I'm not pretending these will solve everything, but they should make
> attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some
of the grandiose ideas in later posts than Richard's list...I think
we're in danger of building castles in the sky while not cleaning up
the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world
off autotools or banning binaries in repos or centralized source depots
authenticated by a committee of Top People, can we remember:
1. We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
COMPULSORY 2FA FOR FEDORA PACKAGERS*.
2. Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the
packager FAS group. Any one of those people - or someone who
compromises any one of those 140 accounts - can grant any other person
on earth Fedora packager status. Our policy on how they should do this
is
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
. The words "trust" and "identity" do not appear in it. There is,
AFAIK, no policy or procedure by which inactive sponsors have this
power removed. There is no mandatory 2FA policy for sponsors.
3. We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder,
*we are a project that allows 1,601 minimally-vetted people to deliver
arbitrary code executed as root on hundreds of thousands of systems*,
and this mechanism allows any one of those people to cause the package
they have complete control over to be automatically pulled in as a
dependency on virtually every single one of those systems.
4. Our main auth system was written years ago by someone who no longer
contributes and nobody is really actively maintaining it[0].
These are just the ones that leap to my tired mind at this moment. I'm
sure we can think of many more things we should probably look at before
we start pontificating (or, worse, lecturing) about how things should
be done upstream of us.
[0]: https://pagure.io/ipsilon/commits/master
10:27 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
BTW, adding comments to myself, the signed commits and their
verification takes care of problems 2FA doesn't. First, ensure
authorship information (2FA doesn't leave a mark in the commit), and
protects the integrity of the downstream source: the build can ensure
that the checked out downstream project doesn't get "dirty" during and
after the build, and/or commits are not modified during and after the build.
But this doesn't solve the specific xz problem. That one requires
different kinds of defense techniques.
On 3/31/24 08:10, Carlos Rodriguez-Fernandez wrote:
Going in that same route, if 2FA becomes mandatory, then we have a
stronger defense of the GPG public key in the user profile. The attacker
would need not only the credentials, but the 2FA device to change the
public GPG.
That then makes one step further possible: enforce commit signing on the
downstream project. That makes authorship of changes non repudiateable
(that is, important changes like `sources`, all patches, specs, etc...),
and also allows the build to check that all commits between the last
release and the current build are signed by the committers of the
project. This can be done by having the build importing the public GPGs
of the committers of the downstream project from pagure/FAS and
checking, again, the commits since the last release. The downstream
project *source code* would be effectively signed, so the signed rpm
then would carry even a higher assurance: "this artifact was built by
Fedora Infrastructure", but also "all the downstream sources used to
make it are assuredly coming from the maintainers."
It is definitely an additional hassle for the maintainers, but there is
value in it, especially in this day and age.
On 3/31/24 01:58, Adam Williamson wrote:
> On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
>> I'm not pretending these will solve everything, but they should make
>> attacks a little harder in future.
>
> I don't disagree with Richard's list. However...more in regards to some
> of the grandiose ideas in later posts than Richard's list...I think
> we're in danger of building castles in the sky while not cleaning up
> the poop in our backyard, here.
>
> Before we start in on the grand fantasies about converting the world
> off autotools or banning binaries in repos or centralized source depots
> authenticated by a committee of Top People, can we remember:
>
> 1. We *still don't have compulsory 2FA for Fedora packagers*. We *still
> don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE
> COMPULSORY 2FA FOR FEDORA PACKAGERS*.
>
> 2. Our process for vetting packagers is, let's face it, from a security
> perspective almost *comically* patchy. There are 140 sponsors in the
> packager FAS group. Any one of those people - or someone who
> compromises any one of those 140 accounts - can grant any other person
> on earth Fedora packager status. Our policy on how they should do this
> is
>
https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a...
> . The words "trust" and "identity" do not appear in it. There
is,
> AFAIK, no policy or procedure by which inactive sponsors have this
> power removed. There is no mandatory 2FA policy for sponsors.
>
> 3. We have no mechanism to flag when J. Random Packager adds
> "Supplements: glibc" to their random leaf node package. As a reminder,
> *we are a project that allows 1,601 minimally-vetted people to deliver
> arbitrary code executed as root on hundreds of thousands of systems*,
> and this mechanism allows any one of those people to cause the package
> they have complete control over to be automatically pulled in as a
> dependency on virtually every single one of those systems.
>
> 4. Our main auth system was written years ago by someone who no longer
> contributes and nobody is really actively maintaining it[0].
>
> These are just the ones that leap to my tired mind at this moment. I'm
> sure we can think of many more things we should probably look at before
> we start pontificating (or, worse, lecturing) about how things should
> be done upstream of us.
>
> [0]: https://pagure.io/ipsilon/commits/master
7:41 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Sat, Mar 30, 2024 at 09:37:44AM +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should
make
attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
I've noted many times before how inconsistent it is that many
of us will happily mock a website saying "curl <url> | sh -",
and then with a straight face tell users to download a tarball
and run the 1 MB auto-generated 'configure' shell script that
is obfuscated to the point of unreadability by autotools. It
is the perfect place to hide malicious code in tarballs.
For most projects, just running "autoreconf -fiv" is
enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
This is challenging to address too, since often the 'm4/' directory
contains a mixture of application provided m4 files, and autotools
provided m4 files. You can't blindly purge the m4/ directory. They
could have provided a "m4/my-build-to-host.m4" file that interferes
or overides the main 'm4/build-to-host.m4'. So even if autoreconf
did regenerate that file as desired, you're still not entirely
safe.
Still, it would be good practice to have a way to purge any files
that are known to be autoconf provided, so at least you know which
remaining files to look at for threats / problems.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib.
It's extremely convoluted and almost no one understands how it really
works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about
porting to non-Linux platforms. There are better ways to do this.
Be honest about which platforms you genuinely need portability
too in the modern world. Cull obsolete / dead platforms.
While it is a fun intellectual challenge to write portable code
against POSIX, most projects are better off not reinventing the
wheel. Something like GLib takes out a large part of the portability
pain if you really must stick with developing C, and is easy to
adopt incrementally. This is what let libvirt cull its use of
gnulib, which in turn lets us cull the use of autotools
In the xz case it was a gnulib-derived script which was modified to
do
the initial injection (original:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
These are just my thoughts on a Saturday morning. Feedback welcome of
course.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:02 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Regarding downstream defense, prohibiting blobs or differences between
tars and git repos may be overkill or difficult at this moment, but a
tool to assist maintainers in the identification and analysis of these
situations where attacks may be hidden into can be very helpful.
Regarding the latter, where you want to find diff between tars and git
repo content, someone may say "let's just use the github API to pull the
tar", but not all projects use github, or have that feature. So, I still
think there is value in these tools.
I started working on a PoC personal minitool named rpmseclint that
utilizes the "VCS" tag to pull the git repo, and the tar, and compares
and flags things for further review. Right now, I'm just focused on
blobs, and files/directories differences. The intention is to use it at
least right before a rebase. I'm just planning to personally use it to
get a "feel" for it (and help myself), but open to anyone wanting to
explore this idea.
In general, I think the idea of codifying security practices expertise
in tools to assist maintainers with detection and analysis has great
value; even if it is not enforcing anything at the moment. The
maintainer could then explicitly add found problems to an "ignore" list
after the analysis, and explain the "why" in the git commit body
message, and perpetuate the reasoning in the git history of the
downstream project. There is already a lot of best practices codified in
the gating pipelines, and I have found them extremely useful myself.
Regards,
Carlos R.F.
On 3/30/24 02:37, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should
make
attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib.
It's extremely convoluted and almost no one understands how it really
works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about
porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do
the initial injection (original:
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
These are just my thoughts on a Saturday morning. Feedback welcome of
course.
Rich.
10:32 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Definitely this attack leveraged places where eyes don't look:
distributed tar.gz and blobs.
I put the PoC to flag those two in github[1]
Example output:
$ ./rpmseclint tests/rpmseclint-test.spec
-----Diff-----
~ test.txt
+ additional.txt
+ blob.txt.gz
-----Blobs----
application/gzip blob.txt.gz
Now, against the PoC itself, the diff part can be totally avoided by
using generated tars. Not all projects are in github, but those that
do... The other way is by using source-git?
Regarding the blobs part, ... an attack could still manage to fool
`file` with a crafted binary. However, that would still increase the
attack complexity.
[1] https://github.com/carlosrodfern/rpmseclint
On 3/31/24 19:02, Carlos Rodriguez-Fernandez wrote:
Regarding downstream defense, prohibiting blobs or differences between
tars and git repos may be overkill or difficult at this moment, but a
tool to assist maintainers in the identification and analysis of these
situations where attacks may be hidden into can be very helpful.
Regarding the latter, where you want to find diff between tars and git
repo content, someone may say "let's just use the github API to pull the
tar", but not all projects use github, or have that feature. So, I still
think there is value in these tools.
I started working on a PoC personal minitool named rpmseclint that
utilizes the "VCS" tag to pull the git repo, and the tar, and compares
and flags things for further review. Right now, I'm just focused on
blobs, and files/directories differences. The intention is to use it at
least right before a rebase. I'm just planning to personally use it to
get a "feel" for it (and help myself), but open to anyone wanting to
explore this idea.
In general, I think the idea of codifying security practices expertise
in tools to assist maintainers with detection and analysis has great
value; even if it is not enforcing anything at the moment. The
maintainer could then explicitly add found problems to an "ignore" list
after the analysis, and explain the "why" in the git commit body
message, and perpetuate the reasoning in the git history of the
downstream project. There is already a lot of best practices codified in
the gating pipelines, and I have found them extremely useful myself.
Regards,
Carlos R.F.
On 3/30/24 02:37, Richard W.M. Jones wrote:
> I'm not pretending these will solve everything, but they should make
> attacks a little harder in future.
>
>
> (1) We should routinely delete autoconf-generated cruft from upstream
> projects and regenerate it in %prep. It is easier to study the real
> source rather than dig through the convoluted, generated shell script
> in an upstream './configure' looking for back doors.
>
> For most projects, just running "autoreconf -fiv" is enough.
>
> Yes, there are some projects that depend on a specific or old version
> of autoconf. We should fix those. But that doesn't need to delay us
> from using autoreconf on many projects today.
>
> In the xz case this wouldn't have been enough, it turns out we would
> also have to delete m4/build-to-host.m4, which then autoreconf
> regenerates. I don't fully understand why that is.
>
>
> (2) We should discourage gnulib as much as possible.
>
> In libvirt we took the decision a few years ago to remove gnulib.
> It's extremely convoluted and almost no one understands how it really
> works. It's written in obscure m4 macros and shell script.
>
> It's also not necessary for Linux since gnulib is mainly about
> porting to non-Linux platforms. There are better ways to do this.
>
> In the xz case it was a gnulib-derived script which was modified to do
> the initial injection (original:
>
https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-ho...).
>
>
> (3) We should have a "security path", like "critical path".
>
> sshd is linked to a lot of libraries:
>
> /lib64/libaudit.so.1 audit-libs
> /lib64/libc.so.6 glibc
> /lib64/libcap-ng.so.0 libcap-ng
> /lib64/libcap.so.2 libcap
> /lib64/libcom_err.so.2 libcom_err
> /lib64/libcrypt.so.2 libxcrypt
> /lib64/libcrypto.so.3 openssl-libs
> /lib64/libeconf.so.0 libeconf
> /lib64/libgcc_s.so.1 libgcc
> /lib64/libgssapi_krb5.so.2 krb5-libs
> /lib64/libk5crypto.so.3 krb5-libs
> /lib64/libkeyutils.so.1 keyutils-libs
> /lib64/libkrb5.so.3 krb5-libs
> /lib64/libkrb5support.so.0 krb5-libs
> /lib64/liblz4.so.1 lz4-libs
> /lib64/liblzma.so.5 xz-libs
> /lib64/libm.so.6 glibc
> /lib64/libpam.so.0 pam-libs
> /lib64/libpcre2-8.so.0 pcre2
> /lib64/libresolv.so.2 glibc
> /lib64/libselinux.so.1 libselinux
> /lib64/libsystemd.so.0 systemd-libs
> /lib64/libz.so.1 zlib / zlib-ng
> /lib64/libzstd.so.1 zstd
>
> Should we have a higher level of attention to these packages? We
> already have "critical path", but that's a broad category now. These
> seem like they are "security path" packages, an intentionally small
> subset associated with very secure services which are enabled by
> default.
>
>
> These are just my thoughts on a Saturday morning. Feedback welcome of
> course.
>
> Rich.
>
12:36 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
(3) We should have a "security path", like "critical
path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
I agree, but that brings us to the question of what to do about them
that's special.
Unrelated to the idea that some packages are special in this way, it's
probably worth writing some static analysis tools we could put into
rpm-inspect to detect when (a) a binary grows new public keys it didn't
have before, and (b) a shared object grows a new ifunc. The latter is
dramatically easier, of course, but both of those should be pretty rare
events, so they're worth further inspection.
Even if it's just RSA keys that we search for, that would add some
benefit, and that's pretty easy if nobody has tried to cover their
tracks: scan a binary for a big power of two sized odd number followed
by a small prime number, and then filtering that with a more rigorous
prime test on the first number will detect RSA keys and probably very
little else. Might be worth grepping for "----- BEGIN" as well.
Just some thoughts, I'm sure we'll all have many more where these come
from.
--
Peter
4:31 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Mon, Apr 01, 2024 at 01:36:48PM -0400, Peter Jones wrote:
Unrelated to the idea that some packages are special in this way,
it's
probably worth writing some static analysis tools we could put into
rpm-inspect to detect when (a) a binary grows new public keys it didn't
have before, and (b) a shared object grows a new ifunc. The latter is
dramatically easier, of course, but both of those should be pretty rare
events, so they're worth further inspection.
I don't see much difference between ifuncs and any other library
constructors from the exploit POV, at least with DT_BIND_NOW both is just
some extra code run during library initialization. Sure, ifunc handlers
affect what ifunc target is later called whenever calling the ifunc
function, but harm can be done already when loading the library or the
library constructor could overwrite function pointers, vtable pointers or
just some data in the library to change its behavior later.
So, in addition to watching for new ifuncs (and more importantly, trying to
figure out if it is possible to statically prove the set of possible
functions it will resolve to and compare to the old set; and if it isn't
possible to statically figure out list of possible targets flag it for more
careful inspection) we should watch for additions of
__attribute__((constructor)) code or C++ namespace scope non-trivial ctors or
dynamic initializers of namespace scope variables.
Jakub
12:40 a.m.
On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
These are just my thoughts on a Saturday morning. Feedback welcome
of
course.
I find the use of the ifunc attribute is really uncommon at this place. I
would expect it in ffmpeg or some media codecs. In xz it looks like it is only
there to hook in the payload. The software I know normally uses target
cloning.
I think the use of the ifunc attribute should be a red flag. Can't we check
for it with rpmlint and let the security team verify it?
Best regards
Andreas
2:52 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 07:40:33AM +0200, Andreas Schneider wrote:
On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
> These are just my thoughts on a Saturday morning. Feedback welcome of
> course.
I find the use of the ifunc attribute is really uncommon at this place. I
would expect it in ffmpeg or some media codecs. In xz it looks like it is only
there to hook in the payload. The software I know normally uses target
cloning.
In hindsight it's suspicious, but it's not generally suspicious for a
project that needs to generate optimal code for different
sub-architectures (eg. something that does fast decompression) to use
the mechanism for that purpose, ifunc.
That said, ifunc is a very complicated, fragile but powerful mechanism
and I'd like to know from the glibc developers what we should
look out for. For example:
- Is it ever valid for ifunc to take control of functions in another
library? Can this be detected by ld.so?
- Can some wrappers be developed to make it both easier and safer?
I think the use of the ifunc attribute should be a red flag.
Can't we check
for it with rpmlint and let the security team verify it?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
3:22 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
* Richard W. M. Jones:
That said, ifunc is a very complicated, fragile but powerful
mechanism
and I'd like to know from the glibc developers what we should
look out for. For example:
- Is it ever valid for ifunc to take control of functions in another
library? Can this be detected by ld.so?
I'm not sure what you mean by “take control”. The IFUNC resolver
returns an address that is used as the symbol value, for the specific
symbol the IFUNC resolver was associated with at build time. It's not
expected to write to ld.so? data structures directly.
I have patches which make key ld.so? data structures read-only, but they
still have to be read-write during relocation processing, when IFUNC
resolvers run. Under the defend-against-application-bugs model, that's
a reasonable trade-off because IFUNC resolvers are unlikely to have
vulnerabilities (currently, they cannot even access environment
variables reliably). Of course, it doesn't work for IFUNC resolvers
written with malicious intent. I don't think there is a reasonable
possibility to defend against that because any ld.so? defense we create
is out there for everyone to see, so they can work around that.
- Can some wrappers be developed to make it both easier and safer?
GCC already provides function multi-versioning/target clones as a
higher-level interface.
Thanks,
Florian
10:09 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
Am 02.04.24 um 10:22 schrieb Florian Weimer:
> - Can some wrappers be developed to make it both easier and
safer?
GCC already provides function multi-versioning/target clones as a
higher-level interface.
Also, upstreams should by default properly mark their stuffs with
restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean
that one can't choose the better option. So, by default one should
choose -fvisibility=hidden and mark the public API with
__attribute__((visibility("protected"))) or, if they really want a
function to be interpositionable (by e.g. LD_PRELOAD) as
__attribute__((visibility("default"))).
As a side effect, if you ever want your library be usable on Windows,
you need to do that anyway since hidden is the default there and your
public API must be marked explicitly. (Also, Windows doesn't support
interposition and also doesn't support cyclic library dependencies
without complicated hacks. So yeah, Windows kinda has the better
defaults here.)
Some newer languages do that anyway already, but we obviously can't just
change it for C and C++ projects.
But depending on the architecture this may not necessarily be possible.
So yeah, only upstream can do that, not us.
Regards
Kilian Hanich
10:25 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 05:09:18PM +0200, Kilian Hanich via devel wrote:
Am 02.04.24 um 10:22 schrieb Florian Weimer:
>> - Can some wrappers be developed to make it both easier and safer?
>GCC already provides function multi-versioning/target clones as a
>higher-level interface.
Also, upstreams should by default properly mark their stuffs with
restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean
that one can't choose the better option. So, by default one should
choose -fvisibility=hidden and mark the public API with
__attribute__((visibility("protected"))) or, if they really want a
function to be interpositionable (by e.g. LD_PRELOAD) as
__attribute__((visibility("default"))).
ISTR this also makes the library faster (faster loading I think?)
Anyway we've done it for all the virt libraries for years.
Rich.
As a side effect, if you ever want your library be usable on
Windows,
you need to do that anyway since hidden is the default there and your
public API must be marked explicitly. (Also, Windows doesn't support
interposition and also doesn't support cyclic library dependencies
without complicated hacks. So yeah, Windows kinda has the better
defaults here.)
Some newer languages do that anyway already, but we obviously can't just
change it for C and C++ projects.
But depending on the architecture this may not necessarily be possible.
So yeah, only upstream can do that, not us.
Regards
Kilian Hanich
--
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
11:03 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
* Kilian Hanich via devel:
Am 02.04.24 um 10:22 schrieb Florian Weimer:
>> - Can some wrappers be developed to make it both easier and safer?
> GCC already provides function multi-versioning/target clones as a
> higher-level interface.
Also, upstreams should by default properly mark their stuffs with
restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean
that one can't choose the better option. So, by default one should
choose -fvisibility=hidden and mark the public API with
__attribute__((visibility("protected"))) or, if they really want a
function to be interpositionable (by e.g. LD_PRELOAD) as
__attribute__((visibility("default"))).
I think protected visibility is still broken on x86-64. For function
symbols, it's more convenient to use -fno-semantic-interposition and
rely on LTO to do the heavy lifting. For data symbols, it may be a
better longer term investment to add explicit export lists (maybe even
with symbol versioning), and again rely on LTO to make everything else
hidden.
Thanks,
Florian
3:30 a.m.
On Tuesday, 2 April 2024 09:52:38 CEST Richard W.M. Jones wrote:
On Tue, Apr 02, 2024 at 07:40:33AM +0200, Andreas Schneider wrote:
> On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
> > These are just my thoughts on a Saturday morning. Feedback welcome of
> > course.
>
> I find the use of the ifunc attribute is really uncommon at this place. I
> would expect it in ffmpeg or some media codecs. In xz it looks like it is
> only there to hook in the payload. The software I know normally uses
> target cloning.
In hindsight it's suspicious, but it's not generally suspicious for a
project that needs to generate optimal code for different
sub-architectures (eg. something that does fast decompression) to use
the mechanism for that purpose, ifunc.
That said, ifunc is a very complicated, fragile but powerful mechanism
and I'd like to know from the glibc developers what we should
look out for. For example:
- Is it ever valid for ifunc to take control of functions in another
library? Can this be detected by ld.so?
- Can some wrappers be developed to make it both easier and safer?
Well, if it would do that. I took a quick look at xz and didn't see any
specific code for an architecture flavor like x86_64-v3 or avx related. It
lacks the implementation for that. All it did was adding the infrastructure
without using it. I guess that the use of ifunc would is still be very rare.
Target clones is what you normally see.
3:59 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
* Richard W. M. Jones:
I'm not pretending these will solve everything, but they should
make
attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream
projects and regenerate it in %prep. It is easier to study the real
source rather than dig through the convoluted, generated shell script
in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version
of autoconf. We should fix those. But that doesn't need to delay us
from using autoreconf on many projects today.
Not shipping the m4 files and other artifacts required for regenerating
autoconf scripts is not exactly rare, unfortunately. I have filed a
bunch of bugs because it's my understanding that this incomplete source
code is against Fedora policies, but in the end, there isn't much we can
do about it.
But I sympathize with this approach, we should build from sources as
much as we can. Maybe not regenerate everything in %prep though, this
really belongs into %build. It's invoking a compiler, after all.
In the xz case this wouldn't have been enough, it turns out we
would
also have to delete m4/build-to-host.m4, which then autoreconf
regenerates. I don't fully understand why that is.
I would expect that's what the serial number is for? But that's just a
guess.
(2) We should discourage gnulib as much as possible.
Or use Git sources without bundled gnulib, and re-bundled at build time
using gnulib-devel. It would make gnulib updates easier. Such updates
are sometimes required because gnulib overrides glibc internals without
much coordination, and gnulib-using software may fail to build (or run)
until gnulib is updated to reflect the internal glibc changes.
However, while I don't particularly like autoconf and gnulib (they are
mainly there to support systems other than GNU/Linux, most of them
proprietary), I don't expect any steps we take there to stop anyone who
targets Fedora specifically. Being different from other distributions
doesn't help at all if Fedora is the target.
(3) We should have a "security path", like "critical
path".
I think there is a strong overlap with a bootstrap package set that is
occasionally discussed. And this leads to the main point I want to
make:
Stopping an attack that has already happened does not make much sense.
Given that we work in the open, preventing future attacks that target
Fedora specifically is likely impossible. Therefore, the focus should
be on builder and buildroot integrity, and generic strategies for
restoring buildroot integrity after an incident. In this particular
case, so far we considered it sufficient to merely downgrade one
package. But the package in question was part of the default buildroot.
That means that in theory, it could have persisted itself outside the
confines of its own package.
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs
/lib64/libc.so.6 glibc
/lib64/libcap-ng.so.0 libcap-ng
/lib64/libcap.so.2 libcap
/lib64/libcom_err.so.2 libcom_err
/lib64/libcrypt.so.2 libxcrypt
/lib64/libcrypto.so.3 openssl-libs
/lib64/libeconf.so.0 libeconf
/lib64/libgcc_s.so.1 libgcc
/lib64/libgssapi_krb5.so.2 krb5-libs
/lib64/libk5crypto.so.3 krb5-libs
/lib64/libkeyutils.so.1 keyutils-libs
/lib64/libkrb5.so.3 krb5-libs
/lib64/libkrb5support.so.0 krb5-libs
/lib64/liblz4.so.1 lz4-libs
/lib64/liblzma.so.5 xz-libs
/lib64/libm.so.6 glibc
/lib64/libpam.so.0 pam-libs
/lib64/libpcre2-8.so.0 pcre2
/lib64/libresolv.so.2 glibc
/lib64/libselinux.so.1 libselinux
/lib64/libsystemd.so.0 systemd-libs
/lib64/libz.so.1 zlib / zlib-ng
/lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We
already have "critical path", but that's a broad category now. These
seem like they are "security path" packages, an intentionally small
subset associated with very secure services which are enabled by
default.
The bootstrap set depends on SWIG, and SWIG pulls in language
implementations which are not bootstrapped from source using the
previous version.
Thanks,
Florian
4:01 a.m.
On Tue, Apr 2, 2024 at 4:59 AM Florian Weimer <fweimer(a)redhat.com> wrote:
* Richard W. M. Jones:
> I'm not pretending these will solve everything, but they should make
> attacks a little harder in future.
>
>
> (1) We should routinely delete autoconf-generated cruft from upstream
> projects and regenerate it in %prep. It is easier to study the real
> source rather than dig through the convoluted, generated shell script
> in an upstream './configure' looking for back doors.
>
> For most projects, just running "autoreconf -fiv" is enough.
>
> Yes, there are some projects that depend on a specific or old version
> of autoconf. We should fix those. But that doesn't need to delay us
> from using autoreconf on many projects today.
Not shipping the m4 files and other artifacts required for regenerating
autoconf scripts is not exactly rare, unfortunately. I have filed a
bunch of bugs because it's my understanding that this incomplete source
code is against Fedora policies, but in the end, there isn't much we can
do about it.
But I sympathize with this approach, we should build from sources as
much as we can. Maybe not regenerate everything in %prep though, this
really belongs into %build. It's invoking a compiler, after all.
We have a %conf stage for this purpose. We should start using it.
--
真実はいつも一つ！/ Always, there's only one truth!
4:27 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Tue, Apr 02, 2024 at 10:59:10AM +0200, Florian Weimer wrote:
* Richard W. M. Jones:
> In the xz case this wouldn't have been enough, it turns out we would
> also have to delete m4/build-to-host.m4, which then autoreconf
> regenerates. I don't fully understand why that is.
I would expect that's what the serial number is for? But that's just a
guess.
Yes, in this case the attacker had set the serial number to 30, but
the latest upstream serial number was 3. autoreconf won't replace the
file in this case unless it is deleted. There really should be an
"always replace if you can" option in autoreconf.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
5:47 p.m.
Richard W.M. Jones wrote:
Yes, in this case the attacker had set the serial number to 30, but
the latest upstream serial number was 3. autoreconf won't replace the
file in this case unless it is deleted. There really should be an
"always replace if you can" option in autoreconf.
Is that not what -f is supposed to do? At least, the documentation claims
so, but the implementation does not actually do what is documented.
Kevin Kofler
10:17 a.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Wed, Apr 03, 2024 at 12:47:39AM +0200, Kevin Kofler via devel wrote:
Richard W.M. Jones wrote:
> Yes, in this case the attacker had set the serial number to 30, but
> the latest upstream serial number was 3. autoreconf won't replace the
> file in this case unless it is deleted. There really should be an
> "always replace if you can" option in autoreconf.
Is that not what -f is supposed to do? At least, the documentation claims
so, but the implementation does not actually do what is documented.
The upstream autoconf discussion says that 'autoreconf -fi' behavior
on which 'serial NN' .m4 files to update is determined by automake,
not autoconf, in part inspired by semantics desired in gnulib. And
the automake and gnulib developers have argued that the upstream
behavior is intentional:
https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
So the only sane way to work around the upstream behavior (if we
insist that working around it is essential, such as if we insist on
running autoreconf in order to pull in modernized config.guess that
will allow us to build on more platforms than were present at the time
the upstream package was cut), is to do something like removing all
.m4 files, then autoreconf -fi to reinstall all of them with known
versions of the same .m4 files that we somehow curate as being more
reliable than the ones that upstream tested their release tarball
with. Lots of busy work, but if we think it will help as another
layer of reassurance to keep our customers happy, then we can sign up
for it.
However, although it is unlikely that upstream automake will accept a
patch to change the existing 'autoreconf -fi' behavior of not
overriding an already-present .m4 with a higher serial, getting some
other patch in that adds a new option such as 'autoreconf
--ignore-serial -fi' might be possible.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
12:48 p.m.
Eric Blake wrote:
The upstream autoconf discussion says that 'autoreconf -fi'
behavior
on which 'serial NN' .m4 files to update is determined by automake,
not autoconf, in part inspired by semantics desired in gnulib. And
the automake and gnulib developers have argued that the upstream
behavior is intentional:
https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
Don't you love intentional bugs? Yet another reason to avoid autotools at
all costs.
By the way, the documentation of the serials "feature" actually warns about
this (and seems to imply that even without serials, --force does not work as
advertised for those files):
https://www.gnu.org/software/automake/manual/html_node/Serials.html
| Finally, note that the --force option of aclocal has absolutely no effect
| on the files installed by --install. For instance, if you have modified
| your local macros, do not expect --install --force to replace the local
| macros by their system-wide versions. If you want to do so, simply erase
| the local macros you want to revert, and run ‘aclocal --install’.
But the documentation of "autoreconf --force" does not include that warning.
And this also makes "--force" pretty much useless as it stands.
We and Debian both need to patch aclocal downstream immediately to make
--force actually work. And then of course Fedora needs to actually always
run autoreconf -i -f as Debian already does, or the patch will not do much
good.
Kevin Kofler
1:37 p.m.
New subject: Three steps we could take to make supply chain attacks a bit
harder
On Wed, Apr 03, 2024 at 07:48:03PM +0200, Kevin Kofler via devel wrote:
Eric Blake wrote:
> The upstream autoconf discussion says that 'autoreconf -fi' behavior
> on which 'serial NN' .m4 files to update is determined by automake,
> not autoconf, in part inspired by semantics desired in gnulib. And
> the automake and gnulib developers have argued that the upstream
> behavior is intentional:
>
> https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
Don't you love intentional bugs? Yet another reason to avoid autotools at
all costs.
By the way, the documentation of the serials "feature" actually warns about
this (and seems to imply that even without serials, --force does not work as
advertised for those files):
https://www.gnu.org/software/automake/manual/html_node/Serials.html
Note - that's in the automake manual (it is the actions taken by aclocal...).
| Finally, note that the --force option of aclocal has absolutely no effect
| on the files installed by --install. For instance, if you have modified
| your local macros, do not expect --install --force to replace the local
| macros by their system-wide versions. If you want to do so, simply erase
| the local macros you want to revert, and run ‘aclocal --install’.
But the documentation of "autoreconf --force" does not include that warning.
And this also makes "--force" pretty much useless as it stands.
...and autoreconf is shipped by autoconf. While the two projects are
related, they are independent (you can use autoconf without automake,
at which point autoreconf --force does update everything that autoconf
touched, since automake's semantics aren't getting in the way; it only
invokes aclocal when it is obvious that automake was also in use).
But I'm sure (speaking as one of the previous upstream autoconf
maintainers, but only as a very infrequent automake patch poster) that
a patch to autoconf's docs to mention the automake pitfalls inherited
from 'aclocal --force' into 'autoreconf --force' is worthwhile.
We and Debian both need to patch aclocal downstream immediately to make
--force actually work. And then of course Fedora needs to actually always
run autoreconf -i -f as Debian already does, or the patch will not do much
good.
Downstream is certainly entitled to have a different idea of best
practices than upstream, especially when downstream is targetting just
one OS, rather than the world of machines out there that the autotools
originally designed for. That's one of the joys of open source - you
don't have to live with upstream decisions.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
20
days inactive
39
days old
209 comments
48 participants
participants (48)
-
Adam Williamson -
Alexander Bokovoy -
Andreas Schneider -
Artem S. Tashkinov -
Arthur Bols -
Ben Beasley -
Carlos Rodriguez Fernandez -
Carlos Rodriguez-Fernandez
-
Chris Adams -
Christoph Erhardt -
Christoph Karl -
Daniel Alley -
Daniel P. Berrangé -
Dmitry Belyavskiy -
Dominique Martinet -
Eric Blake -
Florian Weimer -
François Rigault -
Gabriel Somlo -
Gary Buhrmaster -
Gordon Messmer -
Jakub Jelinek -
Jonathan Steffan -
Kevin Fenzi -
Kevin Kofler -
Kilian Hanich -
Lennart Poettering -
Marcin Juszkiewicz -
Matthew Miller -
Michael Catanzaro -
Michel Lind -
Miroslav Suchý -
Neal Gompa -
Panu Matilainen -
Peter Jones -
Richard W.M. Jones -
Sam Varshavchik -
Scott Schmit -
Simo Sorce -
Simon de Vlieger -
Stephen Gallagher -
Stephen Smoogen -
Steve Cossette -
Steve Grubb -
Tim Landscheidt -
Todd Zullinger -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek