https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary == Fedora users / developers who need to debug/trace distro binaries can make use of the recently activated elfutils-debuginfod servers to automatically fetch debugging data and source code, instead of having to use `# sudo dnf` commands.
== Owner == * Name: [[User:fche| Frank Ch. Eigler]] * Email: fche@redhat.com
== Detailed Description == Numerous fedora debugging-type tools have built-in capabilities to use the [https://sourceware.org/elfutils/Debuginfod.html debuginfod] protocol to fetch debuginfo/source code automatically. We would like to activate a setting so that Fedora's debuginfod servers are automatically used, rather than requiring hand-editing individual users' dot files.
== Feedback ==
There has existed [https://www.spinics.net/lists/fedora-devel/msg279002.html broad interest] in a Fedora debuginfod server since the project was proposed / announced in 2020, and several distros already operate public servers of this nature. Some of the distros configure their installations by default to talk to those servers, some do not.
Turning on this by default has some limited privacy implications. Some Debian users have [https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed concerns] that this facility "calls home" during debugging, so it may expose a limited amount of information about what a user is debugging. The information is limited to the build-id and source code file names of programs being debugged, and is only sent to the servers if their machine lacks locally installed debuginfo. Whether this should be opt-in or opt-out and how has been resolved there via an install-time query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is [https://build.opensuse.org/request/show/879926 simply defaulted-on], and we have heard of no controversy.
We anticipate discussing this topic on the mailing list, and noting it in the release notes either way.
== Benefit to Fedora == This will improve developers' experience.
It may reduce download server burden, as only individual ELF/DWARF/source files are downloaded rather than entire `-debuginfo` and `-debugsource` RPMs.
It would help Fedora catch up to other distros who put up `debuginfod` servers already. :-)
== Scope == * Proposal owners: The work could consist one extra parameter in the `elfutils.spec` `%configure`. Its effect is to arrange for the `elfutils-debuginfod-client` RPM to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS` environment variable automatically to `https://debuginfod.fedoraproject.org/%60. (At the time of this writing, the _staging_ server is getting ready for testing: `https://debuginfod.stg.fedoraproject.org/%60.)
* Other developers: None - relevant code has been previously upstreamed!
* Release engineering: None - our team is operating the `debuginfod[.stg].fedoraproject.org` VMs.
* Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A
== Upgrade/compatibility impact == None.
Note that these servers index all active Fedora releases (32-), all architectures, so users of those versions can already set `DEBUGINFOD_URLS` manually to take advantage.
== How To Test ==
* Install `elfutils-debuginfod-client` * Open arbitrary fedora binary via gdb. ** Admire the immediate downloading of debuginfo and source code. * Run `eu-stack -v -p $pid` for an arbitrary process. ** Admire the immediate downloading of debuginfo to give precise file:line data.
== User Experience == Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual `dnf` commands to install this data.
== Dependencies ==
The `debuginfod` servers at `fedora-infra` need to be up.
== Contingency Plan ==
* Contingency mechanism: change the elfutils-debuginfod-client subrpm to not set the default `DEBUGINFOD_URLS` environment variable for all users. In the case of a server outage, the debugger tools revert to debuginfo-less operation, prior to this feature. * Contingency deadline: shortly before freeze * Blocks release? No
== Documentation == There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling. What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.
== Release Notes ==
TBD.
On Wed, Apr 7, 2021 at 4:33 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary == Fedora users / developers who need to debug/trace distro binaries can make use of the recently activated elfutils-debuginfod servers to automatically fetch debugging data and source code, instead of having to use `# sudo dnf` commands.
== Owner ==
- Name: [[User:fche| Frank Ch. Eigler]]
- Email: fche@redhat.com
== Detailed Description == Numerous fedora debugging-type tools have built-in capabilities to use the [https://sourceware.org/elfutils/Debuginfod.html debuginfod] protocol to fetch debuginfo/source code automatically. We would like to activate a setting so that Fedora's debuginfod servers are automatically used, rather than requiring hand-editing individual users' dot files.
== Feedback ==
There has existed [https://www.spinics.net/lists/fedora-devel/msg279002.html broad interest] in a Fedora debuginfod server since the project was proposed / announced in 2020, and several distros already operate public servers of this nature. Some of the distros configure their installations by default to talk to those servers, some do not.
Turning on this by default has some limited privacy implications. Some Debian users have [https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed concerns] that this facility "calls home" during debugging, so it may expose a limited amount of information about what a user is debugging. The information is limited to the build-id and source code file names of programs being debugged, and is only sent to the servers if their machine lacks locally installed debuginfo. Whether this should be opt-in or opt-out and how has been resolved there via an install-time query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is [https://build.opensuse.org/request/show/879926 simply defaulted-on], and we have heard of no controversy.
We anticipate discussing this topic on the mailing list, and noting it in the release notes either way.
== Benefit to Fedora == This will improve developers' experience.
It may reduce download server burden, as only individual ELF/DWARF/source files are downloaded rather than entire `-debuginfo` and `-debugsource` RPMs.
It would help Fedora catch up to other distros who put up `debuginfod` servers already. :-)
== Scope ==
- Proposal owners:
The work could consist one extra parameter in the `elfutils.spec` `%configure`. Its effect is to arrange for the `elfutils-debuginfod-client` RPM to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS` environment variable automatically to `https://debuginfod.fedoraproject.org/%60. (At the time of this writing, the _staging_ server is getting ready for testing: `https://debuginfod.stg.fedoraproject.org/%60.)
Other developers: None - relevant code has been previously upstreamed!
Release engineering: None - our team is operating the
`debuginfod[.stg].fedoraproject.org` VMs.
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: N/A
== Upgrade/compatibility impact == None.
Note that these servers index all active Fedora releases (32-), all architectures, so users of those versions can already set `DEBUGINFOD_URLS` manually to take advantage.
== How To Test ==
- Install `elfutils-debuginfod-client`
- Open arbitrary fedora binary via gdb.
** Admire the immediate downloading of debuginfo and source code.
- Run `eu-stack -v -p $pid` for an arbitrary process.
** Admire the immediate downloading of debuginfo to give precise file:line data.
== User Experience == Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual `dnf` commands to install this data.
== Dependencies ==
The `debuginfod` servers at `fedora-infra` need to be up.
== Contingency Plan ==
- Contingency mechanism: change the elfutils-debuginfod-client subrpm
to not set the default `DEBUGINFOD_URLS` environment variable for all users. In the case of a server outage, the debugger tools revert to debuginfo-less operation, prior to this feature.
- Contingency deadline: shortly before freeze
- Blocks release? No
== Documentation == There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling. What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.
== Release Notes ==
TBD.
Woohoo! I'm excited for this!
I got a chance to use debuginfod to do some debugging of DNF on openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and everything else!
ngompa13 wrote:
Woohoo! I'm excited for this!
Thanks!
I got a chance to use debuginfod to do some debugging of DNF on openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable that way. (The staging indexer is processing the l* packages, after a few days of churning from 0* through k*. It has a one or two more days to go, but may already be used.)
The purpose of this Change is to configure this environment variable by default, pointing to the forthcoming production version of the server at https://debuginfod.fedoraproject.org/.
- FChE
On Thu, Apr 8, 2021 at 11:20 AM Frank Ch. Eigler fche@redhat.com wrote:
ngompa13 wrote:
Woohoo! I'm excited for this!
Thanks!
I got a chance to use debuginfod to do some debugging of DNF on openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable that way. (The staging indexer is processing the l* packages, after a few days of churning from 0* through k*. It has a one or two more days to go, but may already be used.)
The purpose of this Change is to configure this environment variable by default, pointing to the forthcoming production version of the server at https://debuginfod.fedoraproject.org/.
We should probably have a halfway decent splash page for debuginfod.fedoraproject.org, similar to what the openSUSE one has: https://debuginfod.opensuse.org/
On Thu, Apr 08, 2021 at 11:19:59AM -0400, Frank Ch. Eigler wrote:
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable that way.
Really cool. Thanks for making it happen.
On Fri, Apr 9 2021 at 04:19:31 PM +0200, Lars Seipel ls@slrz.net wrote:
Really cool. Thanks for making it happen.
I started testing the staging server yesterday. It seems a little slow -- I worry for anyone debugging anything that links to WebKit, which on the desktop is a lot -- but otherwise it works *very* well. I'm impressed. Very nice.
mcatanzaro wrote:
I started testing the staging server yesterday. It seems a little slow -- I worry for anyone debugging anything that links to WebKit, which on the desktop is a lot -- but otherwise it works *very* well. I'm impressed. Very nice.
It'd be good to get a sense of what you found slow. The .stg. server is just finishing up ingesting all the koji rpms, so has been fully occupied with that. In the case where a new file is sought from inside some large rpm, yeah there will be some inherent latency in decompressing it (CPU bound), saving the selected file to tmp disk (RAM/storage bound) and then sending you the file (network bound).
At the server, there are some prefetching/caching options available to take advantage of the spacious servers kindly provided by fedora-infra.
There is also aggressive caching on the client side, with an adjustable week-long retention for files. See the CACHE section in [man debuginfod_find_debuginfo]. Plus web caching proxies or federated intermediate debuginfod servers can add site-level caching near you.
- FChE
On Fri, Apr 9, 2021 at 11:01 AM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Fri, Apr 9 2021 at 04:19:31 PM +0200, Lars Seipel ls@slrz.net wrote:
Really cool. Thanks for making it happen.
I started testing the staging server yesterday. It seems a little slow -- I worry for anyone debugging anything that links to WebKit, which on the desktop is a lot -- but otherwise it works *very* well. I'm impressed. Very nice.
Did you notice that it also works for the Fedora Flatpaks (thanks, Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel org.gnome.Aisleriot [📦 org.gnome.Aisleriot ~]$ DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/ DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
- Owen
On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor otaylor@redhat.com wrote:
Did you notice that it also works for the Fedora Flatpaks (thanks, Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel org.gnome.Aisleriot [📦 org.gnome.Aisleriot ~]$ DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/ DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
I think that's OK for a manual debugging workflow, since it's pretty rare to want to do that under flatpak in my experience. Normally what's most important to me is being able to easily generate a backtrace for a previous crash using 'flatpak-coredumpctl'. Ideally 'flatpak-coredumpctl' would handle setting the right environment variables and executing flatpak with the right permissions to make it work. (In the future, ABRT could do something similar.)
On Sat, Apr 10, 2021 at 9:55 AM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor otaylor@redhat.com wrote:
Did you notice that it also works for the Fedora Flatpaks (thanks, Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel org.gnome.Aisleriot [📦 org.gnome.Aisleriot ~]$ DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/ DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
I think that's OK for a manual debugging workflow, since it's pretty rare to want to do that under flatpak in my experience. Normally what's most important to me is being able to easily generate a backtrace for a previous crash using 'flatpak-coredumpctl'. Ideally 'flatpak-coredumpctl' would handle setting the right environment variables and executing flatpak with the right permissions to make it work. (In the future, ABRT could do something similar.)
I think we could store the debuginfo urls in repository metadata (ostree summary / oci json index) and have flatpak automatically set things up for 'flatpak run --devel'. This isn't Fedora specific - e.g. there's an eventual goal to have a debuginfo server for Flathub as well.
For coredump+debuginfod you can actually do the backtrace with a system (or toolbox) gdb instead of downloading the SDK ... if we want to deal with that complexity, we could add a mode like that into flatpak-coredumpctl.
One gap here is that it's a little hard to figure out what exact revision of the Flatpak and runtime coredumped - you can reverse-engineer that out of COREDUMP_PROC_MOUNTINFO information in the journal.
As you say, the eventual goal would be ABRT support.
Regards, Owen
On Sun, Apr 11, 2021 at 1:52 PM Owen Taylor otaylor@redhat.com wrote:
On Sat, Apr 10, 2021 at 9:55 AM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Sat, Apr 10 2021 at 08:03:09 AM -0400, Owen Taylor otaylor@redhat.com wrote:
Did you notice that it also works for the Fedora Flatpaks (thanks, Frank!) - basic proof of concept:
$ flatpak run --command=sh --filesystem=home --share=network --devel org.gnome.Aisleriot [📦 org.gnome.Aisleriot ~]$ DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/ DEBUGINFOD_CACHE_PATH=~/.cache/debuginfod_client gdb /app/bin/sol
(Without the --filesystem=home and DEBUGINFOD_CACHE_PATH, the cache ends up in ~/.var/app/org.gnome.Aisleriot/cache/debuginfod_client/)
I think that's OK for a manual debugging workflow, since it's pretty rare to want to do that under flatpak in my experience. Normally what's most important to me is being able to easily generate a backtrace for a previous crash using 'flatpak-coredumpctl'. Ideally 'flatpak-coredumpctl' would handle setting the right environment variables and executing flatpak with the right permissions to make it work. (In the future, ABRT could do something similar.)
I think we could store the debuginfo urls in repository metadata (ostree summary / oci json index) and have flatpak automatically set things up for 'flatpak run --devel'. This isn't Fedora specific - e.g. there's an eventual goal to have a debuginfo server for Flathub as well.
Filed an upstream pull-request for this (still needs some work):
https://github.com/flatpak/flatpak/pull/4222
- Owen
While trying to collect a backtrace for org.gnome.Tetravex, I got this in gdb:
===
Downloading separate debug info for /lib64/liblzma.so.5... Download failed: Timer expired. Continuing without debug info for /lib64/libbrotlicommon.so.1. Missing separate debuginfo for /lib64/libbrotlicommon.so.1 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/0e/bb3270fdbf40dbe56ea79d6630ac594b897ffe.debug Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1. Missing separate debuginfo for /lib64/libzstd.so.1 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug Downloading separate debug info for /lib64/liblz4.so.1... Download failed: Timer expired. Continuing without debug info for /home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo. Error while reading shared library symbols for /lib64/liblz4.so.1: could not find '.gnu_debugaltlink' file for /home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo Downloading separate debug info for /lib64/libcap.so.2...
===
After running all the "Try" suggestions, the second run of gdb do not complain.
I wanted to know how I can collect the required backtrace so that I can attach it to the bugzilla entry to help developers to looking at the bug.
Thanks a lot!
FUNG Chi Chuen Sampson wrote:
While trying to collect a backtrace for org.gnome.Tetravex, I got this in gdb:
===
Downloading separate debug info for /lib64/liblzma.so.5... Download failed: Timer expired. Continuing without debug info for /lib64/libbrotlicommon.so.1. Missing separate debuginfo for /lib64/libbrotlicommon.so.1 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/0e/bb3270fdbf40dbe56ea79d6630ac594b897ffe.debug Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1. Missing separate debuginfo for /lib64/libzstd.so.1 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug Downloading separate debug info for /lib64/liblz4.so.1... Download failed: Timer expired. Continuing without debug info for /home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo. Error while reading shared library symbols for /lib64/liblz4.so.1: could not find '.gnu_debugaltlink' file for /home/fcc/.var/app/org.gnome.Tetravex/cache/debuginfod_client/a2429c266188acc10181f6936915f35274bb4a38/debuginfo Downloading separate debug info for /lib64/libcap.so.2...
I was wondering what the user experience would be like in such a situation. Could you estimate how long you had to wait in total? Was there a long delay before each "Timer expired" message, or only one delay?
Björn Persson
Björn Persson Bjorn@xn--rombobjrn-67a.se writes:
I was wondering what the user experience would be like in such a situation. Could you estimate how long you had to wait in total? Was there a long delay before each "Timer expired" message, or only one delay?
Each outright-hung request could entail a $DEBUGINFOD_TIMEOUT (default 90s) wait. The common code does not "hold a grudge" against a server that formerly timed out.
- FChE
It is just a short delay then the "Try" suggestion is given.
My first run using my Notebook takes quite some time.
A few hours later, I run again using my Desktop, this time is very fast and no "Try" given.
I have run the same debug session using two different machines.
The first run, giving my "Try"s, takes much longer than the second run, which gives no "Trys".
Just from impression: 1st run: From run to download finish, I will say it takes about 5+ minutes. 2nd run: ~1 minute
For each "Try" given, the delay is not obvious to me.
"Sampson Fung" sampsonfung@gmail.com writes:
The first run, giving my "Try"s, takes much longer than the second run, which gives no "Trys". Just from impression: 1st run: From run to download finish, I will say it takes about 5+ minutes. 2nd run: ~1 minute For each "Try" given, the delay is not obvious to me.
I'm pretty sure Sampson was affected by a proxy.stg.fedoraproject.org misconfiguration problem that was fixed about an hour ago, so those timeouts should not be happening any more. That means we'd be down to normal service latencies ameliorated by caching effects.
A cute demonstration of the cost/benefit of this capability now in the distro, I recently ran
% gdb /usr/bin/gnome-control-center
On a normal machine, you'll get no debuginfo and a suggestion to install a wall-of-text list of RPMs as root.
On a debuginfod configured machine, you'll get gdb downloading ~400MB of debuginfo (HTTP compressed -- decompresses to ~6GB) as rapidly as your network connection allows. This could take some minutes, for the first time. After that time, you get instant visibility into the entire enormous gnome software stack, including LLVM, mesa, x11, samba, gst, opengl, glib, etc. etc. etc. right down to the glibc assembly wrappers for syscalls.
- FChE
Frank Ch. Eigler wrote:
"Sampson Fung" sampsonfung@gmail.com writes:
The first run, giving my "Try"s, takes much longer than the second run, which gives no "Trys". Just from impression: 1st run: From run to download finish, I will say it takes about 5+ minutes. 2nd run: ~1 minute For each "Try" given, the delay is not obvious to me.
I'm pretty sure Sampson was affected by a proxy.stg.fedoraproject.org misconfiguration problem that was fixed about an hour ago, so those timeouts should not be happening any more.
It is however a good illustration of how a network problem can destroy the user experience. Five minutes is a long wait. I'm glad that we now have this information.
Björn Persson
Björn Persson wrote on Thu, Apr 22, 2021 at 04:26:22PM +0200:
It is however a good illustration of how a network problem can destroy the user experience. Five minutes is a long wait. I'm glad that we now have this information.
This is an old post but I just used debuginfod for the first time (on debian actually (debuginfod 0.183-1, gdb 10.1-1.7) in case things have changed - forgive me and please correct me if anything I've said is significantly better on current version on rawhide), so figured I'd share...
- My usecase admitedly wasn't a very nice one (graphical application, info dll says I have 265 linked libraries...), but it felt very slow. Looking at iftop the server was reliably sending me things at 1mbps which isn't great but with the amount of data involved being slow can't be helped, I guess. What makes it really bad is that such applications link extra libraries at runtime, so when I thought it was finally done and it got into the breakpoints I had set, continuing a bit just ran into another wall of downloads, but this also isn't debuginfod's wrongdoings.
Most of these libraries are nothing I care about. I understand gdb tries to find debug infos for libraries it encounters when these are loaded into the program's address space, but could it be possible to not download anything at this point and only download debuginfos when they are really used (the first time an address in such libraries appear in a backtrace to get line number / local variables / etc) ?
I realize this isn't obvious, but it'd make my usage much more appreciable -- downloading 4 or maybe 5 libraries for the area I'm debugging in that monster instead of the full 265 of them.
- None of ^C, ^, ^Z work, when I grew tired of waiting I had to switch to another terminal and kill the gdb process.
It'd be great if e.g. ^C could skip the current download (or future downloads too?) and continue debugging, or enter the (gdb) prompt like it normally does -- iirc it went into that prompt after the download? or batch of downloads I'm not sure, I just remember it took too long.
- unsetting DEBUGINFOD_URLS completely disables the debuginfod client, it would have been nice to use the data available in the client cache anyway. Ultimately I set it to localhost so debuginfod would give up immediately on new requests but still use previously downloaded data. Perhaps there could be an "offline mode" of sorts?
I also agree with the security concerns given, debuginfos are usually seen as trusted data and I wouldn't be surprised new bugs get found in their parsing which could compromise developers since the server can send whatever it wants to send. I'm not going to be very helpful on that respect though as I don't have any bright idea (distributing just checksums of all debug files in the main package, so when debuginfod downloads something it can compare with the checksum that was installed and verified locally by signed rpm? Didn't reread the whole thread but I think something similar was suggested), but that topic didn't get much discussion and it sounds important to me.
All in all I think it's a very valuable tool, but it would be really great if we could minimize the amount of data actually downloaded, and establish some chain of trust.
Thanks,
Hi -
asmadeus wrote:
- My usecase admitedly wasn't a very nice one (graphical application,
info dll says I have 265 linked libraries...), but it felt very slow.
Yes, a first-time download series for such a program is going to take time. (Afterwards, it's cached.)
Looking at iftop the server was reliably sending me things at 1mbps which isn't great but with the amount of data involved being slow can't be helped, I guess.
Right. The fedora debuginfod server seems well situated on the net, so can generally send you compressed DWARF files fast. (I'm getting 5ish megabytes/s across https.)
[...] Most of these libraries are nothing I care about. [...] but could it be possible to not download anything at this point and only download debuginfos when they are really used [...]
This sounds like a worthwhile suggestion for upstream gdb. We will follow up with them.
[...] - None of ^C, ^, ^Z work, when I grew tired of waiting I had to switch to another terminal and kill the gdb process. [...]
OK, in my experiments, ^C did work, so something must have broken here. We will follow up with gdb folks.
- unsetting DEBUGINFOD_URLS completely disables the debuginfod client,
it would have been nice to use the data available in the client cache anyway. Ultimately I set it to localhost so debuginfod would give up immediately on new requests but still use previously downloaded data.
Perhaps there could be an "offline mode" of sorts?
I'll document this trick on the wiki.
By the way, consider operating your own debuginfod (rawhide or 0.185) server, federating to upstream. The gdb-gui-download-series process you experienced could be maybe twice as fast, due to http connection keepalive improvements. Maybe even faster, once gdb itself takes direct advantage (patches there are pending). Plus you get a shared cache for your network.
[... re. security ...] (distributing just checksums of all debug files in the main package, so when debuginfod downloads something it can compare with the checksum that was installed and verified locally by signed rpm?
That sounds like something that could fall out from this effort [1], when/if comes to fruition. We in debuginfod land could naturally follow up with interfacing & checking glue code.
[1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
All in all I think it's a very valuable tool, but it would be really great if we could minimize the amount of data actually downloaded, and establish some chain of trust.
I'm hoping we can make some improvements between now and F35 release, though practically all of these rely on changes to other projects.
- FChE
Frank Ch. Eigler wrote on Tue, May 25, 2021 at 02:57:54PM -0400:
[...] Most of these libraries are nothing I care about. [...] but could it be possible to not download anything at this point and only download debuginfos when they are really used [...]
This sounds like a worthwhile suggestion for upstream gdb. We will follow up with them.
Thanks!
[...] - None of ^C, ^, ^Z work, when I grew tired of waiting I had to switch to another terminal and kill the gdb process. [...]
OK, in my experiments, ^C did work, so something must have broken here. We will follow up with gdb folks.
I just tried on fedora34, this works here so that must be related to the version used on debian, thanks for pointing out it works.
(For others: ^c cancels the download of a single object file, so might need quite a few of these to eventually get to a gdb prompt if there are many objects to download, but it works)
By the way, consider operating your own debuginfod (rawhide or 0.185) server, federating to upstream. The gdb-gui-download-series process you experienced could be maybe twice as fast, due to http connection keepalive improvements. Maybe even faster, once gdb itself takes direct advantage (patches there are pending). Plus you get a shared cache for your network.
Interesting, I wasn'te aware of such a debuginfod proxifying mode. Are there noticeable advantages over a simple caching http proxy (e.g. squid) besides serving off in-house binaries debuginfo along the way? In my usecase this was a one-off thing for now, but I can see the point if using it a lot.
[... re. security ...] (distributing just checksums of all debug files in the main package, so when debuginfod downloads something it can compare with the checksum that was installed and verified locally by signed rpm?
That sounds like something that could fall out from this effort [1], when/if comes to fruition. We in debuginfod land could naturally follow up with interfacing & checking glue code.
[1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
Hm, I guess debuginfod could distribute the signature files together with the dwarf files straight from the rpm, but I'm a bit pessimistic whether everyone will have IMA enabled even at relatively long term.
The change proposal does state that they won't actually enforce anything:
Note explicitly that we do not intend to install a default policy as part of this change, and users will need to deploy their own policy before anything is measured or appraised. This means that after this is done, users will have the option to enable a policy and have that be enforced, but there will be nothing automatic. We will, however, document various example policies people can adapt to their needs.
Even when it does become standard many machines with developers will want to continue letting users compile their own code, and these are users likely to rely on debuggers/debuginfod to debug what they just wrote, so would they be protected? (I'm not familiar with how that works, now I'm reading up it looks possible to integrate with selinux to require appraisal by context, but there are still many many machines without selinux or equivalent in enforcing mode nowadays)
I guess I'd be fine with that if the debuginfod client implements the signature verification in userspace regardless of the policy if configured for it (I almost wrote if the server sends it, but that'd let malicious server just pretend there is no signature available... heh), other distros will probably take a bit more time to deploy but I guess it will come together eventually.
Anyway, thanks a bunch for following up, it's appreciated.
Hi -
[...] Most of these libraries are nothing I care about. [...] but could it be possible to not download anything at this point and only download debuginfos when they are really used [...]
This sounds like a worthwhile suggestion for upstream gdb. We will follow up with them.
https://sourceware.org/bugzilla/show_bug.cgi?id=17547
OK, in my experiments, ^C did work, so something must have broken here. We will follow up with gdb folks.
[...] (For others: ^c cancels the download of a single object file, so might need quite a few of these to eventually get to a gdb prompt if there are many objects to download, but it works)
https://sourceware.org/bugzilla/show_bug.cgi?id=27911
[...] Interesting, I wasn't aware of such a debuginfod proxifying mode.
Yes, we refer to it as "federation", and designed into debuginfod from the beginning, so that one can establish a tree of servers, e.g. along personal/group/corporate/public sorts of lines.
Are there noticeable advantages over a simple caching http proxy (e.g. squid) besides serving off in-house binaries debuginfo along the way? [...]
It should be about the same, except cache management policy is different, and that debuginfod will pass requests to its upstream peers *concurrently*, so this can create more redundancy and possibly performance. By design, squid proxies or CDNs should interoperate just fine with debuginfod and within federation trees.
[...]
That sounds like something that could fall out from this effort [1], when/if comes to fruition. We in debuginfod land could naturally follow up with interfacing & checking glue code.
[1] https://fedoraproject.org/wiki/Changes/Signed_RPM_Contents
Hm, I guess debuginfod could distribute the signature files together with the dwarf files straight from the rpm, but I'm a bit pessimistic whether everyone will have IMA enabled even at relatively long term. The change proposal does state that they won't actually enforce anything:
Right, I'm referring only to that if the per-file signatures were built & packaged, we could trick the debuginfod client/server to use that for manual signature checking, vs. relying on selinux/kernel enforcement.
I guess I'd be fine with that if the debuginfod client implements the signature verification in userspace regardless of the policy if configured for it (I almost wrote if the server sends it, but that'd let malicious server just pretend there is no signature available... heh),
(Good point!)
- FChE
sampsonfung wrote:
While trying to collect a backtrace for org.gnome.Tetravex, I got this in gdb:
[...] Download failed: Timer expired. Continuing without debug info for /lib64/libzstd.so.1. Missing separate debuginfo for /lib64/libzstd.so.1 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/33/70d80a1bf749b3c2baaad0188c864ee9e4bbc4.debug [...]
We have had reports that from some corners of the internet (e.g. behind the "great firewall"), that maintaining a waiting HTTPS connection for 10s of seconds was fragile. Extending the $DEBUGINFOD_TIMEOUT on the other hand was sometimes reported to help.
https://sourceware.org/bugzilla/show_bug.cgi?id=27531
Looking over the system logs for that particular request (3370d80a...), I'm seeing debuginfod.fedoraproject.org answer that particular buildid in about tens of milliseconds (!) a bunch of times, and one failure to transmit (usually a broken HTTP connection). If you could send me (directly) the IP address from where you requested it, and the $DEBUGINFOD_URLS you were using, I could look further in the logs. But so far I see no sign of something triggering timeouts on this side.
- FChE
"FUNG Chi Chuen Sampson" sampsonfung@gmail.com writes:
Downloading separate debug info for /lib64/liblzma.so.5... Download failed: Timer expired. Continuing without debug info for /lib64/libbrotlicommon.so.1. Missing separate debuginfo for /lib64/libbrotlicommon.so.1 [...]
By the way, if you were using the staging (.stg.) debuginfod server, please switch to the main one, which is in full service.
DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/
We're currently tracking down some infrastructure/proxy problem that intermittently affects only the .stg. ones.
- FChE
On Thu, 2021-04-08 at 11:19 -0400, Frank Ch. Eigler wrote:
ngompa13 wrote:
Woohoo! I'm excited for this!
Thanks!
I got a chance to use debuginfod to do some debugging of DNF on openSUSE last Saturday and the experience was fantastic.
I'm looking forward to this being wired up into GDB, ABRT, and everything else
Hey, it already is there in most tools, try it.
% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/
and shortly all F32+ packages/versions/architectures will be debuggable that way. (The staging indexer is processing the l* packages, after a few days of churning from 0* through k*. It has a one or two more days to go, but may already be used.)
The purpose of this Change is to configure this environment variable by default, pointing to the forthcoming production version of the server at https://debuginfod.fedoraproject.org/.
Something that occurred to me about this change is that as well as sources and DWARF data, some of our debuginfo files contain python scripts.
Back in Fedora 13 with: https://fedoraproject.org/wiki/Features/EasierPythonDebugging I added python scripts to python-debuginfo and python3-debuginfo so that if you install them, gdb becomes gains type-specific pretty- printers and extra commands relevant to what you're debugging.
That made sense to me back then, given that if you're debugging a process you had to download the debuginfo package, so it made sense to put the .py scripts in there so they would get autoadded (I don't see any reference in my old wiki page to "add-auto-load-safe-path", so maybe those directories would be regarded as safe automatically).
Having the sources and DWARF on-demand via debuginfod sounds great, and hopefully means never having to manually install debuginfo rpms again - but what about those .py files? I'm assuming debuginfod isn't supplying those (is it?). I'm much more nervous about arbitrary python scripts being supplied over this service, as the barrier to entry for bad guys to do Bad Things would be so much lower as compared to malformed DWARF, so perhaps if people want the .py files, they have to install the debuginfo package in the traditional way? (It's still .py files from rpm payloads, but having them autodownloaded with little user involvement seems troublesome as compared to manually installing debuginfo rpms).
Thoughts? Dave
dmalcolm wrote:
[...] Something that occurred to me about this change is that as well as sources and DWARF data, some of our debuginfo files contain python scripts.
Because these files are stand-alone .py files rather than elf/dwarf files, debuginfod has no way of serving those. If they were embedded inside the DWARF file as a special section, then it could travel implicitly.
[...] Having the sources and DWARF on-demand via debuginfod sounds great, and hopefully means never having to manually install debuginfo rpms again - but what about those .py files?
If we were interested in serving them, we could do it by extending the webapi protocol with a way to refer to a buildid-indexed debuginfo-related file. Like /buildid/BUILDID/debuginfo-gdb.py
[...] I'm much more nervous about arbitrary python scripts being supplied over this service, as the barrier to entry for bad guys to do Bad Things would be so much lower as compared to malformed DWARF [...]
Perhaps ... I'm not sure bad DWARF is inherently safer than bad Python. Nevertheless, all this is based on the proposition that the files are coming from a generally trusted build system, serving trusted artifacts maintained by trusted individuals. If malevolent enough files get in there, the service can be degraded and/or clients can be affected.
- FChE
On Fri, Apr 9 2021 at 04:20:16 PM -0400, David Malcolm dmalcolm@redhat.com wrote:
Back in Fedora 13 with: https://fedoraproject.org/wiki/Features/EasierPythonDebugging I added python scripts to python-debuginfo and python3-debuginfo so that if you install them, gdb becomes gains type-specific pretty- printers and extra commands relevant to what you're debugging.
For glib, we put these pretty printers under the -devel package. That's probably a better place for them.
fche@redhat.com (Frank Ch. Eigler) writes:
The purpose of this Change is to configure this environment variable by default, pointing to the forthcoming production version of the server at https://debuginfod.fedoraproject.org/.
With elfutils-0.184-4.fc35, this is now active for rawhide. If you're interested in testing, please "# yum update", and try debugging things.
- FChE
The change page lacks a discussion of security implications. An informed decision requires answers to questions such as:
· What kinds of attacks might be possible with malicious debuginfo files? (For example debugging tools might have undiscovered bugs that could be exploited by malformed DWARF data.)
· How is it verified that files received from debuginfo servers have not been tampered with?
· Is there any end-to-end authentication from the Fedora build system to my workstation – which there is with signed debuginfo packages – or do the tools blindly trust a whole network of federated debuginfo servers?
Some Debian users have [https://lists.debian.org/debian-devel/2021/02/msg00262.html expressed concerns] that this facility "calls home" during debugging, so it may expose a limited amount of information about what a user is debugging.
To fully understand the privacy implications, one needs to know:
· Does that happen every time, or are downloaded files cached locally?
· If there is a cache, when are old files purged from the cache?
The change page should also mention how a network problem can impact the usability of debugging tools. Could it for example make GDB hang for a minute every time it encounters a new source filename?
Finally, if somebody doesn't like the answers to the above questions, then they'll want to know how to disable the feature.
Björn Persson
Bjorn wrote:
The change page lacks a discussion of security implications. An informed decision requires answers to questions such as: [...]
Thank you for your questions. Added a section and placed initial answers there:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault#Security_FAQ
- FChE
Hi -
Björn Persson writes:
· How is it verified that files received from debuginfo servers have not been tampered with?
Following up further to this, we're planning to add optional client-side hash-verification of cached content, to provide some protection against tampering:
https://sourceware.org/bugzilla/show_bug.cgi?id=27758
- FChE
Frank Ch. Eigler wrote:
Björn Persson writes:
· How is it verified that files received from debuginfo servers have not been tampered with?
Following up further to this, we're planning to add optional client-side hash-verification of cached content, to provide some protection against tampering:
The design you propose there won't improve anything for anyone. If the hash is computed on the debuginfo server, then an attacker who can make the server serve malicious debuginfo can also make it serve hashes that match the malicious files. And as you noted yourself, an attacker who can manipulate cached files client-side has already taken over the user account anyway.
Quote from Sourceware Bugzilla:
As transport over HTTPS protects the content, we can safely assume that immediately during/after the download, the content will be fine. However, what of cached files?
Of course – from your point of view. From my point of view, I can safely assume that nobody has tampered with my cache. However, what of files on the debuginfo server?
I see that debuginfod.fedoraproject.org is currently another name for koji.fedoraproject.org. Given that it serves debuginfo only for Fedora packages, and does not forward requests to any other debuginfo servers, using this server seems equivalent security-wise to downloading unsigned packages from Koji.
As far as I understand, packages are built and signed on separate servers with a smaller attack surface than the web front-end to minimize the risk that an attacker could tamper with them. To make the debuginfo protocol as secure as signed debuginfo packages, the client should verify the files against a hash computed and signed on the signing server.
Perhaps a hash of the debuginfo could be stored in a signed RPM package, either the main package or a separate debughash package?
For those who are concerned about privacy, the proposal would make that problem worse as it would cause the "phoning home" to happen every time they debug something.
By the way, the change page still doesn't say enough about how network problems will affect the user experience. Making a previously offline activity network-dependent also makes it sensitive to network problems. For example, if great packet loss causes TCP timeouts or long delays, will that make GDB hang for minutes at a time, or is it handled asynchronously? Will GDB hang once per process, once per login session, or every time it encounters a new source filename?
Björn Persson
On Wed, Apr 21, 2021 at 11:26:10AM +0200, Björn Persson wrote:
Frank Ch. Eigler wrote:
Björn Persson writes:
· How is it verified that files received from debuginfo servers have not been tampered with?
Following up further to this, we're planning to add optional client-side hash-verification of cached content, to provide some protection against tampering:
The design you propose there won't improve anything for anyone. If the hash is computed on the debuginfo server, then an attacker who can make the server serve malicious debuginfo can also make it serve hashes that match the malicious files. And as you noted yourself, an attacker who can manipulate cached files client-side has already taken over the user account anyway.
Quote from Sourceware Bugzilla:
As transport over HTTPS protects the content, we can safely assume that immediately during/after the download, the content will be fine. However, what of cached files?
Of course – from your point of view. From my point of view, I can safely assume that nobody has tampered with my cache. However, what of files on the debuginfo server?
I see that debuginfod.fedoraproject.org is currently another name for koji.fedoraproject.org. Given that it serves debuginfo only for Fedora packages, and does not forward requests to any other debuginfo servers, using this server seems equivalent security-wise to downloading unsigned packages from Koji.
As far as I understand, packages are built and signed on separate servers with a smaller attack surface than the web front-end to minimize the risk that an attacker could tamper with them. To make the debuginfo protocol as secure as signed debuginfo packages, the client should verify the files against a hash computed and signed on the signing server.
Yes, this is the scenario which I think is worrisome. This was also raised during the FESCo meeting, and I want to clarify a bit.
A hypothetical attack through -debuginfo files would require gdb (or other consumers of the debug data) to incorrectly handle corrupt debug data. Even if we don't know any such cases right now, gdb and the underlying libraries are written in C. We have a long history of buffer overflows and other exploitable memory handling errors, and we should assume that sooner or later some will be discovered in those code paths too.
Currently, the -debuginfo packages are distributed as any other package, i.e. they are built and signed on dedicated machines. A modification anywhere at a later point would cause a signature mismatch. The trust level for -debuginfo data is the same as any other package. A hypothetical attacker who gained access to the package contents *before signing* would probably be better off modifying executable code in those packages, instead of a roundabout attack through debug data.
OTOH, the debuginfo files distributed through the debuginfod server are not signed and there is no direct way to verify that they match the (signed) contents of the debuginfo package. Thus, the debuginfod server becomes a juicy target. There are a few things which make it particularly attractive to an attacker: the debugger is very likely to be ran directly from a developer account. The debugger is executed without any sandboxing, and possibly even with elevated privileges (when debugging system services). The debugger code was not written with security it mind, so it's more likely to be exploitable than say a web browser.
As to the debuginfod code itself, it is in C++, and has SQL and threads, a http server, and also does bunch of low-level string parsing. It is also fairly new code. I don't have any particular knowledge about the code, but some exploit being found is not outside of the realm of possibility.
Thus, to summarize: debuginfo files served over the web provide a new fairly big attack surface, with attacks most likely leading to a compromise of a developer or privileged account.
Zbyszek
Perhaps a hash of the debuginfo could be stored in a signed RPM package, either the main package or a separate debughash package?
For those who are concerned about privacy, the proposal would make that problem worse as it would cause the "phoning home" to happen every time they debug something.
By the way, the change page still doesn't say enough about how network problems will affect the user experience. Making a previously offline activity network-dependent also makes it sensitive to network problems. For example, if great packet loss causes TCP timeouts or long delays, will that make GDB hang for minutes at a time, or is it handled asynchronously? Will GDB hang once per process, once per login session, or every time it encounters a new source filename?
Björn Persson
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
OTOH, the debuginfo files distributed through the debuginfod server are not signed and there is no direct way to verify that they match the (signed) contents of the debuginfo package.
A direct way would be for someone to koji-download the given rpm, and hand-extract/compare the files. (It's obviously not economical.)
Thus, the debuginfod server becomes a juicy target.
Yes. The Changes FAQ section discusses this topic.
Unfortunately, in the absence of per-file signatures generated by the build system, and securely distributed out-of-band, I can't think of any way to provide client-side verifiability of a debuginfod type service. That's independent of any particular level of server code robustness.
Interestingly, if debuginfod were considered *trusted*, then it could be used as the *basis* for such a capability, because the client-side hash verification feature being prototyped. It would serve trusted hashes to RPM-based artifacts on demand.
- FChE
On Wed, Apr 21, 2021 at 4:09 PM Frank Ch. Eigler fche@redhat.com wrote:
A direct way would be for someone to koji-download the given rpm, and hand-extract/compare the files. (It's obviously not economical.)
Thus, the debuginfod server becomes a juicy target.
Yes. The Changes FAQ section discusses this topic.
Unfortunately, in the absence of per-file signatures generated by the build system, and securely distributed out-of-band, I can't think of any way to provide client-side verifiability of a debuginfod type service. That's independent of any particular level of server code robustness.
I think there *are* solutions that reduce the attack surface so that the public facing server no longer needs to be trusted.
Service 1: indexes signed debuginfo files in Fedora, verifying RPM signatures, puts the object IDs and hashes into a Merkle tree [Root node of Merkle tree is signed] Service 2: serves out those debuginfo files to clients, along with root signature and the nodes from the root to the file of interest
But I don't want to see this proposal blocked on implementing something that technically complex - I think it offers big benefits to Fedora users as is. Certainly there are other programs that are typically run without sandboxing by developers and connect to network services - even entirely untrusted network services - and we typically consider that acceptable.
Owen
* Frank Ch. Eigler:
Unfortunately, in the absence of per-file signatures generated by the build system, and securely distributed out-of-band, I can't think of any way to provide client-side verifiability of a debuginfod type service. That's independent of any particular level of server code robustness.
I hat to bring them up, but IMA signatures could *almost* be used for this. They cover individual files.
The problem is that any Fedora developer can get an IMA signature of any file contents. There is nothing in the signature that says that it's been produced by debuginfo generation. So I'm not sure if IMA signatures actually reduce the attack surface in any significant way.
Thanks, Florian
Björn Persson Bjorn@xn--rombobjrn-67a.se writes:
The design you propose there won't improve anything for anyone. If the hash is computed on the debuginfo server, then an attacker who can make the server serve malicious debuginfo can also make it serve hashes that match the malicious files.
Yes, this does not provide protection against a penetrated server. It does not claim to.
And as you noted yourself, an attacker who can manipulate cached files client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for anyone". The proposed client-side verification is roughly analogous to running "rpm -V" on a machine. Yes, if an attacker has control at that moment, it's not reliable. Nevertheless, to detect residue of a -previous attack- or accidental data corruption, it can be worthwhile.
[...] I see that debuginfod.fedoraproject.org is currently another name for koji.fedoraproject.org.
They are separate VMs, if that's what you mean. (You may be confused by use of a number of shared HTTP front-end proxies.)
Given that it serves debuginfo only for Fedora packages, and does not forward requests to any other debuginfo servers, using this server seems equivalent security-wise to downloading unsigned packages from Koji.
Not exactly. All the data is -from- signed packages.
To make the debuginfo protocol as secure as signed debuginfo packages, the client should verify the files against a hash computed and signed on the signing server.
If the threat model includes a -local active attacker-, then this would not help either. An attacker could interfere with the local keystore and/or trust chains and/or signature verification software.
For those who are concerned about privacy, the proposal would make that problem worse as it would cause the "phoning home" to happen every time they debug something.
That's if they wish to rely on live verification.
Note that the privacy being leaked consists of the hex buildid of the program being debugged, and an elfutils#/fedora#/arch, and of course IP address. It's not nothing, but it's nothing more. It's roughly equivalent to the dnf-automatic.timer call-home in this respect.
By the way, the change page still doesn't say enough about how network problems will affect the user experience. [...]
I'm not sure why you say "still" when this question was not posed here before. I will add some text on this.
- FChE
On Wed, Apr 21, 2021 at 03:15:23PM -0400, Frank Ch. Eigler wrote:
Björn Persson Bjorn@xn--rombobjrn-67a.se writes:
The design you propose there won't improve anything for anyone. If the hash is computed on the debuginfo server, then an attacker who can make the server serve malicious debuginfo can also make it serve hashes that match the malicious files.
Yes, this does not provide protection against a penetrated server. It does not claim to.
And as you noted yourself, an attacker who can manipulate cached files client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for anyone". The proposed client-side verification is roughly analogous to running "rpm -V" on a machine. Yes, if an attacker has control at that moment, it's not reliable. Nevertheless, to detect residue of a -previous attack- or accidental data corruption, it can be worthwhile.
We have btrfs now… It's not exactly the same, but it provides protection against the most likely corruption scenario — disk errors.
[...] I see that debuginfod.fedoraproject.org is currently another name for koji.fedoraproject.org.
They are separate VMs, if that's what you mean. (You may be confused by use of a number of shared HTTP front-end proxies.)
Given that it serves debuginfo only for Fedora packages, and does not forward requests to any other debuginfo servers, using this server seems equivalent security-wise to downloading unsigned packages from Koji.
Not exactly. All the data is -from- signed packages.
It is equivalent in the following sense: if the server is compromised, it can serve any data it wants, and the client has no chance of knowing.
Zbyszek
Frank Ch. Eigler wrote:
Björn Persson Bjorn@xn--rombobjrn-67a.se writes:
And as you noted yourself, an attacker who can manipulate cached files client-side has already taken over the user account anyway.
Yes and no, and so I must disagree with your "won't improve ... for anyone". The proposed client-side verification is roughly analogous to running "rpm -V" on a machine. Yes, if an attacker has control at that moment, it's not reliable. Nevertheless, to detect residue of a -previous attack- or accidental data corruption, it can be worthwhile.
I fail to imagine how you believe attackers operate to make the distinction between an attacker who has control and a previous attack relevant.
It can detect accidental data corruption, yes. If you want to checksum the cache to detect accidental data corruption, that's fine by me, but that's better done locally, so that the checksums can be verified without contacting any server.
Given that it serves debuginfo only for Fedora packages, and does not forward requests to any other debuginfo servers, using this server seems equivalent security-wise to downloading unsigned packages from Koji.
Not exactly. All the data is -from- signed packages.
Okay, so it's equivalent to downloading packages that were once signed, but had the signatures removed before the packages were offered for downloading – which is in turn equivalent to downloading unsigned packages.
To make the debuginfo protocol as secure as signed debuginfo packages, the client should verify the files against a hash computed and signed on the signing server.
If the threat model includes a -local active attacker-, then this would not help either. An attacker could interfere with the local keystore and/or trust chains and/or signature verification software.
A local active attacker who can already read, write and execute whatever they want has nothing more to gain from tampering with cached debuginfo.
By the way, the change page still doesn't say enough about how network problems will affect the user experience. [...]
I'm not sure why you say "still" when this question was not posed here before.
Because I posed the question in my first message in this thread, on the 11th.
Björn Persson
On 4/7/21 22:32, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
== Summary == Fedora users / developers who need to debug/trace distro binaries can make use of the recently activated elfutils-debuginfod servers to automatically fetch debugging data and source code, instead of having to use `# sudo dnf` commands.
Now readelf, annobin and hell knows what else started to talk to REMOTE SERVERS, deep out of internals of complicated build infrastructure running on presumably secure build machines of various IT corporations and whatnot!
This is devastatingly insecure, just ONE remote exploit bug, and many IT corporations can be exposed.
Do you understand how many fetches of debuginfo will be attempted by e.g. a kernel build tooling when it runs readelf on 8000 freshly built modules _for every kernel build_? How slow it is?
Now various tools need to forcibly unset the variable to stop this madness.
commit b927c044b8809c4dd892f75737240a20c32c2b90 Author: Panu Matilainen pmatilai@redhat.com Date: Thu Feb 16 12:25:24 2023 +0200
Disable debuginfod server lookups in build and dependency generator scripts
With recent elfutils (0.182 or so) various seemingly innocuous tools such as `readelf` like to do network lookups for ELF symbol information. There's no circumstance where we'd want that to happen during rpmbuild, so disable these lookups during all spec build scriptlets and also dependency generator children.
+ unsetenv("DEBUGINFOD_URLS");
Hi, Denys -
dvlasenk wrote:
[...] Now readelf, annobin and hell knows what else started to talk to REMOTE SERVERS, deep out of internals of complicated build infrastructure running on presumably secure build machines of various IT corporations and whatnot!
(It may not be appropriate for secure build machines to have general internet access, but that's for the operators to decide.)
[...] Do you understand how many fetches of debuginfo will be attempted by e.g. a kernel build tooling when it runs readelf on 8000 freshly built modules _for every kernel build_? How slow it is?
If remote debuginfo is not needed for these particular readelf invocations, then the tools should not be making debuginfod calls. Can you help identify examples?
Now various tools need to forcibly unset the variable to stop this madness.
The defaults are set with normal developers in mind.
- FChE
On 6/4/24 21:43, Frank Ch. Eigler wrote:
Hi, Denys -
dvlasenk wrote:
[...] Now readelf, annobin and hell knows what else started to talk to REMOTE SERVERS, deep out of internals of complicated build infrastructure running on presumably secure build machines of various IT corporations and whatnot!
(It may not be appropriate for secure build machines to have general internet access, but that's for the operators to decide.)
[...] Do you understand how many fetches of debuginfo will be attempted by e.g. a kernel build tooling when it runs readelf on 8000 freshly built modules _for every kernel build_? How slow it is?
If remote debuginfo is not needed for these particular readelf invocations, then the tools should not be making debuginfod calls. Can you help identify examples?
Now various tools need to forcibly unset the variable to stop this madness.
The defaults are set with normal developers in mind.
The problem is that nobody expects a tool this low-level to do internet lookups, and silently at that. It's like 'ls' suddenly started looking up stuff from the net *by default*.
And, not just interactive use but scripts too, scripts that existed years or even decades before this thing and cannot possibly expect such activity. Like the case of https://bugzilla.redhat.com/show_bug.cgi?id=2079600
And, it's made much worse by the packaging which means you cannot just remove the package and be done with it, because so much important tooling links to it. If the library and the configuration to actually enable it were split between different packages, it'd be less offensive at least.
- Panu -
Hi Panu,
On Wed, Jun 05, 2024 at 09:26:14AM +0300, Panu Matilainen wrote:
On 6/4/24 21:43, Frank Ch. Eigler wrote:
dvlasenk wrote:
[...] Do you understand how many fetches of debuginfo will be attempted by e.g. a kernel build tooling when it runs readelf on 8000 freshly built modules _for every kernel build_? How slow it is?
If remote debuginfo is not needed for these particular readelf invocations, then the tools should not be making debuginfod calls. Can you help identify examples?
Now various tools need to forcibly unset the variable to stop this madness.
The defaults are set with normal developers in mind.
The problem is that nobody expects a tool this low-level to do internet lookups, and silently at that. It's like 'ls' suddenly started looking up stuff from the net *by default*.
And, not just interactive use but scripts too, scripts that existed years or even decades before this thing and cannot possibly expect such activity. Like the case of https://bugzilla.redhat.com/show_bug.cgi?id=2079600
This surprises me. You are right that nobody would expect readelf -d (to get the dynamic table entries) would use debuginfod. And it indeed doesn't. Might this just have been an old bug in binutils since fixed?
And, it's made much worse by the packaging which means you cannot just remove the package and be done with it, because so much important tooling links to it. If the library and the configuration to actually enable it were split between different packages, it'd be less offensive at least.
I am sorry you are so offended by the packaging, but nobody has suggested this before. Please do file a bug with this suggestion if you think it makes sense to split the default /etc files and the actual debuginfod-client library.
Cheers,
Mark