Now that I've packaged the GEF gdb extension (package "gdb-gef" in Rawhide) which provides the "got-audit" command, I wanted to ask again whether there's interest in monitoring critical packages for signs of GOT tampering.
The "got-audit" gdb command examines a running process to print a list of symbols, and the file backing the area of memory where each symbol points. For example:
accept@GLIBC_2.2.5 : /usr/lib64/libc.so.6 access@GLIBC_2.2.5 : /usr/lib64/libc.so.6 AES_encrypt@OPENSSL_3.0.0 : /usr/lib64/libcrypto.so.3 AES_set_encrypt_key@OPENSSL_3.0.0 : /usr/lib64/libcrypto.so.3 ... EVP_PKEY_set1_RSA@OPENSSL_3.0.0 : /usr/lib64/liblzma.so.5 :: ERROR EVP_PKEY_set1_RSA not exported by /usr/lib64/liblzma.so.5
In addition to printing the mapping, the command will print an error if it determines that two libraries are both exporting a matching symbol (e.g., if EVP_PKEY_set1_RSA appeared in libcrypto.so and in liblzma.so), and it will print an error if it determines that a symbol has been resolved to a file that doesn't export that symbol (e.g., if EVP_PKEY_set1_RSA points to liblzma.so, but liblzma.so doesn't export a symbol of that name).
In the case of the xz-utils attack, the got-audit command would have logged an error due to the latter check, as illustrated above.
(As a topic for later: the tirpc library exports functions with the same name as functions that appear in libc, so the behavior of erroring on duplicate symbols needs to be rationalized. Maybe an exemption for libtirpc.so? Are there other libraries that do this?)
I have a proof-of-concept test implemented for the openssh package, in this pull request: https://src.fedoraproject.org/rpms/openssh/pull-request/73
If we are interested in building out protection from this class of attack, there are a number of questions I have about the best way to do that:
The current implementation compares the result to an expected result. While I like the idea of keeping historical records, I'm not sure I can justify the developer overhead of updating the expected result file as it evolves due to updates in the program and in the libraries it uses. It would be simpler to simply watch for errors in the output.
If we only watch for errors in the output, it would be much easier to convert this into a generic test and run it against more daemons.
The proof-of-concept only audits symbols in the binary, but shared objects have their own set of dynamic symbols loaded from shared libraries. The output is quite large if we log the full set, and more importantly it will change much more often, but if we only look for errors, we can use "got-audit --all" to also watch symbol resolution within the shared objects.
But it's probably important to mention that if we only watch for errors in the output, the test could very easily invisibly break. If it stopped working properly and printed nothing, we might go on thinking that we had mitigations in place when the test had actually become a NOP.
Is this work interesting? Should I continue working on it?
On Пан, 24 чэр 2024, Gordon Messmer wrote:
Now that I've packaged the GEF gdb extension (package "gdb-gef" in Rawhide) which provides the "got-audit" command, I wanted to ask again whether there's interest in monitoring critical packages for signs of GOT tampering.
The "got-audit" gdb command examines a running process to print a list of symbols, and the file backing the area of memory where each symbol points. For example:
accept@GLIBC_2.2.5 : /usr/lib64/libc.so.6 access@GLIBC_2.2.5 : /usr/lib64/libc.so.6 AES_encrypt@OPENSSL_3.0.0 : /usr/lib64/libcrypto.so.3 AES_set_encrypt_key@OPENSSL_3.0.0 : /usr/lib64/libcrypto.so.3 ... EVP_PKEY_set1_RSA@OPENSSL_3.0.0 : /usr/lib64/liblzma.so.5 :: ERROR EVP_PKEY_set1_RSA not exported by /usr/lib64/liblzma.so.5
In addition to printing the mapping, the command will print an error if it determines that two libraries are both exporting a matching symbol (e.g., if EVP_PKEY_set1_RSA appeared in libcrypto.so and in liblzma.so), and it will print an error if it determines that a symbol has been resolved to a file that doesn't export that symbol (e.g., if EVP_PKEY_set1_RSA points to liblzma.so, but liblzma.so doesn't export a symbol of that name).
In the case of the xz-utils attack, the got-audit command would have logged an error due to the latter check, as illustrated above.
(As a topic for later: the tirpc library exports functions with the same name as functions that appear in libc, so the behavior of erroring on duplicate symbols needs to be rationalized. Maybe an exemption for libtirpc.so? Are there other libraries that do this?)
Most of alternative memory allocation libraries provide their own free/malloc replacements to be used with LD_PRELOAD, e.g. jemalloc.
I have a proof-of-concept test implemented for the openssh package, in this pull request: https://src.fedoraproject.org/rpms/openssh/pull-request/73
If we are interested in building out protection from this class of attack, there are a number of questions I have about the best way to do that:
The current implementation compares the result to an expected result. While I like the idea of keeping historical records, I'm not sure I can justify the developer overhead of updating the expected result file as it evolves due to updates in the program and in the libraries it uses. It would be simpler to simply watch for errors in the output.
If we only watch for errors in the output, it would be much easier to convert this into a generic test and run it against more daemons.
The proof-of-concept only audits symbols in the binary, but shared objects have their own set of dynamic symbols loaded from shared libraries. The output is quite large if we log the full set, and more importantly it will change much more often, but if we only look for errors, we can use "got-audit --all" to also watch symbol resolution within the shared objects.
But it's probably important to mention that if we only watch for errors in the output, the test could very easily invisibly break. If it stopped working properly and printed nothing, we might go on thinking that we had mitigations in place when the test had actually become a NOP.
Is this work interesting? Should I continue working on it?
Yes, it is definitely an interesting test. Thank you for investing your time and resources into this.
On Tue, Jun 25, 2024 at 08:34:38AM GMT, Alexander Bokovoy wrote:
On Пан, 24 чэр 2024, Gordon Messmer wrote:
...snip...
Is this work interesting? Should I continue working on it?
Yes, it is definitely an interesting test. Thank you for investing your time and resources into this.
Yes, seconded.
I wonder if this wouldn't fit in as a CI test? Or something that might be added to rpminspect?
I guess it would need to keep state so it can keep track of changes, but hopefully thats possible.
kevin
On 2024-06-25 10:37 AM, Kevin Fenzi wrote:
I wonder if this wouldn't fit in as a CI test?
Do you mean https://docs.fedoraproject.org/en-US/ci/generic_tests/ ?
Maybe it would? If I misunderstand this, please correct me:
Because Fedora uses "-z,relro" and "-z,now" in %build_ldflags, all binaries should have a fully resolved GOT when they reach main(). That being the case, gdb could be started with any binary, at which point it could set a breakpoint at "main", run the binary, and then audit the GOT at the breakpoint.
Does that sound right?
Or something that might be added to rpminspect?
It's been a few months since I looked at rpminspect. Does it install a package and all of its deps in order to inspect it? The GOT can't be audited unless the process can start.
On 2024-06-26 5:42 PM, Gordon Messmer wrote:
On 2024-06-25 10:37 AM, Kevin Fenzi wrote:
I wonder if this wouldn't fit in as a CI test?
Do you mean https://docs.fedoraproject.org/en-US/ci/generic_tests/ ?
Maybe it would? If I misunderstand this, please correct me:
Because Fedora uses "-z,relro" and "-z,now" in %build_ldflags, all binaries should have a fully resolved GOT when they reach main(). That being the case, gdb could be started with any binary,
Oh, no... now I remember... the compromised library detects startup under gdb and doesn't initialize itself. That's why the proof of concept implementation attaches to a running sshd.
On Wed, Jun 26, 2024 at 05:42:14PM GMT, Gordon Messmer wrote:
On 2024-06-25 10:37 AM, Kevin Fenzi wrote:
I wonder if this wouldn't fit in as a CI test?
Do you mean https://docs.fedoraproject.org/en-US/ci/generic_tests/ ?
yeah...
Maybe it would? If I misunderstand this, please correct me:
Because Fedora uses "-z,relro" and "-z,now" in %build_ldflags, all binaries should have a fully resolved GOT when they reach main(). That being the case, gdb could be started with any binary, at which point it could set a breakpoint at "main", run the binary, and then audit the GOT at the breakpoint.
Does that sound right?
Sounds reasonable to me, but I don't know this area very well.
Or something that might be added to rpminspect?
It's been a few months since I looked at rpminspect. Does it install a package and all of its deps in order to inspect it? The GOT can't be audited unless the process can start.
yeah, I think it is able to start processes to test things, but I am again not sure of the details.
So, hopefully rpminspect / ci folks will chime in...
kevin
On 2024-06-24 10:34 PM, Alexander Bokovoy wrote:
On Пан, 24 чэр 2024, Gordon Messmer wrote:
(As a topic for later: the tirpc library exports functions with the same name as functions that appear in libc, so the behavior of erroring on duplicate symbols needs to be rationalized. Maybe an exemption for libtirpc.so? Are there other libraries that do this?)
Most of alternative memory allocation libraries provide their own free/malloc replacements to be used with LD_PRELOAD, e.g. jemalloc.
Good point. But I'm mostly concerned with the libraries that binaries link directly against, since those are the cases where the got-audit will always see symbol duplication.
Yes, it is definitely an interesting test. Thank you for investing your time and resources into this.
Thanks. I'll keep plugging away at it.
Am 27.06.24 um 02:03 schrieb Gordon Messmer:
On 2024-06-24 10:34 PM, Alexander Bokovoy wrote:
On Пан, 24 чэр 2024, Gordon Messmer wrote:
(As a topic for later: the tirpc library exports functions with the same name as functions that appear in libc, so the behavior of erroring on duplicate symbols needs to be rationalized. Maybe an exemption for libtirpc.so? Are there other libraries that do this?)
Most of alternative memory allocation libraries provide their own free/malloc replacements to be used with LD_PRELOAD, e.g. jemalloc.
Good point. But I'm mostly concerned with the libraries that binaries link directly against, since those are the cases where the got-audit will always see symbol duplication.
Well, jemalloc provides a few more function for e.g. fine-tuning which some applications make use of (and as such need to link against jemalloc directly).
-
Alexander Bokovoy -
Gordon Messmer -
Kevin Fenzi -
Kilian Hanich