= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal vgoyal@redhat.com
Currently kexec/kdump is disabled on machines with secureboot enabled. This feature aims to enable kexec/kdump on such machines.
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And upon successful signature verification, allow it to load kernel, initramfs, and binary blob. /sbin/kexec will verify signatures of kernel being loaded before it asks running kernel to load it.
There are quite a few pieces to the puzzle. I am listing the some of the things which need to be done.
=== Build and ship ima-evm-utils package === /sbin/kexec will be signed by evmctl. This utility will put an xattr security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in kernel to verify signature of /sbin/kexec upon execution.
* There is a bz open 807476 for inclusion of this package since long time. Not sure what it is stuck on though.
* There are some patches which are not upstream yet (like lock down executable in memory) which we need to carry in this patckage till patches get upstream.
=== Changes to kexec-tools === Build /sbin/kexec statically. There is no infrastructure to sign and verify signature of shared libraries. Even if we can sign and verify shared libraries, currently kernel allows writing to shared libraries while these are mapped. So one can overwrite the library after signature verification. So build /sbin/kexec statically. kexec does not use nss related code, so there should not be any issues w.r.t glibc calling into other shared libraries.
* Generate detached /sbin/kexec signautre at build time and install these signature on target in seucurity.ima xattr when kexec-tools is installed.
* Modify kexec-tools so that it can call keyctl() and verify signature of a new kernel being loaded.
=== Kernel Changes === Kernel needs to carry additional patches to do verify elf binary signature. * There are patches to extend keyctl() so that user space can use it to verify signature of a user buffer (vmlinuz in this case). * These patches are not upstream, so these need to be carried in fedora till patches get upstream. * Kernel need to be signed using evmctl and detached signature need to be generated. These signatures need to be installed on vmlinuz upon kernel rpm installation in security.ima xattr.
=== Signing Key Management === Yet to be figured out. There are couple of ideas on table.
* Embed few keys in kernel and one of these keys will be used to sign /sbin/kexec. In case of a key is revoked, use a new key from set of embedded keys. * Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF binary should be signed by appropriate authority so it can be loaded in system keyring.
== Scope == Proposal owners: * Provide patches for kernel package. * Provide patches for kexec-tools package. * Provide patches for ima-evm-utils package. * ima-evm-utils package will be new. It is not clear who will maintain that package.
Other developers: * Signing Process, Signing key management, Signing key loading on target after boot and Signature installation on target are unknown areas right now. It is not very clear how these things will be done. Once details have been figured out, it might require work from other developers. Not sure though at this point of time. * Can't think any other major work required by other developers yet. We will have to figure out how signing will take place and how signatures will be installed on target system. And depending on how we do it, it might require some work from other developers. Like making use of rpm hooks to install signature etc. Given the fact that we are signing only one file /sbin/kexec, we might get away doing changes in kexec-tools spec file.
Release engineering: * We need to sort out how signing will take place. I will require release- engineering help on this. * For issues related to key management I will need help of security/release engineering.
Policies and guidelines: * I think to begin with we might not have to update any packaging guidelines for this. Anything magic w.r.t signing can probably be limited to kexec-tools spec file.
_______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On 07/11/2013 01:40 PM, Jaroslav Reznik wrote:
=== Build and ship ima-evm-utils package === /sbin/kexec will be signed by evmctl. This utility will put an xattr security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in kernel to verify signature of /sbin/kexec upon execution.
- There is a bz open 807476 for inclusion of this package since long time. Not
sure what it is stuck on though.
- There are some patches which are not upstream yet (like lock down executable
in memory) which we need to carry in this patckage till patches get upstream.
Is there a chance this (and the other patches mentioned below) actually makes it in the kernel? Are at least the VM changes part of upstream already?
I don't think it would make sense to add more and more Fedora-specific patches which implement security functionality. I don't want Fedora to become the next Android.
=== Kernel Changes === Kernel needs to carry additional patches to do verify elf binary signature.
- There are patches to extend keyctl() so that user space can use it to verify
signature of a user buffer (vmlinuz in this case).
- These patches are not upstream, so these need to be carried in fedora till
patches get upstream.
- Kernel need to be signed using evmctl and detached signature need to be
generated. These signatures need to be installed on vmlinuz upon kernel rpm installation in security.ima xattr.
Does this mean your implementation of signature checking will be completely independent of UEFI Secure Boot (unless you decide to use that to obtain the trust root)?
=== Signing Key Management === Yet to be figured out. There are couple of ideas on table.
- Embed few keys in kernel and one of these keys will be used to sign
/sbin/kexec. In case of a key is revoked, use a new key from set of embedded keys.
How do you intend to handle revocation?
- Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF binary
should be signed by appropriate authority so it can be loaded in system keyring.
Who is the appropriate authority?
On Thu, Jul 11, 2013 at 03:57:38PM +0200, Florian Weimer wrote:
On 07/11/2013 01:40 PM, Jaroslav Reznik wrote:
=== Build and ship ima-evm-utils package === /sbin/kexec will be signed by evmctl. This utility will put an xattr security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in kernel to verify signature of /sbin/kexec upon execution.
- There is a bz open 807476 for inclusion of this package since long time. Not
sure what it is stuck on though.
- There are some patches which are not upstream yet (like lock down executable
in memory) which we need to carry in this patckage till patches get upstream.
Is there a chance this (and the other patches mentioned below) actually makes it in the kernel? Are at least the VM changes part of upstream already?
I don't think it would make sense to add more and more Fedora-specific patches which implement security functionality. I don't want Fedora to become the next Android.
I don't see those patches going upstream in near term. First of all base secureboot patches need to go upstream. And at this point of time upsteram does not seem to be eager to do anything more in kernel for secureboot.
Secondly, there are disagreements upstream w.r.t how locking down executable should happen. IMA folks want some functionality behind security hooks (as opposed to what I have done). So I am expecting that once patches do get merged upstream, they might be in little different shape altogether.
I think now we can not back out. Merging secureboot patches without these being upstream broke other subsystems (kdump, systemtap,..). And we should now allow other non-upstream patches to go in to make these subsystems work again. Or we should remove kernel lockdown functionality in secureboot mode altogether (as upstream does not have it).
=== Kernel Changes === Kernel needs to carry additional patches to do verify elf binary signature.
- There are patches to extend keyctl() so that user space can use it to verify
signature of a user buffer (vmlinuz in this case).
- These patches are not upstream, so these need to be carried in fedora till
patches get upstream.
- Kernel need to be signed using evmctl and detached signature need to be
generated. These signatures need to be installed on vmlinuz upon kernel rpm installation in security.ima xattr.
Does this mean your implementation of signature checking will be completely independent of UEFI Secure Boot (unless you decide to use that to obtain the trust root)?
Yes. I am going to use IMA for signature verification. These signatures formats are very close to what is used for module signing (PKCS1.5 with some metadata).
For trust chain, we will still use secureboot trust chain and trust an executable only if it has been signed by a key in .system_keyring.
=== Signing Key Management === Yet to be figured out. There are couple of ideas on table.
- Embed few keys in kernel and one of these keys will be used to sign
/sbin/kexec. In case of a key is revoked, use a new key from set of embedded keys.
How do you intend to handle revocation?
I have not written the code to check blacklist yet but I plan to do that later.
If a key is revoked, I am expecting that we will request M$ for an update. And also push out new version of /sbin/kexec signed with a new key.
We will need to have this new key signed with either redhat certificate or signed with M$ so that it can be loaded in already running kernel. That will make sure old /sbin/kexec instances don't run while new instances signed with new key do run.
- Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF binary
should be signed by appropriate authority so it can be loaded in system keyring.
Who is the appropriate authority?
I am not sure yet. I was thinking that can we embed fedora/redhat certificate in kernel (like shim) and use that as CA key to sign /sbin/kexec key.
Signing a key using CA key will require loading of that key every reboot automatically. I am not sure how that can be handled. May be rpm packages can drop those keys in some directory and a system service scans for keys and loads these every reboot?
Thanks Vivek
On 07/11/2013 04:33 PM, Vivek Goyal wrote:
I don't think it would make sense to add more and more Fedora-specific patches which implement security functionality. I don't want Fedora to become the next Android.
I don't see those patches going upstream in near term. First of all base secureboot patches need to go upstream. And at this point of time upsteram does not seem to be eager to do anything more in kernel for secureboot.
The IMA stuff looks pretty independent of Secure Boot to me. It seems upstream picked up some of it in 2.6.30.
Secondly, there are disagreements upstream w.r.t how locking down executable should happen. IMA folks want some functionality behind security hooks (as opposed to what I have done). So I am expecting that once patches do get merged upstream, they might be in little different shape altogether.
The important question is whether we can drop our own patches and switch to whatever upstream does when the time comes.
I think now we can not back out. Merging secureboot patches without these being upstream broke other subsystems (kdump, systemtap,..).
Sure. But systemtap (and things like PCI passthrough) are fundamentally incompatible with our approach to Secure Boot. There are conceptual challenges like irreversible software updates, too. At a certain point, we will have added so much inconvenience that only very few users will use this feature. Upstream is not totally crazy in rejecting Fedora's restrictive approach.
The proposed change goes in the right direction (more user control), but we're still missing things on the restrictive side (dbx updates, Fedora-local revocation checking, and others I'm not aware of). And as long as Fedora and the more lenient distributions are under the same trust root, Fedora users get pretty close to zero additional security benefit, despite all the effort we put into this.
Yes. I am going to use IMA for signature verification. These signatures formats are very close to what is used for module signing (PKCS1.5 with some metadata).
For trust chain, we will still use secureboot trust chain and trust an executable only if it has been signed by a key in .system_keyring.
Okay, that's a dependency on the Secure Boot patchset, but not Secure Boot as a technology. Good to know.
I have not written the code to check blacklist yet but I plan to do that later.
There's a challenge to obtain the blacklist in a way that is safe against rollback.
I don't think we have indirect (and proven) write access to the dbx variable yet.
If a key is revoked, I am expecting that we will request M$ for an update. And also push out new version of /sbin/kexec signed with a new key.
Then you better add a separate CA root for /sbin/kexec, so that we don't have to blacklist the kernel. (An alternative would be to blacklist binaries based on their hashes.)
Revocation as the result of a software bug seems far more likely to me than revocation because of CA compromise (although I don't know what Fedora does with these keys, but at least this is a something that's solvable in principle with suitable hardware and processes).
Signing a key using CA key will require loading of that key every reboot automatically. I am not sure how that can be handled. May be rpm packages can drop those keys in some directory and a system service scans for keys and loads these every reboot?
Not sure I follow. You can't add additional root (that is, CA) keys after building. What you could do is add intermediate CAs, but the way X.509 works, this is unnecessary if you put the entire chain (perhaps excluding the root) into the binary. So this shouldn't be required at all.
On Thu, Jul 11, 2013 at 05:19:58PM +0200, Florian Weimer wrote:
On 07/11/2013 04:33 PM, Vivek Goyal wrote:
I don't think it would make sense to add more and more Fedora-specific patches which implement security functionality. I don't want Fedora to become the next Android.
I don't see those patches going upstream in near term. First of all base secureboot patches need to go upstream. And at this point of time upsteram does not seem to be eager to do anything more in kernel for secureboot.
The IMA stuff looks pretty independent of Secure Boot to me. It seems upstream picked up some of it in 2.6.30.
It is but it implements stuff which is needed to meet TCB requirements. Current implementation is nowhere near to require secureboot requirements.
For example, executables are not locked down in memory. That means after signature verification, if executable gets swapped out, it can be tweaked by root.
Any direct changes to disk to a file are not detected. So modify the disk blocks after signature verification and IMA will not notice it.
Without secureboot we don't have any requirement where we need the capability to extend root of trust to user space application. Current IMA is not sufficient to meet secureboot requirements. So pushing more patches in because they are required to meet secureboot requirements is hard. (Because lot of people don't bite into the theory of locking down kernel in secureboot mode).
So there is very little direct dependecny on secureboot. It is more of why do you need these enahancements. And answer is because secureboot will enforce that. Well, secureboot locking is not even in kernel, and kdump is not even broken upstream, so why are you bothering to push something in now.
Secondly, there are disagreements upstream w.r.t how locking down executable should happen. IMA folks want some functionality behind security hooks (as opposed to what I have done). So I am expecting that once patches do get merged upstream, they might be in little different shape altogether.
The important question is whether we can drop our own patches and switch to whatever upstream does when the time comes.
I think we should be able to do that. We are expecting only /sbin/kexec to use this functionality and if things change we can change /sbin/kexec and signing process as we control everything.
I think important thing will be to emphasize that other applications should not try to latch on to new keyctl() ioctl option to verify signature of a user space buffer or IMA functionality to put extra data in signature which locks down executable in memory. These are not stable interfaces and might disappear in next fedora release.
I think now we can not back out. Merging secureboot patches without these being upstream broke other subsystems (kdump, systemtap,..).
Sure. But systemtap (and things like PCI passthrough) are fundamentally incompatible with our approach to Secure Boot. There are conceptual challenges like irreversible software updates, too. At a certain point, we will have added so much inconvenience that only very few users will use this feature. Upstream is not totally crazy in rejecting Fedora's restrictive approach.
Is it time to re-visit the decision of locking down kernel on secureboot machine given the fact that upstream does not seem to like the idea.
What are other distributions planning to do? Are they carrying additional patches or they have decided to go upstream way of not locking down kernel.
If others are going upstream way, then is it worth that fedora continues to be different and in the process we continue to add more out of the tree patches to make things like kdump work with secureboot.
The proposed change goes in the right direction (more user control), but we're still missing things on the restrictive side (dbx updates, Fedora-local revocation checking, and others I'm not aware of). And as long as Fedora and the more lenient distributions are under the same trust root, Fedora users get pretty close to zero additional security benefit, despite all the effort we put into this.
Yes. I am going to use IMA for signature verification. These signatures formats are very close to what is used for module signing (PKCS1.5 with some metadata).
For trust chain, we will still use secureboot trust chain and trust an executable only if it has been signed by a key in .system_keyring.
Okay, that's a dependency on the Secure Boot patchset, but not Secure Boot as a technology. Good to know.
I have not written the code to check blacklist yet but I plan to do that later.
There's a challenge to obtain the blacklist in a way that is safe against rollback.
I don't think we have indirect (and proven) write access to the dbx variable yet.
Ok, I am not aware of details here but if blacklisting does not work already, then this concern is not specific to kdump. Once it starts working and I can blacklisted keys in blacklist keyring, I should be able to check for key in that keyring and disallow sys_kexec().
If a key is revoked, I am expecting that we will request M$ for an update. And also push out new version of /sbin/kexec signed with a new key.
Then you better add a separate CA root for /sbin/kexec, so that we don't have to blacklist the kernel. (An alternative would be to blacklist binaries based on their hashes.)
To keep things simple, I was thinking if we use same CA root which we use for signing kernel, it will become easier and I don't have to start a new CA root only for /sbin/kexec.
Downside is that if CA root used for signing kernel ever gets compromised, then we will have to issue kexec-tools update too (in addition to shim, kernel, grub). I guess issuing kexec-tools update is simpler.
Revocation as the result of a software bug seems far more likely to me than revocation because of CA compromise (although I don't know what Fedora does with these keys, but at least this is a something that's solvable in principle with suitable hardware and processes).
Signing a key using CA key will require loading of that key every reboot automatically. I am not sure how that can be handled. May be rpm packages can drop those keys in some directory and a system service scans for keys and loads these every reboot?
Not sure I follow. You can't add additional root (that is, CA) keys after building. What you could do is add intermediate CAs, but the way X.509 works, this is unnecessary if you put the entire chain (perhaps excluding the root) into the binary. So this shouldn't be required at all.
I was hoping that CA certificate will be embedded in kernel at build time. And using that CA certificate we will sign keys which in turn will sign /sbin/kexec. And that key will be loaded dynamically at boot time in system_keyring. (And it should be allowed because CA key is already there and it can verify the new key being loaded).
IIUC, PKCS7 signatures can carry all the signign chain info and hence intermediate certifiates don't have to be loaded separately.
Like modules, /sbin/kexec will not use PKCS7 format. So it only knows about the key it has been signed with and expects that key to be loaded already.
Did I understand your question correctly?
Thanks Vivek
On 07/11/2013 06:03 PM, Vivek Goyal wrote:
It is but it implements stuff which is needed to meet TCB requirements. Current implementation is nowhere near to require secureboot requirements.
For example, executables are not locked down in memory. That means after signature verification, if executable gets swapped out, it can be tweaked by root.
Ah, okay, so for Trusted Boot, the onus is on the system administrator to make sure that he can provide attestation (by not running any dodgy binaries which would invalidate that), but for Security Boot, we have to prevent the system administrator to run such binaries. Makes sense.
The important question is whether we can drop our own patches and switch to whatever upstream does when the time comes.
I think we should be able to do that. We are expecting only /sbin/kexec to use this functionality and if things change we can change /sbin/kexec and signing process as we control everything.
I think important thing will be to emphasize that other applications should not try to latch on to new keyctl() ioctl option to verify signature of a user space buffer or IMA functionality to put extra data in signature which locks down executable in memory. These are not stable interfaces and might disappear in next fedora release.
Okay.
There some potential ABI impact from the VM-related changes, but this shouldn't be a problem for Fedora.
Is it time to re-visit the decision of locking down kernel on secureboot machine given the fact that upstream does not seem to like the idea.
What are other distributions planning to do? Are they carrying additional patches or they have decided to go upstream way of not locking down kernel.
I think they don't do any lockdown after boot. CAP_COMPROMISE_KERNEL (or what's it called these days) is a Fedora thing.
Ok, I am not aware of details here but if blacklisting does not work already, then this concern is not specific to kdump. Once it starts working and I can blacklisted keys in blacklist keyring, I should be able to check for key in that keyring and disallow sys_kexec().
There are some potential approaches to blacklisting which would be incompatible with certain ways of signing binaries. So it's not immediately clear if an unrelated blacklisting service could be made to work with userspace signatures.
So let's skip the blacklisting discussion for now. If we ever have to implement it, it's going to be messy, kexec or no kexec. (See the discussion about post-release media respins.)
Like modules, /sbin/kexec will not use PKCS7 format. So it only knows about the key it has been signed with and expects that key to be loaded already.
Ah, okay, that's an unfortunate upstream design decision. In this light, what you're planning to do seems fairly unavoidable.
Have you considered a non-cryptographic solution, like a physical presence check to (temporarily) disable Secure Boot so that the kexec restriction no longer applies? This could be a fallback option if the original plan turns out to be too brittle/complex.
On Fri, Jul 19, 2013 at 06:08:48PM +0200, Florian Weimer wrote:
[..]
Have you considered a non-cryptographic solution, like a physical presence check to (temporarily) disable Secure Boot so that the kexec restriction no longer applies? This could be a fallback option if the original plan turns out to be too brittle/complex.
I think kyle has a patch which will allow disabling secureboot restriction if one is on console. I will have to look into details and see how can I make use of it in kexec code to relax signature restrictions if user is on physical console.
[CC kyle]
Thanks Vivek
On Mon, Jul 22, 2013 at 02:54:41PM -0400, Vivek Goyal wrote:
On Fri, Jul 19, 2013 at 06:08:48PM +0200, Florian Weimer wrote:
[..]
Have you considered a non-cryptographic solution, like a physical presence check to (temporarily) disable Secure Boot so that the kexec restriction no longer applies? This could be a fallback option if the original plan turns out to be too brittle/complex.
I think kyle has a patch which will allow disabling secureboot restriction if one is on console. I will have to look into details and see how can I make use of it in kexec code to relax signature restrictions if user is on physical console.
http://pkgs.fedoraproject.org/cgit/kernel.git/tree/devel-sysrq-secure-boot-2...
It still needs a bit of work for edge cases, but seems to work ok in some simple VM testing.
--Kyle
vgoyal wrote:
[...]
Have you considered a non-cryptographic solution, like a physical presence check to (temporarily) disable Secure Boot so that the kexec restriction no longer applies? [...]
I think kyle has a patch which will allow disabling secureboot restriction if one is on console. [...]
Considering that kexec/kdump events get triggered asynchronously (whenevery the kernel panics), someone cannot be assumed to be sitting physically at the terminal, ready to press a sysrq to make that secureboot-disabling transition. (One wouldn't want to press sysrq-foo too early, since AIUI it's a one-way transition.)
- FChE
On Thursday, July 11, 2013 10:33:05 AM Vivek Goyal wrote:
Secondly, there are disagreements upstream w.r.t how locking down executable should happen. IMA folks want some functionality behind security hooks (as opposed to what I have done). So I am expecting that once patches do get merged upstream, they might be in little different shape altogether.
I don't know if the average person has played with IMA. It hashes all files being accessed depending on its policy. This is CPU intensive and will cause the system fans to run faster and the system uses more power. It also runs slower because of all the time spent hashing files. I reported this to upstream IMA developers a while back. I doubt anything has changed.
-Steve
On Thu, Jul 11, 2013 at 11:45:34AM -0400, Steve Grubb wrote:
On Thursday, July 11, 2013 10:33:05 AM Vivek Goyal wrote:
Secondly, there are disagreements upstream w.r.t how locking down executable should happen. IMA folks want some functionality behind security hooks (as opposed to what I have done). So I am expecting that once patches do get merged upstream, they might be in little different shape altogether.
I don't know if the average person has played with IMA. It hashes all files being accessed depending on its policy. This is CPU intensive and will cause the system fans to run faster and the system uses more power. It also runs slower because of all the time spent hashing files. I reported this to upstream IMA developers a while back. I doubt anything has changed.
This overhead shows up only one loads an IMA policy to do so. In my case I have exported some appraisal functions from IMA code so these can be directly called by other kernel components. And I call these functions from elf loader code.
That way, in regular configuration no hashing of all the files will take place. Executables will be hashed only if they are signed and only if user has asked to run executable locked down in memory. (I have created a way so that in security.ima attribute one can put additional info to run executable locked down in memory).
So we just need to enable IMA but for regular users I am not expecting any significant overhead to show up. It will show up only if users choose to load some IMA policy in the system.
Thanks Vivek
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal vgoyal@redhat.com
Currently kexec/kdump is disabled on machines with secureboot enabled. This feature aims to enable kexec/kdump on such machines.
As a minor language nit, I would change the feature title itself to 'Allow kdump...' ; a quick reading of the title otherwise might imply that it would be enabled OOTB on such machines. Or maybe I'm being overly anal.
Bill
On Thu, Jul 11, 2013 at 10:53:42AM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik@redhat.com) said:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal vgoyal@redhat.com
Currently kexec/kdump is disabled on machines with secureboot enabled. This feature aims to enable kexec/kdump on such machines.
As a minor language nit, I would change the feature title itself to 'Allow kdump...' ; a quick reading of the title otherwise might imply that it would be enabled OOTB on such machines. Or maybe I'm being overly anal.
I changed the feature title to "Allow kdump on secureboot machines".
Thanks Vivek
On 11 July 2013 05:40, Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal vgoyal@redhat.com
Currently kexec/kdump is disabled on machines with secureboot enabled. This feature aims to enable kexec/kdump on such machines.
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
My question is "Does kdump work even without secureboot?" In trying to debug a crashing bug with older kernels and F19 I enabled kdump to try and get a crashing image for the developers to work. I ran into a ton of issues and when asking on #fedora-kernel was given the strong impression that kdump was not expected to work by the various kernel developers.
Issues I ran into was:
1) kdump needs to write to an unencrypted disk space. I tried a USB disk and various other places but the best ability I got was reinstalling the laptop and making a /var/crash partition. 2) kdump didn't seem to dump for anything than the forced dump in the instruction manual.
In the end the kernel developers got me a kernel with enough oops detection to help find the problems but how do we get kdump to be more useful?
On Thu, Jul 11, 2013 at 11:58:56AM -0600, Stephen John Smoogen wrote:
On 11 July 2013 05:40, Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal vgoyal@redhat.com
Currently kexec/kdump is disabled on machines with secureboot enabled. This feature aims to enable kexec/kdump on such machines.
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
My question is "Does kdump work even without secureboot?"
Yes it works. Kdump user space bits rotted for a while in Fedora and it did not get enough care and attention. But things have changed now. We first put everything in Fedora and test it before we try to pull the same bits in rhel.
In trying to debug a crashing bug with older kernels and F19 I enabled kdump to try and get a crashing image for the developers to work. I ran into a ton of issues and when asking on #fedora-kernel was given the strong impression that kdump was not expected to work by the various kernel developers.
We have been testing F18 and F19 in our virtual machines and things work for us.
I think this is a wrong impression. Kdump should work in Fedora. For a long time I got the feedback that fedora users don't care about kdump working. But I think kdump is an important debugging facility and is very useful for enterprise distro. So at times it should be useful for fedora users too.
So now I am trying to make sure things work well in Fedora.
Issues I ran into was:
- kdump needs to write to an unencrypted disk space. I tried a USB disk
and various other places but the best ability I got was reinstalling the laptop and making a /var/crash partition.
Is your root encrypted? USB should have worked. Otherwise try dumping to NFS partition. Or ssh the dump out to a different machine. All of these should work.
- kdump didn't seem to dump for anything than the forced dump in the
instruction manual.
You mean dump did not trigger after panic or it did not complete after panic?
If kdump kernel is loaded, and panic happens or oops happens and panic_on_oops is set, we should transition in to second kernel and capture dump.
If it did not work, there must have been some kernel issue. Please open bugs for these issues.
In the end the kernel developers got me a kernel with enough oops detection to help find the problems but how do we get kdump to be more useful?
I think testing and bug reporting will help. I would love to have kdump enabled by default in Fedora. But it eats around 128MB of memory by default which keeps sitting and not used. So lot of people might not like it.
Thanks Vivek
On Thu, Jul 11, 2013 at 02:13:05PM -0400, Vivek Goyal wrote:
[..]
how do we get kdump to be more useful?
I think testing and bug reporting will help. I would love to have kdump enabled by default in Fedora. But it eats around 128MB of memory by default which keeps sitting and not used. So lot of people might not like it.
Right now there is no fedora specific kdump list to discuss issues. If there is enough interest, we can create one list just to discuss fedora kdump issues and fixes.
Thanks Vivek
On Thu, 2013-07-11 at 14:13 -0400, Vivek Goyal wrote:
I think this is a wrong impression. Kdump should work in Fedora. For a long time I got the feedback that fedora users don't care about kdump working. But I think kdump is an important debugging facility and is very useful for enterprise distro. So at times it should be useful for fedora users too.
So now I am trying to make sure things work well in Fedora.
FWIW, dedoimedo (a notoriously tough distro reviewer) mentioned to me in private correspondence that he's been hitting a kernel crash with F19 with one of his test systems and he'd want to use kdump to try and diagnose it, only it doesn't work on live images, he says. Is that accurate? If so, can it be made to work on live images?
On Thu, Jul 11, 2013 at 02:09:54PM -0700, Adam Williamson wrote:
On Thu, 2013-07-11 at 14:13 -0400, Vivek Goyal wrote:
I think this is a wrong impression. Kdump should work in Fedora. For a long time I got the feedback that fedora users don't care about kdump working. But I think kdump is an important debugging facility and is very useful for enterprise distro. So at times it should be useful for fedora users too.
So now I am trying to make sure things work well in Fedora.
FWIW, dedoimedo (a notoriously tough distro reviewer) mentioned to me in private correspondence that he's been hitting a kernel crash with F19 with one of his test systems and he'd want to use kdump to try and diagnose it, only it doesn't work on live images, he says. Is that accurate? If so, can it be made to work on live images?
I have never tested kdump on live images. What's the problem he is facing?
Thanks Vivek
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And upon successful signature verification, allow it to load kernel, initramfs, and binary blob. /sbin/kexec will verify signatures of kernel being loaded before it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be possible to generate all relevant blobs (kdump kernel/initrd, ...) at kernel build time and sign them using essentially the existing module signing mechanism, and let the _kernel_ do all signature verification? Then /sbin/kexec wouldn't have to be trusted at all.
=== Build and ship ima-evm-utils package === /sbin/kexec will be signed by evmctl. This utility will put an xattr security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in kernel to verify signature of /sbin/kexec upon execution.
(My motivation for the above question is that I view IMA (and any approach based on verifying only a pre-specified subset of files) as rather suspect, and that dm-verity makes much more sense to me for enforcing a "trusted base OS". This doesn't automatically mean that kdump shouldn't be using IMA, however I fear that once we start for one binary, calls to verify more of userspace using IMA will be difficult to resist.) Mirek
On Thu, Jul 18, 2013 at 08:51:36PM +0200, Miloslav Trmač wrote:
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And upon successful signature verification, allow it to load kernel, initramfs, and binary blob. /sbin/kexec will verify signatures of kernel being loaded before it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be possible to generate all relevant blobs (kdump kernel/initrd, ...) at kernel build time and sign them using essentially the existing module signing mechanism, and let the _kernel_ do all signature verification? Then /sbin/kexec wouldn't have to be trusted at all.
The short version of that is no, because kdump needs to build some code at runtime. Upstream have refused to revisit that design decision.
On Thu, Jul 18, 2013 at 08:51:36PM +0200, Miloslav Trmač wrote:
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines = https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
== Detailed description == /sbin/kexec prepares a binary blob, called purgatory. This code runs at priviliged level between kernel transition. With secureboot enabled, no unsigned code should run at privilige level 0, hence kexec/kdump is currently disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And upon successful signature verification, allow it to load kernel, initramfs, and binary blob. /sbin/kexec will verify signatures of kernel being loaded before it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be possible to generate all relevant blobs (kdump kernel/initrd, ...) at kernel build time and sign them using essentially the existing module signing mechanism, and let the _kernel_ do all signature verification? Then /sbin/kexec wouldn't have to be trusted at all.
=== Build and ship ima-evm-utils package === /sbin/kexec will be signed by evmctl. This utility will put an xattr security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in kernel to verify signature of /sbin/kexec upon execution.
(My motivation for the above question is that I view IMA (and any approach based on verifying only a pre-specified subset of files) as rather suspect, and that dm-verity makes much more sense to me for enforcing a "trusted base OS".
IIUC, dm-verity will just ensure that what was loaded from disk matches signature. It does not enforce any restrictions after that. That is make sure once signatures are verified, nothing unsgined can change the address space.
So disable ptrace() from unsigned process, run executable run locked in from memory etc.
IMA does not enforce this too. But it can possibly be enhanced to put some metadata in signature which says file needs to be executed locked in memory.
And IMA will allow me to do it per file and we don't have to sign everything which is on disk.
So for this use case where we want to sign only one thing, there is no need to make everything signed.
Thanks Vivek
-
Adam Williamson -
Bill Nottingham -
Florian Weimer -
Frank Ch. Eigler -
Jaroslav Reznik -
Kyle McMartin -
Matthew Garrett -
Miloslav Trmač -
Stephen John Smoogen -
Steve Grubb -
Vivek Goyal