6:40 a.m.
= Proposed System Wide Change: Enable kdump on secureboot machines =
https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal <vgoyal(a)redhat.com>
Currently kexec/kdump is disabled on machines with secureboot enabled. This
feature aims to enable kexec/kdump on such machines.
== Detailed description ==
/sbin/kexec prepares a binary blob, called purgatory. This code runs at
priviliged level between kernel transition. With secureboot enabled, no
unsigned code should run at privilige level 0, hence kexec/kdump is currently
disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And
upon successful signature verification, allow it to load kernel, initramfs, and
binary blob. /sbin/kexec will verify signatures of kernel being loaded before
it asks running kernel to load it.
There are quite a few pieces to the puzzle. I am listing the some of the
things which need to be done.
=== Build and ship ima-evm-utils package ===
/sbin/kexec will be signed by evmctl. This utility will put an xattr
security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in
kernel to verify signature of /sbin/kexec upon execution.
* There is a bz open 807476 for inclusion of this package since long time. Not
sure what it is stuck on though.
* There are some patches which are not upstream yet (like lock down executable
in memory) which we need to carry in this patckage till patches get upstream.
=== Changes to kexec-tools ===
Build /sbin/kexec statically. There is no infrastructure to sign and verify
signature of shared libraries. Even if we can sign and verify shared
libraries, currently kernel allows writing to shared libraries while these are
mapped. So one can overwrite the library after signature verification. So build
/sbin/kexec statically. kexec does not use nss related code, so there should
not be any issues w.r.t glibc calling into other shared libraries.
* Generate detached /sbin/kexec signautre at build time and install these
signature on target in seucurity.ima xattr when kexec-tools is installed.
* Modify kexec-tools so that it can call keyctl() and verify signature of a
new kernel being loaded.
=== Kernel Changes ===
Kernel needs to carry additional patches to do verify elf binary signature.
* There are patches to extend keyctl() so that user space can use it to verify
signature of a user buffer (vmlinuz in this case).
* These patches are not upstream, so these need to be carried in fedora till
patches get upstream.
* Kernel need to be signed using evmctl and detached signature need to be
generated. These signatures need to be installed on vmlinuz upon kernel rpm
installation in security.ima xattr.
=== Signing Key Management ===
Yet to be figured out. There are couple of ideas on table.
* Embed few keys in kernel and one of these keys will be used to sign
/sbin/kexec. In case of a key is revoked, use a new key from set of embedded
keys.
* Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF binary
should be signed by appropriate authority so it can be loaded in system
keyring.
== Scope ==
Proposal owners:
* Provide patches for kernel package.
* Provide patches for kexec-tools package.
* Provide patches for ima-evm-utils package.
* ima-evm-utils package will be new. It is not clear who will maintain that
package.
Other developers:
* Signing Process, Signing key management, Signing key loading on target after
boot and Signature installation on target are unknown areas right now. It is
not very clear how these things will be done. Once details have been figured
out, it might require work from other developers. Not sure though at this
point of time.
* Can't think any other major work required by other developers yet. We will
have to figure out how signing will take place and how signatures will be
installed on target system. And depending on how we do it, it might require
some work from other developers. Like making use of rpm hooks to install
signature etc. Given the fact that we are signing only one file /sbin/kexec, we
might get away doing changes in kexec-tools spec file.
Release engineering:
* We need to sort out how signing will take place. I will require release-
engineering help on this.
* For issues related to key management I will need help of security/release
engineering.
Policies and guidelines:
* I think to begin with we might not have to update any packaging guidelines
for this. Anything magic w.r.t signing can probably be limited to kexec-tools
spec file.
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
8:57 a.m.
On 07/11/2013 01:40 PM, Jaroslav Reznik wrote:
=== Build and ship ima-evm-utils package ===
/sbin/kexec will be signed by evmctl. This utility will put an xattr
security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in
kernel to verify signature of /sbin/kexec upon execution.
* There is a bz open 807476 for inclusion of this package since long time. Not
sure what it is stuck on though.
* There are some patches which are not upstream yet (like lock down executable
in memory) which we need to carry in this patckage till patches get upstream.
Is there a chance this (and the other patches mentioned below) actually
makes it in the kernel? Are at least the VM changes part of upstream
already?
I don't think it would make sense to add more and more Fedora-specific
patches which implement security functionality. I don't want Fedora to
become the next Android.
=== Kernel Changes ===
Kernel needs to carry additional patches to do verify elf binary signature.
* There are patches to extend keyctl() so that user space can use it to verify
signature of a user buffer (vmlinuz in this case).
* These patches are not upstream, so these need to be carried in fedora till
patches get upstream.
* Kernel need to be signed using evmctl and detached signature need to be
generated. These signatures need to be installed on vmlinuz upon kernel rpm
installation in security.ima xattr.
Does this mean your implementation of signature checking will be
completely independent of UEFI Secure Boot (unless you decide to use
that to obtain the trust root)?
=== Signing Key Management ===
Yet to be figured out. There are couple of ideas on table.
* Embed few keys in kernel and one of these keys will be used to sign
/sbin/kexec. In case of a key is revoked, use a new key from set of embedded
keys.
How do you intend to handle revocation?
* Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF
binary
should be signed by appropriate authority so it can be loaded in system
keyring.
Who is the appropriate authority?
--
Florian Weimer / Red Hat Product Security Team
9:33 a.m.
On Thu, Jul 11, 2013 at 03:57:38PM +0200, Florian Weimer wrote:
On 07/11/2013 01:40 PM, Jaroslav Reznik wrote:
>=== Build and ship ima-evm-utils package ===
>/sbin/kexec will be signed by evmctl. This utility will put an xattr
>security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in
>kernel to verify signature of /sbin/kexec upon execution.
>
>* There is a bz open 807476 for inclusion of this package since long time. Not
>sure what it is stuck on though.
>
>* There are some patches which are not upstream yet (like lock down executable
>in memory) which we need to carry in this patckage till patches get upstream.
Is there a chance this (and the other patches mentioned below)
actually makes it in the kernel? Are at least the VM changes part
of upstream already?
I don't think it would make sense to add more and more
Fedora-specific patches which implement security functionality. I
don't want Fedora to become the next Android.
I don't see those patches going upstream in near term. First of all
base secureboot patches need to go upstream. And at this point of
time upsteram does not seem to be eager to do anything more in kernel
for secureboot.
Secondly, there are disagreements upstream w.r.t how locking down
executable should happen. IMA folks want some functionality behind
security hooks (as opposed to what I have done). So I am expecting
that once patches do get merged upstream, they might be in little
different shape altogether.
I think now we can not back out. Merging secureboot patches without
these being upstream broke other subsystems (kdump, systemtap,..). And
we should now allow other non-upstream patches to go in to make
these subsystems work again. Or we should remove kernel lockdown
functionality in secureboot mode altogether (as upstream does not
have it).
>=== Kernel Changes ===
>Kernel needs to carry additional patches to do verify elf binary signature.
>* There are patches to extend keyctl() so that user space can use it to verify
>signature of a user buffer (vmlinuz in this case).
>* These patches are not upstream, so these need to be carried in fedora till
>patches get upstream.
>* Kernel need to be signed using evmctl and detached signature need to be
>generated. These signatures need to be installed on vmlinuz upon kernel rpm
>installation in security.ima xattr.
Does this mean your implementation of signature checking will be
completely independent of UEFI Secure Boot (unless you decide to use
that to obtain the trust root)?
Yes. I am going to use IMA for signature verification. These signatures
formats are very close to what is used for module signing (PKCS1.5 with
some metadata).
For trust chain, we will still use secureboot trust chain and trust
an executable only if it has been signed by a key in .system_keyring.
>=== Signing Key Management ===
>Yet to be figured out. There are couple of ideas on table.
>
>* Embed few keys in kernel and one of these keys will be used to sign
>/sbin/kexec. In case of a key is revoked, use a new key from set of embedded
>keys.
How do you intend to handle revocation?
I have not written the code to check blacklist yet but I plan to do
that later.
If a key is revoked, I am expecting that we will request M$ for an
update. And also push out new version of /sbin/kexec signed with
a new key.
We will need to have this new key signed with either redhat certificate
or signed with M$ so that it can be loaded in already running kernel. That
will make sure old /sbin/kexec instances don't run while new instances
signed with new key do run.
>* Ship a PE/COFF wrapped key in kexec-tools package. This PE/COFF binary
>should be signed by appropriate authority so it can be loaded in system
>keyring.
Who is the appropriate authority?
I am not sure yet. I was thinking that can we embed fedora/redhat
certificate in kernel (like shim) and use that as CA key to sign
/sbin/kexec key.
Signing a key using CA key will require loading of that key every
reboot automatically. I am not sure how that can be handled. May
be rpm packages can drop those keys in some directory and a system
service scans for keys and loads these every reboot?
Thanks
Vivek
10:19 a.m.
On 07/11/2013 04:33 PM, Vivek Goyal wrote:
> I don't think it would make sense to add more and more
> Fedora-specific patches which implement security functionality. I
> don't want Fedora to become the next Android.
I don't see those patches going upstream in near term. First of all
base secureboot patches need to go upstream. And at this point of
time upsteram does not seem to be eager to do anything more in kernel
for secureboot.
The IMA stuff looks pretty independent of Secure Boot to me. It seems
upstream picked up some of it in 2.6.30.
Secondly, there are disagreements upstream w.r.t how locking down
executable should happen. IMA folks want some functionality behind
security hooks (as opposed to what I have done). So I am expecting
that once patches do get merged upstream, they might be in little
different shape altogether.
The important question is whether we can drop our own patches and switch
to whatever upstream does when the time comes.
I think now we can not back out. Merging secureboot patches without
these being upstream broke other subsystems (kdump, systemtap,..).
Sure. But systemtap (and things like PCI passthrough) are fundamentally
incompatible with our approach to Secure Boot. There are conceptual
challenges like irreversible software updates, too. At a certain point,
we will have added so much inconvenience that only very few users will
use this feature. Upstream is not totally crazy in rejecting Fedora's
restrictive approach.
The proposed change goes in the right direction (more user control), but
we're still missing things on the restrictive side (dbx updates,
Fedora-local revocation checking, and others I'm not aware of). And as
long as Fedora and the more lenient distributions are under the same
trust root, Fedora users get pretty close to zero additional security
benefit, despite all the effort we put into this.
Yes. I am going to use IMA for signature verification. These
signatures
formats are very close to what is used for module signing (PKCS1.5 with
some metadata).
For trust chain, we will still use secureboot trust chain and trust
an executable only if it has been signed by a key in .system_keyring.
Okay, that's a dependency on the Secure Boot patchset, but not Secure
Boot as a technology. Good to know.
I have not written the code to check blacklist yet but I plan to do
that later.
There's a challenge to obtain the blacklist in a way that is safe
against rollback.
I don't think we have indirect (and proven) write access to the dbx
variable yet.
If a key is revoked, I am expecting that we will request M$ for an
update. And also push out new version of /sbin/kexec signed with
a new key.
Then you better add a separate CA root for /sbin/kexec, so that we don't
have to blacklist the kernel. (An alternative would be to blacklist
binaries based on their hashes.)
Revocation as the result of a software bug seems far more likely to me
than revocation because of CA compromise (although I don't know what
Fedora does with these keys, but at least this is a something that's
solvable in principle with suitable hardware and processes).
Signing a key using CA key will require loading of that key every
reboot automatically. I am not sure how that can be handled. May
be rpm packages can drop those keys in some directory and a system
service scans for keys and loads these every reboot?
Not sure I follow. You can't add additional root (that is, CA) keys
after building. What you could do is add intermediate CAs, but the way
X.509 works, this is unnecessary if you put the entire chain (perhaps
excluding the root) into the binary. So this shouldn't be required at all.
--
Florian Weimer / Red Hat Product Security Team
11:03 a.m.
On Thu, Jul 11, 2013 at 05:19:58PM +0200, Florian Weimer wrote:
On 07/11/2013 04:33 PM, Vivek Goyal wrote:
>>I don't think it would make sense to add more and more
>>Fedora-specific patches which implement security functionality. I
>>don't want Fedora to become the next Android.
>
>I don't see those patches going upstream in near term. First of all
>base secureboot patches need to go upstream. And at this point of
>time upsteram does not seem to be eager to do anything more in kernel
>for secureboot.
The IMA stuff looks pretty independent of Secure Boot to me. It
seems upstream picked up some of it in 2.6.30.
It is but it implements stuff which is needed to meet TCB requirements.
Current implementation is nowhere near to require secureboot requirements.
For example, executables are not locked down in memory. That means
after signature verification, if executable gets swapped out, it can
be tweaked by root.
Any direct changes to disk to a file are not detected. So modify
the disk blocks after signature verification and IMA will not
notice it.
Without secureboot we don't have any requirement where we need
the capability to extend root of trust to user space application. Current
IMA is not sufficient to meet secureboot requirements. So pushing
more patches in because they are required to meet secureboot
requirements is hard. (Because lot of people don't bite into the
theory of locking down kernel in secureboot mode).
So there is very little direct dependecny on secureboot. It is
more of why do you need these enahancements. And answer is because
secureboot will enforce that. Well, secureboot locking is not
even in kernel, and kdump is not even broken upstream, so why
are you bothering to push something in now.
>Secondly, there are disagreements upstream w.r.t how locking
down
>executable should happen. IMA folks want some functionality behind
>security hooks (as opposed to what I have done). So I am expecting
>that once patches do get merged upstream, they might be in little
>different shape altogether.
The important question is whether we can drop our own patches and
switch to whatever upstream does when the time comes.
I think we should be able to do that. We are expecting only /sbin/kexec
to use this functionality and if things change we can change /sbin/kexec
and signing process as we control everything.
I think important thing will be to emphasize that other applications
should not try to latch on to new keyctl() ioctl option to verify
signature of a user space buffer or IMA functionality to put extra
data in signature which locks down executable in memory. These
are not stable interfaces and might disappear in next fedora
release.
>I think now we can not back out. Merging secureboot patches without
>these being upstream broke other subsystems (kdump, systemtap,..).
Sure. But systemtap (and things like PCI passthrough) are
fundamentally incompatible with our approach to Secure Boot. There
are conceptual challenges like irreversible software updates, too.
At a certain point, we will have added so much inconvenience that
only very few users will use this feature. Upstream is not totally
crazy in rejecting Fedora's restrictive approach.
Is it time to re-visit the decision of locking down kernel on secureboot
machine given the fact that upstream does not seem to like the idea.
What are other distributions planning to do? Are they carrying additional
patches or they have decided to go upstream way of not locking down
kernel.
If others are going upstream way, then is it worth that fedora continues
to be different and in the process we continue to add more out of the
tree patches to make things like kdump work with secureboot.
The proposed change goes in the right direction (more user control),
but we're still missing things on the restrictive side (dbx updates,
Fedora-local revocation checking, and others I'm not aware of). And
as long as Fedora and the more lenient distributions are under the
same trust root, Fedora users get pretty close to zero additional
security benefit, despite all the effort we put into this.
>Yes. I am going to use IMA for signature verification. These signatures
>formats are very close to what is used for module signing (PKCS1.5 with
>some metadata).
>
>For trust chain, we will still use secureboot trust chain and trust
>an executable only if it has been signed by a key in .system_keyring.
Okay, that's a dependency on the Secure Boot patchset, but not
Secure Boot as a technology. Good to know.
>I have not written the code to check blacklist yet but I plan to do
>that later.
There's a challenge to obtain the blacklist in a way that is safe
against rollback.
I don't think we have indirect (and proven) write access to the dbx
variable yet.
Ok, I am not aware of details here but if blacklisting does not work
already, then this concern is not specific to kdump. Once it starts
working and I can blacklisted keys in blacklist keyring, I should be
able to check for key in that keyring and disallow sys_kexec().
>If a key is revoked, I am expecting that we will request M$ for an
>update. And also push out new version of /sbin/kexec signed with
>a new key.
Then you better add a separate CA root for /sbin/kexec, so that we
don't have to blacklist the kernel. (An alternative would be to
blacklist binaries based on their hashes.)
To keep things simple, I was thinking if we use same CA root which
we use for signing kernel, it will become easier and I don't have
to start a new CA root only for /sbin/kexec.
Downside is that if CA root used for signing kernel ever gets
compromised, then we will have to issue kexec-tools update too (in
addition to shim, kernel, grub). I guess issuing kexec-tools update
is simpler.
Revocation as the result of a software bug seems far more likely to
me than revocation because of CA compromise (although I don't know
what Fedora does with these keys, but at least this is a something
that's solvable in principle with suitable hardware and processes).
>Signing a key using CA key will require loading of that key every
>reboot automatically. I am not sure how that can be handled. May
>be rpm packages can drop those keys in some directory and a system
>service scans for keys and loads these every reboot?
Not sure I follow. You can't add additional root (that is, CA) keys
after building. What you could do is add intermediate CAs, but the
way X.509 works, this is unnecessary if you put the entire chain
(perhaps excluding the root) into the binary. So this shouldn't be
required at all.
I was hoping that CA certificate will be embedded in kernel at build
time. And using that CA certificate we will sign keys which in turn
will sign /sbin/kexec. And that key will be loaded dynamically at
boot time in system_keyring. (And it should be allowed because CA key
is already there and it can verify the new key being loaded).
IIUC, PKCS7 signatures can carry all the signign chain info and
hence intermediate certifiates don't have to be loaded separately.
Like modules, /sbin/kexec will not use PKCS7 format. So it only knows
about the key it has been signed with and expects that key to be
loaded already.
Did I understand your question correctly?
Thanks
Vivek
11:08 a.m.
On 07/11/2013 06:03 PM, Vivek Goyal wrote:
It is but it implements stuff which is needed to meet TCB
requirements.
Current implementation is nowhere near to require secureboot requirements.
For example, executables are not locked down in memory. That means
after signature verification, if executable gets swapped out, it can
be tweaked by root.
Ah, okay, so for Trusted Boot, the onus is on the system administrator
to make sure that he can provide attestation (by not running any dodgy
binaries which would invalidate that), but for Security Boot, we have to
prevent the system administrator to run such binaries. Makes sense.
> The important question is whether we can drop our own patches
and
> switch to whatever upstream does when the time comes.
I think we should be able to do that. We are expecting only /sbin/kexec
to use this functionality and if things change we can change /sbin/kexec
and signing process as we control everything.
I think important thing will be to emphasize that other applications
should not try to latch on to new keyctl() ioctl option to verify
signature of a user space buffer or IMA functionality to put extra
data in signature which locks down executable in memory. These
are not stable interfaces and might disappear in next fedora
release.
Okay.
There some potential ABI impact from the VM-related changes, but this
shouldn't be a problem for Fedora.
Is it time to re-visit the decision of locking down kernel on
secureboot
machine given the fact that upstream does not seem to like the idea.
What are other distributions planning to do? Are they carrying additional
patches or they have decided to go upstream way of not locking down
kernel.
I think they don't do any lockdown after boot. CAP_COMPROMISE_KERNEL
(or what's it called these days) is a Fedora thing.
Ok, I am not aware of details here but if blacklisting does not work
already, then this concern is not specific to kdump. Once it starts
working and I can blacklisted keys in blacklist keyring, I should be
able to check for key in that keyring and disallow sys_kexec().
There are some potential approaches to blacklisting which would be
incompatible with certain ways of signing binaries. So it's not
immediately clear if an unrelated blacklisting service could be made to
work with userspace signatures.
So let's skip the blacklisting discussion for now. If we ever have to
implement it, it's going to be messy, kexec or no kexec. (See the
discussion about post-release media respins.)
Like modules, /sbin/kexec will not use PKCS7 format. So it only
knows
about the key it has been signed with and expects that key to be
loaded already.
Ah, okay, that's an unfortunate upstream design decision. In this
light, what you're planning to do seems fairly unavoidable.
Have you considered a non-cryptographic solution, like a physical
presence check to (temporarily) disable Secure Boot so that the kexec
restriction no longer applies? This could be a fallback option if the
original plan turns out to be too brittle/complex.
--
Florian Weimer / Red Hat Product Security Team
1:54 p.m.
On Fri, Jul 19, 2013 at 06:08:48PM +0200, Florian Weimer wrote:
[..]
Have you considered a non-cryptographic solution, like a physical
presence check to (temporarily) disable Secure Boot so that the
kexec restriction no longer applies? This could be a fallback
option if the original plan turns out to be too brittle/complex.
I think kyle has a patch which will allow disabling secureboot
restriction if one is on console. I will have to look into details
and see how can I make use of it in kexec code to relax signature
restrictions if user is on physical console.
[CC kyle]
Thanks
Vivek
1:57 p.m.
On Mon, Jul 22, 2013 at 02:54:41PM -0400, Vivek Goyal wrote:
On Fri, Jul 19, 2013 at 06:08:48PM +0200, Florian Weimer wrote:
[..]
> Have you considered a non-cryptographic solution, like a physical
> presence check to (temporarily) disable Secure Boot so that the
> kexec restriction no longer applies? This could be a fallback
> option if the original plan turns out to be too brittle/complex.
I think kyle has a patch which will allow disabling secureboot
restriction if one is on console. I will have to look into details
and see how can I make use of it in kexec code to relax signature
restrictions if user is on physical console.
http://pkgs.fedoraproject.org/cgit/kernel.git/tree/devel-sysrq-secure-boo...
It still needs a bit of work for edge cases, but seems to work ok
in some simple VM testing.
--Kyle
4:26 p.m.
vgoyal wrote:
[...]
> Have you considered a non-cryptographic solution, like a physical
> presence check to (temporarily) disable Secure Boot so that the
> kexec restriction no longer applies? [...]
I think kyle has a patch which will allow disabling secureboot
restriction if one is on console. [...]
Considering that kexec/kdump events get triggered asynchronously
(whenevery the kernel panics), someone cannot be assumed to be sitting
physically at the terminal, ready to press a sysrq to make that
secureboot-disabling transition. (One wouldn't want to press
sysrq-foo too early, since AIUI it's a one-way transition.)
- FChE
10:45 a.m.
On Thursday, July 11, 2013 10:33:05 AM Vivek Goyal wrote:
Secondly, there are disagreements upstream w.r.t how locking down
executable should happen. IMA folks want some functionality behind
security hooks (as opposed to what I have done). So I am expecting
that once patches do get merged upstream, they might be in little
different shape altogether.
I don't know if the average person has played with IMA. It hashes all files
being accessed depending on its policy. This is CPU intensive and will cause
the system fans to run faster and the system uses more power. It also runs
slower because of all the time spent hashing files. I reported this to upstream
IMA developers a while back. I doubt anything has changed.
-Steve
11:07 a.m.
On Thu, Jul 11, 2013 at 11:45:34AM -0400, Steve Grubb wrote:
On Thursday, July 11, 2013 10:33:05 AM Vivek Goyal wrote:
> Secondly, there are disagreements upstream w.r.t how locking down
> executable should happen. IMA folks want some functionality behind
> security hooks (as opposed to what I have done). So I am expecting
> that once patches do get merged upstream, they might be in little
> different shape altogether.
I don't know if the average person has played with IMA. It hashes all files
being accessed depending on its policy. This is CPU intensive and will cause
the system fans to run faster and the system uses more power. It also runs
slower because of all the time spent hashing files. I reported this to upstream
IMA developers a while back. I doubt anything has changed.
This overhead shows up only one loads an IMA policy to do so. In my case
I have exported some appraisal functions from IMA code so these can be
directly called by other kernel components. And I call these functions
from elf loader code.
That way, in regular configuration no hashing of all the files will
take place. Executables will be hashed only if they are signed and
only if user has asked to run executable locked down in memory. (I
have created a way so that in security.ima attribute one can
put additional info to run executable locked down in memory).
So we just need to enable IMA but for regular users I am not
expecting any significant overhead to show up. It will show up
only if users choose to load some IMA policy in the system.
Thanks
Vivek
9:53 a.m.
Jaroslav Reznik (jreznik(a)redhat.com) said:
= Proposed System Wide Change: Enable kdump on secureboot machines =
https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal <vgoyal(a)redhat.com>
Currently kexec/kdump is disabled on machines with secureboot enabled. This
feature aims to enable kexec/kdump on such machines.
As a minor language nit, I would change the feature title itself to 'Allow
kdump...' ; a quick reading of the title otherwise might imply that it would
be enabled OOTB on such machines. Or maybe I'm being overly anal.
Bill
10:06 a.m.
On Thu, Jul 11, 2013 at 10:53:42AM -0400, Bill Nottingham wrote:
Jaroslav Reznik (jreznik(a)redhat.com) said:
> = Proposed System Wide Change: Enable kdump on secureboot machines =
> https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
>
> Change owner(s): Vivek Goyal <vgoyal(a)redhat.com>
>
> Currently kexec/kdump is disabled on machines with secureboot enabled. This
> feature aims to enable kexec/kdump on such machines.
As a minor language nit, I would change the feature title itself to 'Allow
kdump...' ; a quick reading of the title otherwise might imply that it would
be enabled OOTB on such machines. Or maybe I'm being overly anal.
I changed the feature title to "Allow kdump on secureboot machines".
Thanks
Vivek
12:58 p.m.
On 11 July 2013 05:40, Jaroslav Reznik <jreznik(a)redhat.com> wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines =
https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
Change owner(s): Vivek Goyal <vgoyal(a)redhat.com>
Currently kexec/kdump is disabled on machines with secureboot enabled. This
feature aims to enable kexec/kdump on such machines.
== Detailed description ==
/sbin/kexec prepares a binary blob, called purgatory. This code runs at
priviliged level between kernel transition. With secureboot enabled, no
unsigned code should run at privilige level 0, hence kexec/kdump is
currently
disabled if secureboot is enabled.
My question is "Does kdump work even without secureboot?" In trying to
debug a crashing bug with older kernels and F19 I enabled kdump to try and
get a crashing image for the developers to work. I ran into a ton of issues
and when asking on #fedora-kernel was given the strong impression that
kdump was not expected to work by the various kernel developers.
Issues I ran into was:
1) kdump needs to write to an unencrypted disk space. I tried a USB disk
and various other places but the best ability I got was reinstalling the
laptop and making a /var/crash partition.
2) kdump didn't seem to dump for anything than the forced dump in the
instruction manual.
In the end the kernel developers got me a kernel with enough oops detection
to help find the problems but how do we get kdump to be more useful?
--
Stephen J Smoogen.
1:13 p.m.
On Thu, Jul 11, 2013 at 11:58:56AM -0600, Stephen John Smoogen wrote:
On 11 July 2013 05:40, Jaroslav Reznik <jreznik(a)redhat.com>
wrote:
> = Proposed System Wide Change: Enable kdump on secureboot machines =
> https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
>
> Change owner(s): Vivek Goyal <vgoyal(a)redhat.com>
>
> Currently kexec/kdump is disabled on machines with secureboot enabled. This
> feature aims to enable kexec/kdump on such machines.
>
> == Detailed description ==
> /sbin/kexec prepares a binary blob, called purgatory. This code runs at
> priviliged level between kernel transition. With secureboot enabled, no
> unsigned code should run at privilige level 0, hence kexec/kdump is
> currently
> disabled if secureboot is enabled.
>
>
My question is "Does kdump work even without secureboot?"
Yes it works. Kdump user space bits rotted for a while in Fedora and
it did not get enough care and attention. But things have changed now.
We first put everything in Fedora and test it before we try to pull
the same bits in rhel.
In trying to debug a crashing bug with older kernels and F19 I
enabled kdump
to try and
get a crashing image for the developers to work. I ran into a ton of issues
and when asking on #fedora-kernel was given the strong impression that
kdump was not expected to work by the various kernel developers.
We have been testing F18 and F19 in our virtual machines and things work
for us.
I think this is a wrong impression. Kdump should work in Fedora. For a
long time I got the feedback that fedora users don't care about kdump
working. But I think kdump is an important debugging facility and is
very useful for enterprise distro. So at times it should be useful for
fedora users too.
So now I am trying to make sure things work well in Fedora.
Issues I ran into was:
1) kdump needs to write to an unencrypted disk space. I tried a USB disk
and various other places but the best ability I got was reinstalling the
laptop and making a /var/crash partition.
Is your root encrypted? USB should have worked. Otherwise try dumping
to NFS partition. Or ssh the dump out to a different machine. All of
these should work.
2) kdump didn't seem to dump for anything than the forced dump in
the
instruction manual.
You mean dump did not trigger after panic or it did not complete after
panic?
If kdump kernel is loaded, and panic happens or oops happens and
panic_on_oops is set, we should transition in to second kernel and capture
dump.
If it did not work, there must have been some kernel issue. Please open
bugs for these issues.
In the end the kernel developers got me a kernel with enough oops detection
to help find the problems but how do we get kdump to be more useful?
I think testing and bug reporting will help. I would love to have kdump
enabled by default in Fedora. But it eats around 128MB of memory by
default which keeps sitting and not used. So lot of people might not like
it.
Thanks
Vivek
1:23 p.m.
On Thu, Jul 11, 2013 at 02:13:05PM -0400, Vivek Goyal wrote:
[..]
how do we get kdump to be more useful?
I think testing and bug reporting will help. I would love to have kdump
enabled by default in Fedora. But it eats around 128MB of memory by
default which keeps sitting and not used. So lot of people might not like
it.
Right now there is no fedora specific kdump list to discuss issues. If
there is enough interest, we can create one list just to discuss fedora
kdump issues and fixes.
Thanks
Vivek
4:09 p.m.
On Thu, 2013-07-11 at 14:13 -0400, Vivek Goyal wrote:
I think this is a wrong impression. Kdump should work in Fedora. For
a
long time I got the feedback that fedora users don't care about kdump
working. But I think kdump is an important debugging facility and is
very useful for enterprise distro. So at times it should be useful for
fedora users too.
So now I am trying to make sure things work well in Fedora.
FWIW, dedoimedo (a notoriously tough distro reviewer) mentioned to me in
private correspondence that he's been hitting a kernel crash with F19
with one of his test systems and he'd want to use kdump to try and
diagnose it, only it doesn't work on live images, he says. Is that
accurate? If so, can it be made to work on live images?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
9:55 a.m.
On Thu, Jul 11, 2013 at 02:09:54PM -0700, Adam Williamson wrote:
On Thu, 2013-07-11 at 14:13 -0400, Vivek Goyal wrote:
> I think this is a wrong impression. Kdump should work in Fedora. For a
> long time I got the feedback that fedora users don't care about kdump
> working. But I think kdump is an important debugging facility and is
> very useful for enterprise distro. So at times it should be useful for
> fedora users too.
>
> So now I am trying to make sure things work well in Fedora.
FWIW, dedoimedo (a notoriously tough distro reviewer) mentioned to me in
private correspondence that he's been hitting a kernel crash with F19
with one of his test systems and he'd want to use kdump to try and
diagnose it, only it doesn't work on live images, he says. Is that
accurate? If so, can it be made to work on live images?
I have never tested kdump on live images. What's the problem he is facing?
Thanks
Vivek
1:51 p.m.
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik <jreznik(a)redhat.com> wrote:
= Proposed System Wide Change: Enable kdump on secureboot machines =
https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
== Detailed description ==
/sbin/kexec prepares a binary blob, called purgatory. This code runs at
priviliged level between kernel transition. With secureboot enabled, no
unsigned code should run at privilige level 0, hence kexec/kdump is currently
disabled if secureboot is enabled.
One proposed way to solve the problem is that sign /sbin/kexec utility. And
upon successful signature verification, allow it to load kernel, initramfs, and
binary blob. /sbin/kexec will verify signatures of kernel being loaded before
it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be
possible to generate all relevant blobs (kdump kernel/initrd, ...) at
kernel build time and sign them using essentially the existing module
signing mechanism, and let the _kernel_ do all signature verification?
Then /sbin/kexec wouldn't have to be trusted at all.
=== Build and ship ima-evm-utils package ===
/sbin/kexec will be signed by evmctl. This utility will put an xattr
security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in
kernel to verify signature of /sbin/kexec upon execution.
(My motivation for the above question is that I view IMA (and any
approach based on verifying only a pre-specified subset of files) as
rather suspect, and that dm-verity makes much more sense to me for
enforcing a "trusted base OS". This doesn't automatically mean that
kdump shouldn't be using IMA, however I fear that once we start for
one binary, calls to verify more of userspace using IMA will be
difficult to resist.)
Mirek
1:57 p.m.
On Thu, Jul 18, 2013 at 08:51:36PM +0200, Miloslav Trmač wrote:
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik
<jreznik(a)redhat.com> wrote:
> = Proposed System Wide Change: Enable kdump on secureboot machines =
> https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
> == Detailed description ==
> /sbin/kexec prepares a binary blob, called purgatory. This code runs at
> priviliged level between kernel transition. With secureboot enabled, no
> unsigned code should run at privilige level 0, hence kexec/kdump is currently
> disabled if secureboot is enabled.
>
> One proposed way to solve the problem is that sign /sbin/kexec utility. And
> upon successful signature verification, allow it to load kernel, initramfs, and
> binary blob. /sbin/kexec will verify signatures of kernel being loaded before
> it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be
possible to generate all relevant blobs (kdump kernel/initrd, ...) at
kernel build time and sign them using essentially the existing module
signing mechanism, and let the _kernel_ do all signature verification?
Then /sbin/kexec wouldn't have to be trusted at all.
The short version of that is no, because kdump needs to build some code
at runtime. Upstream have refused to revisit that design decision.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
10:52 a.m.
On Thu, Jul 18, 2013 at 08:51:36PM +0200, Miloslav Trmač wrote:
On Thu, Jul 11, 2013 at 1:40 PM, Jaroslav Reznik
<jreznik(a)redhat.com> wrote:
> = Proposed System Wide Change: Enable kdump on secureboot machines =
> https://fedoraproject.org/wiki/Changes/Kdump_with_secureboot
> == Detailed description ==
> /sbin/kexec prepares a binary blob, called purgatory. This code runs at
> priviliged level between kernel transition. With secureboot enabled, no
> unsigned code should run at privilige level 0, hence kexec/kdump is currently
> disabled if secureboot is enabled.
>
> One proposed way to solve the problem is that sign /sbin/kexec utility. And
> upon successful signature verification, allow it to load kernel, initramfs, and
> binary blob. /sbin/kexec will verify signatures of kernel being loaded before
> it asks running kernel to load it.
For someone like me unfamiliar with kdump architecture, wouldn't it be
possible to generate all relevant blobs (kdump kernel/initrd, ...) at
kernel build time and sign them using essentially the existing module
signing mechanism, and let the _kernel_ do all signature verification?
Then /sbin/kexec wouldn't have to be trusted at all.
> === Build and ship ima-evm-utils package ===
> /sbin/kexec will be signed by evmctl. This utility will put an xattr
> security.ima on /sbin/kexec file and kernel will leverage IMA infrastructure in
> kernel to verify signature of /sbin/kexec upon execution.
(My motivation for the above question is that I view IMA (and any
approach based on verifying only a pre-specified subset of files) as
rather suspect, and that dm-verity makes much more sense to me for
enforcing a "trusted base OS".
IIUC, dm-verity will just ensure that what was loaded from disk matches
signature. It does not enforce any restrictions after that. That is make
sure once signatures are verified, nothing unsgined can change the
address space.
So disable ptrace() from unsigned process, run executable run locked in
from memory etc.
IMA does not enforce this too. But it can possibly be enhanced to put
some metadata in signature which says file needs to be executed locked
in memory.
And IMA will allow me to do it per file and we don't have to sign
everything which is on disk.
So for this use case where we want to sign only one thing, there is
no need to make everything signed.
Thanks
Vivek
3993
days inactive
4004
days old
20 comments
11 participants
participants (11)
-
Adam Williamson -
Bill Nottingham -
Florian Weimer -
Frank Ch. Eigler -
Jaroslav Reznik -
Kyle McMartin -
Matthew Garrett -
Miloslav Trmač -
Stephen John Smoogen -
Steve Grubb -
Vivek Goyal