2:24 p.m.
https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
By design, ostree does not manage bootloader updates as they can not
(yet) happen in a transactional, atomic and safe fashion. Thus bootupd
(https://github.com/coreos/bootupd) was created to solve this issue
and enable admin-lead bootloader updates. This change is about
enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
to make bootloader updates easier.
== Owner ==
* [[User:Siosm| Timothée Ravier]] <siosm(a)fedoraproject.org>
* [[User:Tpopela| Tomáš Popela]] <tpopela(a)fedoraproject.org>
* [[User:Walters| Colin Walters]] <walters(a)fedoraproject.org>
== Detailed Description ==
Adding bootupd full bootupd support has two immediate benefits:
* User will be able to easily update the bootloader on their system.
This will let them apply Secure Boot dbx updates that block old
bootloaders with known vulnerabilities from loading. See:
** https://github.com/fedora-silverblue/issue-tracker/issues/355
** https://bugzilla.redhat.com/show_bug.cgi?id=2127995
* We can start planning for the removal of the ostree-grub2 package
from the images to solve the "all deployments are shown twice in GRUB"
issue. This bug comes from the fact that old GRUB versions that do not
have BLS support and we thus can not only rely on BLS support in
ostree to generate boot and have to also update the grub configuration
for every updates via the scripts in the ostree-grub2 package. This
has already (indirectly) caused a major upgrade issue on
Silverblue/Kinoite requiring manual interventions from all users. See:
**
https://fedoramagazine.org/manual-action-required-to-update-fedora-silver...
** https://github.com/fedora-silverblue/issue-tracker/issues/120
Note that we can not yet enable unattended bootloader updates as even
though bootupd tries hard hard to make those updates as safe as
possible, it is currently not possible that they are safe if the
system crashes (or loses power) at the wrong time. The following
change in shim (https://github.com/rhboot/shim/pull/502) should help
with that.
Thus bootloaders updates will remain a manually user triggered
operation for now.
Also note that this change currently relies on the image being
composed via rpm-ostree in unified core, which is the subject of the
following change also proposed for Fedora 38:
https://fedoraproject.org/wiki/Changes/FedoraSilverblueUnifiedCore
== Feedback ==
None so far.
<!-- Summarize the feedback from the community and address why you
chose not to accept proposed alternatives. This section is optional
for all change proposals but is strongly suggested. Incorporating
feedback here as it is raised gives FESCo a clearer view of your
proposal and leaves a good record for the future. If you get no
feedback, that is useful to note in this section as well. For
innovative or possibly controversial ideas, consider collecting
feedback before you file the change proposal. -->
== Benefit to Fedora ==
Fedora Silverblue and Fedora Kinoite users can easily do bootloaders
updates (that includes security fixes) and we can remove support for
legacy GRUB versions thus simplify the upgrade process and making it
more reliable.
== Scope ==
* Proposal owners: Testing of the integration and new builds. The code
changes are mostly done and the integration changes are mostly already
ready as bootupd has already been integrated in a similar fashion in
Fedora CoreOS.
* Other developers: N/A
* Release engineering: N/A
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
There should be not visible change for users when upgrading. The
change only impacts the way the images are composed on the server.
== How To Test ==
We will extend the test instructions once the unified core changes
have landed. You can follow:
https://github.com/fedora-silverblue/issue-tracker/issues/120 and
https://github.com/fedora-silverblue/issue-tracker/issues/355.
== User Experience ==
For now, users will have to update their bootloader manually via the
command line. Integration to GNOME Software and Plasma Discover might
be interesting to make that easier.
Once the fallback EFI feature is available in shim (and support
implemented in bootupd), we can consider implementing automated
updates.
== Dependencies ==
N/A
== Contingency Plan ==
* Contingency mechanism: Revert the change in the rpm-ostree
manifests. Owners will do it. Nothing to do for users.
* Contingency deadline: Can happen anytime.
* Blocks release? No
== Documentation ==
We will write docs to let users update their bootloaders manually.
They will look very similar to
https://docs.fedoraproject.org/en-US/fedora-coreos/bootloader-updates/.
== Release Notes ==
Will have to be written.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
5:08 p.m.
Ben Cotton <bcotton(a)redhat.com> writes:
By design, ostree does not manage bootloader updates as they can not
(yet) happen in a transactional, atomic and safe fashion.
As we've talked about before, it's not possible to make updates
transactional. It involves, per spec and depending on processor
architecture, updating multiple files in different directories,
potentially on different filesystems entirely, one of which is fat32.
* User will be able to easily update the bootloader on their system.
This will let them apply Secure Boot dbx updates that block old
bootloaders with known vulnerabilities from loading. See:
** https://github.com/fedora-silverblue/issue-tracker/issues/355
** https://bugzilla.redhat.com/show_bug.cgi?id=2127995
What's the plan to apply the outstanding security updates (shim, grub2,
and dbx push from June) to fedora silverblue 36 + 37 that aren't covered
by this change?
Be well,
--Robbie
10:40 a.m.
As we've talked about before, it's not possible to make
updates
transactional. It involves, per spec and depending on processor
architecture, updating multiple files in different directories,
potentially on different filesystems entirely, one of which is fat32.
I should probably have used only 'safe' here. My understanding of the
"fallback work" was that with it, we could do updates automatically and retry
them after failures without risking ending up in a state where we have no working
bootloader. The update process would look like this:
1. Verify current bootloader hash
2. Fix it if not good
3. Copy current version to fallback
4. Update bootloader to new version
What I've indeed forgotten to specify is that this only applies to EFI (so probably
only x86 & aarch64) for now as that's what's implemented in bootupd.
Am I missing something?
What's the plan to apply the outstanding security updates (shim,
grub2,
and dbx push from June) to fedora silverblue 36 + 37 that aren't covered
by this change?
We definitely want to backport all that to F36/37. This will only be possible for changes
not in Anaconda, as we don't respin the installers. I'm not 100% sur yet we'll
need Anaconda changes but mentioning it here in case it turns out we do.
Thanks for the comments, I'll update the proposal.
12:32 p.m.
Timothée Ravier <siosm(a)fedoraproject.org> writes:
> As we've talked about before, it's not possible to make
updates
> transactional. It involves, per spec and depending on processor
> architecture, updating multiple files in different directories,
> potentially on different filesystems entirely, one of which is fat32.
I should probably have used only 'safe' here. My understanding of the
"fallback work" was that with it, we could do updates automatically and retry
them after failures without risking ending up in a state where we have no working
bootloader. The update process would look like this:
1. Verify current bootloader hash
2. Fix it if not good
3. Copy current version to fallback
4. Update bootloader to new version
What I've indeed forgotten to specify is that this only applies to EFI (so probably
only x86 & aarch64) for now as that's what's implemented in bootupd.
Am I missing something?
Bootloaders are not single files. Consider UEFI:
For grub2, there's both a .efi and some configuration that I'll handwave
for purposes of this conversation. For shim, it's more like 4 things -
the main shim*64.efi, fallback.efi, boot.efi, and boot.csv. These all
serve different purposes, and need to get loaded from specific parts of
the ESP. (Recall here that fat32 doesn't have symlinks, either.)
While I think it will surprise no one that I don't agree with doing so,
bootupd claims the additional goal of supporting Legacy BIOS. For that,
you also need to consider updates to the MBR, which isn't a file at all.
Be well,
--Robbie
1:31 p.m.
Bootloaders are not single files. Consider UEFI:
For grub2, there's both a .efi and some configuration that I'll handwave
for purposes of this conversation. For shim, it's more like 4 things -
the main shim*64.efi, fallback.efi, boot.efi, and boot.csv. These all
serve different purposes, and need to get loaded from specific parts of
the ESP. (Recall here that fat32 doesn't have symlinks, either.)
I don't think I understand the problem here. Do shim updates usually change all of
those files at once? What's blocking us from updating them one by one?
Note that bootupd is not trying to solve the "safe" part of bootloader updates:
we're aware that the system should not crash while the bootloader is being updated.
See https://github.com/coreos/bootupd#questions-and-answers
Thus that's why we're requiring interactions from an admin to apply the update as
only they can now when the system is the less likely to crash.
While I think it will surprise no one that I don't agree with
doing so,
bootupd claims the additional goal of supporting Legacy BIOS. For that,
you also need to consider updates to the MBR, which isn't a file at all.
While we do have that as a goal, we don't support legacy BIOS right now thus the
reason why this proposal is focusing on EFI systems:
https://github.com/coreos/bootupd/issues/53
11 a.m.
Timothée Ravier <siosm(a)fedoraproject.org> writes:
> Bootloaders are not single files. Consider UEFI:
>
> For grub2, there's both a .efi and some configuration that I'll handwave
> for purposes of this conversation. For shim, it's more like 4 things -
> the main shim*64.efi, fallback.efi, boot.efi, and boot.csv. These all
> serve different purposes, and need to get loaded from specific parts of
> the ESP. (Recall here that fat32 doesn't have symlinks, either.)
I don't think I understand the problem here. Do shim updates usually
change all of those files at once? What's blocking us from updating
them one by one?
Conceptually, we need to be able to update most of them in the event of
a security issue. Shim releases are infrequent due to needing signing,
which means that as a signed unit they update more at once. In the
issue you linked, I described what would be a safe order to apply
updates to them in, but that's not *transactional* (a system that takes
a reboot during the small window of moving files around could see some
from old and some from new).
Note that bootupd is not trying to solve the "safe" part of
bootloader
updates: we're aware that the system should not crash while the
bootloader is being updated. See
https://github.com/coreos/bootupd#questions-and-answers
If your model doesn't permit the system to cease execution during
bootloader updates, then I'm not sure why you need bootupd at all -
traditional RPM updating will work just fine (assuming the A/B change
we've been talking about). But the "Questions and answers" section
doesn't read that way to me: it mentions that "forcibly pulling the
power during OS updates" is a case ostree wants to support and doesn't
explicitly negate that for the bootloader.
I'm happy to send a PR to update that text if that's not what's meant.
Thus that's why we're requiring interactions from an admin to
apply
the update as only they can now when the system is the less likely to
crash.
Are you referring to
https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecommen...
? I didn't think that was generally something admins expected to be
doing, but would be happy to be wrong about that.
Be well,
--Robbie
1:08 p.m.
On Tue, Nov 15, 2022, at 12:00 PM, Robbie Harwood wrote:
If your model doesn't permit the system to cease execution
during
bootloader updates, then I'm not sure why you need bootupd at all -
traditional RPM updating will work just fine (assuming the A/B change
we've been talking about). But the "Questions and answers" section
doesn't read that way to me: it mentions that "forcibly pulling the
power during OS updates" is a case ostree wants to support and doesn't
explicitly negate that for the bootloader.
There's lots of nuance going on here. What both bootupd and your shim prototype are
doing is effectively moving the "payload" into /usr and not have RPM directly
writing to /efi (or /boot/efi, wherever it's mounted).
This also then directly leads into a next issue:
https://github.com/coreos/bootupd/issues/50
Basically, `rpm -q shim` becomes a lie - or at least, starts to mean something else[1].
So part of the role of bootupd here is just to become the API to query and manage state.
It is also kind of aiming to abstract out the differences across platforms, i.e. we were
discussing bringing BIOS and PReP under management too in
https://github.com/coreos/bootupd/issues/398
So basically the rationale for bootupd is to become a (relatively thin) layer for managing
this stuff since it is decoupled from the kernel+rootfs lifecycle (but, delivered inside
that).
[1] In a similar vein of course, `rpm -q kernel` can often be a lie after live updates
but before rebooting, and similar for userspace services that aren't restarted or old
libraries loaded. But, admins have known about those for a long time, or if they
don't they learn the hard way.
1:55 p.m.
"Colin Walters" <walters(a)verbum.org> writes:
On Tue, Nov 15, 2022, at 12:00 PM, Robbie Harwood wrote:
> If your model doesn't permit the system to cease execution during
> bootloader updates, then I'm not sure why you need bootupd at all -
> traditional RPM updating will work just fine (assuming the A/B change
> we've been talking about). But the "Questions and answers" section
> doesn't read that way to me: it mentions that "forcibly pulling the
> power during OS updates" is a case ostree wants to support and doesn't
> explicitly negate that for the bootloader.
There's lots of nuance going on here. What both bootupd and your shim
prototype are doing is effectively moving the "payload" into /usr and
not have RPM directly writing to /efi (or /boot/efi, wherever it's
mounted).
No, the install script install script in an RPM trigger, so the write is
still carried out by RPM.
This also then directly leads into a next issue:
https://github.com/coreos/bootupd/issues/50 Basically, `rpm -q shim`
becomes a lie - or at least, starts to mean something else[1].
I don't agree. Just because a user can mess with files on the system
doesn't mean the rpmdb is a lie, nor is it reasonable to go recheck all
paths on the filesystem just in case they've done so, or create a daemon
to provide an interface for doing that.
Be well,
--Robbie
11:35 a.m.
No, the install script install script in an RPM trigger, so the write
is
still carried out by RPM.
I don't agree. Just because a user can mess with files on the system
doesn't mean the rpmdb is a lie, nor is it reasonable to go recheck all
paths on the filesystem just in case they've done so, or create a daemon
to provide an interface for doing that.
Note that this change here is only about Fedora Silverblue & Kinoite (and maybe IoT
later). In those variants, /usr is read only and not expected to be changed by users. The
rpmdb is thus pretty much guaranteed to match the content of the system by design for
/usr.
On rpm-ostree based systems, system composes (image builds), package installation and
updates are done "offline" so /boot is not available. The bootloader is not
updated by an RPM trigger and this is why we need an additional tool to perform the update
at another time. We've created a daemon because we need the update to be reliable so
it should not fail if your SSH connections fails or if you Ctrl-C it in the middle, etc.
This daemon is really small and socket activated so it's mostly never running.
12:10 p.m.
On Fri, Nov 18, 2022, at 12:35 PM, Timothée Ravier wrote:
> No, the install script install script in an RPM trigger, so the
write is
> still carried out by RPM.
>
> I don't agree. Just because a user can mess with files on the system
> doesn't mean the rpmdb is a lie, nor is it reasonable to go recheck all
> paths on the filesystem just in case they've done so, or create a daemon
> to provide an interface for doing that.
Note that this change here is only about Fedora Silverblue & Kinoite
(and maybe IoT later). In those variants, /usr is read only and not
expected to be changed by users. The rpmdb is thus pretty much
guaranteed to match the content of the system by design for /usr.
On rpm-ostree based systems, system composes (image builds), package
installation and updates are done "offline" so /boot is not available.
The bootloader is not updated by an RPM trigger and this is why we need
an additional tool to perform the update at another time. We've created
a daemon because we need the update to be reliable so it should not
fail if your SSH connections fails or if you Ctrl-C it in the middle,
etc. This daemon is really small
I wouldn't even call it a daemon, at least not really. More in:
https://github.com/coreos/bootupd/pull/401/files
That said, I think Robbie is effectively saying "bootupd seems over-engineered"
and that's not *entirely* wrong. It certainly could be simpler; yes, in theory we
don't strictly need `bootupctl validate`. But...things like having status information
going to the journal in a reliable fashion have proven *very* useful in the past. Most
classically of course, dnf/apt type systems don't do this and it definitely makes
things harder to debug after the fact.
The discussion about offline versus live installs touches on
https://github.com/coreos/bootupd/issues/108
and we probably should have an option to do that.
5:04 p.m.
On 11/14/22 13:32, Robbie Harwood wrote:
Timothée Ravier <siosm(a)fedoraproject.org> writes:
>> As we've talked about before, it's not possible to make updates
>> transactional. It involves, per spec and depending on processor
>> architecture, updating multiple files in different directories,
>> potentially on different filesystems entirely, one of which is fat32.
>
> I should probably have used only 'safe' here. My understanding of the
"fallback work" was that with it, we could do updates automatically and retry
them after failures without risking ending up in a state where we have no working
bootloader. The update process would look like this:
> 1. Verify current bootloader hash
> 2. Fix it if not good
> 3. Copy current version to fallback
> 4. Update bootloader to new version
>
> What I've indeed forgotten to specify is that this only applies to EFI (so
probably only x86 & aarch64) for now as that's what's implemented in bootupd.
>
> Am I missing something?
Bootloaders are not single files. Consider UEFI:
For grub2, there's both a .efi and some configuration that I'll handwave
for purposes of this conversation. For shim, it's more like 4 things -
the main shim*64.efi, fallback.efi, boot.efi, and boot.csv. These all
serve different purposes, and need to get loaded from specific parts of
the ESP. (Recall here that fat32 doesn't have symlinks, either.)
While I think it will surprise no one that I don't agree with doing so,
bootupd claims the additional goal of supporting Legacy BIOS. For that,
you also need to consider updates to the MBR, which isn't a file at all.
For at least UEFI, this can be solved by using *two* boot entries. Once
everything is working in the new location, switch to it.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
10:41 p.m.
On Thu, Nov 10, 2022, at 6:08 PM, Robbie Harwood wrote:
Ben Cotton <bcotton(a)redhat.com> writes:
> By design, ostree does not manage bootloader updates as they can not
> (yet) happen in a transactional, atomic and safe fashion.
As we've talked about before, it's not possible to make updates
transactional. It involves, per spec and depending on processor
architecture, updating multiple files in different directories,
potentially on different filesystems entirely, one of which is fat32.
EFI/FedoraA
EFI/FedoraB
NVRAM bootorder uses A then B
Update the bootloader in EFI/FedoraB
At any point of failure, only the EFI/FedoraA bootloader path is used. Once everything in
EFI/FedoraB is committed to stable media, set bootnext FedoraB. If the boot fails,
automatic failback to FedoraA. If the boot succeeds, bootupd can change bootorder. B then
A.
?
--
Chris Murphy
11:33 a.m.
If I understood things correctly, that's not how this would work. We would not touch
the boot order at all.
See https://github.com/rhboot/shim/pull/502 &
https://mivehind.net/2022/08/17/shim-ab-booting-poc/.
1:01 p.m.
On Fri, Nov 11, 2022, at 11:41 PM, Chris Murphy wrote:
On Thu, Nov 10, 2022, at 6:08 PM, Robbie Harwood wrote:
> Ben Cotton <bcotton(a)redhat.com> writes:
>
>> By design, ostree does not manage bootloader updates as they can not
>> (yet) happen in a transactional, atomic and safe fashion.
>
> As we've talked about before, it's not possible to make updates
> transactional. It involves, per spec and depending on processor
> architecture, updating multiple files in different directories,
> potentially on different filesystems entirely, one of which is fat32.
EFI/FedoraA
EFI/FedoraB
NVRAM bootorder uses A then B
Update the bootloader in EFI/FedoraB
At any point of failure, only the EFI/FedoraA bootloader path is used.
Once everything in EFI/FedoraB is committed to stable media, set
bootnext FedoraB. If the boot fails, automatic failback to FedoraA. If
the boot succeeds, bootupd can change bootorder. B then A.
I think it makes sense for us to make some use of BootNext indeed. However, this heavily
overlaps with a potential move to UKIs, which effectively obviate this whole thread by
dropping shim and grub out of the equation. It also overlaps with
https://github.com/ostreedev/ostree/issues/2725 where ostree could potentially start using
BootNext itself - and this is I guess strongly pushing things to be more integrated.
(I'd tried to keep the two independent, but...)
(There's also an overlap in your proposal with fully redundant EFI partitions across
multiple disks and how that would need to be handled, also something I'd hoped to
support in bootupd explicitly)
8:57 p.m.
On Tue, Nov 15, 2022 at 7:02 PM Colin Walters <walters(a)verbum.org> wrote:
On Fri, Nov 11, 2022, at 11:41 PM, Chris Murphy wrote:
> On Thu, Nov 10, 2022, at 6:08 PM, Robbie Harwood wrote:
>> Ben Cotton <bcotton(a)redhat.com> writes:
>>
>>> By design, ostree does not manage bootloader updates as they can not
>>> (yet) happen in a transactional, atomic and safe fashion.
>>
>> As we've talked about before, it's not possible to make updates
>> transactional. It involves, per spec and depending on processor
>> architecture, updating multiple files in different directories,
>> potentially on different filesystems entirely, one of which is fat32.
>
> EFI/FedoraA
> EFI/FedoraB
>
> NVRAM bootorder uses A then B
>
> Update the bootloader in EFI/FedoraB
>
> At any point of failure, only the EFI/FedoraA bootloader path is used.
> Once everything in EFI/FedoraB is committed to stable media, set
> bootnext FedoraB. If the boot fails, automatic failback to FedoraA. If
> the boot succeeds, bootupd can change bootorder. B then A.
I think it makes sense for us to make some use of BootNext indeed. However, this heavily
overlaps with a potential move to UKIs, which effectively obviate this whole thread by
dropping shim and grub out of the equation. It also overlaps with
https://github.com/ostreedev/ostree/issues/2725 where ostree could potentially start using
BootNext itself - and this is I guess strongly pushing things to be more integrated.
(I'd tried to keep the two independent, but...)
The problem as it stands with UEFI BootNext booting the kernel
directly, as opposed to using grub2 or similar, is that it doesn't
currently deal with loading initrd/kernel command line strings etc.
There's people looking at solving those issues in a few different ways
though so it may be less of a problem in the not too distant future.
> (There's also an overlap in your proposal with fully redundant EFI partitions
across multiple disks and how that would need to be handled, also something I'd hoped
to support in bootupd explicitly)
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
5:16 a.m.
On Thu, Nov 10, 2022 at 03:24:07PM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
By design, ostree does not manage bootloader updates as they can not
(yet) happen in a transactional, atomic and safe fashion. Thus bootupd
(https://github.com/coreos/bootupd) was created to solve this issue
and enable admin-lead bootloader updates. This change is about
enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
to make bootloader updates easier.
This is missing a sentence that actually says what "bootupd" is and does.
Also "admin-lead" → "admin-led". (Though "admin-controlled"
would be
probably better.)
Zbyszek
2:52 p.m.
https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
== Summary ==
By design, ostree does not manage bootloader updates as they can not
(yet) happen in a transactional, atomic and safe fashion. Thus bootupd
(https://github.com/coreos/bootupd) was created to solve this issue
and enable admin-lead bootloader updates. This change is about
enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
to make bootloader updates easier.
This change does strikes me as something that shouldn't be necessary.
The boot loader needs to updated, this is pretty clear, but I think we should
have the technology to just update the boot loader in atomic (*) fashion without
involving the admin.
In particular, two reasons why an upgrade might be interrupted were raised:
power being cut and the system crashing. Bootupd (or any other daemon) cannot
do much about crashes so this isn't a good motivation. For power, we have
partial solutions: software-initited poweroffs or reboots should be delayed by
systemd inhibitors. Similarly, when on battery, the upgrade could be delayed if
battery power is below some fraction. The only case that we really need to
worry about is the user unplugging the power cord or pressing a button for force
a hard poweroff or reboot. But I think it should be enough to just pop up a
notification message: "critical system update in progress, do not reboot or
power-off" for the duration.
Bootupd+bootupctl creates a lot of interface for the admin
(status, update, adopt-and-update, validate). This is additional stuff
to learn. It is also additional logic to implement: bootupd must understand
EFI and boot partitions, mount points, what to do during upgrades, etc.
I took a brief look at the code and it makes various assumptions about
how the partitions are named (instead of using part-type uuids!),
that ESP is mounted on /boot/efi, etc. I think this will be problematic
in the long term because it hardcodes assumptions about the system
and duplicates logic that is already implemented in other places.
Also, bootupd does up-calls into the package manager to query state.
Information should flow from the package management system into lower-level
components, and not the other way around. The package management system
should just call into lower-level helpers with specific component
paths and versions to upgrade, and those components shouldn't ever need
to look at the big picture. Mixing the paradigms results in fragility.
The raison d'être for bootupd seems to be updates of grub. I guess there isn't
much that can be done in the short term: grub doesn't provide a way to do
updates atomically, and we need to do those updates, and bootupd seems to be a
reasonable interim solution to wrap them. But I hope this will stop being
necessary, and either grub will provide such functionality and/or we'll use a
different bootloader. In other words, I understand and won't block this Change,
but doesn't make me particularly happy. It seems that it's code that will be
used for some time and then go away.
Zbyszek
(*) The issue of atomic updates was raised. Things don't really need
to be "atomic" in the strict sense. Even if multiple files need to be
written, it is enough if there's one final write (usually a file rename)
which makes all the earlier changes visible. In particular I'm thinking
about BLS entries that consist of multiple files (kernel, initrd, .conf
snippet). As long as we put the .conf snippet last, the existence of
the other files on disk is not a problem because the boot loader will
never see a partial loader entry. Similar principle should be used
for multi-file updates of the boot loader.
2:55 p.m.
On Mon, Nov 21, 2022 at 3:53 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
> == Summary ==
>
> By design, ostree does not manage bootloader updates as they can not
> (yet) happen in a transactional, atomic and safe fashion. Thus bootupd
> (https://github.com/coreos/bootupd) was created to solve this issue
> and enable admin-lead bootloader updates. This change is about
> enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
> to make bootloader updates easier.
This change does strikes me as something that shouldn't be necessary.
The boot loader needs to updated, this is pretty clear, but I think we should
have the technology to just update the boot loader in atomic (*) fashion without
involving the admin.
In particular, two reasons why an upgrade might be interrupted were raised:
power being cut and the system crashing. Bootupd (or any other daemon) cannot
do much about crashes so this isn't a good motivation. For power, we have
partial solutions: software-initited poweroffs or reboots should be delayed by
systemd inhibitors. Similarly, when on battery, the upgrade could be delayed if
battery power is below some fraction. The only case that we really need to
worry about is the user unplugging the power cord or pressing a button for force
a hard poweroff or reboot. But I think it should be enough to just pop up a
notification message: "critical system update in progress, do not reboot or
power-off" for the duration.
Bootupd+bootupctl creates a lot of interface for the admin
(status, update, adopt-and-update, validate). This is additional stuff
to learn. It is also additional logic to implement: bootupd must understand
EFI and boot partitions, mount points, what to do during upgrades, etc.
I took a brief look at the code and it makes various assumptions about
how the partitions are named (instead of using part-type uuids!),
that ESP is mounted on /boot/efi, etc. I think this will be problematic
in the long term because it hardcodes assumptions about the system
and duplicates logic that is already implemented in other places.
Also, bootupd does up-calls into the package manager to query state.
Information should flow from the package management system into lower-level
components, and not the other way around. The package management system
should just call into lower-level helpers with specific component
paths and versions to upgrade, and those components shouldn't ever need
to look at the big picture. Mixing the paradigms results in fragility.
The raison d'être for bootupd seems to be updates of grub. I guess there isn't
much that can be done in the short term: grub doesn't provide a way to do
updates atomically, and we need to do those updates, and bootupd seems to be a
reasonable interim solution to wrap them. But I hope this will stop being
necessary, and either grub will provide such functionality and/or we'll use a
different bootloader. In other words, I understand and won't block this Change,
but doesn't make me particularly happy. It seems that it's code that will be
used for some time and then go away.
We could do the same thing SUSE does and switch to calling
scripts/tools to install into /boot and /boot/efi rather than doing it
directly from RPM. That would simplify the logic of bootupd and allow
it to just call these tools instead of having to go back and forth
between the package manager and the system environment.
--
真実はいつも一つ！/ Always, there's only one truth!
3:17 p.m.
On Mon, Nov 21, 2022 at 03:55:45PM -0500, Neal Gompa wrote:
On Mon, Nov 21, 2022 at 3:53 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> > https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
> > == Summary ==
> >
> > By design, ostree does not manage bootloader updates as they can not
> > (yet) happen in a transactional, atomic and safe fashion. Thus bootupd
> > (https://github.com/coreos/bootupd) was created to solve this issue
> > and enable admin-lead bootloader updates. This change is about
> > enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
> > to make bootloader updates easier.
>
> This change does strikes me as something that shouldn't be necessary.
> The boot loader needs to updated, this is pretty clear, but I think we should
> have the technology to just update the boot loader in atomic (*) fashion without
> involving the admin.
>
> In particular, two reasons why an upgrade might be interrupted were raised:
> power being cut and the system crashing. Bootupd (or any other daemon) cannot
> do much about crashes so this isn't a good motivation. For power, we have
> partial solutions: software-initited poweroffs or reboots should be delayed by
> systemd inhibitors. Similarly, when on battery, the upgrade could be delayed if
> battery power is below some fraction. The only case that we really need to
> worry about is the user unplugging the power cord or pressing a button for force
> a hard poweroff or reboot. But I think it should be enough to just pop up a
> notification message: "critical system update in progress, do not reboot or
> power-off" for the duration.
>
> Bootupd+bootupctl creates a lot of interface for the admin
> (status, update, adopt-and-update, validate). This is additional stuff
> to learn. It is also additional logic to implement: bootupd must understand
> EFI and boot partitions, mount points, what to do during upgrades, etc.
> I took a brief look at the code and it makes various assumptions about
> how the partitions are named (instead of using part-type uuids!),
> that ESP is mounted on /boot/efi, etc. I think this will be problematic
> in the long term because it hardcodes assumptions about the system
> and duplicates logic that is already implemented in other places.
>
> Also, bootupd does up-calls into the package manager to query state.
> Information should flow from the package management system into lower-level
> components, and not the other way around. The package management system
> should just call into lower-level helpers with specific component
> paths and versions to upgrade, and those components shouldn't ever need
> to look at the big picture. Mixing the paradigms results in fragility.
>
> The raison d'être for bootupd seems to be updates of grub. I guess there
isn't
> much that can be done in the short term: grub doesn't provide a way to do
> updates atomically, and we need to do those updates, and bootupd seems to be a
> reasonable interim solution to wrap them. But I hope this will stop being
> necessary, and either grub will provide such functionality and/or we'll use a
> different bootloader. In other words, I understand and won't block this Change,
> but doesn't make me particularly happy. It seems that it's code that will
be
> used for some time and then go away.
>
We could do the same thing SUSE does and switch to calling
scripts/tools to install into /boot and /boot/efi rather than doing it
directly from RPM. That would simplify the logic of bootupd and allow
it to just call these tools instead of having to go back and forth
between the package manager and the system environment.
We already have such scripts: kernel-install and the plugins. The
kernel %post (or %posttrans?) just calls kernel-install. Doing that
kernel-install invocation from another context should just work.
Zbyszek
3:39 p.m.
On Mon, Nov 21, 2022 at 4:17 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Mon, Nov 21, 2022 at 03:55:45PM -0500, Neal Gompa wrote:
> On Mon, Nov 21, 2022 at 3:53 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
> >
> > > https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
> > > == Summary ==
> > >
> > > By design, ostree does not manage bootloader updates as they can not
> > > (yet) happen in a transactional, atomic and safe fashion. Thus bootupd
> > > (https://github.com/coreos/bootupd) was created to solve this issue
> > > and enable admin-lead bootloader updates. This change is about
> > > enabling bootupd integration in Fedora Silverblue and Fedora Kinoite
> > > to make bootloader updates easier.
> >
> > This change does strikes me as something that shouldn't be necessary.
> > The boot loader needs to updated, this is pretty clear, but I think we should
> > have the technology to just update the boot loader in atomic (*) fashion
without
> > involving the admin.
> >
> > In particular, two reasons why an upgrade might be interrupted were raised:
> > power being cut and the system crashing. Bootupd (or any other daemon) cannot
> > do much about crashes so this isn't a good motivation. For power, we have
> > partial solutions: software-initited poweroffs or reboots should be delayed by
> > systemd inhibitors. Similarly, when on battery, the upgrade could be delayed
if
> > battery power is below some fraction. The only case that we really need to
> > worry about is the user unplugging the power cord or pressing a button for
force
> > a hard poweroff or reboot. But I think it should be enough to just pop up a
> > notification message: "critical system update in progress, do not reboot
or
> > power-off" for the duration.
> >
> > Bootupd+bootupctl creates a lot of interface for the admin
> > (status, update, adopt-and-update, validate). This is additional stuff
> > to learn. It is also additional logic to implement: bootupd must understand
> > EFI and boot partitions, mount points, what to do during upgrades, etc.
> > I took a brief look at the code and it makes various assumptions about
> > how the partitions are named (instead of using part-type uuids!),
> > that ESP is mounted on /boot/efi, etc. I think this will be problematic
> > in the long term because it hardcodes assumptions about the system
> > and duplicates logic that is already implemented in other places.
> >
> > Also, bootupd does up-calls into the package manager to query state.
> > Information should flow from the package management system into lower-level
> > components, and not the other way around. The package management system
> > should just call into lower-level helpers with specific component
> > paths and versions to upgrade, and those components shouldn't ever need
> > to look at the big picture. Mixing the paradigms results in fragility.
> >
> > The raison d'être for bootupd seems to be updates of grub. I guess there
isn't
> > much that can be done in the short term: grub doesn't provide a way to do
> > updates atomically, and we need to do those updates, and bootupd seems to be a
> > reasonable interim solution to wrap them. But I hope this will stop being
> > necessary, and either grub will provide such functionality and/or we'll use
a
> > different bootloader. In other words, I understand and won't block this
Change,
> > but doesn't make me particularly happy. It seems that it's code that
will be
> > used for some time and then go away.
> >
>
> We could do the same thing SUSE does and switch to calling
> scripts/tools to install into /boot and /boot/efi rather than doing it
> directly from RPM. That would simplify the logic of bootupd and allow
> it to just call these tools instead of having to go back and forth
> between the package manager and the system environment.
We already have such scripts: kernel-install and the plugins. The
kernel %post (or %posttrans?) just calls kernel-install. Doing that
kernel-install invocation from another context should just work.
We don't. We have it for the kernel, but not for shim, grub2, or anything else.
--
真実はいつも一つ！/ Always, there's only one truth!
4:31 p.m.
On Mon, 2022-11-21 at 16:39 -0500, Neal Gompa wrote:
> >
> > We could do the same thing SUSE does and switch to calling
> > scripts/tools to install into /boot and /boot/efi rather than doing it
> > directly from RPM. That would simplify the logic of bootupd and allow
> > it to just call these tools instead of having to go back and forth
> > between the package manager and the system environment.
>
> We already have such scripts: kernel-install and the plugins. The
> kernel %post (or %posttrans?) just calls kernel-install. Doing that
> kernel-install invocation from another context should just work.
>
We don't. We have it for the kernel, but not for shim, grub2, or anything else.
It's a stretch to call what we have for kernels "simple", of
course...:D
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
4:08 p.m.
On Mon, Nov 21, 2022, at 3:52 PM, Zbigniew Jędrzejewski-Szmek wrote:
In particular, two reasons why an upgrade might be interrupted were
raised:
power being cut and the system crashing. Bootupd (or any other daemon) cannot
do much about crashes so this isn't a good motivation. For power, we have
partial solutions: software-initited poweroffs or reboots should be delayed by
systemd inhibitors.
Oh yes, definitely an obvious omission from the current code. Filed
https://github.com/coreos/bootupd/issues/403 - thanks!
Bootupd+bootupctl creates a lot of interface for the admin
(status, update, adopt-and-update, validate). This is additional stuff
to learn.
Yeah, totally valid comment; though `adopt-and-update` is not something most admins will
need to know. I've been thinking lately that `rpm-ostree upgrade` should at least
*also* display information when bootupd needs to be invoked too. (And if we did that then
combined with the "dnf image" bit then typing `dnf update` would show this too,
which should help a lot. Plus having clients like gnome-software also become aware of
bootupd was part of the idea)
It is also additional logic to implement: bootupd must understand
EFI and boot partitions, mount points, what to do during upgrades, etc.
I took a brief look at the code and it makes various assumptions about
how the partitions are named (instead of using part-type uuids!),
Part of the rationale of for this is that in order to do redundant disk EFI, we can't
use the discoverable UUIDs. Or at least, it'd need to be queried per disk and not
globally.
Also, bootupd does up-calls into the package manager to query state.
No - at least, not in the way you're thinking. bootupd has a separation between
"image build phase" and the client side. The package management query only
happens during image builds (e.g. rpm-ostree compose image/tree today) which are normally
server side.
Information should flow from the package management system into
lower-level
components
Yes...though did you read https://github.com/coreos/bootupd/issues/50 and the sub-thread
with Robbie on this? If we want to support lifecycling bootloader updates separately from
the RPM database, that inherently calls for having the "package manager" (or
more generally, the OS updater, which may not actually "manage packages" at
least by default) *not* invoke bootloader updates - at least by default.
To connect this with the previous comment - on the client side, bootupd has its own notion
of "update payload" which is just a bit of JSON that today captures the NEVRA of
the component RPMs (but could obviously support content not from RPM too).
To state this all another way, remember *today* with systems using MBR/BIOS and grub2,
`dnf update` does *not* update the MBR and hence `rpm -q grub2` is misleading. So we
already have a situation in which the RPM database is not the same thing as the bootloader
state.
The raison d'être for bootupd seems to be updates of grub. I
guess there isn't
much that can be done in the short term: grub doesn't provide a way to do
updates atomically, and we need to do those updates, and bootupd seems to be a
reasonable interim solution to wrap them. But I hope this will stop being
necessary, and either grub will provide such functionality and/or we'll use a
different bootloader. In other words, I understand and won't block this Change,
but doesn't make me particularly happy. It seems that it's code that will be
used for some time and then go away.
Thanks, I agree with all of this in general; though, there's going to be a really long
tail on "go away", particularly when one tries to scope in actually switching
bootloaders...
11:14 a.m.
On Mon, Nov 21, 2022 at 05:08:40PM -0500, Colin Walters wrote:
On Mon, Nov 21, 2022, at 3:52 PM, Zbigniew Jędrzejewski-Szmek wrote:
> In particular, two reasons why an upgrade might be interrupted were raised:
> power being cut and the system crashing. Bootupd (or any other daemon) cannot
> do much about crashes so this isn't a good motivation. For power, we have
> partial solutions: software-initited poweroffs or reboots should be delayed by
> systemd inhibitors.
Oh yes, definitely an obvious omission from the current code. Filed
https://github.com/coreos/bootupd/issues/403 - thanks!
> Bootupd+bootupctl creates a lot of interface for the admin
> (status, update, adopt-and-update, validate). This is additional stuff
> to learn.
Yeah, totally valid comment; though `adopt-and-update` is not something most admins will
need to know. I've been thinking lately that `rpm-ostree upgrade` should at least
*also* display information when bootupd needs to be invoked too. (And if we did that then
combined with the "dnf image" bit then typing `dnf update` would show this too,
which should help a lot. Plus having clients like gnome-software also become aware of
bootupd was part of the idea)
Hmm, so this seems like the wrong direction: gnome-software shouldn't
need to know about such low-level details. If gnome-software needs to handle
bootloader updates in a special way, it means that bootloader updates become
visible to the user, and this shouldn't be necessary.
> It is also additional logic to implement: bootupd must
understand
> EFI and boot partitions, mount points, what to do during upgrades, etc.
> I took a brief look at the code and it makes various assumptions about
> how the partitions are named (instead of using part-type uuids!),
Part of the rationale of for this is that in order to do redundant disk EFI, we can't
use the discoverable UUIDs. Or at least, it'd need to be queried per disk and not
globally.
I filed https://github.com/systemd/systemd/issues/25540
(/dev/disk/by-partuuid should have a variant which includes the disk)
for this.
> Also, bootupd does up-calls into the package manager to query
state.
No - at least, not in the way you're thinking. bootupd has a separation between
"image build phase" and the client side. The package management query only
happens during image builds (e.g. rpm-ostree compose image/tree today) which are normally
server side.
Ah, OK. Thanks for for the explanation.
> Information should flow from the package management system into
lower-level
> components
Yes...though did you read https://github.com/coreos/bootupd/issues/50 and the sub-thread
with Robbie on this? If we want to support lifecycling bootloader updates separately from
the RPM database, that inherently calls for having the "package manager" (or
more generally, the OS updater, which may not actually "manage packages" at
least by default) *not* invoke bootloader updates - at least by default.
To connect this with the previous comment - on the client side, bootupd has its own
notion of "update payload" which is just a bit of JSON that today captures the
NEVRA of the component RPMs (but could obviously support content not from RPM too).
To state this all another way, remember *today* with systems using MBR/BIOS and grub2,
`dnf update` does *not* update the MBR and hence `rpm -q grub2` is misleading. So we
already have a situation in which the RPM database is not the same thing as the bootloader
state.
> The raison d'être for bootupd seems to be updates of grub. I guess there
isn't
> much that can be done in the short term: grub doesn't provide a way to do
> updates atomically, and we need to do those updates, and bootupd seems to be a
> reasonable interim solution to wrap them. But I hope this will stop being
> necessary, and either grub will provide such functionality and/or we'll use a
> different bootloader. In other words, I understand and won't block this Change,
> but doesn't make me particularly happy. It seems that it's code that will
be
> used for some time and then go away.
Thanks, I agree with all of this in general; though, there's going to be a really
long tail on "go away", particularly when one tries to scope in actually
switching bootloaders...
Yeah, this is all a thorny problem. I'm not pretending to have any idea
how to solve this comprehensively, but as I wrote before, building tools
that interact with the user seems like the wrong direction. I think it's fine
to add some temporary tooling (if a few years can be considered temporary)
to make the update safer. But the end goal should be for those updates to happen
in the background without any interaction.
Zbyszek
494
days inactive
530
days old
23 comments
10 participants
participants (10)
-
Adam Williamson -
Ben Cotton -
Chris Murphy -
Colin Walters -
Demi Marie Obenour -
Neal Gompa -
Peter Robinson -
Robbie Harwood -
Timothée Ravier -
Zbigniew Jędrzejewski-Szmek