10:27 p.m.
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
I want this change to succeed but I'm experiencing a regression, and
while trying to troubleshoot it I'm finding it difficult to understand
the myriad differences:
- I can't find the code. I assume all of it is in grub as blscfg.mod
and grub2-mkconfig and its associated scripts in /etc/grub.d but I'm
not seeing code or code reference in the change or here
https://apps.fedoraproject.org/packages/grub2-efi-x64-modules/
- I can't find any documentation. In particular documentation that
explains how it differs from upstream logic and what files the user
should be interacting with to do typical tasks (typical for those who
tend to interact with the bootloader which should be a minority of
user); I know those differences are really substantial, not least of
which is how to change boot parameters. This is important because
there's a lot of grub2 documentation in Fedora and outside Fedora, all
of which conflicts with this feature. And Fedora's experts in QA and
those who interface with user as volunteer support agents, will need
to thoroughly understand the change, otherwise I'm concerned the
common advice will become "reinstall" or even just disable this
feature.
-How does the blscfg module find /boot/loader/entries? Is this path
hard wired somehow? I've got three different installations, all three
have identical grub environment variables shown by "set" command, and
yet one of the installations doesn't find /boot/loader/entries so the
only menu entry is to enter firmware.
- Is upstream expected to eventually accept this change? Have they
given feedback on the change? Are there any other distributions
planning on adopting it, maybe downstreams of ostree?
--
Chris Murphy
2:08 a.m.
Hello Chris,
On Wed, Feb 6, 2019 at 5:28 AM Chris Murphy <lists(a)colorremedies.com> wrote:
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
I want this change to succeed but I'm experiencing a regression, and
while trying to troubleshoot it I'm finding it difficult to understand
the myriad differences:
Is this on F29 or Rawhide? Also, it's on an EFI or BIOS install?
- I can't find the code. I assume all of it is in grub as
blscfg.mod
and grub2-mkconfig and its associated scripts in /etc/grub.d but I'm
That's correct, most of changes are in /etc/grub.d/10_linux and
blscfg.mod as you said:
https://github.com/rhboot/grub2/blob/fedora-30/util/grub.d/10_linux.in
https://github.com/rhboot/grub2/blob/fedora-30/grub-core/commands/blscfg.c
not seeing code or code reference in the change or here
https://apps.fedoraproject.org/packages/grub2-efi-x64-modules/
- I can't find any documentation. In particular documentation that
explains how it differs from upstream logic and what files the user
should be interacting with to do typical tasks (typical for those who
tend to interact with the bootloader which should be a minority of
user); I know those differences are really substantial, not least of
which is how to change boot parameters. This is important because
It depends on how you do it. Changing GRUB_CMDLINE_LINUX in
/etc/default/grub and running grub2-mkconfig should have the same
effect that with a non-bls setup. Using the grubby options (through
the BLS wrapper) is also supported for users using grubby.
there's a lot of grub2 documentation in Fedora and outside
Fedora, all
of which conflicts with this feature. And Fedora's experts in QA and
Could you please elaborate a little bit? If you see a big mismatch
between some documentation and how the BLS support behaves, please
point out so we can try to make it more transparent for users.
those who interface with user as volunteer support agents, will need
to thoroughly understand the change, otherwise I'm concerned the
common advice will become "reinstall" or even just disable this
feature.
-How does the blscfg module find /boot/loader/entries? Is this path
hard wired somehow? I've got three different installations, all three
It tries to look for them in ($root)/loader/entries/ and
($root)/boot/loader/entries/ for EFI and ($boot)/loader/entries/ and
($boot)/boot/loader/entries/ for BIOS. The blsdir variable in grubenv
can also be if you want a custom BLS directory path.
have identical grub environment variables shown by "set"
command, and
yet one of the installations doesn't find /boot/loader/entries so the
only menu entry is to enter firmware.
Is there any difference between the systems that work and the one who
doesn't? Could you please fill a bug for it, that will make the issue
easier to track.
On Rawhide the blscfg command now supports to load BLS entries to
troubleshoot these cases. For example you should be able to do:
blscfg (hd0,gpt2)/loader/entries/
to load all the BLS snippets from a directory or:
blscfg
(hd0,gpt2)/loader/entries/loader/entries/036f3661be4047b6b678d491f76df6f4-5.0.0-0.rc4.git3.1.fc30.x86_64.conf
to load a single BLS snippet.
- Is upstream expected to eventually accept this change? Have they
given feedback on the change? Are there any other distributions
Not yet, it's hard to engage with upstream GRUB and even to upstream
simple patches take time.
planning on adopting it, maybe downstreams of ostree?
Besides Fedora and RHEL, Endless OS is using our GRUB blscfg patches.
We have also been talking with the ostree folks to use the module so
they can drop the ostree admin instutil grub2-generate command that
genrates a GRUB config from the generated BLS snippets.
I'll be on PTO since today and without good connectivity, so I may not
be able to answer more emails until next week. Please ping Peter Jones
if you need help with this.
--
Chris Murphy
Best regards,
Javier
1:05 p.m.
On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
On Wed, Feb 6, 2019 at 5:28 AM Chris Murphy <lists(a)colorremedies.com> wrote:
>
> https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
>
> I want this change to succeed but I'm experiencing a regression, and
> while trying to troubleshoot it I'm finding it difficult to understand
> the myriad differences:
>
Is this on F29 or Rawhide? Also, it's on an EFI or BIOS install?
Rawhide UEFI. I thought it was a regression compared to grubby and
grub2-mkconfig, because the problem doesn't happen with them. But
those tools work directly on the grub.cfg which on UEFI only ever
appears on FAT. Meanwhile, blscfg's .conf files are located at
/boot/loader/entries which on this particular test setup is subject to
zstd compression, and GRUB doesn't yet support zstd compression. GRUB
couldn't read the file contents, therefore it's not a regression, it's
a missing feature. Removing the compression solved the problem.
> there's a lot of grub2 documentation in Fedora and outside
Fedora, all
> of which conflicts with this feature. And Fedora's experts in QA and
Could you please elaborate a little bit? If you see a big mismatch
between some documentation and how the BLS support behaves, please
point out so we can try to make it more transparent for users.
If a user has bootloader related issues, I ordinarily would only ask
for a copy of the grub.cfg. But with bls systems, I need to ask for
the grub.cfg, all the .conf files, and the grubenv file. All of them
need to be interpreted. Each of those files are only described in
generic terms by upstream GRUB, and no one else is using those files
in the specific way they're being used in Fedora.
Between this feature for F30, and the F29 feature to hide the grub
menu which comes with boot success+fail marking by using the grubenv,
there are substantial changes in bootloading on Fedora that exist no
where else and near as I can tell there is no documentation at all. I
can't really call specs we don't fully follow, or feature pages, to be
documentation.
On Rawhide the blscfg command now supports to load BLS entries to
troubleshoot these cases. For example you should be able to do:
blscfg (hd0,gpt2)/loader/entries/
to load all the BLS snippets from a directory or:
blscfg
(hd0,gpt2)/loader/entries/loader/entries/036f3661be4047b6b678d491f76df6f4-5.0.0-0.rc4.git3.1.fc30.x86_64.conf
to load a single BLS snippet.
Ok that is a useful troubleshooting technique. In my case this was
totally silent, ergo it found nothing even though there were files
there. Adding 'set debug=all' and then following that up with blscfg
exposed what directories it was looking at and the contents weren't
readable - and that's when I clued in.
--
Chris Murphy
6:40 a.m.
Le 2019-02-10 20:05, Chris Murphy a écrit :
On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
Between this feature for F30, and the F29 feature to hide the grub
menu which comes with boot success+fail marking by using the grubenv,
there are substantial changes in bootloading on Fedora that exist no
where else and near as I can tell there is no documentation at all. I
can't really call specs we don't fully follow, or feature pages, to be
documentation.
FYI I had to rescue two EFI rawhide system this week-end borked by grub
changes. As far as I could reconstruct:
1. the new grub needs the env file to be regenerated or kernel scriplets
will fail "environment block too small"
2. there are *two* versions of this file, one in EFI directory space
another in /boot/grub2
3. half our tools think the correct path if the first one, the other the
second is the correct one
4. they all depend on things written by the other half in a "common"
file
5. it only works because the boot/grub2 is a symlink to the EFI version,
syncing all the tools
6. but nothing makes sure it is
7. you you follow net advice blindly, you will break the symlink while
regenerating the env file and fixing the error in 1.
8. the result is unbootable, it will miss the kernelopts line in the
file the EFI bootloader reads. No kernelopts line, no root to pivot to
in initramfs
Regards,
--
Nicolas Mailhot
1:52 p.m.
On Mon, Feb 11, 2019 at 5:40 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
Le 2019-02-10 20:05, Chris Murphy a écrit :
> On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
> Between this feature for F30, and the F29 feature to hide the grub
> menu which comes with boot success+fail marking by using the grubenv,
> there are substantial changes in bootloading on Fedora that exist no
> where else and near as I can tell there is no documentation at all. I
> can't really call specs we don't fully follow, or feature pages, to be
> documentation.
FYI I had to rescue two EFI rawhide system this week-end borked by grub
changes. As far as I could reconstruct:
1. the new grub needs the env file to be regenerated or kernel scriplets
will fail "environment block too small"
The new behavior only applies to Fedora 30 clean installs; upgraded
Fedora 29 and older should not get the new behavior, as I understand
it.
I did a clean install to UEFI, and then installed a newer kernel and
didn't get this error so I'm wondering exactly when you saw it? During
installation? From what install media? Or during a kernel update? Was
it BIOS or UEFI? Default partitioning or what was the file system
where the real grubenv is located?
2. there are *two* versions of this file, one in EFI directory space
another in /boot/grub2
3. half our tools think the correct path if the first one, the other the
second is the correct one
4. they all depend on things written by the other half in a "common"
file
5. it only works because the boot/grub2 is a symlink to the EFI version,
syncing all the tools
6. but nothing makes sure it is
7. you you follow net advice blindly, you will break the symlink while
regenerating the env file and fixing the error in 1.
8. the result is unbootable, it will miss the kernelopts line in the
file the EFI bootloader reads. No kernelopts line, no root to pivot to
in initramfs
The symlink business is confusing. I think that's for grubby's
benefit. It is a self describing method rather than hardwiring it in.
But I don't really like that the real grubenv ends up being in
/boot/efi/EFI/fedora/grubenv even when on BIOS systems where that path
should even exist but... ya not a hill I want to die on today I think.
--
Chris Murphy
2:14 p.m.
Le lundi 11 février 2019 à 12:52 -0700, Chris Murphy a écrit :
On Mon, Feb 11, 2019 at 5:40 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
> Le 2019-02-10 20:05, Chris Murphy a écrit :
> > On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
> > Between this feature for F30, and the F29 feature to hide the grub
> > menu which comes with boot success+fail marking by using the
> > grubenv,
> > there are substantial changes in bootloading on Fedora that exist
> > no
> > where else and near as I can tell there is no documentation at
> > all. I
> > can't really call specs we don't fully follow, or feature pages,
> > to be
> > documentation.
>
> FYI I had to rescue two EFI rawhide system this week-end borked by
> grub
> changes. As far as I could reconstruct:
>
> 1. the new grub needs the env file to be regenerated or kernel
> scriplets
> will fail "environment block too small"
The new behavior only applies to Fedora 30 clean installs; upgraded
Fedora 29 and older should not get the new behavior, as I understand
it.
I did a clean install to UEFI, and then installed a newer kernel and
didn't get this error so I'm wondering exactly when you saw it? During
installation? From what install media? Or during a kernel update? Was
it BIOS or UEFI? Default partitioning or what was the file system
where the real grubenv is located?
That was during dnf update to the result of the latest rawhide mass
rebuild, on two UEFI systems, one initially installed in september 2015,
the other in january 2019, with whatever was most current then and then
switched to rawhide and continually updated via dnf since.
Both systems use lvm, one with md raid below lvm, the other without.
Both have the separate /boot/efi vfat mount, one with a separate ext4
/boot below it
The symlink business is confusing. I think that's for grubby's
benefit. It is a self describing method rather than hardwiring it in.
But I don't really like that the real grubenv ends up being in
/boot/efi/EFI/fedora/grubenv even when on BIOS systems where that path
should even exist but... ya not a hill I want to die on today I think.
There's no "real" grubenv, when the symlink gets broken you see that
part of the tools write in one file location, and the others in the
other one. IIRC boot_succes is a pathological case, the thing that sets
it to 0 writes in one grubenv location, and the thing that sets it to 1
uses the other one.
--
Nicolas Mailhot
2:39 p.m.
On Mon, Feb 11, 2019 at 1:14 PM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
That was during dnf update to the result of the latest rawhide mass
rebuild, on two UEFI systems, one initially installed in september 2015,
the other in january 2019, with whatever was most current then and then
switched to rawhide and continually updated via dnf since.
Weird, this feature landed in Rawhide a long time ago, September or
October last year? Why would the mass rebuild trigger the problem
you're seeing?
Both systems use lvm, one with md raid below lvm, the other without.
Both have the separate /boot/efi vfat mount, one with a separate ext4
/boot below it
>
> The symlink business is confusing. I think that's for grubby's
> benefit. It is a self describing method rather than hardwiring it in.
> But I don't really like that the real grubenv ends up being in
> /boot/efi/EFI/fedora/grubenv even when on BIOS systems where that path
> should even exist but... ya not a hill I want to die on today I think.
There's no "real" grubenv, when the symlink gets broken you see that
part of the tools write in one file location, and the others in the
other one. IIRC boot_succes is a pathological case, the thing that sets
it to 0 writes in one grubenv location, and the thing that sets it to 1
uses the other one.
The thing that sets it to 0 is the pre-boot GRUB code, the
bootloader/manager itself, it happens has it reads the grub.cfg. In
your UEFI cases, this should always only ever point to the grubenv in
the same location as the grub EFI binary. I don't know offhand if
pre-boot GRUB follows symlinks... and therefore if BIOS GRUB knows to
write to the grubenv in /boot/efi/EFI/fedora/grubenv - but I don't see
how it can overwrite the a symlink anyway, it should just fail. But
I'm not sure what such a fail looks like. There are caveats with the
bootloader writing to grubenv on md raid, LVM, XFS and Btrfs; in your
case that doesn't sound true except the possibility the raid1 system:
is the EFI system partition on raid1, or is it a plain partition as is
the usual case with a single drive installation?
The thing that sets it to 1 is a systemd unit, which is on a 2 minute
timer (I think starting from when gdm launches). This unit is going
through the file system, so it's always a legitimate write through all
the various storage stack layers, and is definitely following the
symlink.
--
Chris Murphy
3:42 a.m.
Le 2019-02-11 21:39, Chris Murphy a écrit :
On Mon, Feb 11, 2019 at 1:14 PM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
>
> That was during dnf update to the result of the latest rawhide mass
> rebuild, on two UEFI systems, one initially installed in september
> 2015,
> the other in january 2019, with whatever was most current then and
> then
> switched to rawhide and continually updated via dnf since.
Weird, this feature landed in Rawhide a long time ago, September or
October last year? Why would the mass rebuild trigger the problem
you're seeing?
Really, I have no idea, I just fix the systems after the dnf update
borks them
--
Nicolas Mailhot
6:36 a.m.
On Mon, Feb 11, 2019 at 8:53 PM Chris Murphy <lists(a)colorremedies.com> wrote:
On Mon, Feb 11, 2019 at 5:40 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
>
[snip]
>
> FYI I had to rescue two EFI rawhide system this week-end borked by grub
> changes. As far as I could reconstruct:
>
> 1. the new grub needs the env file to be regenerated or kernel scriplets
> will fail "environment block too small"
The new behavior only applies to Fedora 30 clean installs; upgraded
Fedora 29 and older should not get the new behavior, as I understand
it.
The plan is to switch the GRUB configuration to BLS on a F29 -> F30 upgrade.
Best regards,
Javier
6:32 a.m.
Hello Nicolas,
On Mon, Feb 11, 2019 at 1:41 PM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
Le 2019-02-10 20:05, Chris Murphy a écrit :
> On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
> Between this feature for F30, and the F29 feature to hide the grub
> menu which comes with boot success+fail marking by using the grubenv,
> there are substantial changes in bootloading on Fedora that exist no
> where else and near as I can tell there is no documentation at all. I
> can't really call specs we don't fully follow, or feature pages, to be
> documentation.
FYI I had to rescue two EFI rawhide system this week-end borked by grub
changes. As far as I could reconstruct:
1. the new grub needs the env file to be regenerated or kernel scriplets
will fail "environment block too small"
Yes, grub2-editenv is too fragile. The problem is when the grubenv
file is modified without grub2-editenv, that breaks the assumption
grub2-editenv makes that the file will be filled with # characters to
always have a size of 1024 bytes. This was also the case on non-BLS
configuration, but in that case only the saved_entry variable was set
while for BLS also kernelopts is set.
2. there are *two* versions of this file, one in EFI directory space
another in /boot/grub2
3. half our tools think the correct path if the first one, the other the
second is the correct one
4. they all depend on things written by the other half in a "common"
file
5. it only works because the boot/grub2 is a symlink to the EFI version,
syncing all the tools
6. but nothing makes sure it is
7. you you follow net advice blindly, you will break the symlink while
regenerating the env file and fixing the error in 1.
I didn't get how it will break the symlink. IIRC grub2-editenv create
will just follow the symlink and "empty" the grubenv file (where empty
for GRUB means adding the '# GRUB Environment Block' header and
filling with # characters up to 1024 bytes).
8. the result is unbootable, it will miss the kernelopts line in the
file the EFI bootloader reads. No kernelopts line, no root to pivot to
in initramfs
I also wonder how you got in this situation, the
grub2-switch-to-blscfg script should had restored the original
(non-BLS) grub2.cfg file if grub2-editenv set failed. I tried to
reproduce your issue with a borked grubenv but couldn't, I'll try to
dig deeper on this.
Regards,
--
Nicolas Mailhot
Best regards,
Javier
7:58 a.m.
Le 2019-02-12 13:32, Javier Martinez Canillas a écrit :
Hi Javier
I didn't get how it will break the symlink.
You get a "environment block too small" on kernel update
You type "environment block too small" in google because you want the
update to work
You get some ubuntu advice that says "rm grubenv grub-editenv grubenv"
You try it. That fails. You intuit the Fedora incantation uses grub2 not
grub.
You type it. No error
You ask dnf the kernel. That works! The kernel scriptlet is happy and
does not error out anymore
Reboot. BOOM
Regards,
--
Nicolas Mailhot
8:10 a.m.
Hello Nicolas,
On Tue, Feb 12, 2019 at 2:58 PM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net> wrote:
Le 2019-02-12 13:32, Javier Martinez Canillas a écrit :
Hi Javier
> I didn't get how it will break the symlink.
You get a "environment block too small" on kernel update
You type "environment block too small" in google because you want the
update to work
You get some ubuntu advice that says "rm grubenv grub-editenv grubenv"
I see, I would had thought that the advice would had been to use
grub-editenv create instead. But still grub2-editenv create will only
prevent removing the symlink, it won't help with the issue of the
kernelopts variable not being set.
We should make editing the grubenv more reliable to avoid these issues.
Best regards,
Javier
8:32 a.m.
On 12/02/2019 14:10, Javier Martinez Canillas wrote:
I see, I would had thought that the advice would had been to use
grub-editenv create instead. But still grub2-editenv create will only
prevent removing the symlink, it won't help with the issue of the
kernelopts variable not being set.
We should make editing the grubenv more reliable to avoid these issues.
Am I the only person that considers grub2-editenv terribly
misnamed, because it doesn't really let you edit anything.
Could it not just, you know, open the environment in an
editor and let you edit it ;-) Would be a lot nicer than
the current recipe of:
sudo grub2-editenv list | fgrep kernelopts > x
vi x
sudo grub2-editenv - set "$(cat x)"
every time I want to change the kernel options.
Oh and why does "set" need an explicit hyphen to use
the default file while "list" doesn't?!?!?
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
3:18 p.m.
On Tue, Feb 12, 2019, 7:10 AM Javier Martinez Canillas <javier(a)dowhile0.org
wrote:
I see, I would had thought that the advice would had been to use
grub-editenv create instead. But still grub2-editenv create will only
prevent removing the symlink, it won't help with the issue of the
kernelopts variable not being set.
We should make editing the grubenv more reliable to avoid these issues.
There is another complication with writing to grubenv from the pre-boot
environment. It was designed for simple filesystems like FAT and ext2. It
can't work on software/firmware RAID and Btrfs - because modifications
outside kernel code immediately makes those volumes invalid. GRUB disallows
writes to Btrfs volumes, and perhaps also mdadm and LVM (but I haven't
tested them). GRUB permits writes to XFS but XFS upstream devs consider it
proscribed to write to an XFS volume outside xfs-progs or kernel code. The
ext4 developers are more tolerant, but my take is that they'd really prefer
GRUB write to the reserve areas for the bootloader instead. These reserve
areas for bootloaders are in different locations and are of different sizes
depending on the file system.
In the short term it means any features depending on writing to grubenv
from pre-boot GRUB (as opposed to Linux user space GRUB tools), can only be
expected on UEFI (grubenv is always on FAT), or if on BIOS only with
default (automatic) partitioning. That might be an acceptable limitation.
---
Chris Murphy
3:34 p.m.
On Tue, Feb 12, 2019 at 2:18 PM Chris Murphy <lists(a)colorremedies.com> wrote:
In the short term it means any features depending on writing to grubenv from pre-boot
GRUB (as opposed to Linux user space GRUB tools), can only be expected on UEFI (grubenv is
always on FAT), or if on BIOS only with default (automatic) partitioning. That might be an
acceptable limitation.
The only feature I'm thinking of that depends on pre-boot GRUB writing
to grubenv is resetting boot success to zero, which is part of the
hide boot menu by default feature.
If grubenv can be change from pre-boot environment, we can't correctly
determine if boot has succeeded or failed. So what's the best
fallback? Always show grub menu or always hide it? I think probably
the former.
If the grub.cfg code first reads grubenv, and sees boot_success=0 and
therefore no need to write to grubenv to reset it back to 0, that's
one first step. But then to make sure the systemd unit that marks boot
successful by overwriting grubenv with boot_success=1, is there a way
to disable (or not enable) that systemd unit if the user has chosen a
custom installation in anaconda?
--
Chris Murphy
3:37 p.m.
On Tue, Feb 12, 2019 at 2:34 PM Chris Murphy <lists(a)colorremedies.com> wrote:
If grubenv can be change from pre-boot environment, we can't correctly
determine if boot has succeeded or failed.
s/can/can't
s/change/changed
(grubenv can't be changed in certain custom partitioning cases e.g.
/boot on Btrfs or /boot on XFS or /boot on software/firmware RAID.)
--
Chris Murphy
12:49 p.m.
I glossed over the fact upgrades to Fedora 30 will be converted to the
"bls way" of things. So I want to be sure I understand feature scope:
- all Fedora 30 editions and spins and archs, *except* 32-bit ARM
- new Fedora 30 installations, and upgrades of Fedora 28 and 29 to Fedora 30
- does not apply to install media (right now I'm seeing install media
still contains monolithic grub.conf, grub.cfg and isolinux.cfg that
have all menu entries)
Thanks,
---
Chris Murphy
4:33 a.m.
Hello Chris,
On Wed, Feb 13, 2019 at 7:49 PM Chris Murphy <lists(a)colorremedies.com> wrote:
I glossed over the fact upgrades to Fedora 30 will be converted to the
"bls way" of things. So I want to be sure I understand feature scope:
- all Fedora 30 editions and spins and archs, *except* 32-bit ARM
Yes, because for ARMv7 we are still booting from u-boot directly, so
it needs the old grubby tool to update the extlinux.conf that's used
by u-boot.
There's a plan to use the u-boot uEFI stub to chain-load grub2 as it's
already done for aarch64 boards AFAIK. If that's the case we could
also use BLS for ARMv7.
The Changes wiki page for that feature says that it's targeted for
F30, but AFAIU it probably won't happen in time and we will have to
keep a non-BLS configuration for ARMv7 in F30.
https://fedoraproject.org/wiki/Changes/uEFIforARMv7
- new Fedora 30 installations, and upgrades of Fedora 28 and 29 to
Fedora 30
Correct. Although I would just say upgrades from F29 to F30. Or can
you skip a release when doing a system upgrade?
- does not apply to install media (right now I'm seeing install
media
still contains monolithic grub.conf, grub.cfg and isolinux.cfg that
have all menu entries)
Yes, I'm not that familiar with how the install media is created and
its boot configuration. But I think that BLS makes less sense there
since the menu entries will always be a static configuration anyways.
Thanks,
---
Chris Murphy
Best regards,
Javier
4:38 a.m.
On Thu, Feb 14, 2019 at 10:34 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
Hello Chris,
On Wed, Feb 13, 2019 at 7:49 PM Chris Murphy <lists(a)colorremedies.com> wrote:
>
> I glossed over the fact upgrades to Fedora 30 will be converted to the
> "bls way" of things. So I want to be sure I understand feature scope:
>
> - all Fedora 30 editions and spins and archs, *except* 32-bit ARM
Yes, because for ARMv7 we are still booting from u-boot directly, so
it needs the old grubby tool to update the extlinux.conf that's used
by u-boot.
Correct.
There's a plan to use the u-boot uEFI stub to chain-load grub2 as
it's
already done for aarch64 boards AFAIK. If that's the case we could
also use BLS for ARMv7.
Yes aarch64 uses UEFI/grub2 exclusively.
The Changes wiki page for that feature says that it's targeted
for
F30, but AFAIU it probably won't happen in time and we will have to
keep a non-BLS configuration for ARMv7 in F30.
We're almost there, I'm in the trying to smash all the bugs out of it
phase, but you're correct we do need a fallback :-/
> https://fedoraproject.org/wiki/Changes/uEFIforARMv7
>
> > - new Fedora 30 installations, and upgrades of Fedora 28 and 29 to Fedora 30
>
> Correct. Although I would just say upgrades from F29 to F30. Or can
> you skip a release when doing a system upgrade?
>
> > - does not apply to install media (right now I'm seeing install media
> > still contains monolithic grub.conf, grub.cfg and isolinux.cfg that
> > have all menu entries)
> >
>
> Yes, I'm not that familiar with how the install media is created and
> its boot configuration. But I think that BLS makes less sense there
> since the menu entries will always be a static configuration anyways.
>
> > Thanks,
> > ---
> > Chris Murphy
>
> Best regards,
> Javier
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
2:12 p.m.
On Thu, Feb 14, 2019 at 3:33 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
> - new Fedora 30 installations, and upgrades of Fedora 28 and 29
to Fedora 30
Correct. Although I would just say upgrades from F29 to F30. Or can
you skip a release when doing a system upgrade?
Fedora 28 folks will see an option to upgrade to Fedora 30 in Gnome
Software, and it is a direct upgrade.
--
Chris Murphy
2:27 p.m.
On Thu, Feb 14, 2019 at 9:12 PM Chris Murphy <lists(a)colorremedies.com> wrote:
On Thu, Feb 14, 2019 at 3:33 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
> > - new Fedora 30 installations, and upgrades of Fedora 28 and 29 to Fedora 30
>
> Correct. Although I would just say upgrades from F29 to F30. Or can
> you skip a release when doing a system upgrade?
Fedora 28 folks will see an option to upgrade to Fedora 30 in Gnome
Software, and it is a direct upgrade.
I see, I was not aware of that. It should also work but I'll test to
make sure that's the case.
Best regards,
Javier
6:07 a.m.
Hello Chris,
Sorry for the late response but I was on vacation last week.
On Sun, Feb 10, 2019 at 8:06 PM Chris Murphy <lists(a)colorremedies.com> wrote:
On Wed, Feb 6, 2019 at 1:08 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
>
> On Wed, Feb 6, 2019 at 5:28 AM Chris Murphy <lists(a)colorremedies.com> wrote:
> >
> > https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
> >
> > I want this change to succeed but I'm experiencing a regression, and
> > while trying to troubleshoot it I'm finding it difficult to understand
> > the myriad differences:
> >
>
> Is this on F29 or Rawhide? Also, it's on an EFI or BIOS install?
Rawhide UEFI. I thought it was a regression compared to grubby and
grub2-mkconfig, because the problem doesn't happen with them. But
those tools work directly on the grub.cfg which on UEFI only ever
appears on FAT. Meanwhile, blscfg's .conf files are located at
/boot/loader/entries which on this particular test setup is subject to
zstd compression, and GRUB doesn't yet support zstd compression. GRUB
couldn't read the file contents, therefore it's not a regression, it's
a missing feature. Removing the compression solved the problem.
I see, thanks a lot for the explanation.
> > there's a lot of grub2 documentation in Fedora and outside Fedora, all
> > of which conflicts with this feature. And Fedora's experts in QA and
>
> Could you please elaborate a little bit? If you see a big mismatch
> between some documentation and how the BLS support behaves, please
> point out so we can try to make it more transparent for users.
If a user has bootloader related issues, I ordinarily would only ask
for a copy of the grub.cfg. But with bls systems, I need to ask for
the grub.cfg, all the .conf files, and the grubenv file. All of them
need to be interpreted. Each of those files are only described in
generic terms by upstream GRUB, and no one else is using those files
in the specific way they're being used in Fedora.
Yes, a BLS configuration comes with a different set of trade offs. Not
having the boot entries in the grub2.cfg has the advantage that the
file doesn't have to be modified on kernel install / removal, but as
you said it also means that it doesn't contain all the information of
your boot configuration.
Between this feature for F30, and the F29 feature to hide the grub
menu which comes with boot success+fail marking by using the grubenv,
there are substantial changes in bootloading on Fedora that exist no
where else and near as I can tell there is no documentation at all. I
can't really call specs we don't fully follow, or feature pages, to be
documentation.
You are correct, we will work on more documentation and also better
tooling to manage the BLS snippets, grubenv file, etc. For example
grub2-editenv is very fragile as was mentioned in this thread.
> On Rawhide the blscfg command now supports to load BLS entries to
> troubleshoot these cases. For example you should be able to do:
>
> blscfg (hd0,gpt2)/loader/entries/
>
> to load all the BLS snippets from a directory or:
>
> blscfg
(hd0,gpt2)/loader/entries/loader/entries/036f3661be4047b6b678d491f76df6f4-5.0.0-0.rc4.git3.1.fc30.x86_64.conf
>
> to load a single BLS snippet.
Ok that is a useful troubleshooting technique. In my case this was
totally silent, ergo it found nothing even though there were files
there. Adding 'set debug=all' and then following that up with blscfg
exposed what directories it was looking at and the contents weren't
readable - and that's when I clued in.
Got it, I'm glad that you figured out the issue.
--
Chris Murphy
Best regards,
Javier
10:06 p.m.
On Tue, Feb 12, 2019 at 5:08 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
You are correct, we will work on more documentation and also better
tooling to manage the BLS snippets, grubenv file, etc. For example
grub2-editenv is very fragile as was mentioned in this thread.
I'm looking at this:
https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-gui...
I'm noticing that grubby on F30 is actually grubby-bls. On the one hand:
# grubby --set-default /boot/vmlinuz-5.0.0-0.rc5.git0.1.fc30.x86_64
# cat /boot/efi/EFI/fedora/grubenv
# GRUB Environment Block
saved_entry=8a6804fc8b1447b49bf2b522e419f2ae-5.0.0-0.rc5.git0.1.fc30.x86_64
[snip]
That appears to work. However,
[chris@localhost ~]$ sudo grubby --remove-args=quiet
[chris@localhost ~]$ sudo cat /boot/efi/EFI/fedora/grubenv
[snip]
kernelopts=root=/dev/mapper/fedora_localhost--live-root ro
resume=/dev/mapper/fedora_localhost--live-swap
rd.lvm.lv=fedora_localhost-live/root
rd.luks.uuid=luks-8dc728e5-4420-4f90-a1c5-aaf87a69d88b
rd.lvm.lv=fedora_localhost-live/swap rhgb quiet
boot_indeterminate=0
That doesn't work. Nor does --remove-args="quiet" - nor is there an
error message. It just fails silently. And the --args argument doesn't
appear to add arguments to the grubenv (or the bls snippets), also
fails silently. Is it a bug?
Is grubby-bls intended as the primary tool for changing these things?
Long term or short term?
--
Chris Murphy
5:05 a.m.
On Thu, Feb 14, 2019 at 5:06 AM Chris Murphy <lists(a)colorremedies.com> wrote:
On Tue, Feb 12, 2019 at 5:08 AM Javier Martinez Canillas
<javier(a)dowhile0.org> wrote:
> You are correct, we will work on more documentation and also better
> tooling to manage the BLS snippets, grubenv file, etc. For example
> grub2-editenv is very fragile as was mentioned in this thread.
I'm looking at this:
https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-gui...
I'm noticing that grubby on F30 is actually grubby-bls. On the one hand:
Yes. The grubby-bls script supports all the grubby options for
backward compatibility, but under the hood it just manages the BLS
snippets.
# grubby --set-default /boot/vmlinuz-5.0.0-0.rc5.git0.1.fc30.x86_64
# cat /boot/efi/EFI/fedora/grubenv
# GRUB Environment Block
saved_entry=8a6804fc8b1447b49bf2b522e419f2ae-5.0.0-0.rc5.git0.1.fc30.x86_64
[snip]
That appears to work. However,
[chris@localhost ~]$ sudo grubby --remove-args=quiet
That's not a valid grubby command, even for the old grubby tool:
$ sudo grubby --info=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64 | grep args
args="ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb quiet"
$ sudo grubby --remove-args=quiet
grubby: no action specified
$ sudo grubby --update-kernel=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64
--remove-args=quiet
$ sudo grubby --info=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64 | grep args
args="ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb"
[chris@localhost ~]$ sudo cat /boot/efi/EFI/fedora/grubenv
[snip]
kernelopts=root=/dev/mapper/fedora_localhost--live-root ro
resume=/dev/mapper/fedora_localhost--live-swap
rd.lvm.lv=fedora_localhost-live/root
rd.luks.uuid=luks-8dc728e5-4420-4f90-a1c5-aaf87a69d88b
rd.lvm.lv=fedora_localhost-live/swap rhgb quiet
boot_indeterminate=0
That doesn't work. Nor does --remove-args="quiet" - nor is there an
error message. It just fails silently. And the --args argument doesn't
appear to add arguments to the grubenv (or the bls snippets), also
fails silently. Is it a bug?
Yes, the tool not printing an error message is a bug. I've fixed it
now, thanks a lot for reporting this.
The following will work with the new grubby script (and also the old
grubby tool):
$ sudo grubby --info=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64 | grep args
args="ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb quiet"
$ sudo grubby --update-kernel=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64
--remove-args=quiet
$ grubby --info=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64 | grep args
args="ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb"
Now, the grubby script doesn't work as you expected it. The
$kernelopts variable in grubenv is used to set a kernel command line
that's shared by all the BLS snippets, that allows you to have a
single place where you can update the cmdline for all the entries.
You have two ways to update this:
1) Using grub2-editenv set option
2) Setting GRUB_CMDLINE_LINUX in /etc/default/grub and running
grub2-mkconfig again.
Option (2) is what most people are used to and what we have documented
in many places, so we supported for backward compatibility. The
grub2-mkconfig tool will set $kernelopts with the value in
GRUB_CMDLINE_LINUX.
The grubby script (and the old grubby tool) are used to change
individual boot entries. So you need to specify a kernel with
--update-kernel (you can also specify DEFAULT for the default entry or
ALL to update all the entries).
Since the $kernelopts is shared by all the entries, the grubby script
can't change, since that will affect all the others. So what we
decided in this case is to make the grubby-bls script to copy-on-write
the kernel cmdline. That is, when you change an cmdline for an entry
it will expand the $kernelopts variable, do the changes and store the
modified cmdline in the BLS options field instead of $kernelopts. So
from now on the modified entry won't share the $kernelopts anymore and
will have it's own cmdline. Any changes to $kernrelopts won't affect
this entry anymore and the grubby script should be used for any future
updates.
As an example:
$ sudo grep kernelopts /boot/grub2/grubenv
kernelopts=root=/dev/mapper/fedora-root ro
resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb quiet
$ sudo grep options
/boot/loader/entries/036f3661be4047b6b678d491f76df6f4-5.0.0-0.rc6.git0.1.fc30.x86_64.conf
options $kernelopts
$ sudo grubby --update-kernel=/boot/vmlinuz-5.0.0-0.rc6.git0.1.fc30.x86_64
--remove-args=quiet
$ sudo grep kernelopts /boot/grub2/grubenv
kernelopts=root=/dev/mapper/fedora-root ro
resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root
rd.lvm.lv=fedora/swap rhgb quiet
$ sudo grep options
/boot/loader/entries/036f3661be4047b6b678d491f76df6f4-5.0.0-0.rc6.git0.1.fc30.x86_64.conf
options root=/dev/mapper/fedora-root ro resume=/dev/mapper/fedora-swap
rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb
Is grubby-bls intended as the primary tool for changing these things?
In the short term yes. And provide better docs that clearly explain
how all this works.
Long term or short term?
In the medium term our plan is to provide better tooling for this.
Having two tools to update the kernel cmdline (grub2-editenv and
grubby) is not intuitive. So our plan is to have a library + command
line tool for configuring everything related to the bootloader. We
will keep grub2-editenv and the grubby script for backward
compatibility, maybe they could just be a wrapper on top of this new
tool.
--
Chris Murphy
Best regards,
Javier
11:57 a.m.
On Thu, Feb 14, 2019, at 6:05 AM, Javier Martinez Canillas wrote:
So our plan is to have a library + command
line tool for configuring everything related to the bootloader.
FWIW, "rpm-ostree kargs" is already a CLI and DBus API to change the kernel
arguments that we expect people to use on rpm-ostree based systems. At some point
we're likely to mount /boot read-only by default to enforce these sorts of things.
And probably start wrapping traditional tools to redirect them.
6:50 a.m.
How can i disable this new feature complete?
I am using a multiboot system with 3 different root partitions for testing my packages.
Sadly reading the kernel vars with root=UUID=....from
[root@mother rave]# cat /boot/grub2/grubenv
# GRUB Environment Block
boot_success=1
boot_indeterminate=0
saved_entry=xxxxxxxxxxxxxxxxxxx-5.0.0-0.rc4.git2.2.fc30.x86_64
kernelopts=root=UUID=xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx ro
rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1
rd.md.uuid=xxxxxxx:xxxxxxx:xxxxxxx:xxxxxxx rd.md.uuid=xxxxxxx:xxxxxxx:xxxxxxx:xxxxxxx
selinux=0
doesn't make any sense for f28 and f29, which have their own root partition.
grub2-mkconfig produces a lot of non working (garbage) entries with one root=UUID, for all
kernels from 3 fedora Installations
greetz
Wolfgang
7:05 a.m.
Hello Wolfgang,
On Thu, Feb 14, 2019 at 1:51 PM Wolfgang Ulbrich <fedora(a)raveit.de> wrote:
How can i disable this new feature complete?
You can disable it by removing the GRUB_ENABLE_BLSCFG=true from
/etc/default/grub and re-generating your grub2.cfg with grub2-mkconfig
(i.e: grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg). You need to
install the grubby-deprecated package so the old grubby updates the
menu entries in the grub2.cfg file.
Best regards,
Javier
1949
days inactive
1957
days old
26 comments
7 participants
participants (7)
-
Chris Murphy -
Colin Walters -
Javier Martinez Canillas -
Nicolas Mailhot -
Peter Robinson -
Tom Hughes -
Wolfgang Ulbrich