7:36 p.m.
Hey folks! So I looked into an interesting bug today where with systemd
250, kernel-install started installing kernels and config snippets to
the wrong place.
The bug happened on my system because it had, for some reason, these
directories:
/boot/efi/loader/entries
/boot/efi/(machine_id)
where (machine_id) is the machine ID from /etc/machine-id.
With systemd 250+, if either of those directories exists, kernel-
install will write the boot loader config snippets to
/boot/efi/loader/entries, but our grub2 config will not find them
there. It will *try* to write the kernel files themselves to a
subdirectory of /boot/efi/(machine_id); but if that directory doesn't
exist, it won't write them anywhere at all. Either way, you won't be
able to boot the new kernel.
What's supposed to happen is that the config snippets should get
written to /boot/loader/entries , and the kernel/initrd files installed
to /boot . If you don't have either of the directories listed above,
this should still happen and things should work OK.
So, my question is...do other Rawhide users have this problem, or is my
system an outlier for some reason? I have not been able to figure out
*what* created the problematic directories on my system. An earlier
systemd rc may have created /boot/efi/(machine_id) if
/boot/efi/loader/entries existed, but I don't know what might have
created /boot/efi/loader/entries . I'm pretty sure I didn't do it
myself, though.
I have a patch I can send for systemd to make this behave more like it
did previously (before 250, it checked /boot before /boot/efi when
deciding which layout was in use; 250 checks /boot/efi before /boot).
But it'd be useful to know how many people run into this. It'd also be
great if anyone knows how /boot/efi/loader/entries may have come to
exist on my system.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
9:47 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Wed, Jan 05, 2022 at 05:36:56PM -0800, Adam Williamson wrote:
... snip ...
So, my question is...do other Rawhide users have this problem, or is
my
system an outlier for some reason? I have not been able to figure out
*what* created the problematic directories on my system. An earlier
systemd rc may have created /boot/efi/(machine_id) if
/boot/efi/loader/entries existed, but I don't know what might have
created /boot/efi/loader/entries . I'm pretty sure I didn't do it
myself, though.
I don't have those here at all on 2 rawhide laptops... one installed in
aug 2020 and one aug 2021.
I wonder if it's something to do with when they were installed?
kevin
2:23 a.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Wed, Jan 05, 2022 at 05:36:56PM -0800, Adam Williamson wrote:
So, my question is...do other Rawhide users have this problem, or is
my
system an outlier for some reason? I have not been able to figure out
*what* created the problematic directories on my system. An earlier
systemd rc may have created /boot/efi/(machine_id) if
/boot/efi/loader/entries existed, but I don't know what might have
created /boot/efi/loader/entries . I'm pretty sure I didn't do it
myself, though.
My rawhide system seem to have both and boot even other kernel:
tree /boot/efi
/boot/efi
├── 6a1b72cbb94d4c95a4ac9df2d581e371
│ ├── 0-rescue
│ │ ├── initrd
│ │ └── linux
│ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
│ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
│ ├── initrd
│ └── linux
├── EFI
│ ├── BOOT
│ │ ├── BOOTX64.EFI
│ │ └── fbx64.efi
│ └── fedora
│ ├── BOOTX64.CSV
│ ├── grub.cfg
│ ├── grub.cfg.rpmsave
│ ├── grubenv.rpmsave
│ ├── grubx64.efi
│ ├── mmx64.efi
│ ├── shim.efi
│ └── shimx64.efi
└── loader
└── entries
└──
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
# uname -r
5.16.0-0.rc4.20211207gitcd8c917a56f2.30.fc36.x86_64
--
Tomasz Torcz 72->| 80->|
tomek(a)pipebreaker.pl 72->| 80->|
10:44 a.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, 2022-01-06 at 09:23 +0100, Tomasz Torcz wrote:
On Wed, Jan 05, 2022 at 05:36:56PM -0800, Adam Williamson wrote:
> So, my question is...do other Rawhide users have this problem, or is my
> system an outlier for some reason? I have not been able to figure out
> *what* created the problematic directories on my system. An earlier
> systemd rc may have created /boot/efi/(machine_id) if
> /boot/efi/loader/entries existed, but I don't know what might have
> created /boot/efi/loader/entries . I'm pretty sure I didn't do it
> myself, though.
My rawhide system seem to have both and boot even other kernel:
tree /boot/efi
/boot/efi
├── 6a1b72cbb94d4c95a4ac9df2d581e371
│ ├── 0-rescue
│ │ ├── initrd
│ │ └── linux
│ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
│ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
│ ├── initrd
│ └── linux
├── EFI
│ ├── BOOT
│ │ ├── BOOTX64.EFI
│ │ └── fbx64.efi
│ └── fedora
│ ├── BOOTX64.CSV
│ ├── grub.cfg
│ ├── grub.cfg.rpmsave
│ ├── grubenv.rpmsave
│ ├── grubx64.efi
│ ├── mmx64.efi
│ ├── shim.efi
│ └── shimx64.efi
└── loader
└── entries
└──
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
# uname -r
5.16.0-0.rc4.20211207gitcd8c917a56f2.30.fc36.x86_64
Ah, yeah, so you can see, you've run into the bug - kernel 5.16.0-
0.rc5.20211216git2b14864acbaa.37.fc36 has been installed into /boot/efi
and your system has booted to the older rc4 kernel. Unless my systemd
patch gets merged, kernels won't install properly for you unless you
wipe out /boot/efi/6a1b72cbb94d4c95a4ac9df2d581e371 and
/boot/efi/loader . (After doing that you can run:
/bin/kernel-install add 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
/lib/modules/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/vmlinuz
to reinstall the rc5 kernel correctly, or just update to a newer one).
It's interesting that you have those very old fc29 folders in your
/boot/efi/(machineid); now I think about it, my folder might have had
very old entries like that in it too, but I stupidly wiped it without
keeping a note. So perhaps there was a bug some time ago that led to
these directories being created. I initially deployed this system as
F25, it looks like.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
12:31 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, Jan 6, 2022 at 1:23 AM Tomasz Torcz <tomek(a)pipebreaker.pl> wrote:
My rawhide system seem to have both and boot even other kernel:
tree /boot/efi
/boot/efi
├── 6a1b72cbb94d4c95a4ac9df2d581e371
│ ├── 0-rescue
│ │ ├── initrd
│ │ └── linux
│ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
│ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
│ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
│ ├── initrd
│ └── linux
├── EFI
│ ├── BOOT
│ │ ├── BOOTX64.EFI
│ │ └── fbx64.efi
│ └── fedora
│ ├── BOOTX64.CSV
│ ├── grub.cfg
│ ├── grub.cfg.rpmsave
│ ├── grubenv.rpmsave
│ ├── grubx64.efi
│ ├── mmx64.efi
│ ├── shim.efi
│ └── shimx64.efi
└── loader
└── entries
└──
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
Does
/boot/efi/loader/entries/6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
contain reference to the files
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/linux
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/initrd
And does this rc5 boot entry appear in GRUB but just isn't the
default? Or is it missing?
If it's missing, I wonder if it's a bug? I thought Fedora's blscfg
looks in all of:
1. ESP//loader/entries
2. XBOOTLDR//loader/entries
3. GRUB's "root"//loader/entries
In default installs, GRUB root is the 1G ext4 /boot volume.
--
Chris Murphy
12:41 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, 2022-01-06 at 11:31 -0700, Chris Murphy wrote:
On Thu, Jan 6, 2022 at 1:23 AM Tomasz Torcz
<tomek(a)pipebreaker.pl> wrote:
> My rawhide system seem to have both and boot even other kernel:
>
> tree /boot/efi
> /boot/efi
> ├── 6a1b72cbb94d4c95a4ac9df2d581e371
> │ ├── 0-rescue
> │ │ ├── initrd
> │ │ └── linux
> │ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
> │ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
> │ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
> │ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
> │ ├── initrd
> │ └── linux
> ├── EFI
> │ ├── BOOT
> │ │ ├── BOOTX64.EFI
> │ │ └── fbx64.efi
> │ └── fedora
> │ ├── BOOTX64.CSV
> │ ├── grub.cfg
> │ ├── grub.cfg.rpmsave
> │ ├── grubenv.rpmsave
> │ ├── grubx64.efi
> │ ├── mmx64.efi
> │ ├── shim.efi
> │ └── shimx64.efi
> └── loader
> └── entries
> └──
>
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
Does
/boot/efi/loader/entries/6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
contain reference to the files
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/linux
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/initrd
In my case it did.
And does this rc5 boot entry appear in GRUB but just isn't the
default? Or is it missing?
For me I think it didn't appear at all, but I'm not 100% sure. Even if
it appeared, would it work? Would grub know to look for the files on
the ESP?
It should be easy to experiment with this, anyway - all you need to do
is have systemd 250 installed, and create the directories
/boot/efi/loader/entries and /boot/efi/(yourmachineid). Doing so will
reproduce the bug, and you can run kernel-install yourself and have it
create this style of entry and see what it looks like and how it works.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
12:44 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, Jan 6, 2022 at 11:31 AM Chris Murphy <lists(a)colorremedies.com> wrote:
If it's missing, I wonder if it's a bug? I thought Fedora's blscfg
looks in all of:
1. ESP//loader/entries
2. XBOOTLDR//loader/entries
3. GRUB's "root"//loader/entries
https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/0021-blscfg-add-b...
Pretty sure blscfg is only looking in /boot/loader/entries and
/loader/entries on whatever is the GRUB $root, which is never the EFI
System partition.
*shrug* Maybe blscfg.mod should also look for /loader/entries by
partition type GUID. Both ESP and XBOOTLDR.
--
Chris Murphy
12:45 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, Jan 6, 2022 at 5:46 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Thu, 2022-01-06 at 09:23 +0100, Tomasz Torcz wrote:
> On Wed, Jan 05, 2022 at 05:36:56PM -0800, Adam Williamson wrote:
> > So, my question is...do other Rawhide users have this problem, or is my
> > system an outlier for some reason? I have not been able to figure out
> > *what* created the problematic directories on my system. An earlier
> > systemd rc may have created /boot/efi/(machine_id) if
> > /boot/efi/loader/entries existed, but I don't know what might have
> > created /boot/efi/loader/entries . I'm pretty sure I didn't do it
> > myself, though.
>
> My rawhide system seem to have both and boot even other kernel:
>
> tree /boot/efi
> /boot/efi
> ├── 6a1b72cbb94d4c95a4ac9df2d581e371
> │ ├── 0-rescue
> │ │ ├── initrd
> │ │ └── linux
> │ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
> │ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
> │ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
> │ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
> │ ├── initrd
> │ └── linux
> ├── EFI
> │ ├── BOOT
> │ │ ├── BOOTX64.EFI
> │ │ └── fbx64.efi
> │ └── fedora
> │ ├── BOOTX64.CSV
> │ ├── grub.cfg
> │ ├── grub.cfg.rpmsave
> │ ├── grubenv.rpmsave
> │ ├── grubx64.efi
> │ ├── mmx64.efi
> │ ├── shim.efi
> │ └── shimx64.efi
> └── loader
> └── entries
> └──
>
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
>
> # uname -r
> 5.16.0-0.rc4.20211207gitcd8c917a56f2.30.fc36.x86_64
Ah, yeah, so you can see, you've run into the bug - kernel 5.16.0-
0.rc5.20211216git2b14864acbaa.37.fc36 has been installed into /boot/efi
and your system has booted to the older rc4 kernel. Unless my systemd
patch gets merged, kernels won't install properly for you unless you
wipe out /boot/efi/6a1b72cbb94d4c95a4ac9df2d581e371 and
/boot/efi/loader . (After doing that you can run:
/bin/kernel-install add 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
/lib/modules/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/vmlinuz
to reinstall the rc5 kernel correctly, or just update to a newer one).
It's interesting that you have those very old fc29 folders in your
/boot/efi/(machineid); now I think about it, my folder might have had
very old entries like that in it too, but I stupidly wiped it without
keeping a note. So perhaps there was a bug some time ago that led to
these directories being created. I initially deployed this system as
F25, it looks like.
I seem to remember that /boot/efi/loader and /boot/efi/$machine-id
folders were used for gummiboot / early systemd-boot file snippets.
Back in those days, bootloader config file snippets for new kernels
were only created for those bootloaders if those directories existed,
IIRC.
Fabio
12:48 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, Jan 6, 2022 at 11:41 AM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Thu, 2022-01-06 at 11:31 -0700, Chris Murphy wrote:
> On Thu, Jan 6, 2022 at 1:23 AM Tomasz Torcz <tomek(a)pipebreaker.pl> wrote:
>
> > My rawhide system seem to have both and boot even other kernel:
> >
> > tree /boot/efi
> > /boot/efi
> > ├── 6a1b72cbb94d4c95a4ac9df2d581e371
> > │ ├── 0-rescue
> > │ │ ├── initrd
> > │ │ └── linux
> > │ ├── 4.16.0-0.rc5.git2.1.fc29.x86_64
> > │ ├── 4.16.0-0.rc6.git2.1.fc29.x86_64
> > │ ├── 4.16.0-0.rc7.git1.1.fc29.x86_64
> > │ └── 5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64
> > │ ├── initrd
> > │ └── linux
> > ├── EFI
> > │ ├── BOOT
> > │ │ ├── BOOTX64.EFI
> > │ │ └── fbx64.efi
> > │ └── fedora
> > │ ├── BOOTX64.CSV
> > │ ├── grub.cfg
> > │ ├── grub.cfg.rpmsave
> > │ ├── grubenv.rpmsave
> > │ ├── grubx64.efi
> > │ ├── mmx64.efi
> > │ ├── shim.efi
> > │ └── shimx64.efi
> > └── loader
> > └── entries
> > └──
> >
6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
>
> Does
>
/boot/efi/loader/entries/6a1b72cbb94d4c95a4ac9df2d581e371-5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64.conf
> contain reference to the files
>
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/linux
>
/6a1b72cbb94d4c95a4ac9df2d581e371/5.16.0-0.rc5.20211216git2b14864acbaa.37.fc36.x86_64/initrd
In my case it did.
> And does this rc5 boot entry appear in GRUB but just isn't the
> default? Or is it missing?
For me I think it didn't appear at all, but I'm not 100% sure. Even if
it appeared, would it work? Would grub know to look for the files on
the ESP?
Yes. BLS snippets don't support references to other file systems. If
GRUB blscfg looks on the ESP at all for /loader/entries, and thus can
read the conf files there, then it can also find the kernel and
initrd, since those are referenced relative to the file system the
conf files are located on. But I suspect from the code that blscfg
doesn't look on the ESP at all.
It should be easy to experiment with this, anyway - all you need to
do
is have systemd 250 installed, and create the directories
/boot/efi/loader/entries and /boot/efi/(yourmachineid). Doing so will
reproduce the bug, and you can run kernel-install yourself and have it
create this style of entry and see what it looks like and how it works.
True. But you know how lazy I am.
--
Chris Murphy
12:49 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, 2022-01-06 at 19:45 +0100, Fabio Valentini wrote:
>
> It's interesting that you have those very old fc29 folders in your
> /boot/efi/(machineid); now I think about it, my folder might have had
> very old entries like that in it too, but I stupidly wiped it without
> keeping a note. So perhaps there was a bug some time ago that led to
> these directories being created. I initially deployed this system as
> F25, it looks like.
I seem to remember that /boot/efi/loader and /boot/efi/$machine-id
folders were used for gummiboot / early systemd-boot file snippets.
Back in those days, bootloader config file snippets for new kernels
were only created for those bootloaders if those directories existed,
IIRC.
I've never used either of those.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1:18 p.m.
New subject: Rawhide / systemd 250: kernel-install may place kernels and
config in the wrong place
On Thu, Jan 6, 2022 at 11:46 AM Fabio Valentini <decathorpe(a)gmail.com> wrote:
I seem to remember that /boot/efi/loader and /boot/efi/$machine-id
folders were used for gummiboot / early systemd-boot file snippets.
Back in those days, bootloader config file snippets for new kernels
were only created for those bootloaders if those directories existed,
IIRC.
Yes. It's a bit of a saga.
Fedora takes an al la carte approach to the Boot Loader Spec, where we
effectively have two $BOOT volumes. One for the bootloader and a
forwarding configuration file, a second for the rest of the
configuration files plus kernel and initramfs.
Boot Loader Spec defines only one $BOOT where everything goes:
bootloader, configs, kernel, initramfs.
I suppose the main rationale for the approach we took is the path of
least change. The main change was getting away from constant grub.cfg
modification, and using drop-in configuration files that ostensibly
any bootloader could support. But we didn't commit to the bigger
changes: discovery by partition type code/UUID instead of filesystem
UUID baked into a configuration file which has to be searched for by
the bootloader; and the idea of putting kernel/initramfs on FAT was
never popular.
Possibly a big step toward better BLS compliance that also wouldn't be
too controversial:
* use XBOOTLDR partition type code for our ext4 /boot, and discover it
by that type code rather than than by fs UUID
* mount XBOOTLDR at /boot
* mount EFI System at /efi
But now I'm in the weeds, and it should be a different thread to
discuss doing something different.
--
Chris Murphy
857
days inactive
857
days old
10 comments
5 participants
participants (5)
-
Adam Williamson -
Chris Murphy -
Fabio Valentini -
Kevin Fenzi -
Tomasz Torcz