Some Fedora CoreOS nodes may fail to upgrade
by Benjamin Gilbert
Hi all,
We've recently encountered two separate problems causing nodes to get stuck
on an old release of Fedora CoreOS. Both issues should primarily affect
machines which have been offline for an extended period. Nevertheless, you
may wish to check whether any of your nodes are unexpectedly running a
CoreOS release older than 20200715. If so, see below for mitigation
instructions.
Details
-------
There are two issues:
1. Due to an interaction with ostree downgrade protection, older versions
of Fedora CoreOS could fail to apply updates [1] when there are update
barriers [2] in the update graph. We applied a server-side workaround [3]
for several releases after the bug was fixed, but some nodes that were
offline during that period will not successfully update to current releases.
2. In April we began migrating the Cincinnati update service from the
Fedora staging cluster at updates.coreos.stg.fedoraproject.org to the
production cluster at updates.coreos.fedoraproject.org. The staging cluster
was removed from service as part of the Fedora datacenter move [4], and
there are no short-term plans [5] to restore it. Any nodes that have not
updated since the migration will fail to update to current releases.
Mitigation
----------
If you have nodes affected by either problem, the following commands should
correct the issue:
sudo tee /run/zincati/config.d/99-fixes.toml << EOF
[cincinnati]
base_url = "https://updates.coreos.fedoraproject.org"
[updates]
allow_downgrade = true
EOF
sudo systemctl restart zincati.service
This procedure does not make any persistent configuration changes to the
system, so no further action is necessary after the node updates.
If you have any questions, you can connect with us on the CoreOS mailing
list [6], Fedora Discourse [7], or #fedora-coreos on Freenode.
Thanks,
--Benjamin Gilbert
[1]: https://github.com/coreos/fedora-coreos-tracker/issues/481
[2]: https://github.com/coreos/fedora-coreos-tracker/blob/master/metadata/upda...
[3]: https://github.com/coreos/fedora-coreos-tracker/issues/481#issuecomment-6...
[4]: https://lists.fedoraproject.org/archives/list/announce@lists.fedoraprojec...
[5]: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
[6]: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject....
[7]: https://discussion.fedoraproject.org/c/server/coreos
3 years, 8 months
Live PXE boot will soon require a third artifact
by Benjamin Gilbert
Hi all,
Over the next couple Fedora CoreOS release cycles, we’ll be changing
<https://github.com/coreos/fedora-coreos-tracker/issues/390> how the live
PXE image is booted. So far, we’ve been shipping two live PXE artifacts, a
kernel and an initramfs. Starting with today’s releases, we’ll be shipping
three: a kernel, an initramfs, and a new rootfs. To boot a live PXE system,
you’ll need to add both the initramfs and the rootfs to your PXE
configuration.
At first, we’ll be shipping a stub rootfs that isn’t required for boot. If
Fedora CoreOS is PXE booted without the stub rootfs, it’ll print a message
at login, but will otherwise work normally. Upcoming releases will begin
requiring the rootfs image on the following dates:
- next: August 25
- testing: September 22
- stable: October 6
Why are we making this change?
In some preboot environments, fetching the large live initrd can take five
minutes or more. With this change, you can fetch a smaller image from the
preboot environment, and the majority of the image using the network stack
of the booted system. Fetching the rootfs from the booted system also
reduces RAM requirements during early boot.
What do I need to do?
If you boot with the live PXE artifacts, you’ll need to fetch the
additional rootfs artifact (coreos-installer download -f pxe will do this
for you) and do one of three things:
- Specify only the initramfs file as the initrd, and pass an HTTP(S) URL
to the rootfs using the coreos.live.rootfs_url= kernel argument. This
approach is preferred if you host the artifacts on a web server.
- Specify both initramfs and rootfs files as initrds in your PXE
configuration. In PXELINUX, put both file paths in the initrd directive
or initrd= kernel argument, separated by commas. In GRUB, put both file
paths in the initrd directive, separated by spaces.
- Concatenate the initramfs and rootfs files together, and specify the
combined file as the initrd. This is useful for QEMU’s -initrd option.
For more information on booting Fedora CoreOS via live PXE, see the Fedora
CoreOS documentation
<https://docs.fedoraproject.org/en-US/fedora-coreos/live-booting-ipxe/>. If
you have any questions, you can connect with us on the CoreOS mailing list
<https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject....>,
Fedora Discourse <https://discussion.fedoraproject.org/c/server/coreos>, or
#fedora-coreos on Freenode.
Thanks,
–Benjamin Gilbert
3 years, 8 months