Hi, Firecracker[0] is a minimal virtual machine manager (a la QEMU) written in Rust that uses KVM to start Linux VMs extremely quickly and securely. It is used by AWS Lambda and Fargate among other things to make VM startup time comparable to containers. I've built it for Fedora x86_64 and shared it in a Copr repository[1] which includes some example commands for starting VMs. Making it build for Fedora required changes across a few components, so I'm writing to ask if this is acceptable for inclusion in Fedora. The Copr specs are all dumped in a Git repository[2] for readability. Changes include: - The musl package adds /usr paths for compatibility with the compiler --sysroot option.

- The rust compiler adds musl target subpackages.

- The kernel must set CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y to be usable as a guest.

- About two dozen Rust crates must be added to Fedora (but a handful are just new versions of existing packages). - Unrelated, but in the Copr repo anyway: The musl package is fixed to allow multilib installs, and Rust includes both 32- and 64-bit targets. I used upstream-preferred settings when adding things, but they may be in conflict with Fedora guidelines. Here are some concerns: - Firecracker can be built with Fedora's libc (glibc), but it is officially unsupported upstream[3]. Functionality would be harmed by not using musl, e.g. seccomp filters are not used.

- Upstream Rust wants musl targets to be statically linked by default[4]. It can be changed by patching (Gentoo does this) if dynamic linking is still a priority with Rust binaries, but I haven't tested that.

- Firecracker uses two crates forked from crates.io, but they are not vendored/bundled nor published to a registry. I'm currently manually bundling them as if they were vendored to avoid package name conflicts since nothing else uses them, but I don't know the preferred way to deal with those.

So does any of that sound like a showstopper for being included in Fedora? Is there any other interest in the project from the community?

On Sat, Mar 4, 2023 at 5:51 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > On Sat, Mar 4, 2023 at 12:41 PM David Michael <fedora.dm0(a)gmail.com&gt; wrote: > > Hi, > > > > Firecracker[0] is a minimal virtual machine manager (a la QEMU) > > written in Rust that uses KVM to start Linux VMs extremely quickly and > > securely. It is used by AWS Lambda and Fargate among other things to > > make VM startup time comparable to containers. I've built it for > > Fedora x86_64 and shared it in a Copr repository[1] which includes > > some example commands for starting VMs. > > > > Making it build for Fedora required changes across a few components, > > so I'm writing to ask if this is acceptable for inclusion in Fedora. > > The Copr specs are all dumped in a Git repository[2] for readability. > > Changes include: > > > > - The musl package adds /usr paths for compatibility with the > > compiler --sysroot option. > > As the musl package maintainer, I don't particularly want Fedora > packages depending on it unless they absolutely have to. I originally > maintained it in Copr, but brought it into Fedora for the Enarx folks. > I still maintain that generally you don't want to use this unless you > *really* know what you're doing and can deal with the consequences of > using an alternative libc. It sounds like there are enough issues with musl in Fedora that glibc will have to be used. > That said, the changes you make are not currently compatible with the > packaging guidelines. Can't you just use --prefix /usr/<target> > instead? It was for using --sysroot directly on gcc calls. > > - The rust compiler adds musl target subpackages. > > This will require discussion with the Fedora Rust SIG. > > > - The kernel must set CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y to be > > usable as a guest. > > This will require talking to the kernel maintainers to add the flag. Okay, I'll ask about getting this applied even if Firecracker is not included in Fedora. It still might be useful to use for a quick guest kernel, and the option is just to support a command-line parameter[0], so it shouldn't be disruptive. [0] https://github.com/torvalds/linux/blob/v6.2/Documentation/admin-guide/ker... > > - About two dozen Rust crates must be added to Fedora (but a handful > > are just new versions of existing packages). > > - Unrelated, but in the Copr repo anyway: The musl package is fixed > > to allow multilib installs, and Rust includes both 32- and 64-bit > > targets. > > > > I used upstream-preferred settings when adding things, but they may be > > in conflict with Fedora guidelines. Here are some concerns: > > > > - Firecracker can be built with Fedora's libc (glibc), but it is > > officially unsupported upstream[3]. Functionality would be harmed by > > not using musl, e.g. seccomp filters are not used. > > Why not drive this to be fixed upstream? Also, openSUSE builds > Firecracker against glibc and I see the seccomp stuff is also produced > and shipped[1]. I am not sure whether this means openSUSE's package is > broken or something else. > > [1]: https://code.opensuse.org/package/firecracker/blob/master/f/firecracker.spec I can look into what they're doing more thoroughly later, but I would assume they are just running with the empty seccomp filter. I am aware of two issues with glibc: The jailer program allegedly does not work with non-musl (although it builds) and is disabled upstream[1], and there is no non-musl seccomp filter[2][3]. [1] https://github.com/firecracker-microvm/firecracker/pull/2125 [2] https://github.com/firecracker-microvm/firecracker/blob/v1.3.0/src/vmm/bu... [3] https://github.com/firecracker-microvm/firecracker/tree/v1.3.0/resources/... > > - Upstream Rust wants musl targets to be statically linked by > > default[4]. It can be changed by patching (Gentoo does this) if > > dynamic linking is still a priority with Rust binaries, but I haven't > > tested that. > > As in musl is statically linked into the binaries? Or the Rust code is > statically linked. The former is not okay, the latter is what we do > already. The former, the musl targets produce static binaries by default. > > - Firecracker uses two crates forked from crates.io, but they are > > not vendored/bundled nor published to a registry. I'm currently > > manually bundling them as if they were vendored to avoid package name > > conflicts since nothing else uses them, but I don't know the preferred > > way to deal with those. > > > > That's probably the right way to deal with it. > > > So does any of that sound like a showstopper for being included in > > Fedora? Is there any other interest in the project from the > > community? > > > > It'd be interesting to have in Fedora, for sure, but this should be > done as an opportunity to bring our integration concerns upstream. Okay, thanks for all the feedback. I interpret this as essentially requiring the use of the glibc Rust target for inclusion in Fedora, so the changes on the Fedora side would be reduced to adding a couple dozen crates and ideally supporting the kernel as a guest. That shifts the bulk of the work to getting glibc fully supported upstream, and running a not-quite-as-secure build until then. I'll investigate this, and if there are no other blockers, maybe I can start sending review requests for the glibc build.

- Firecracker can be built with Fedora's libc (glibc), but it is officially unsupported upstream[3]. Functionality would be harmed by not using musl, e.g. seccomp filters are not used.

David Michael wrote: > - Firecracker can be built with Fedora's libc (glibc), but it is > officially unsupported upstream[3]. Functionality would be harmed by > not using musl, e.g. seccomp filters are not used. Upstream's refusal to write seccomp filters that work with glibc should be a red flag. It is definitely possible to sandbox glibc applications with seccomp, e.g., Chromium does it. It does need updates/fixes to the seccomp rules with almost every new version of glibc, but it is possible.

On Sun, Mar 05, 2023 at 12:18:18AM +0100, Kevin Kofler via devel wrote: > David Michael wrote: >> - Firecracker can be built with Fedora's libc (glibc), but it is >> officially unsupported upstream[3]. Functionality would be harmed by >> not using musl, e.g. seccomp filters are not used. > > Upstream's refusal to write seccomp filters that work with glibc should be a > red flag. It is definitely possible to sandbox glibc applications with > seccomp, e.g., Chromium does it. It does need updates/fixes to the seccomp > rules with almost every new version of glibc, but it is possible. And since we're talking hypervisors, qemu also manages to use glibc & implement a seccomp filter.

On Mar 5, 2023, at 10:19 AM, Kevin Kofler via devel < devel(a)lists.fedoraproject.org&gt; wrote: > > > David Michael wrote: >> - Firecracker can be built with Fedora's libc (glibc), but it is >> officially unsupported upstream[3]. Functionality would be harmed by >> not using musl, e.g. seccomp filters are not used. > > Upstream's refusal to write seccomp filters that work with glibc should be a > red flag. It is definitely possible to sandbox glibc applications with > seccomp, e.g., Chromium does it. It does need updates/fixes to the seccomp > rules with almost every new version of glibc, but it is possible. I’m happy to engage with the Firecracker team and get everyone together to talk through the issues. We did used to package Firecracker for Amazon Linux (in an AL2 Extra), but it had literally zero users from our repos (lambda and others build their own). This could be due to just Firecracker by itself isn’t too useful without some other easy integration with something like containerd. That being said, I’d be interested in what use cases people have for it packaged in fedora.

Hi, Following up on this, Firecracker has been accepted and submitted to Fedora. Thanks to Fabio for all of the Rust reviews. F37 https://bodhi.fedoraproject.org/updates/FEDORA-2023-dca8124d3b F38 https://bodhi.fedoraproject.org/updates/FEDORA-2023-edcbcf18e0 Some quick comments on the TODO from the original e-mail: On Sat, Mar 4, 2023 at 12:40 PM David Michael <fedora.dm0(a)gmail.com&gt; wrote: > - The musl package adds /usr paths for compatibility with the compiler --sysroot option. > - The rust compiler adds musl target subpackages. Targeting musl was dropped after the initial discussion, so there is no sandbox or seccomp filter in Fedora until that can be implemented with dynamic linking glibc upstream. I'll keep the Copr repo[0] active for a while to provide musl builds.

> Would it be possible to add a warning to this effect? Without any form > of sandboxing Firecracker is not suitable for production use. Where would such a warning be placed? The sandboxing is done by a standalone program[0] which is not built in the package, so it should be clear that it isn't available.

On Fri, Apr 21, 2023 at 10:02 PM Demi Marie Obenour <demiobenour(a)gmail.com&gt; wrote: > On 4/21/23 11:13, David Michael wrote: >> Hi, >> >> Following up on this, Firecracker has been accepted and submitted to >> Fedora. Thanks to Fabio for all of the Rust reviews. >> >> F37 https://bodhi.fedoraproject.org/updates/FEDORA-2023-dca8124d3b >> F38 https://bodhi.fedoraproject.org/updates/FEDORA-2023-edcbcf18e0 >> >> Some quick comments on the TODO from the original e-mail: >> >> On Sat, Mar 4, 2023 at 12:40 PM David Michael <fedora.dm0(a)gmail.com&gt; wrote: >>> - The musl package adds /usr paths for compatibility with the compiler --sysroot option. >>> - The rust compiler adds musl target subpackages. >> >> Targeting musl was dropped after the initial discussion, so there is >> no sandbox or seccomp filter in Fedora until that can be implemented >> with dynamic linking glibc upstream. I'll keep the Copr repo[0] >> active for a while to provide musl builds. > > Would it be possible to add a warning to this effect? Without any form > of sandboxing Firecracker is not suitable for production use. Where would such a warning be placed? The sandboxing is done by a standalone program[0] which is not built in the package, so it should be clear that it isn't available.

> Does the > sandboxed portion of Firecracker do anything that would require an NSS > (name service switch) lookup, such as DNS or username resolution? I don't think NSS lookups are an issue since the program takes numerical UID/GID values as command-line arguments. The main breaking issue with the jailer is that it requires Firecracker to be a single static binary[1] (which is the musl target's default output upstream). Their documentation also says glibc isn't supported, but I haven't tried making a static glibc binary to see what fails. The seccomp filter being unimplemented for glibc is a separate issue from the jailer.

> If > not, then I don’t see why musl (which Fedora already ships!) would be a > problem. There is no problem technically; the Copr repo[2] is building Firecracker RPMs with musl. Maintainers of both Rust and musl seemed to be against it in Fedora. From this thread:

> > There is no problem technically; the Copr repo[2] is building > > Firecracker RPMs with musl. Maintainers of both Rust and musl seemed > > to be against it in Fedora. From this thread: > Why does Fedora not want to ship Firecracker statically linked to musl? > That is the supported and tested configuration upstream. Using glibc > or dynamic linking is not supported for production use. Because glibc is Fedora's supported libc implementation and supporting two different implementations at the same time is complex

On 4/24/23 08:33, Neal Gompa wrote: > On Mon, Apr 24, 2023 at 4:19 AM Peter Robinson <pbrobinson(a)gmail.com&gt; wrote: >> >>>> There is no problem technically; the Copr repo[2] is building >>>> Firecracker RPMs with musl. Maintainers of both Rust and musl seemed >>>> to be against it in Fedora. From this thread: >>> Why does Fedora not want to ship Firecracker statically linked to musl? >>> That is the supported and tested configuration upstream. Using glibc >>> or dynamic linking is not supported for production use. >> >> Because glibc is Fedora's supported libc implementation and supporting >> two different implementations at the same time is complex > > And importantly, as the musl maintainer, I recommended against it. We > should take the opportunity to engage with upstream to fully support > glibc instead. Can they support glibc without either taking on a huge maintenance burden or weakening the sandbox?