On Thu, Apr 1, 2021 at 2:23 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote: > > https://fedoraproject.org/wiki/Changes/SmallerContainerBase > > == Summary == > This change proposes to remove 3 packages (sssd-client, util-linux, > shadow-utils) from the Container Base Image (including the minimal > image). The Fedora Base Image is still quite large compared to other > distributions and the tools offered by these packages are not > essential in base image. > > == Owner == > * Name: [[User:cverna| Clément Verna]] > * Email: <cverna-at-fedoraproject.org> > > > == Detailed Description == > This is a proposal to make the Fedora Container Base image smaller by > remove the following 3 packages: > * sssd-client > * util-linux > * shadow-utils > > Current size of the base image and minimal base image : > {| class="wikitable" > |- > ! REPOSITORY !! TAG !! IMAGE ID !! CREATED !! SIZE > |- > | registry.fedoraproject.org/fedora || 34 || eede0db319cc || 2 days > ago || 187 MB > |- > | registry.fedoraproject.org/fedora-minimal || 34 || 4ff120184ee4 || > 2 days ago || 122 MB > |} > > The installed size of each package is : > > {| class="wikitable" > |- > ! Package !! Installed Size > |- > | util-linux || 13018140 > |- > | shadow-utils || 3876259 > |- > | sssd-client || 317948 > |} > > Removing these packages would allow to gain around 17MB in both images. > > Each of these packages provides useful tools but the main goal of the > base image is for building layered images. Each of these packages can > easily be added in a layered image if needed. > > More info and discussion happened for each package in the Container SIG tracker > > sssd-client : https://pagure.io/ContainerSIG/container-sig/issue/44 > > util-linux : https://pagure.io/ContainerSIG/container-sig/issue/45 > > shadow-utils : https://pagure.io/ContainerSIG/container-sig/issue/46 > > > == Benefit to Fedora == > Reducing the size of the base image makes it a more interesting choice > for users to build layered images using Fedora. The base image is also > heavily used by CI systems so reducing the size makes it faster to be > pulled. > Removing packages from the base image also reduces the number of CVEs > our users have to care about. > > > == Scope == > * Proposal owners: > Explicitly remove the 3 packages from the base image kickstart : > https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base.ks > > * Release engineering: > Approve and Merge the kickstart change. > > * Policies and guidelines: N/A (not needed for this Change) > > * Trademark approval: N/A (not needed for this Change) > > * Alignment with Objectives: N/A > > == Upgrade/compatibility impact == > > Some layered images that relied on these packages being provided by > the base image will fail to build. These images will now have to make > sure to install the required package in their Container/Dockerfile. > > In most cases that will results in adding the following : > > RUN dnf -y install sssd-client shadow-utils util-linux && dnf clean all > > > == How To Test == > Once implemented, one can test this change by pulling the rawhide > image and verify that none of the above packages are present in the > image. > > == User Experience == > See Upgrade/compatibility impact > > == Dependencies == > > == Contingency Plan == > Kickstart changes can simply be reverted and packages added back in > the base image. > The only one of these I have a major problem with removing is shadow-utils. Without those tools, it's impossible to create and modify users, and that's an extremely common pattern for containers. I also don't think freeing 4MB on the unpacked rootfs is much of a gain for the pain you're about to cause by dropping shadow-utils from the base image. The overhead of having to install that makes it considerably less attractive to use. Unless OpenShift and RKE recently changed so that containers can run as root by default (as of yesterday, they didn't), this is solidly a bad idea, since it makes it much more unintuitive to set up secure containers conforming with the guidelines for these Kubernetes platforms.

Unless OpenShift and RKE recently changed so that containers can run as root by default (as of yesterday, they didn't), this is solidly a bad idea, since it makes it much more unintuitive to set up secure containers conforming with the guidelines for these Kubernetes platforms.

On Fri, Apr 2, 2021 at 5:18 PM Lars Seipel <ls(a)slrz.net&gt; wrote: > > On Thu, Apr 01, 2021 at 02:36:48PM -0400, Neal Gompa wrote: > >Unless OpenShift and RKE recently changed so that containers can run > >as root by default (as of yesterday, they didn't), this is solidly a > >bad idea, since it makes it much more unintuitive to set up secure > >containers conforming with the guidelines for these Kubernetes > >platforms. > > In my experience, containers trying to run stuff from shadow-utils in > their entrypoint/startup scripts tend to be a reason for containers to > *not* run on OpenShift/OKD without additional adjustments. > > A related (and more common) issue are images that expect to run with a > particular named user (or UID) determined during the build process > (again, most likely created using shadow-utils). > > I'm not familiar with Rancher but at least for OpenShift, I don't think > the availability of shadow-utils is very useful. At run time, you can't > use the shadow-utils at all and whatever you do with it during build > time is unlikely to be helpful (and actively harmful more often than > not) at run time when OpenShift assigns you an arbitrary UID. It's basically required for building containers that will work at runtime where OpenShift assigns an arbitrary UID. For example, in my containers, I *build* and create a "runtime user" with the UID 1000, and then set things up to use that context at the end. OpenShift uses that for its dynamic UID assignment.

On 4/3/21 02:34, Tomasz Torcz wrote: > Dnia Fri, Apr 02, 2021 at 05:30:30PM -0400, Neal Gompa napisał(a): >> On Fri, Apr 2, 2021 at 5:18 PM Lars Seipel <ls(a)slrz.net&gt; wrote: >>> On Thu, Apr 01, 2021 at 02:36:48PM -0400, Neal Gompa wrote: >>>> Unless OpenShift and RKE recently changed so that containers can run >>>> as root by default (as of yesterday, they didn't), this is solidly a >>>> bad idea, since it makes it much more unintuitive to set up secure >>>> containers conforming with the guidelines for these Kubernetes >>>> platforms. >>> In my experience, containers trying to run stuff from shadow-utils in >>> their entrypoint/startup scripts tend to be a reason for containers to >>> *not* run on OpenShift/OKD without additional adjustments. >>> >>> A related (and more common) issue are images that expect to run with a >>> particular named user (or UID) determined during the build process >>> (again, most likely created using shadow-utils). >>> >>> I'm not familiar with Rancher but at least for OpenShift, I don't think >>> the availability of shadow-utils is very useful. At run time, you can't >>> use the shadow-utils at all and whatever you do with it during build >>> time is unlikely to be helpful (and actively harmful more often than >>> not) at run time when OpenShift assigns you an arbitrary UID. >> It's basically required for building containers that will work at >> runtime where OpenShift assigns an arbitrary UID. >> >> For example, in my containers, I *build* and create a "runtime user" >> with the UID 1000, and then set things up to use that context at the >> end. OpenShift uses that for its dynamic UID assignment. > But you do not need shadow-utils for that. Even OpenShift > documentation shows simple echo is enough: > > if ! whoami &> /dev/null; then > if [ -w /etc/passwd ]; then > echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd > fi > fi > https://docs.openshift.com/container-platform/3.10/creating_images/guidel... > (yeah, I know it's an old and obsolete version of docs) > What about all of the users of Docker and Podman who do?

``` from fedora run useradd XYZ user XYZ ... ``` Do you just break them out of the box?

_______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

On Tue, Apr 6, 2021 at 3:23 AM Clement Verna <cverna(a)fedoraproject.org&gt; wrote: > > > > On Mon, 5 Apr 2021 at 20:30, Daniel Walsh <dwalsh(a)redhat.com&gt; wrote: >> >> On 4/3/21 02:34, Tomasz Torcz wrote: >> > Dnia Fri, Apr 02, 2021 at 05:30:30PM -0400, Neal Gompa napisał(a): >> >> On Fri, Apr 2, 2021 at 5:18 PM Lars Seipel <ls(a)slrz.net&gt; wrote: >> >>> On Thu, Apr 01, 2021 at 02:36:48PM -0400, Neal Gompa wrote: >> >>>> Unless OpenShift and RKE recently changed so that containers can run >> >>>> as root by default (as of yesterday, they didn't), this is solidly a >> >>>> bad idea, since it makes it much more unintuitive to set up secure >> >>>> containers conforming with the guidelines for these Kubernetes >> >>>> platforms. >> >>> In my experience, containers trying to run stuff from shadow-utils in >> >>> their entrypoint/startup scripts tend to be a reason for containers to >> >>> *not* run on OpenShift/OKD without additional adjustments. >> >>> >> >>> A related (and more common) issue are images that expect to run with a >> >>> particular named user (or UID) determined during the build process >> >>> (again, most likely created using shadow-utils). >> >>> >> >>> I'm not familiar with Rancher but at least for OpenShift, I don't think >> >>> the availability of shadow-utils is very useful. At run time, you can't >> >>> use the shadow-utils at all and whatever you do with it during build >> >>> time is unlikely to be helpful (and actively harmful more often than >> >>> not) at run time when OpenShift assigns you an arbitrary UID. >> >> It's basically required for building containers that will work at >> >> runtime where OpenShift assigns an arbitrary UID. >> >> >> >> For example, in my containers, I *build* and create a "runtime user" >> >> with the UID 1000, and then set things up to use that context at the >> >> end. OpenShift uses that for its dynamic UID assignment. >> > But you do not need shadow-utils for that. Even OpenShift >> > documentation shows simple echo is enough: >> > >> > if ! whoami &> /dev/null; then >> > if [ -w /etc/passwd ]; then >> > echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd >> > fi >> > fi >> > https://docs.openshift.com/container-platform/3.10/creating_images/guidel... >> > (yeah, I know it's an old and obsolete version of docs) >> > >> What about all of the users of Docker and Podman who do? >> >> >> >> ``` >> >> from fedora >> >> run useradd XYZ >> >> user XYZ >> >> ... >> >> ``` >> >> Do you just break them out of the box? > > > Yes and that's the point of the Change Proposal (ie make this more widely known and allow people to change their Dockerfile). This change would only be applied starting from the Fedora 35 base image, I don't think it is unreasonable to have breaking change between major version of the container base image. > I think it would be unreasonable to break such a commonly established pattern, though. That's enough of a reason for people to stop using the Fedora base container.

On Tue, 6 Apr 2021 at 12:58, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote: > > On Tue, Apr 6, 2021 at 11:36 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > > > > On Tue, Apr 6, 2021 at 3:23 AM Clement Verna <cverna(a)fedoraproject.org&gt; wrote: > > > > > > > > > > > > On Mon, 5 Apr 2021 at 20:30, Daniel Walsh <dwalsh(a)redhat.com&gt; wrote: > > >> > > >> On 4/3/21 02:34, Tomasz Torcz wrote: > > >> > Dnia Fri, Apr 02, 2021 at 05:30:30PM -0400, Neal Gompa napisał(a): > > >> >> On Fri, Apr 2, 2021 at 5:18 PM Lars Seipel <ls(a)slrz.net&gt; wrote: > > >> >>> On Thu, Apr 01, 2021 at 02:36:48PM -0400, Neal Gompa wrote: > > >> >>>> Unless OpenShift and RKE recently changed so that containers can run > > >> >>>> as root by default (as of yesterday, they didn't), this is solidly a > > >> >>>> bad idea, since it makes it much more unintuitive to set up secure > > >> >>>> containers conforming with the guidelines for these Kubernetes > > >> >>>> platforms. > > >> >>> In my experience, containers trying to run stuff from shadow-utils in > > >> >>> their entrypoint/startup scripts tend to be a reason for containers to > > >> >>> *not* run on OpenShift/OKD without additional adjustments. > > >> >>> > > >> >>> A related (and more common) issue are images that expect to run with a > > >> >>> particular named user (or UID) determined during the build process > > >> >>> (again, most likely created using shadow-utils). > > >> >>> > > >> >>> I'm not familiar with Rancher but at least for OpenShift, I don't think > > >> >>> the availability of shadow-utils is very useful. At run time, you can't > > >> >>> use the shadow-utils at all and whatever you do with it during build > > >> >>> time is unlikely to be helpful (and actively harmful more often than > > >> >>> not) at run time when OpenShift assigns you an arbitrary UID. > > >> >> It's basically required for building containers that will work at > > >> >> runtime where OpenShift assigns an arbitrary UID. > > >> >> > > >> >> For example, in my containers, I *build* and create a "runtime user" > > >> >> with the UID 1000, and then set things up to use that context at the > > >> >> end. OpenShift uses that for its dynamic UID assignment. > > >> > But you do not need shadow-utils for that. Even OpenShift > > >> > documentation shows simple echo is enough: > > >> > > > >> > if ! whoami &> /dev/null; then > > >> > if [ -w /etc/passwd ]; then > > >> > echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd > > >> > fi > > >> > fi > > >> > https://docs.openshift.com/container-platform/3.10/creating_images/guidel... > > >> > (yeah, I know it's an old and obsolete version of docs) > > >> > > > >> What about all of the users of Docker and Podman who do? > > >> > > >> > > >> > > >> ``` > > >> > > >> from fedora > > >> > > >> run useradd XYZ > > >> > > >> user XYZ > > >> > > >> ... > > >> > > >> ``` > > >> > > >> Do you just break them out of the box? > > > > > > > > > Yes and that's the point of the Change Proposal (ie make this more widely known and allow people to change their Dockerfile). This change would only be applied starting from the Fedora 35 base image, I don't think it is unreasonable to have breaking change between major version of the container base image. > > > > > > > I think it would be unreasonable to break such a commonly established > > pattern, though. That's enough of a reason for people to stop using > > the Fedora base container. > > We do have the Base container and a Base Minimal, so maybe do it in > the later and not the former? Yes that's a good suggestion :-), I can probably make another Change for that tho. Based on the feedback received, I will update the change proposal to exclude shadow-utils from the packages proposed to be removed. That way we should be able to move on and at least remove sssd-client and util-linux ;-)

It's basically required for building containers that will work at runtime where OpenShift assigns an arbitrary UID.

On Thu, Apr 01, 2021 at 02:22:31PM -0400, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/SmallerContainerBase > > == Summary == > This change proposes to remove 3 packages (sssd-client, util-linux, > shadow-utils) from the Container Base Image (including the minimal > image). The Fedora Base Image is still quite large compared to other > distributions and the tools offered by these packages are not > essential in base image. I do not understand how do you want to use any system without for example mount(8), umount(8), ... ;-)

Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: > The container runtime in the host OS will have configured most mount > points before the container starts. It would be relatively uncommon > for processes inside the container image to need to mount additional > volumes later. That's fair, but util-linux contains many other tools that may be useful at runtine within a containerized tool (logger, flock, lsmem, rename, taskset, whereis, others?). Some those be moved somewhere else? /usr/bin/* files from f33's util-linux: cal chmem choom chrt col colcrt colrm column dmesg eject fallocate fincore findmnt flock getopt hardlink hexdump i386 ionice ipcmk ipcrm ipcs irqtop isosize kill last lastb linux32 linux64 logger login look lsblk lscpu lsipc lsirq lslocks lslogins lsmem lsns mcookie mesg more mount mountpoint namei nsenter prlimit raw rename renice rev script scriptlive scriptreplay setarch setpriv setsid setterm su taskset ul umount uname26 unshare utmpdump uuidgen uuidparse wall wdctl whereis write x86_64

- FChE _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

IMHO even then we would need the default "fedora" image to be the minimal one, as that's what a casual user will compare, unless they happen to know "fedora-minimal" exists.

On Wed, May 19, 2021 at 09:04:08AM +0200, Clement Verna wrote: > On Mon, 17 May 2021 at 16:40, Frank Ch. Eigler <fche(a)redhat.com&gt; wrote: > >> Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: >> >>> The container runtime in the host OS will have configured most mount >>> points before the container starts. It would be relatively uncommon >>> for processes inside the container image to need to mount additional >>> volumes later. >> That's fair, but util-linux contains many other tools that may be useful >> at runtine within a containerized tool (logger, flock, lsmem, rename, >> taskset, whereis, others?). Some those be moved somewhere else? >> >> /usr/bin/* files from f33's util-linux: >> >> cal chmem choom chrt col colcrt colrm column dmesg eject fallocate >> fincore findmnt flock getopt hardlink hexdump i386 ionice ipcmk ipcrm >> ipcs irqtop isosize kill last lastb linux32 linux64 logger login look >> lsblk lscpu lsipc lsirq lslocks lslogins lsmem lsns mcookie mesg more >> mount mountpoint namei nsenter prlimit raw rename renice rev script >> scriptlive scriptreplay setarch setpriv setsid setterm su taskset ul >> umount uname26 unshare utmpdump uuidgen uuidparse wall wdctl whereis >> write x86_64 >> > It is all about tradeoff between what **may** be useful and the size of the > base image. In the container base image space, size is very important (see > how successful is Alpine) and we have to make tradeoff in terms of what > tools are included by default in the image. This is very true, but to be brutally honest when you look at these differences below, I can't see any of the traditional distros ever winning on size, because Alpine is a different order of magnitude compared to them. If minimal size is the user's primary goal we're not even in the same contest as Alpine. That means we have to win on some other metric with the remaining users, for whom size is not the primary driving factor. We have to at least not look terrible size-wise compared to other traditional distros, which means we need to be as close as practical to Ubuntu / Debian (slim). IMHO even then we would need the default "fedora" image to be the minimal one, as that's what a casual user will compare, unless they happen to know "fedora-minimal" exists. > To get an idea on how Fedora does compared to some other distros > > REPOSITORY TAG IMAGE ID > CREATED SIZE > registry.fedoraproject.org/fedora rawhide 5e91f1acac7d 47 > hours ago 187 MB > registry.fedoraproject.org/fedora-minimal latest 438d4fec7485 5 > days ago 119 MB > docker.io/library/debian latest 4a7a1f401734 7 > days ago 119 MB FWIW, there's also debian:stable-slim at 72 MB > registry.opensuse.org/opensuse/leap latest 1a798c6c690f 5 > days ago 108 MB > docker.io/library/ubuntu latest 7e0aa2d69a15 3 > weeks ago 75.1 MB So we need to cull 45 MB / 36% of 'fedora-minimal' to reach this target, or 112 MB / 60% of 'fedora'. util-linux is a decent sized chunk of that, so makes sense to question its need. > docker.io/library/alpine latest 6dbb9cc54074 4 > weeks ago 5.88 MB Regards, Daniel

On Wed, May 19, 2021 at 04:37:55PM -0400, Daniel Walsh wrote: > The sad thing with these types of slimming is that it is horrible in > production use case. > > I often describe layered images in the form of a wedding cake, where you > have a large base > > and then smaller  mid section say with httpd or java engine and then a small > layer on top which has the actual application. > > This gives applications the ability to share most of the content of the base > image and those who share the mid section ability to share.  This shrinks > the overall disk space and more importantly allows the kernel to realize > that the same glibc or jre is being used so it is not loaded into memory for > each application run. I wonder how much of a difference it actually makes a difference in the practice. Lets say the Fedora base image is refreshed with updated RPMs on a weekly basis. Each application republishes their app containers on an arbitrarily different schedule, maybe fortnightly, monthly, whatever. Thus out of 10 different apps deployed, you could easily find yourself shipping 10 different versions of the base image. IOW we only get the space benefit of sharing a large image, if a large number of apps are aligned with using the same point in time version base image.

If we assume that apps in general will NOT align on the same point in time version of a base image, and the base image gets refreshed frequently then a small base image feels more important. If you can control all your apps such that they get built using the exact same version of the base, then a large base would get the benefits you describe. It is easier to achieve sharing with a slow refreshed container - IOW perhaps it matter more for RHEL than for Fedora.

It feels like we probably can cope with both scenarios at the same time though. The large base to maximize sharing doesn't actually have to be the bottom most layer to get sharing. We just need a large base /somewhere/ below the end user applications. It could easily be the second layer.

Lets say we have a "fedora" image that's tiny with the absolute bare minimum - just package manager and things that needs. Then we could have a "fedora-max" that inherits from this and throws in the kitchen sink. Or you could have more targetted layers "fedora-python", "fedora-java", etc. That way if you are deployed a bunch of python apps you can derive from the large-ish "fedora-python" container image and get maximum sharing of python stack, while we still offer a very small base image for those who don't care about sharing and just want to win on size.

Regards, Daniel

On Thu, May 20, 2021, at 8:21 AM, Daniel P. Berrangé wrote: > Lets say the Fedora base image is refreshed with updated RPMs on a weekly > basis. Each application republishes their app containers on an arbitrarily > different schedule, maybe fortnightly, monthly, whatever. Thus out of > 10 different apps deployed, you could easily find yourself shipping 10 > different versions of the base image. This of course is the big difference between flatpak and docker/podman/kubernetes as they work today. flatpak dynamically links base images (by having apps go into `/app`). I can't think of a hard reason that podman couldn't learn to dynamically link base images too with the same approach of having the app bits live in `/app`. Or honestly, in many cases we could just forcibly dynamically link layers, for most of the OpenShift platform we just have big statically linked Go binaries that don't overlay anything on the host (no package installs etc). One could imagine something like this in a Containerfile: ``` FROM quay.io/fedora:rust as buildroot COPY . /build RUN cargo build --release FROM DYNAMIC quay.io/fedora COPY --from-buildroot /build/target/release/myapp /usr/bin/myapp ``` Where the `DYNAMIC` part here opts-in to dynamic linking like flatpak, and the overlayfs is dynamic instead of static.

FWIW, there's also debian:stable-slim at 72 MB > registry.opensuse.org/opensuse/leap latest 1a798c6c690f 5 > days ago 108 MB > docker.io/library/ubuntu latest 7e0aa2d69a15 3 > weeks ago 75.1 MB So we need to cull 45 MB / 36% of 'fedora-minimal' to reach this target, or 112 MB / 60% of 'fedora'.

On Thu, Apr 01, 2021 at 02:22:31PM -0400, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/SmallerContainerBase > > == Summary == > This change proposes to remove 3 packages (sssd-client, util-linux, > shadow-utils) from the Container Base Image (including the minimal > image). The Fedora Base Image is still quite large compared to other > distributions and the tools offered by these packages are not > essential in base image. I do not understand how do you want to use any system without for example mount(8), umount(8), ... ;-) > The installed size of each package is : > > {| class="wikitable" > |- > ! Package !! Installed Size > |- > | util-linux || 13018140 My plan is to create more sub-packages from util-linux and create util-linux-core where will be essential tools like mount, losetup, blkid, lsblk, findmnt, etc. The change will be backwardly compatible. The classic util-linux.rpm will depend on this small util-linux-core package , so for all use-cases where is hardcoded util-linux we will not see a change. For use-cases where minimal installation is important will be possible to use alone util-linux-core. I also plan to move some less often used tools, like mcookie mesg raw isosize namei hardlink fincore wall readprofile ctrlaltdel fsck.cramfs fsck.minix mkfs.cramfs mkfs.minix fdformat to util-linux-optional package. Does it make sense?

Karel -- Karel Zak <kzak(a)redhat.com&gt; http://karelzak.blogspot.com _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure