On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> This document represents a proposed Change. As part of the Changes
> process, proposals are publicly announced in order to receive
> community feedback. This proposal will only be implemented if approved
> by the Fedora Engineering Steering Committee.
>
>
> == Summary ==
>
> The `systemd-udev` package installs
> `"/usr/lib/systemd/network/99-default.link"`,
> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
> This is particularly important for bridge and bond devices.
>
> This change can either only apply to bridge/bond devices, or to
> various software devices. That is to be discussed.
Hi!
I already participated in the upstream discussion, so what I write
here will be a restatement to some extent, but with a look from the
side of Fedora specifically.
The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
or 2. limiting the change to bridge and bond devices.
Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have
a hardware address. The proposal as written (blanket MACAddressPolicy=none)
would change behaviour for all kinds of devices, incl. e.g. software
devices like veths, and cheap hardware devices without a fixed MAC.
The proposal doesn't provide any justification for this (except for
simplicity of implementation) and this variant seems pretty bad and
I'm strongly opposed.
Re variant 2: the proposal limited to brige/bond devices seems much more
reasonable. In particular, the case described below of a server (virtualized
or not) in a big datacenter is the one case where the benefits of
MACAddressPolicy=none are clearly visible. I still don't think it's worth
changing the default, but here the cost:benefit ratio is much closer.
> == Benefit to Fedora ==
>
> Pros:
>
> - Consistent behavior with RHEL8 and RHEL9.
>
> - Avoid race of udev and the tool that creates the interface.
The race will happen if the creation is done in a specific way. But at
the same time, even the Change proposal describes how to avoid the
race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation
can be summarized as "we have a bunch of 'big' tools that create
devices like NetworkManager or systemd-networkd, which already know or
can be easily fixed to avoid the race, and a manual tool which can be invoked
in a way that avoids the race". Instead of changing the default in udev we
could educate people how to invoke it better.
This comes across as blaming every networking tool out there for
having their previously correct & working behaviour be broken by
a systemd change imposing new requirements on them.
Are there any notable tools which actually follow the usage pattern
described, or are we in effect having to fix *everything*, including
an uncountable amount of documentation, blogs, examples, most of
which will prove impossible to fix in practice.
> - Bridge and bond interfaces can get the MAC addresses from
their first port.
>
> In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
> get a MAC related to one of its physical NIC devices. In the case of
> provisioning
> new systems (especially in a large datacenter) information about the server
> can be used to configure the network environment (DHCP, Firewall, etc) before
> the server is ever even powered on. This does mean that you may have to update
> your network environment configuration if you replace a NIC (con), but that you
> won't have to update your network environment configuration if you re-install
> your server (pro), which is what you'd have to do now with
> `MACAddressPolicy=persistent`.
Yep. This is *the* case.
Having the bridge/bond get the MAC addr of tht first physical NIC is pretty
compelling from an automation POV, especially when the virtualziation host is
doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to
a guest, it is not uncommon to apply firewall rules providing anti-spoofing
protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP
addresses. Libvirt provides this functionality with its 'clean-traffic'
network filter, and other virt/cloud mgmt apps do similar.
> Cons:
>
> - Deviate from upstream systemd.
This is disappointing, as a great benefit of systemd is the level of
consistency it brought to distro behaviour, and I think this change
would be useful to all distros (in the context of bridge/bond devices)
It is also important to mention that Fedora will "deviate"
from itself
(it's former self). We would be changing a default in place since ~2013 [1].
[1]
https://github.com/systemd/systemd/commit/16b9b87aee
Yes, that is quite unfortunate.
> It is desirable that RHEL and Fedora behaves similar. A possible
outcome
> could be the current behavior stays and RHEL 10 would change behavior. On the
> other hand, different distributions (or even Fedora spins) have different
> uses and needs. Deviating might be fine. In the same vein, there is also
> a desire to stay close to upstream systemd behavior. But the uses of systemd
> project go beyond Fedora/RHEL, so deviating here may also be fine.
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race
when 'ip' is invoked.
IMHO The more compelling reason is compliance with MAC address filtering
applied to VMs.
- We could improve the tools: 'ip link' could be taught to
wait until
udev has stopped processing the device, users can be taught to use
better invocations.
Teaching users is a impractical solution IMHO. The 'ip' tool already
has poor usability/learnability IME, such that getting users to learn
differences is pretty challenging, plus there's way too much docs out
there in the wild that will never be updatable.
- For "small" users (individual machine admins who just
install some
server and configure it after turning it on and/or swap network
cards between machines), having stable MAC addresses that doesn't
depend on the order of device discovery or removal of a single
network card seems better.
- For "big" users (the datacenter case), changing the policy make sense,
but at the same time, those folks can just insert a policy override,
they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case.
Sometimes RHEL and Fedora have different defaults, sometimes RHEL
takes years to follow what Fedora does.
So overall, I think this proposal would make most sense when limited
to specific types of hardware (bridge and bond?), and also to specific
spins: it's a better fit for CoreOS than Workstation…
This doesn't feel like a spin-specific problem to me, given it appears
useful to both host & guest scenarios alike, and nested VM means
host+guest are often the same thing.
But I haven't seen discussion of other approaches: would making
'ip'
better be an option? 'udev' already locks some devices when processing
using a simple BSD lock. Should we do the same for network devices
and just teach 'ip' to do the same? (This would be much simpler and
maybe more appealing that teaching it to wait for udev to report
that it's finished).
With regards,
Daniel
--
|:
https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|:
https://libvirt.org -o-
https://fstop138.berrange.com :|
|:
https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|