On 6/27/22 07:34, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jun 27, 2022 at 09:40:53AM +0100, Daniel P. Berrangé wrote:
> On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
>> On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>>> This document represents a proposed Change. As part of the Changes
>>> process, proposals are publicly announced in order to receive
>>> community feedback. This proposal will only be implemented if approved
>>> by the Fedora Engineering Steering Committee.
>>>
>>>
>>> == Summary ==
>>>
>>> The `systemd-udev` package installs
>>> `"/usr/lib/systemd/network/99-default.link"`,
>>> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>>> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC
address.
>>> This is particularly important for bridge and bond devices.
>>>
>>> This change can either only apply to bridge/bond devices, or to
>>> various software devices. That is to be discussed.
>>
>> Hi!
>>
>> I already participated in the upstream discussion, so what I write
>> here will be a restatement to some extent, but with a look from the
>> side of Fedora specifically.
>>
>> The proposal has two variants: 1. just changing the policy to
MACAddressPolicy=none
>> or 2. limiting the change to bridge and bond devices.
>>
>> Re variant 1: MACAddressPolicy=persistent applies to all devices that don't
have
>> a hardware address. The proposal as written (blanket MACAddressPolicy=none)
>> would change behaviour for all kinds of devices, incl. e.g. software
>> devices like veths, and cheap hardware devices without a fixed MAC.
>> The proposal doesn't provide any justification for this (except for
>> simplicity of implementation) and this variant seems pretty bad and
>> I'm strongly opposed.
>>
>> Re variant 2: the proposal limited to brige/bond devices seems much more
>> reasonable. In particular, the case described below of a server (virtualized
>> or not) in a big datacenter is the one case where the benefits of
>> MACAddressPolicy=none are clearly visible. I still don't think it's
worth
>> changing the default, but here the cost:benefit ratio is much closer.
>>
>>> == Benefit to Fedora ==
>>>
>>> Pros:
>>>
>>> - Consistent behavior with RHEL8 and RHEL9.
>>>
>>> - Avoid race of udev and the tool that creates the interface.
>>
>> The race will happen if the creation is done in a specific way. But at
>> the same time, even the Change proposal describes how to avoid the
>> race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
situation
>> can be summarized as "we have a bunch of 'big' tools that create
>> devices like NetworkManager or systemd-networkd, which already know or
>> can be easily fixed to avoid the race, and a manual tool which can be invoked
>> in a way that avoids the race". Instead of changing the default in udev we
>> could educate people how to invoke it better.
>
> This comes across as blaming every networking tool out there for
> having their previously correct & working behaviour be broken by
> a systemd change imposing new requirements on them.
I was a bit sloppy in my phrasing. NetworkManager and systemd-networkd
already do the right thing. 'ip link' is the odd one out. Various other
tools (e.g. netplan or Debian's network scripts) don't implement this
internally,
but call e.g. NetworkManager or 'ip'. So if we changed 'ip', this would
go
a long way to solving the problem (I'm may be wrong on some details here,
please correct me if necessary.)
I'm not "blaming" the tools, I completely understand that they were
written a long time ago. But in fact the issue is fairly generic: any
software which interacts with devices that udev also touches MUST wait
for udev to be done with the device. It's not just the MAC address
policy, but also other rules that users may configure locally, sysctl
configuration, etc. Without synchronization, one runs into races and
errors from the tools when they try to configure things in parallel.
> Are there any notable tools which actually follow the usage pattern
> described, or are we in effect having to fix *everything*, including
> an uncountable amount of documentation, blogs, examples, most of
> which will prove impossible to fix in practice.
>
>>> - Bridge and bond interfaces can get the MAC addresses from their first
port.
>>>
>>> In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
>>> get a MAC related to one of its physical NIC devices. In the case of
>>> provisioning
>>> new systems (especially in a large datacenter) information about the server
>>> can be used to configure the network environment (DHCP, Firewall, etc)
before
>>> the server is ever even powered on. This does mean that you may have to
update
>>> your network environment configuration if you replace a NIC (con), but that
you
>>> won't have to update your network environment configuration if you
re-install
>>> your server (pro), which is what you'd have to do now with
>>> `MACAddressPolicy=persistent`.
>>
>> Yep. This is *the* case.
>
> Having the bridge/bond get the MAC addr of tht first physical NIC is pretty
> compelling from an automation POV, especially when the virtualziation host is
> doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to
> a guest, it is not uncommon to apply firewall rules providing anti-spoofing
> protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP
> addresses. Libvirt provides this functionality with its 'clean-traffic'
> network filter, and other virt/cloud mgmt apps do similar.
>
>>> Cons:
>>>
>>> - Deviate from upstream systemd.
>
> This is disappointing, as a great benefit of systemd is the level of
> consistency it brought to distro behaviour, and I think this change
> would be useful to all distros (in the context of bridge/bond devices)
That is a good point. If this is done, I too would prefer to change it
upstream rather than just downstream in Fedora.
Big +1 from me on getting upstream to change so that we are in alignment. I don't
know if it was made clear in the upstream discussion, but maybe upstream would be
more accepting for just changing the policy for bond/bridge/team devices.
Similarly, we might be able to get the next RHEL to adopt the approach of
MACAddressPolicy=persistent for non bond/bridge/team and MACAddressPolicy=none
for bond/bridge/team. IOW we may be able to get upstream systemd, Fedora, and
downstream (RHEL/CentOS,Alma,Rocky) to all be in alignment.
Dusty