https://bugzilla.redhat.com/show_bug.cgi?id=1122700

Bug ID: 1122700 Summary: sysctl.conf / sysctl.conf.d settings not read when modules are loaded (one result being that libvirt bridged networking with NetworkManager does not work correctly) Product: Red Hat Enterprise Linux 7 Version: 7.0 Component: kmod Severity: high Priority: low Assignee: dshea@redhat.com Reporter: awilliam@redhat.com QA Contact: kernel-qe@redhat.com CC: arapov@gmail.com, awilliam@redhat.com, bburke264@gmail.com, dcbw@redhat.com, djasa@redhat.com, dwmw2@infradead.org, edgar.hoch@ims.uni-stuttgart.de, extras-qa@fedoraproject.org, hancock@sedsystems.ca, hiroshi01.itani@g.softbank.co.jp, jonathan@jonmasters.org, jwboyer@redhat.com, kmod-maint@lists.fedoraproject.org, laine@redhat.com, m.koshelev@gmail.com, mschuppe@redhat.com, msivak@redhat.com, notting@splat.cc, rc556677@outlook.com, sjenning@redhat.com, vpavlin@redhat.com, zbyszek@in.waw.pl Depends On: 634736

+++ This bug was initially created as a clone of Bug #634736 +++

Description of problem:

On a system that uses NetworkManager and has bridges, when system boot is complete the settings of net.bridge.bridge-nf-call-iptables , net.bridge.bridge-nf-call-ip6tables and net.bridge.bridge-nf-call-arptables are always "1", despite /usr/lib/sysctl.d/00-system.conf attempting to set them to "0". This prevents bridged connections to virtual machines working almost at all (if you set a static IP address a VM can ping numeric IPs, but that's about all).

The problem, which is rather well known by now, is that the settings only actually exist once the 'bridge' module is loaded, but the systemd service which applies these settings, systemd-sysctl.service , runs before the module is loaded. There is no mechanism in the kernel, kmod, or systemd which would cause sysctl settings to be re-read and re-applied when a kernel module is loaded.

The old 'network' service has a hack for this. It runs the function apply_sysctl() from /etc/init.d/functions a couple of times when bringing up interfaces. Hence folks using bridging with the old network service didn't see this problem. NetworkManager does not have an equivalent hack, however.

The 'bridge' incarnation of this problem is the one people are always running into, but in theory at least it is a generic problem, it could affect any other module with tunable settings.

There are various proposals for fixing this in the Fedora bug, notably from Edgar Hoch:

EH#1: Implement a hack in NetworkManager similar to the one in the old network service: "I have created a dispatcher file for NetworkManager that reloads the kernel parameter after change of a network interface." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c5

EH#2: "A real solution - the best? - should be that every kernel module will check for parameters in sysctl config files immediate after loading the kernel module. This would avoid the con listed above." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c7

EH#3: "For solving the general problem, I have the following idea:

- The kernel should remember parameters set by sysctl even if this parameter does not exist in the kernel at the time when the parameter was set. When the parameter is created later (for example by loading a module), the kernel should check the list of previous loaded parameters and if the parameter is found, this value should be used as default instead of other defaults.

This solution needs changes in the kernel. But it sets the values at the right time (only the kernel knows when a parameter is created) and it has less overhead than other solutions (for example like checking sysctl config files after every module load)." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c9

The sysctl.d manpage explicitly notes the 'bridge' incarnation of this bug, and suggests two other approaches:

SYSCTL#1: create a udev rule ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge"

SYSCTL#2: cause the 'bridge' module to be loaded statically very early in boot by listing it in a file in /usr/lib/modules-load.d/

Both of these approaches are specific to the 'bridge' incarnation of the problem.

However we decide to do it, at *least* the bridge incarnation of this issue really needs to be solved; if you're trying to set up libvirt bridged networking with NetworkManager you *will* trip over this bug, and it's quite frustrating to diagnose until you stumble across an old Fedora guide that mentions the sysctl parameters or something.

RHEL 7 and all now-supported Fedora versions are basically identical so far as this bug goes, so the same fix can and probably should be used for both. There's a whole other mess related to these parameters for RHEL 6 - see https://bugzilla.redhat.com/show_bug.cgi?id=919472 , but don't dive in too deep - but thankfully that should not be at all relevant to RHEL 7, as our attempt to override the defaults is no longer in /etc/sysctl.conf .

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=634736 [Bug 634736] sysctl.conf / sysctl.conf.d settings not read when modules are loaded