5:35 a.m.
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been
happy with them so far, although I've recently run into an issue where I've
broken
what seems to be DNS for internal running containers, and I'm trying to isolate the
reason for that. Recently I created a bridge with my primary static ip address
and added my physical NIC as a slave on the bridge so that I'd be able to run
libvirt VMs that appeared to be machines directly on my local network. That
has all worked as expected:
---
# ipb link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s25 UP d0:50:99:a5:08:49 <BROADCAST,MULTICAST,UP,LOWER_UP>
exbr UP 9a:5f:b2:1e:94:a7 <BROADCAST,MULTICAST,UP,LOWER_UP>
virbr0 DOWN 52:54:00:4f:ca:69
<NO-CARRIER,BROADCAST,MULTICAST,UP>
virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST>
docker0 DOWN 02:42:dd:21:67:40
<NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding
priority 32 cost 100
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority
32 cost 100
# ipb a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s25 UP
exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64
virbr0 DOWN 192.168.100.1/24
virbr0-nic DOWN
docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64
---
My problem has suddenly become: My docker containers do not appear to be capable
of dns requests unless I *disable* firwalld, something I obviously do not want to
do. So, I've taken the approach of turning off the firewalld, restarting docker,
and capturing the known good iptables dump, and then tried to compare it to a known
bad dump that is produced after I turn on firewalld and restart docker. Interestingly
*they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the
traffic, and I'm seeing some very confusing info out of the kernel logs:
When things are configured in the "bad" state, the last log I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0
OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad"
configuration,
and according to rule 5 on the filter FOREWARD chain....that's an accept all rule?
So I'm very configured why this would be a problem.
In the known working case, here are my traces:
Failed trace output on filter rule 5
Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0
OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
So obviously something about disabling firewalld allows for the packet to pass
through the forward chain and hit the mangle POSTROUTING. It's completely unclear
to me how to diagnose what might be going on here, any tips appreciated. I have the
fully verbose outputs in this gist:
[1] https://gist.github.com/gunnarniels/6826a7ad6f8b800f4c0b68eca98e071e
GN