5:35 a.m.
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been
happy with them so far, although I've recently run into an issue where I've
broken
what seems to be DNS for internal running containers, and I'm trying to isolate the
reason for that. Recently I created a bridge with my primary static ip address
and added my physical NIC as a slave on the bridge so that I'd be able to run
libvirt VMs that appeared to be machines directly on my local network. That
has all worked as expected:
---
# ipb link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s25 UP d0:50:99:a5:08:49 <BROADCAST,MULTICAST,UP,LOWER_UP>
exbr UP 9a:5f:b2:1e:94:a7 <BROADCAST,MULTICAST,UP,LOWER_UP>
virbr0 DOWN 52:54:00:4f:ca:69
<NO-CARRIER,BROADCAST,MULTICAST,UP>
virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST>
docker0 DOWN 02:42:dd:21:67:40
<NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding
priority 32 cost 100
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority
32 cost 100
# ipb a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s25 UP
exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64
virbr0 DOWN 192.168.100.1/24
virbr0-nic DOWN
docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64
---
My problem has suddenly become: My docker containers do not appear to be capable
of dns requests unless I *disable* firwalld, something I obviously do not want to
do. So, I've taken the approach of turning off the firewalld, restarting docker,
and capturing the known good iptables dump, and then tried to compare it to a known
bad dump that is produced after I turn on firewalld and restart docker. Interestingly
*they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the
traffic, and I'm seeing some very confusing info out of the kernel logs:
When things are configured in the "bad" state, the last log I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0
OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad"
configuration,
and according to rule 5 on the filter FOREWARD chain....that's an accept all rule?
So I'm very configured why this would be a problem.
In the known working case, here are my traces:
Failed trace output on filter rule 5
Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0
OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
So obviously something about disabling firewalld allows for the packet to pass
through the forward chain and hit the mangle POSTROUTING. It's completely unclear
to me how to diagnose what might be going on here, any tips appreciated. I have the
fully verbose outputs in this gist:
[1] https://gist.github.com/gunnarniels/6826a7ad6f8b800f4c0b68eca98e071e
GN
9:32 a.m.
New subject: Enabling firewalld is breaking my docker connectivity, need some
more eyes on the problem
On Tue, Sep 08, 2020 at 06:35:05AM -0400, Gunnar Niels wrote:
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been
happy with them so far, although I've recently run into an issue where I've
broken
what seems to be DNS for internal running containers, and I'm trying to isolate the
reason for that. Recently I created a bridge with my primary static ip address
and added my physical NIC as a slave on the bridge so that I'd be able to run
libvirt VMs that appeared to be machines directly on my local network. That
has all worked as expected:
---
# ipb link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s25 UP d0:50:99:a5:08:49
<BROADCAST,MULTICAST,UP,LOWER_UP>
exbr UP 9a:5f:b2:1e:94:a7
<BROADCAST,MULTICAST,UP,LOWER_UP>
virbr0 DOWN 52:54:00:4f:ca:69
<NO-CARRIER,BROADCAST,MULTICAST,UP>
virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST>
docker0 DOWN 02:42:dd:21:67:40
<NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding
priority 32 cost 100
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority
32 cost 100
# ipb a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s25 UP
exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64
virbr0 DOWN 192.168.100.1/24
virbr0-nic DOWN
docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64
---
My problem has suddenly become: My docker containers do not appear to be capable
of dns requests unless I *disable* firwalld, something I obviously do not want to
do. So, I've taken the approach of turning off the firewalld, restarting docker,
and capturing the known good iptables dump, and then tried to compare it to a known
bad dump that is produced after I turn on firewalld and restart docker. Interestingly
*they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the
traffic, and I'm seeing some very confusing info out of the kernel logs:
Looks like you're using a firewalld version that defaults to the
nftables backend. Check "nft list ruleset".
When things are configured in the "bad" state, the last log
I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0
OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad"
configuration,
and according to rule 5 on the filter FOREWARD chain....that's an accept all rule?
So I'm very configured why this would be a problem.
iptables and nftables are _independent_. That means it must pass through
both firewalls. If a packets is accepted in iptables it still has to
pass through the nftables (firewalld) rules.
In the known working case, here are my traces:
Failed trace output on filter rule 5
Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0
OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr
PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2
DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53
LEN=40
So obviously something about disabling firewalld allows for the packet to pass
through the forward chain and hit the mangle POSTROUTING. It's completely unclear
to me how to diagnose what might be going on here, any tips appreciated. I have the
fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0
docker (moby) very recently gained integration with firewalld [2].
[2]: https://github.com/moby/libnetwork/pull/2548
5:06 a.m.
New subject: Enabling firewalld is breaking my docker connectivity, need some
more eyes on the problem
> So obviously something about disabling firewalld allows for the
packet to pass
> through the forward chain and hit the mangle POSTROUTING. It's completely
unclear
> to me how to diagnose what might be going on here, any tips appreciated. I have the
> fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0
docker (moby) very recently gained integration with firewalld [2].
Hey Eric, that seems to have perfectly resolved my problem! I'm confused because it
previously
worked until I started messing about with my other interfaces, creating a bridge and
assigning
it master to the physical NIC. I never really touched the docker0 interface, and adding it
to
trusted suddenly made it start working. My understanding are firewalld zones are INPUT
only? Could you elaborate on what traffic was getting blocked prefiously, but by adding
the
docker0 iface to the trusted zone, I had suddenly whitelisted it?
Thanks again! You're always a huge help.
6:50 a.m.
New subject: Enabling firewalld is breaking my docker connectivity, need some
more eyes on the problem
On Thu, Sep 10, 2020 at 06:06:26AM -0400, Gunnar Niels wrote:
> > So obviously something about disabling firewalld allows for the packet to pass
> > through the forward chain and hit the mangle POSTROUTING. It's completely
unclear
> > to me how to diagnose what might be going on here, any tips appreciated. I have
the
> > fully verbose outputs in this gist:
>
> Try adding the docker bridge to the "trusted" firewalld zone.
>
> # firewall-cmd --zone trusted --add-interface docker0
>
> docker (moby) very recently gained integration with firewalld [2].
Hey Eric, that seems to have perfectly resolved my problem! I'm confused because it
previously
worked until I started messing about with my other interfaces, creating a bridge and
assigning
it master to the physical NIC. I never really touched the docker0 interface, and adding
it to
trusted suddenly made it start working. My understanding are firewalld zones are INPUT
only?
Mostly true. There are exceptions:
- masquerade
- forward ports
- --set-target also sets the catch-all target for the FORWARD chain.
- this is why using the "trusted" zone above works
Could you elaborate on what traffic was getting blocked prefiously,
but by adding the
docker0 iface to the trusted zone, I had suddenly whitelisted it?
FORWARD traffic. See note above.
1324
days inactive
1326
days old
firewalld-users@lists.fedorahosted.org
3 comments
2 participants
participants (2)
-
Eric Garver -
Gunnar Niels