I'm having a bit of an odd issue with firewalld interfering with network routing. Everything here is running CentOS 7; the physical host has kernel version 3.10.0-957.10.1.el7.x86_64. Here is what's going on:
1. I have a physical host, that has several KVM virtual machines. The physical host's eno1 ethernet interface is on my 192.168.2 network. The .2 network is hard wired to other devices and the internet at large via gigabit switch.
2. The physical host also has an interface and network internal to KVM, virbr2, which is 192.168.4 network and used for all of the VM's.
3. I have a VM which acts as a VPN server. It gives out addresses in the 192.168.8 network.
4. For clients in the 192.168.8 network, they can reach servers in the .4 network. Also, servers in the .4 network are able to reach clients with open ports in the .8 network.
5. Clients in the .8 network can NOT reach other devices on the .2 network. Likewise, things on the .2 network can NOT reach anything on the .8 network. The gateway for .8 is properly configured in the physical host as the .4 address of the VPN server.
6. If I turn off firewalld on the physical host, then clients in the .8 network CAN reach things in .2, and vice versa.
6.1. IP v4 forwarding is enabled in both the VPN VM and the physical host.
6.2. Enabling and/or disabling firewalld on the VPN VM does not change any of this behavior.
7. I have tried to put both virbr0 and eth0 in the same network Zone in firewalld- the "trusted" zone. I have also tried to put them in different zones and explicitly configure firewalld. Nothing works.
To make matters more interesting, if I enable logging of dropped packets in firewalld, I get nothing when I attempt to cross the networks... so I can't debug what's going on.
What should my next course of action be?
I have public IPs and am using firewalld for a router to provide Internet access for the internal network as well as forward ports for the public IPs to internal servers.
I have masquerade enabled on the external network, and no problem accessing the internet internally. The public internet has no problem reaching internal servers via port forwarding.
But, I cannot access anything via the public IPs from the internal network unless the internal network also has masquerade. While I can access servers via their internal IP, there are plenty of links using public host names, preventing this from being an acceptable limitation.
If I enable masquerade on the internal network, all servers can be accessed internally via their public IP, but the SMTP server becomes an open relay as it sees all incoming external traffic as originating from the router and trusts it. Nothing can properly log or control access via source external IPs.