Setting the correct firewall for docker and desktop communication

zL0ki

Hi there,

I was hoping someone could shed some light on this.

I have proxmox with pfsense/pihole and a docker VM. I've successfully set it up or 90% there. I'm having some issues getting the exact firewall permissions.
Currently I've managed to have the docker containers communicate with my desktop but then I loose internet access from the desktop. The docker containers also cannot ping outside.
If I switch the firewall settings I have I loose communication with the containers but the docker containers can ping outside via IP but cannot resolve by name.

I've tried to make the rules as broard as possible to get it working.

For the docker interface I have

I have allowed any traffic and using VPN.

For the LAN which is where my desktop reside on I have

The is set to allow me to access internet. I have more specific rules to lock things down but I didn't want them to muddly the waters.

If I toggle these around I can talk to the containers but loose the internet.

I've disabled dns resolve on pfsense as pihole takes care of that. I can even see the container IPs in pihole when I switch the firewall rules over.

I missing something but been stuck on this for awhile. Any help would be appreciated.

Thanks

viragomann

@zL0ki
These pass rules are policy routing rules. That means, they direct any matching traffic (and this is any in your case) to the VPN gateway. Hence the packets do not reach the other subnet at all.

You have to apply them only to the upstream traffic instead.
Best way to do this is to add an alias of type networks and add all private network ranges to it (I call the alias RFC1918).
Then edit your policy routing rules, at destination check "invert match" and state the alias.

zL0ki

@viragomann Hi, thanks for your reply.

Since you gave the advice ive been trying to understand it to apply it.

I've applied the following to the LAN interface:

I can now access the containers which is great thank you!

As I only partially understand what's going on I'm not sure if any changes need to be done on the DOCKER firewall?

viragomann

@zL0ki
Firewall rules have to be defined on the interface, where the traffic is coming in. And rules are probed and applied if match from the top of the rule set to the bottom.

So to access the docker containers from LAN, you don't need any rule on the docker interface.
But the dockers need at least access to the internet, and as I understand, you want to direct them over NordVPN as well. If so you need the same policy routing rule on the docker interface as well. For internal access, e.g. DNS, you need to add an additional rule without the gateway stated.
For further security you can limit the source to "LAN net" respectively "DOCKER net".

Your LAN rule set allow any access now, but direct all access to destinations except the RFC1918 alias to NordVPN.

zL0ki

@viragomann
I think that first sentence clinched it for me. It helped me picture it. I've been getting confused with the source and destination on each interface and not picturing what the interface denotes.

My first install of pfsense was all over the place with rules. I'm glad I was forced to re-install it as package manager and updating would no longer work.

So I have this now:

So limiting the source ensures it blocks/ignores any traffic other than on those subnets?

I tested a docker container and can see VPN works. Which is awesome. I started going with a intermediary vpn container using gluetun. Got it to work but knew there was a solution which I just couldn't see. Turns out quite a simple one as well!

Thanks for the help, very much appreciated.

viragomann

@zL0ki said in Setting the correct firewall for docker and desktop communication:

So limiting the source ensures it blocks/ignores any traffic other than on those subnets?

This basically is useful to block traffic which is routed through a connected device. For instance if a device or VM has a VPN established, it would be possible to route traffic from the connected remote network through it. However, pfSense would not route response packets back, as long as it has no static route for it pointing to the device.

Your docker subnet has access to any local network like the LAN now per the second rule. Is this desired?

johnpoz

@zL0ki personally I wouldn't do ! rule like that.. I would have rule that allows traffic to rfc1918, and then on your last rule that allows internet I would force it out the gateway.

There are always multiple ways to skin a cat, but there have been problems with ! rules in the past when you use say vips, etc. I personally think its cleaner to be explicit in what is allowed, vs allowing the opposite of the destination.

it would still be only 2 rules. And at least for me its cleaner to see an explicit allow with a specific list of IPs that is allowed, vs hey if opposite of this destination then go out this gateway.

zL0ki

@viragomann said in Setting the correct firewall for docker and desktop communication:

Your docker subnet has access to any local network like the LAN now per the second rule. Is this desired?

It was more the fact I wanted to go with the blanket approach during setup and slowly narrow the parameters when I had a better understanding. I think restricting this to the devices that actually need access to it is preferrable.

@johnpoz said in Setting the correct firewall for docker and desktop communication:

personally I wouldn't do ! rule like that.. I would have rule that allows traffic to rfc1918, and then on your last rule that allows internet I would force it out the gateway.

I like trying different approaches to help with understanding so I will try this method. And now that I have a setup that works it's going to be easier to test.