Gateway firewall rule not applying to some traffic
-
I have a interesting issue with the gateway firewall rule option not applying to some traffic.
Some backround on the setup - I have two pfSense routers connected together through a OpenVPN tunnel, routed mode. The core end is on 192.168.0.0/24, and the remote side is on 192.168.1.0/24. The core end has multiple static IPs associated with it through IP aliases, and then 1:1 NAT rules from each static IP to an internal IP on the 192.168.1.0/24 (remote) subnet. The interface attached to the OpenVPN tunnel is named OVPN.
I have a firewall rule for the device with the static IP with the OVPN_VPNV4 gateway on the remote pfSense router, on the LAN table. This is supposed to send all traffic originating from the device with the static IP out the OpenVPN tunnel.
If I try to access one of the static IPs, this is what I get looking at the OVPN interface on the remote side:
12:22:11.309438 IP 107.72.164.130.51629 > 192.168.1.45.80: tcp 0
12:22:11.319933 IP 107.72.164.130.37885 > 192.168.1.45.80: tcp 0
12:22:12.305938 IP 107.72.164.130.51629 > 192.168.1.45.80: tcp 0
12:22:12.318125 IP 107.72.164.130.37885 > 192.168.1.45.80: tcp 0
12:22:14.309732 IP 107.72.164.130.51629 > 192.168.1.45.80: tcp 0
12:22:14.322105 IP 107.72.164.130.37885 > 192.168.1.45.80: tcp 0Wireshark on the destination server also confirms that those packets are getting to the server. However, there are no packets returning to the core over this tunnel.
Looking at the LAN interface on the remote side:
12:24:40.437243 IP 107.72.164.130.2843 > 192.168.1.45.80: tcp 0
12:24:40.438803 IP 192.168.1.45.80 > 107.72.164.130.2843: tcp 0
12:24:40.440643 IP 107.72.164.130.35142 > 192.168.1.45.80: tcp 0
12:24:40.442206 IP 192.168.1.45.80 > 107.72.164.130.35142: tcp 0
12:24:41.433608 IP 107.72.164.130.2843 > 192.168.1.45.80: tcp 0
12:24:41.437794 IP 107.72.164.130.35142 > 192.168.1.45.80: tcp 0
12:24:43.429357 IP 192.168.1.45.80 > 107.72.164.130.2843: tcp 0
12:24:43.437798 IP 107.72.164.130.2843 > 192.168.1.45.80: tcp 0
12:24:43.439312 IP 192.168.1.45.80 > 107.72.164.130.35142: tcp 0
12:24:43.442284 IP 107.72.164.130.35142 > 192.168.1.45.80: tcp 0Here, the packets that the server are putting out are visible as well.
Now, here is where things get interesting…
Looking at the WAN interface on the remote side:12:27:52.819181 IP 192.168.1.45.80 > 107.72.164.130.9049: tcp 0
12:27:55.821165 IP 192.168.1.45.80 > 107.72.164.130.9049: tcp 0For some reason, these packets are going out the WAN interface, not the OpenVPN tunnel.
Now, here is where things get REALLY interesting...
If I make a connection from the server with the associated firewall rule, this is what I see on the OVPN interface on the remote side:12:31:43.297100 IP 192.168.1.45.53649 > 64.182.208.181.80: tcp 0
12:31:43.297613 IP 192.168.1.45.53650 > 64.182.208.181.80: tcp 0
12:31:43.360256 IP 64.182.208.181.80 > 192.168.1.45.53650: tcp 0
12:31:43.360446 IP 64.182.208.181.80 > 192.168.1.45.53649: tcp 0
12:31:43.360521 IP 192.168.1.45.53650 > 64.182.208.181.80: tcp 0
12:31:43.360637 IP 192.168.1.45.53649 > 64.182.208.181.80: tcp 0
12:31:43.361139 IP 192.168.1.45.53649 > 64.182.208.181.80: tcp 402
12:31:43.421463 IP 64.182.208.181.80 > 192.168.1.45.53649: tcp 0
12:31:43.421607 IP 64.182.208.181.80 > 192.168.1.45.53649: tcp 403
12:31:43.423199 IP 192.168.1.45.53649 > 64.182.208.181.80: tcp 0
12:31:43.461909 IP 192.168.1.45.53650 > 64.182.208.181.80: tcp 323
12:31:43.480867 IP 64.182.208.181.80 > 192.168.1.45.53649: tcp 0
12:31:43.524600 IP 64.182.208.181.80 > 192.168.1.45.53650: tcp 0
12:31:43.524722 IP 64.182.208.181.80 > 192.168.1.45.53650: tcp 402
12:31:43.526124 IP 192.168.1.45.53650 > 64.182.208.181.80: tcp 0
12:31:43.584187 IP 64.182.208.181.80 > 192.168.1.45.53650: tcp 0Doing that, I am able to establish a connection just fine and all associated traffic passes through the OpenVPN tunnel (matching the firewall rule in order to do this).
And if I look at the WAN interface at the remote side when I do this, there are no packets from this device.
Both sides ran 2.2.6, but I updated the remote side to 2.3.2 just to rule out a problem with the 2.2.6 version. The problem still persisted after this update.
Any ideas?
-
What you really need is reply-to on the remote side with the actual 192.168.1.0/24 network. If there is an assigned interface on that side it should be automatic. This will direct reply traffic back out the interface it came in on. You don't want to use policy routing for that (it won't work) and it should be removed.
You probably have rules matching the traffic on the OpenVPN tab instead of or in addition to the OVPN interface tab. Interface groups are processed first (Which is what the OpenVPN tab is under the hood), first match wins, and there is no reply-to possible there.
See this example. SSH with a port translation but the same thing you are trying to do: https://forum.pfsense.org/index.php?topic=82732.msg453269#msg453269
Everything should be the same on 2.3.2. Now you're upgraded so that's a bonus. ;)
-
That worked!
I did in fact have an allow any/any rule on the OpenVPN rules tab, as well as the OVPN tab. Removed that rule on the OpenVPN tab, and changed the rule on the OVPN tab for TCP/80 and a destination IP of 192.168.1.45. Also removed the rule in the LAN tab with the gateway manually assigned.
I can now get to that device through the external IP.
Thank you very much!