Simple Problem: Static route not working

[pwnell]

I have 4 interfaces: WAN, LAN1, LAN2, LAN3.  Lets ignore LAN2 and LAN3 as they are not relevant.

WAN is standard interface with gateway to internet.  I am using NAT.

LAN1 is standard LAN interface, no default GW set.  Subnet 192.168.0.0/24.

I have a Linux router on LAN1 at IP 192.168.0.50.  It has been configured to forward packets to another network accessible via VPN.  On that machine I establish the VPN, and have packets route to that VPN's subnet.  I know this works perfectly because I just replaced my Unifi USG firewall with a pfSense, and with the USG and a static route it worked fine.

I added this gateway to pfSense:

Interface: LAN1, Address: IPv4, Gateway: 192.168.0.50 and everything else unchecked.

And this static route:

Destination network:  10.0.0.0/24
Gateway: 192.168.0.50

I dial up to VPN on 192.168.0.50.  I can connect from 192.168.0.50 to a machine on the VPN's subnet 10.0.0.10 just fine, and everything works.

I try to connect to 10.0.0.10 from another machine on my LAN1, say 192.168.0.30.  I can connect fine, and the traffic flows for about 20 seconds or so.  Then it just stops.  Firewall logs show (I have an allow any from LAN1 to any rule):

Oct 24 13:35:13 LAN1 Default deny rule IPv4 (1000000103) 192.168.0.30:55961 10.0.0.10:22 TCP:R
Oct 24 13:35:03 LAN1 Default deny rule IPv4 (1000000103) 192.168.0.30:51082 10.0.0.10:22 TCP:A

etc.  No rules that I add to the firewall seems to fix this.

Please help?

[pwnell]

After doing some diagnostic tracing I have an idea what is going on:

1. Host 192.168.0.30 sends packet to 10.0.0.10 port 22.  Since that subnet is not local, the firewall gets it (i.e. default GW).
2. The firewall has a routing table and gateway, it forwards the packet out on same interface but to host 192.168.0.50, the GW to the VPN
3. GW receives it, looks up its masquerading table and sends on the packet to the VPN.
4. Host on VPN gets packet, and replies to 192.168.0.30 via 192.168.0.50.
5. Packet arrives on tun0 interface on 192.168.0.50 GW, and GW sees the target (192.168.0.30) is on same subnet, so it does NOT go through firewall, but gets sent via the switch directly to 192.168.0.30.
6. After 30 seconds or so the state table times out on pfSense and it tears down the connection.  192.168.0.30 can no longer send to 10.0.0.10 until a new connection is established.

Fine - two questions:

1. Why did the USG appliance not have this issue?  That is the only variable here - the only item I changed.
2. How do I fix this?

[pwnell]

For anyone else as stupid as me - I ran into the same n00b mistake as I did a while ago, the USG clearly handled this some how and hid the truth from me:

https://forum.pfsense.org/index.php?topic=135256.msg740959

[pwnell]

PS: For anyone with similar issue - I fixed it by checking the option under Advanced - Firewall and Networking: Bypass firewall rules for traffic on the same interface.

Not 100% sure whether this is safe for my configuration but it works.

[johnpoz]

Wouldn't it be simpler to just create the vpn client connection on pfsense directly… Vs what is a hairpin and asymmetrical routing mess that you have to bypass rules on your interface, etc..

Other solution is to put this vpn endpoint on transit network connected to pfsense, so you remove the asymmetrical routing..  You could still have hairpins depending on where you put the transit vlan or its own physical interface and what other vlans are using the transit to get to this downstream machine.