PfSense 2.2 WAN Default gateway issues
[Cloud]–---------[WAN-pfSense] –------- [switch] –------------- [LAB-pfSense]
Hopefully the graphic lines up....
Boxes are connected via a vSwitch on ESXi 6.
All NICs are VMXNET3.
I discovered after doing a lot of digging, that when the default gateway is set on LAB-pfSense, it tries to send all packets there, even for directly connected networks on the WAN.
The network connected to that central switch is a /27, and I run OSPF in that segment.
If I try and connect PC -> LAB-pfSense wan address, it fails. If I connect [Cloud] (VPN) to pfSense WAN, it works.
I did a lot of tcpdumping, and discovered that the ACKs were dropping at the Sophos. I did a traceroute from LAB-pfSense, back to the Sophos, and found that it was going to the WAN-pfSense first. Even though the Sophos and LAB-pfSense are on the same switch / subnet. Checked arp tables, and see the appropriate entry for the Sophos interface.
Now, if I remove the default gateway from the WAN and add a static route back to the PC network, via Sophos, everything works. If I leave the static, and add the default GW, it stops working again.
Is there something I need to change on the pfSense to tell it to send via a local interface first, if it can, then try default gw?
I wanted to add, that the WAN-pfSense, on the traceroute, would respond with ICMP redirect. I disabled ICMP redirect, and the packets would just go straight to WAN-pfSense.
Just not sure why it would send the packet to WAN-pfSense, if the 'WAN' interface is on the same network as it's target.
Alright. Looks like removing default gateway from the WAN interface, then, going command line and running:
route add -net [network] [gateway]
Fixes the issue. Now able to access via devices on the sophos LAN, and it has internet connectivity.
Tried advertising a default route from the WAN pfSense, instead of setting static via command line, but for whatever reason, it didn't want to work. Even though the OSPF routing showed a 0.0.0.0/0 route learned, it wouldn't use it.