Static Route not applying to traffic
I've got two PFSenses (A, and B) and my ISP's gateway sitting on the same internet subnet. Both PFsenses can ping each other and otherwise communicate happily.
PFSense A has the WAN IP blah.blah.blah.230 and the LAN subnet 10.0.0.0/24
PFsense B has the WAN IP blah.blah.blah.231 and the LAN subnet 192.168.210.0/24
The ISP's gateway has the IP blah.blah.blah.225
All private networking firewall blocks have been removed from PFsense A and B.
There are no firewall rules that apply any gateway options to traffic.
PFsense A has its default gateway set to the ISP's gateway (.225), and a static route to send 192.168.210.0/24 traffic to PFsense B which it has added as a gateway.
Currently all traffic from PFsense A to 192.168.210.0/24 is wrongly going by the default gateway, ignoring the static route that has been set, does anyone know why this would be?
This is the output of netstat -rn showing the default and static routes on pfsense A (em0 being the WAN interface)
default blah.blah.blah.225 UGS em0
192.168.210.0/24 blah.blah.blah.231 UGS em0
If traffic appears to ignore a route, it's probably because something else is making the traffic do that. For example, putting a gateway on the LAN rule matching the traffic will force the traffic to that gateway – If such a rule were to match your traffic going to the other box, then it would get sent out the WAN.
Depending on the nature of the traffic, allowing private network traffic to move between those units across the WAN segment is likely still a bad idea. Consider a dedicated interface between them if you have to pass LAN-to-LAN traffic. Even if the WAN segment is "local" to you, it's still an untrustworthy network.
I've added a rule to catch the traffic going to the 192.168.210.0 network from the LAN and used that rule to log the packet and try and force it to use the other gateway. I can see the rule geting hit by the firewall logs but the packet still ends up going via the default gateway and not the gateway supplied in either the static route or the firewall rule.
I know our setup currently isn't ideal and in a perfect world we'd have it going over a dedicated interface, but there isn't anything technical that should be stopping this.
What sort of traffic is it?
You are better off not policy routing (rules on gateway set to "default", not a specific gateway) in these cases. Though routing on WAN can be complicated sometimes due to how the outbound WAN rules have route-to on them. Maybe try an floating rule, outbound, quick, on WAN, matching the destination network and without a gateway set on the rule. You might also have to add a "do not NAT" type rule so that your private-to-private traffic does not get source NAT applied.
If the two internal networks are on separate VLANs within the same switch setup, also consider using a tagged VLAN as a means of handing off traffic between the firewall rules to avoid using WAN.