Routing policy igored for icmp reply
In order to implement redundant connections, I'm setting up policy based routing and get some unexpected behaviour. Setup is like this:
em0 - WAN - default gateway
em1 - LAN - network NetA connection to test machine A
em2 - CONN - connection to another router CONN-RT, serving test machine B in network NetB
For the start, there are no gateway groups.
LAN ICMP src:any dst: <netb>gateway CONN-RT
CONN ICMP any any
Testing ping on machine A to machine B:
Observing the traffic using tcpdump on em0 and em2, I can see icmp echo requests from machineA to B on interface em2, that's fine and just what is expected from the rule on LAN.
Testing ping on machine B to machine A:
The icmp echo requests from machine B arrives on interface em2, but the icmp reply from machine A is routed to em0, ignoring the LAN gateway, and using the default gw instead.
The same will happen with other tcp traffic as well.
I believe this is caused by the implicitely generated reverse rule overriding the explicit rule. Setting the statetype to none on CONN, the return traffic is dumped alltogether, so I'm stumped now.
How can I make the back-traffic obey the gateway policy?
Digging a little in the documentation, I found that I probably need the reply-to option being set on the CONN interface, which works in the opposite direction as pfSense's gateway option, which actually sets route-to.
https://redmine.pfsense.org/issues/2676 already requests this option.
We automatically set reply-to in a way that's proper for virtually all use cases, that feature request is just to make it configurable for those who might have several gateways on one interface. Guessing you don't have a gateway specified under Interfaces>(whatever em2 is called) page. Where an interface isn't configured properly as an Internet connection (has gateway chosen on interface page), no reply-to is added.
Ok, assigning the gw to the interface did the trick, thanks!