Failover and routing issue with multi (dual) WAN with multi (dual) LAN

[robinxyz]

Hi Everyone, I have been banging my head on this one for a few evenings and can't seem to figure out why my LAN1/2 can't talk to each other when I change gateway to anything but default.

Here is the setup, tested with both 2.2.2 and 2.2.4 release

2 X WAN
2 X LAN
LAN1 FW IP 10.0.1.3
LAN2 FW IP 10.2.1.3

LAN1 host IP 10.0.1.6
LAN2 host IP 10.2.1.6

Here are the Gateway Groups

WAN1 and WAN2 in a Gateway Group call LB with both WAN set to tier 1 for load balance.
WAN1 and WAN2 in a Gateway Group call 1over2 with WAN1 set to tier 1 and WAN2 set to tier 2 for WAN1 failover to WAN2
WAN1 and WAN2 in a Gateway Group call 2over1 with WAN1 set to tier 2 and WAN2 set to tier 1 for WAN2 failover to WAN1

SCENARIO 1

When I set both LAN1 and LAN2 gateway to LB. Hosts on LAN1 and LAN2 are able to get internet, LB round robin seems to working
LAN1 host 10.0.1.6 can ping FW LAN2 IP 10.2.1.3
LAN2 host 10.2.1.6 can ping FW LAN1 IP 10.0.1.3

but

LAN1 host 10.0.1.6 can NOT ping LAN2 host 10.2.1.3 and vise versa, I don't see anything in firewall log showing blocked

SCENARIO 2

When I set both LAN1 and LAN2 gateway to 1over2. Hosts on LAN1 and LAN2 are able to get internet with WAN1 FW IP

but same issue with scenario 1 LAN1/2 hosts not able to ping each other

SCENARIO 3

When I set both LAN1 and LAN2 gateway to 2over1. Hosts on LAN1 and LAN2 are able to get internet with WAN2 FW IP

but same issue with scenario 1 LAN1/2 hosts not able to ping each other

a few notes

outbound NAT is set to auto
when both LAN1 and LAN2 gateway are set to default, Hosts on LAN1 and LAN2 are able to get internet via WAN1 only, which is expected, there is no problem with LAN1/2 hosts talking to each other.
I reset state tables after each test
for both LAN1 and LAN2, there are no other FW rules other than the single rule of IPv4* LAN1(2) net, * port, * destination, * port, and Gateway set to each scenario.

I am not good any visio, but I can attach some diagrams if it helps.

thanks

Robin

[Derelict]

https://doc.pfsense.org/index.php/What_is_policy_routing

And this is what you need to do:

https://doc.pfsense.org/index.php/Bypassing_Policy_Routing

[robinxyz]

awesome, Derelict

thanks, looks like that's what I am missing. I just tested by creating a IP alias named LocalNetwork with values of 10.0.1.0/24 and 10.2.1.0/24.

then add firewall rule of IPv4* LAN1(2) net, * port, LocalNetwork destination, * port, * Gateway to both LAN1 and LAN2

insert that rule before the LB rule. and like magic, it's working correctly now.