Bad routing to multiple edge routers
I would like to see if anyone has any ideas. I've tried all I can think of.
I have multiple separate edge routers. For example 3 routers with one ISP wan link on each.
Edge Router (ER)
ER LAN 1 - 192.168.0.11
ER LAN 2 - 192.168.0.12
ER LAN 3 - 192.168.0.13
Core Router (CR) WAN - 192.168.0.1
Default gateway 192.168.0.11
Inside router LAN - 10.0.0.1
I have an public routed IP ex. 184.108.40.206 routed incoming on ER2 to CR WAN.
220.127.116.11 has a Virtual IP alias on CR1.
18.104.22.168 is port forwarded to internal machine 10.0.0.10 on CR1
10.0.0.10 has a policy LAN rule with gateway ER2 set.
Packet comes in properly on ER2 and the reply packet gets sent out ER1. Why wouldn't it go out ER2 as directed in the policy rule. Is it because it's a reply and not the original packet? If so, is there any way around this to make the reply return through it's original entry router?
That's not a proper Multi-WAN setup. You should connect each router to a seperate pfSense interface and set the respective router as gateway in the interface settings. Alternativly you can set up VLANs on WAN, one for each router.
The policy routing rule you've set is for outbound traffic only, which may work properly. But it doesn't influence inbound traffic.
Ah, so the reply packet is currently on a "shared" WAN and therefore will always follow the default route. If I understand, you are saying if I put the edge router LANs to separate interfaces on the core, the replies will automatically return properly as the core knows what interface the conversation is on. Correct?
If that is the case, then as an alternative, I should be able to setup separate openvpn links with interfaces and virtually achieve the same thing. Sound right?
Correct. That is handled by the pf reply-to function. But this works on interface base.
So it replies packets to the gateway assigned to the interface either in the interface settings or by automatical configuration like OpenVPN or DHCP.
Note that the reply-to also requires that the firewall rule which is allowing the respective incoming traffic is defined on an interface tab, not on an interface group tab or in floating rules!
So for OpenVPN you have set your rules on the proper VPN interface tab, which you get after assinging an interface to the OpenVPN instance.
The OpenVPN tab is an interface group including all OpenVPN instances you're running.
Thanks so much Virgomann. It makes perfect sense.