HA, CARP, Multi WAN w/o CARP reply-to issue
I have a setup where 2 pfSense systems are setup on different routers for WAN so CARP is not an option for the WAN interfaces. For the internal interfaces they are connected to the switch fabric so carp is enabled. Ultimately we are using BGP so outbound NAT is configured for BGP IPs and public nets on LAN interfaces are managed via BGP.
This is the issue I had with this setup. By default to deal with Multi WAN pfSence adds a reply-to <gateway ip="">to each WAN rule and the specific reply-to IP from the primary is transferred to the secondary pfSense system which has a different gateway. I was able to resolve this issue by enabling the "Disable reply-to on WAN rules " option in "System > Advanced > Firewall / NAT".
Therefore the problem is solved in this case, but then I thought what if you had this setup with Multi WAN where each pfSense system has two WAN connections and all four WAN connections have a different gateway. See the diagram below which is not uncommon for setups in Data Centers.
> PFSense <
ISP2----WAN2----/ --- > LAGGs > Trunks > LANs > CARP > ------
> Switch <
ISP3---WAN1----\ --- > LAGGs > Trunks > LANs > CARP > ------
> PFSense <
Now you might ask why would you have 4 ISPs. Well some business would, but more likely they have two ISPs with redundant connections and if you have a diverse path for each ISP connection, then your gateway for each connection could be different.
In order to reach the pfSense system in the case of a BGP failure the interface IPs need to be available and without reply-to, there could be issues depending on which WAN interface you connect through.
So what is the solution with this setup?
Obviously I can disable pfsync on WAN rules, but I have to do this for each rule, which is tedious. It would be better if there was a way to disable pfSync for an interface. It would be even better if when a rule is transferred with the reply-to setting, the secondary pfSense system would replace it with the appropriate gateway for that interface. Perhaps this could be set using an alias that references the interface gateway?
OK my explanation of the issue was wrong. I ended it with this statement "Perhaps this could be set using an alias that references the interface gateway?" referring to the reply-to setting for WAN rules. Well that was not the issue as the system does exactly what i asked. The issue I was having is that the default gateway on the secondary firewall was being changed via the XMLRPC sync. Not sure how I could still reach the secondary firewall unless the route was not being change in the routing table which may be due to a bug I have reported before.
Anyway the fix for my issue would to simply allow specific routes to be ignored for config sync. Otherwise I can enter routes on each pfSense system in the clutter manually. So i guess this has really become a feature request.