Multi WAN and multiple gateway issue

Troutpocket · Mar 27, 2025, 4:45 PM

This issue is resolved, but I would like help understanding why it happened.

Last night our multi wan setup started blocking all inbound traffic with "pfsense Default deny rule IPv4 (1000000103)". Outbound traffic seemed to be unaffected. As such, I wasn't able to access any on-prem resources from off-site, but I was able to get out of the network without issue on our primary uplink (WAN-1).

My gateway group has a tier1 (WAN-1) and tier2 (WAN-2) configuration that is supposed to switch based on packet loss. There was some intense thunderstorms at the time, but nothing went down on-site or nearby upstream.

Searching for this behavior I learned it can happen if you have a two gateways and inbound traffic somehow gets routed back out the wrong gateway - thus generating the deny. I checked with netstat -nr and determined that only one gateway was currently active.

I fixed the issue by going into the routing->gateways and checking the box for Force State - Mark Gateway as Down on the backup interface (WAN-2). Immediately everything popped back to life and traffic was once again allowed in on WAN-1.

I have >50 of these setups out in the world and have never seen this behavior before. Any suggestions what else I could check to understand why this happened on a multi WAN setup?

viragomann · Mar 27, 2025, 9:44 PM

@Troutpocket
The failover group and its status has no impact on incoming traffic normally. Request packets are directed out to the gateway, which is defined on the interface, the request came in, as long as your rules are set correctly.

Do have any interface group or floating rules allowing incoming WAN traffic?

Troutpocket · Mar 27, 2025, 10:43 PM

@viragomann No floating rules. This HA pair is built the same as all the others. I double checked my outbound NAT and it looks clean to me.

The RFC1918 is just our LAN networks.

I arrived at the possibility that it was asymmetric routing after finding threads with similar issues. I tcpdumped inbound traffic and saw it arriving on the HA WAN but not returning on that interface. I couldn't find it going anywhere else so I assumed it was being dropped due to the default rule. It wasn't until I forced WAN2GW down that immediately traffic started behaving normally.

viragomann · Mar 27, 2025, 11:09 PM

@Troutpocket
Outbound NAT rule also have now impact on the respond packets on incoming requests. pfSense just translates the source address back into the origin destination address the request was sent to.

Yes, that the traffic is blocked by the default deny rule might indicate an asymmetric routing issue. But why should it happen if the rules are set properly.

The proper rerouting is controlled by the reply-to tag. Did you disable it in System > Advanced > Firewall & NAT or in the rule by any chance?

If pfSense passes an incoming packet it add a state with the source IP, the WAN IP and the destination IP. You can check the state table if the state is set with the correct WAN IP.

Also you can just state different descriptions for the WAN rules and enable logging, presuming you have separate rules on both WAN interfaces. Then check the log to see which interface the traffic is coming in.

Troutpocket · Mar 27, 2025, 11:19 PM

@viragomann said in Multi WAN and multiple gateway issue:

The proper rerouting is controlled by the reply-to tag. Did you disable it in System > Advanced > Firewall & NAT or in the rule by any chance?

Not disabled.

I didn't look closely enough when reviewing the state tables to see if WAN2 was referenced when WAN1 should have been.

Hopefully it never happens again, but I have some things to look into if I ever come across this again. Thanks for discussing it with me!