HA Multi-Wan + outbound NAT

[globule_655]

Greetings everyone !

I have a question regarding outbound NAT rules in a clustered pfsense configuration with multi-wan.

Our network consists of multiple user subnets who need to access the Internet.
Also, we work with clients who need to authorize the specific public IP(s) from witch we access ressources on their side.
So to describe the setup more in details, we have a pool of public IPs from each ISP and 2 of them are dedicated for Internet access. These are configured as CARPs virtual IPs and I also configured outbound NAT rules to force traffic from users to those CARPs.

A gateway group is also configured so Internet access should failover in case one of our Internet access goes down.

Here is a quick draw of the setup :

Here is an example of outbound NAT rule:

Those rules are repeated for each WAN interface.

Now to the issue that brought me here.

The other day we lost one of our Internet access. I figured I was golden but in fact I wasn't. All user networks couldn't access the Internet anymore as if only the first outbound NAT rule was processed, forcing outbound traffic to a gateway that was down.

I thought (obviously mistakenly) the other rule would be processed when seeing traffic and all would go seemlessly. My sentiment is I f*cked up something somewhere...

What could be wrong in my setup ? How should I set outbound NAT rules so that when one ISP goes down traffic goes to the other CARP as intended ?

Thank you very much in advance