Multi-wan failover only triggering in certain situations.

[Ximerian]

If I pull the power on my primary WAN connections modem the pfSense doesn't send traffic out the backup WAN interface until a link is back on the primary WAN connection ( For example, I plug the modem back into power and the link light comes back on, but the modem is still booting so the connection out that gateway is still down for a few minutes. The traffic will then fail over for a few minutes until the modem on the primary gateway finishes it's boot process and then fails back) when this happens the notifications get generated that the primary is down, but traffic doesn't go out the backup WAN automatically. If the pfsense is power cycled the traffic will go out the backup WAN connection after the pfsense boots back up..

My test was a constant ping to a public IP that is not one of the one monitor IP addresses. Also I just tried to do dns lookups from command prompt and browse websites. Primarily though I was just using the constant ping as my test.

If I remove the fiber from the modem everything fails over properly and fails back properly.

I think the issue may be with my floating rule, because no matter how I tried to set it up nothing worked if I had one setup. I see mention of how to create the floating rule but never what to specify for which interfaces/direction. So I've tried what made sense to me, but if the floating rule is the answer please let me know which interfaces/direction to select.

I have my outbound LAN rule setup to use the gateway group. I do get an error in the notifications I'm not sure if it's relevant or not, it can be seen in the notification below.

I've tried using different monitor IPs for the primary gateway, it doesn't change the behavior if I leave it as the ISP upstream gateway or the google DNS as seen below. 1.1.1.1 is the IP for primary and 2.2.2.2 is the backup WAN IP

This is the notification when the network cable is unplugged, what's interesting is that it still does update dyndns but client traffic isn't routed out the backup gateway. I did mess with some of the advanced/misc settings under system but at this point I couldn't tell you exactly when I had what on/off. I did have it set to automatically flush states on an gateway failure, but that is currently off. I also tried the default gateway switching option, but that is also now off.

23:06:35 MONITOR: FIOSGW is down, omitting from routing group WANFAILOVER 
8.8.8.8|1.1.1.1|FIOSGW|17.919ms|0.347ms|23%|down 
23:06:36 There were error(s) loading the rules: /tmp/rules.debug:47: errors in queue definition - The line in question reads [47]: queue qACK on igb0 priority 6 priq ( ecn ) 

23:06:37 There were error(s) loading the rules: /tmp/rules.debug:47: errors in queue definition - The line in question reads [47]: queue qACK on igb0 priority 6 priq ( ecn ) 

23:06:37 DynDNS updated IP Address on OPT1 (igb2) to 2.2.2.2

This is when the fiber is pulled out the back of the primary modem.

23:03:43 There were error(s) loading the rules: /tmp/rules.debug:47: errors in queue definition - The line in question reads [47]: queue qACK on igb0 priority 6 priq ( ecn ) 

23:03:52 MONITOR: FIOSGW is down, omitting from routing group WANFAILOVER 
8.8.8.8|1.1.1.1|FIOSGW|17.849ms|0.349ms|24%|down

The DynDNS notification went out as a separate notification when the fiber was unplugged from the modem. Not sure if that is a relevant fact but thought I would include it.

23:03:54 DynDNS updated IP Address on OPT1 (igb2) to 2.2.2.2

I've tried 'member down' in addition to the 'packet loss' show below for trigger level.

I have tried disabling this floating rule below. It was generated by the traffic shaping wizard for voip.

I would like the internal resources be available even when on the backup internet connection, so I have rules setup for both interfaces, I imagine this would be cleaner accomplished with floating rules also?

If any other screenshots are needed please let me know, and thank you for any assistance.