Multiwan setup is constantly swapping my gateways

[arengifo]

Hello guys:

I'm running pfSense 2.3.1 in a Multi WAN scenario with 2 ISPs. I can barely describe my pfsense configuration:

• GW_WAN: Gateway for ISP1 (this is my default gateway)
• GW_WAN2: Gateway for ISP2 (backup)
• GW_WAN is using 8.8.8.8 as monitoring IP, while GW_WAN2 is using 8.8.4.4 as monitoring IP.
• Both gateways belong to a gateway group called "GW_HA" in which GW_WAN is Tier1 and GW_WAN2 is Tier2 (Trigger level: member down)
• I have the "Enable default gateway switching" feature enabled.
• I have the "State killing on gateway failure" feature disabled.

Some days ago when both ISPs were working fine I decided to make a simple and short test. So I forced my ISP1 (through GW_WAN1) to fail and I saw how pfSense automatically switched the default gateway to ISP2 (through GW_WAN2) by using "netstat -nr | grep default". I thought this worked successfully as expected so I felt happy :). Also, all my firewall rules (previously configured to use GW_HA as gateway) worked fine too which allowed my clients to reach the Internet without issues (i.e. POP3/IMAP or SMTP connections continued working fine).
IMPORTANT: This test was really fast. It took me no more than 1 or 2 minutes at most.

But just yesterday my ISP1 had a real failure scenario (someone forgot to pay the bill, duh!) and I noticed that…

• My firewall rules with GW_HA as gateway continued working fine, so Mail (IMAP, POP, SMTP) and other traffic switched fine through ISP2. This worked fine as expected.
• My pfSense box started to swap the default gateway every 3 minutes. Most of the time I was able to see that even when ISP1 was down (according to Status->Gateways), the default gateway was yet pointing to GW_WAN instead of GW_WAN2.

I've found these suspicious logs:

Aug 12 08:11:36 fw check_reload_status: updating dyndns GW_WAN
Aug 12 08:11:36 fw check_reload_status: Restarting ipsec tunnels
Aug 12 08:11:36 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
Aug 12 08:11:36 fw check_reload_status: Reloading filter
Aug 12 08:11:37 fw php-fpm[2693]: /rc.dyndns.update: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
Aug 12 08:11:37 fw php-fpm[2693]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use GW_WAN.2 08:11:38 fw php-fpm[2693]: /rc.filter_configure_sync: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
Aug 12 08:11:38 fw xinetd[14577]: Starting reconfiguration
Aug 12 08:11:38 fw xinetd[14577]: Swapping defaults
Aug 12 08:11:38 fw xinetd[14577]: readjusting service 6969-udp
Aug 12 08:11:38 fw xinetd[14577]: Reconfigured: new=0 old=1 dropped=0 (services)
Aug 12 08:12:06 fw check_reload_status: updating dyndns GW_WAN
Aug 12 08:12:06 fw check_reload_status: Restarting ipsec tunnels
Aug 12 08:12:06 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
Aug 12 08:12:06 fw check_reload_status: Reloading filter
Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: MONITOR: GW_WAN is down, omitting from routing group GW_HA
Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: Gateways status could not be determined, considering all as up/active. (Group: GW_HA)
Aug 12 08:12:08 fw php-fpm[37361]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel CLOGmined, considering all as up/active. (Group: GW_HA)

The "swapping defaults" and "Gateways status could not be determined, considering all as up/active" messages confused me and made think pfSense wasn't working fine on the "Default gateway switching" feature.

I'd like someone can let me know if I'm missing an important configuration to fix this or it's maybe a bug.

Thanks in advance for your time.

[arengifo]

nobody? any ideas? am I facing a bug, maybe?

[jimp]

Check System > General Setup. make sure you don't have 8.8.8.8 / 8.8.4.4 there set to the opposite WANs as the monitor IP addresses. They have to match in both places.

[arengifo]

Sorry for the long delay. I'm not sure I got your point… Do you mean I should use the same address (e.g. 8.8.8.8) as monitoring IP for both gateways? I thought this would create a routing issue due to have the same route using different gateways... how is it supposed to monitor Internet availability if pfSense doesn't know which gateway to use for reaching 8.8.8.8?

Thanks in advance.

[jimp]

No I mean just the opposite: Make sure you don't have anything that might be causing the same IP address to be used as a monitor for both WANs, which can confuse routing.

[arengifo]

Oh thanks. That's exactly what I did: I used different monitoring IPs for both gateways.