Multi-WAN gateway failover not switching back to tier 1 gw after back online

David127

I have perhaps found my problem in pfSense 2.3.4.

My setup in System > Routing > Gateway Groups:

OpenVPN Client1 = tier1
OpenVPN Client2 = tier2

I have not set 'Default gateway switching' or anything else.

In System > Routing > Gateways I have set:

Monitor IP of OpenVPN Client1 = 8.8.8.8
Monitor IP of OpenVPN Client2 = 8.8.4.4

If I disable OpenVPN Client1, then pfsense switches to OpenVPN Client2 correctly.
But only after I have activated 'Apply Settings' in "System > Routing > Gateways"

If I activate OpenVPN Client1, then pfsense switches back to OpenVPN Client1.
But only after I have activated 'Apply Settings' in "System > Routing > Gateways" again.

Does somebody has any idea, which settings should I make?

lobi

I have the same issue with my failover WAN. If WAN1 goes down=Offline switching to WAN2 works correctly but it don't switch back to WAN1 if its available again. I have to deactivate and activate WAN1 manually.

Anyone have a solution?

I've set multi WAN as the screenshots below:

13:37 WAN1 goes offline
14:00 WAN1 is available again but don't switch back to WAN2

System Logs:

Feb 27 13:37:48 rc.gateway_alarm 91471 >>> Gateway alarm: WAN1GW (Addr:8.8.4.4 Alarm:1 RTT:18713ms RTTsd:4408ms Loss:22%)
Feb 27 13:37:48 check_reload_status updating dyndns WAN1GW
Feb 27 13:37:48 check_reload_status Restarting ipsec tunnels
Feb 27 13:37:48 check_reload_status Restarting OpenVPN tunnels/interfaces
Feb 27 13:37:48 check_reload_status Reloading filter
Feb 27 13:37:50 php-fpm 5341 /rc.dyndns.update: Default gateway down setting WAN2_PPPOE as default!
Feb 27 13:37:50 php-fpm 5341 /rc.dyndns.update: MONITOR: WAN1GW is down, omitting from routing group DualWAN 8.8.4.4|192.168.100.2|WAN1GW|18.747ms|4.479ms|25%|down
Feb 27 13:37:50 php-fpm 5341 /rc.dyndns.update: Default gateway down setting WAN2_PPPOE as default!
Feb 27 13:37:50 php-fpm 65471 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN1GW.
Feb 27 13:37:50 php-fpm 65471 /rc.openvpn: Default gateway down setting WAN2_PPPOE as default!
Feb 27 13:37:50 php-fpm 65471 /rc.filter_configure_sync: Default gateway down setting WAN2_PPPOE as default!
Feb 27 13:40:20 php-fpm 11774 /services_dyndns.php: Default gateway down setting WAN2_PPPOE as default!
Feb 27 13:40:24 check_reload_status Syncing firewall
Feb 27 13:40:24 php-fpm 11697 /services_dyndns.php: Default gateway down setting WAN2_PPPOE as default!
Feb 27 14:11:47 check_reload_status Syncing firewall
Feb 27 14:11:47 check_reload_status Reloading filter
Feb 27 14:11:48 php-fpm 23963 /rc.filter_configure_sync: Default gateway down setting WAN2_PPPOE as default!
Feb 27 14:20:59 check_reload_status Syncing firewall
Feb 27 14:20:59 check_reload_status Reloading filter
Feb 27 14:21:00 php-fpm 34668 /rc.filter_configure_sync: Default gateway down setting WAN2_PPPOE as default!
Feb 27 14:24:10 check_reload_status Syncing firewall
Feb 27 14:24:10 check_reload_status Reloading filter
Feb 27 14:24:11 php-fpm 5833 /rc.filter_configure_sync: Default gateway down setting WAN2_PPPOE as default!

Thanks

markn455

I am having a similar issue. However, I am unable to get it to work simply failing over from WAN1 to WAN2. The logs show that when WAN1 goes down the default it appears things switch. Whoever, traffic does not flow and the pfSense UI hangs (actually becomes very slow). Then Bringing WAN1 back on-line does to resume traffic flow. The quickest way to get things going again is to restart the box. I have configured the dual wan configuration in the simplest way similar to your. I have also tried the suggested configurations that does not use the automatic gateway switching building gateway groups in accordance with the configuration suggestions. I have tried using different hardware and rebuilding pfSense from scratch. The frustrating part for me is I can take a commercial firewall that supports multi-wan and configure things in a similar fashion and it works perfect every time. I apologize for not having a proven solution. You getting it to work this far is great. The only thing different is that many of the multi-wan configuration recommendations is that You have an additional gateway group that handles flipping connections back the other way.

This is one of many configuration examples out there https://www.cyberciti.biz/faq/howto-configure-dual-wan-load-balance-failover-pfsense-router/. I found this one to be helpful as most are. I think in my case it just my limited experience or perhaps I have a glaring simple issue preventing mine from working that I am just missing.

satadru

@markn455 Did you ever find a working configuration for failing back to a primary connection once it comes back up?

matronyx

Goal: Have auto fail-over to 2nd ISP when 1st ISP is down. When 1st ISP comes back, re-enable as primary in routes.
My ISP setup all use static ip config, have not tested with a dynamic interface.

Here is what works for me, various sites, multiple ISP fail-overs.

Step 1:
Navigate to: System - > Advanced -> Miscellaneous
Make sure "Default gateway switching" is UNCHECKED.

Step 2:
Configure your gateway group accordingly. Tier1 is highest priority.
I use Member Down as trigger.

Step 3:
Choose the gateway group in your firewall rules setup.
You find this under Advanced Options for each rule you want to make use of the gateway group.

Simple test if working, plug out Tier1 cable from firewall. Should fail-over to Tier2.
Plug back in Tier1 cable, should become the default route almost instantly.

satadru

@matronyx

Just for clarification:

Default gateway switching is unchecked.
A single gateway group with Tier 1 gateway being highest priority, and Tier 2 being lower priority, and member down is the trigger.
Firewall rules use that gateway group.

And that works for failover? If you pull the cord for gateway 1 it switches to gateway 2? And if you reconnect gateway 1 it switches back?

matronyx

@satadru
Yes that is correct. The switch back and forth between tiers is fully automatic.

nleaudio

Although this is an older thread, I have the same issue happening with the very latest version, as of September, 2019. I have three WAN connections, and one of the gateways I have configured has two of the gateways on it. My PFSense will failover to my Tier 2 connection automatically; but when it comes back up, it will not go back to the Tier 1. I even tried clearing the states - no change. I tried changing the gateway set as Tier 2, and it just routed all the traffic thru that gateway, instead of the Tier 1. All gateways are up, and show as up.

What more can I do to debug this? I did not find the "Default Gateway Switching" option where indicated. Indeed, my "default" gateway is the Tier 1 gateway that seems not to be being used by the Gateway group.

My config is a bit complex, but I'm happy to try to debug this. Just need direction. Thanks.

Bob

gniting

I ended up writing a script and running it via cron to achieve the "switch." Yes, it is not elegant, but it gets the job done.

Here's what I have and I run this as a 5-minute cron job.

#!/bin/sh

# get active gateway and current time
CURRENT_TIME="$(date +"%c")"
CURRENT_GW="$(netstat -rn | grep default | awk '{print $4}')"

if [ $CURRENT_GW = "em2" ]; then
	#check if WAN1 is up or not
	WAN1_STATUS="$(pfSsh.php playback gatewaystatus brief | grep WANGW | awk '{print $2}')"
	if [ $WAN1_STATUS = "none" ]; then
		#WAN1 is back online, stop/start WAN2
		echo "$CURRENT_TIME: Bringing down WAN2"
		ifconfig em2 down
		echo "$CURRENT_TIME: Sleeping for 30s"
		sleep 30
		echo "$CURRENT_TIME: Bringing up WAN2"
		ifconfig em2 up
	else
		echo "$CURRENT_TIME: WAN1 is still down"
	fi
else
	echo "$CURRENT_TIME: Nothing to do!"
fi

C_C

Hey. Thanks @ibbetsion for the script.

Here is a slightly modified version that kills firewall states when there are connections remaining on WAN2 and WAN1 is back online.

Works great for my needs ( LTE failover ).

I set it as a cron, every minute:

*/1 * * * * /root/clear_state_back_from_failover_cron.sh >> /root/clear_state_back_from_failover_cron.log

I also checked "Flush all states when a gateway goes down" in System / Advanced / Miscellaneous.
The LTE gateway has monitoring disabled "Disable Gateway Monitoring" in System / Routing / Gateways. Otherwise states will be created on the interface and the script becomes wrong. Also, monitoring would consume data and I did not want that.

Code:

#!/bin/sh
# *** kills firewall states on failover WAN when WAN1 is up ***

WAN1_NAME="WAN_DHCP"
WAN2_IF=ue0
WAN2_GW_IP=192.168.3.1

CURRENT_TIME="$(date +"%c")"
WAN1_STATUS=`pfSsh.php playback gatewaystatus brief | grep "$WAN1_NAME" | awk '{print $2}'`

if [ "$WAN1_STATUS" = "none" ]; then
	# the following line may need to be tweaked depending on your needs
	WAN2_NSTATES=`pfctl -s state | grep "$WAN2_IF" | grep -v " -> $WAN2_GW_IP" | wc -l`
	if [ "$WAN2_NSTATES" -gt 0 ]; then
		echo "$CURRENT_TIME: WAN1 is online, but connections remain on $WAN2_IF. Killing states."
		pfctl -F state
	fi
else
	echo "$CURRENT_TIME: WAN1 is down"
fi

cmb991

I'm really surprised pfSense has nothing built in to handle this yet. This has been ongoing since 2017. In my case, my LTE modem (unlimited data) is still in gateway monitoring mode, so I'll be using @ibbetsion script. Thanks @ibbetsion

mo10

EDIT2
Issues not fixed. If i pull cable and put it right back in it will mess up Multi-WAN. Will not switch back correctly.

EDIT:
~~Resetting to defaults and setting everything up again seems to have fixed my issues.~~

Old:
I think i found the cause:

Seems Multi WAN is not working properly (or maybe dpinger) if the Interface goes down and back up (unplugging and re plugging).
In my tests i was just unplugging the cable on the WAN-Port.

I think the same happens with PPPOE or anytime the link is down and up again (physically).

This should not happen In my opinion. If modems reboot and so on: MultiWan would stop working.

I use "Paket Loss" as trigger level on Gateway-Group.

Would love to hear from you, thanks.

serbus

Hello!

I have several sites using multi wan and gateway groups with a mixture of static, dhcp, and pppoe. They all behave as expected.

Are you policy routing all of your WAN bound traffic?

"Defining gateway groups is only part of the story. Traffic must be assigned to these gateways using the Gateway setting on firewall rules."

https://docs.netgate.com/pfsense/en/latest/routing/multi-wan.html#firewall-rules

My experience is that you cant depend on the system routing table having your "preferred" (tier1) default route.

John

mo10

@serbus said in Multi-WAN gateway failover not switching back to tier 1 gw after back online:

Hello!

I have several sites using multi wan and gateway groups with a mixture of static, dhcp, and pppoe. They all behave as expected.

Are you policy routing all of your WAN bound traffic?

"Defining gateway groups is only part of the story. Traffic must be assigned to these gateways using the Gateway setting on firewall rules."

https://docs.netgate.com/pfsense/en/latest/routing/multi-wan.html#firewall-rules

My experience is that you cant depend on the system routing table having your "preferred" (tier1) default route.

John

Thank you,

strange is: if i don't unplug a cable on testing and switch off internet without pulling the cable, everything works just as expected. Every time. Soon as i unplug and replug i have to save interface settings for example to get switiching back to default Tier back working.

I checked almost every configuration before and nothing really helped.

serbus

Hello!

Ahhhh, gotcha.

I am having a problems following the thread. It is long and old, and seems to cover different (resolved?) problems. Yours could be yet another issue. Maybe a new thread?

John

nleaudio

As far as I know, this is still problematic! Some PFSense boxes I have on dual-wan setups will switch from tier 1 to tier 2 connections without issue, but going back when the tier 1 is restored does not always work... At least not in the timeframe I would consider usable.

Bob

mo10

@nleaudio said in Multi-WAN gateway failover not switching back to tier 1 gw after back online:

As far as I know, this is still problematic! Some PFSense boxes I have on dual-wan setups will switch from tier 1 to tier 2 connections without issue, but going back when the tier 1 is restored does not always work... At least not in the timeframe I would consider usable.

Bob

i think i had those issues because i imported a configuration to different hardware. Did you do the same?
After i did a reset to defaults and set up everything again it is now switching back fine.

gniting

@mo10 to be clear... in your dual-WAN setup, if WAN1 (default gateway) goes down and pfsense ends up making WAN2 the default, then upon recovery of WAN1, pfsense automatically marks WAN1 as default?

mo10

@ibbetsion

This was never a problem for me. It maked the gateway as default fine but still was sending traffic the wrong way. Saving an interface fixed it until i unplugged (physically) a cable again.
Now after resetting everything everything runs as expected.

Do you have problems with Multi-Wan? What exactly?

gniting

@mo10 my problem is that post recovery, WAN1 never goes back to being default. I have to use a script to bring down WAN2 so that WAN1 becomes default again. Not an ideal solution but it works.