Dual WAN Failover doesn't failover back to WAN 1 [Resolved]

pfrickroll

Does this feature work or don't? Should be a simple set up and work without problems. It doesn't work in any of my firewalls, not a single one. I put a brand new Protectli at my office without any extra configuration but Dual WAN Failover. It switches to WAN 2 when WAN 1 goes down but doesn't switch to WAN 1 from WAN 2 when WAN 1 is up. What is the problem?

serbus

Hello!

I have a simple dual-wan setup and it works without problems.

John

johnpoz

Going to move this to multiwan section. If you want help to why your setup is not working. Some details would be most helpful

noplan

A simple tip to check if there a states alive when the wan1 comes back alive

New states will be on WAN1

pfrickroll

This is a testing firewall no other configuratios besides Failover. WAN 1 Comcast DHCP, WAN 2 (OPT1) Cellular modem DHCP

Raffi_

There are many discussions on this. Here is one example with a script to help kill states on the backup wan when the main wan comes back up.
https://forum.netgate.com/topic/84269/multi-wan-gateway-failover-not-switching-back-to-tier-1-gw-after-back-online/87

Unless you have some other problem, the failover to WAN2 and switch back WAN1 should be occurring. A problem many have seen, including myself is that the states which were on the backup WAN2 connection remain unless they are manually killed or naturally die over time. This could make it appear like the main WAN1 is not being used, but that's not the case. As mentioned by @noplan the new states should be on the primary WAN1 if it's back up and running.

Edit: This doesn't really impact my scenario all that much since we typically have very little traffic. So when an event does occur and the primary WAN is back up, I go to Diagnostic => states and filter for my secondary WAN interface IP. Then I kill all those states to make sure no states are remaining on the link I don't want to be used.

johnpoz

@Raffi_ said in Dual WAN Failover doesn't failover back to WAN 1:

states which were on the backup WAN2 connection remain unless they are manually killed or naturally die over time

Yeah this would be common misconception about not switching back to primary..

pfrickroll

Why they wouldn't be killed automatically, lets say in 2 mins? I just got busy with something else can't check the states. Yesterday when I checked it wouldn't switch even after about an hour.

Raffi_

@pfrickroll said in Dual WAN Failover doesn't failover back to WAN 1:

Why they wouldn't be killed automatically, lets say in 2 mins? I just got busy with something else can't check the states. Yesterday when I checked it wouldn't switch even after about an hour.

Because killing states is typically something you don't want to do unless the client/server connection is truly dead. You should be able to setup an automatic method to kill those states with the script mentioned in that thread. I never tried it, but you can read what other are saying about it there.

Edit: That thread was only one example. If that doesn't solve the issue some searching will show other threads on the topic, possibly with other scripts if I recall.

johnpoz

@pfrickroll said in Dual WAN Failover doesn't failover back to WAN 1:

Yesterday when I checked it wouldn't switch even after about an hour.

Did you check that new states where using the wan 1? A State is really not going to die unless traffic stops for a long time, or the session is ended by the server or client with fin or rst, etc.

So if you were checking via say a browser or something and what your IP was like going to whatsmyip . com or something.. That state would be still using the wan 2, and traffic would continue to route out that connection.

You would need to make sure you shutdown any existing states using wan 2, or make sure you bring up a new session to validate which wan path you were talking.

pfrickroll

So, in real setup in small offices where i have pfsense. I got static IP WAN 1 and DHCP WAN 2.
I have IPsec, and IP Phones service that we pay for. When WAN 1 comes up, I would like Failover to switch to it lets say in 5 minutes. So that IPsec would up and running as fast as possible. I am not script savvy.

netblues

@pfrickroll If phones or vpn keep sending keepalives, these connections will only switch to wan1 either by manually killing them, or wan2 goes down.
This is a wanted feature. Connections should be not dropped while exchanging data.

Raffi_

@pfrickroll said in Dual WAN Failover doesn't failover back to WAN 1:

So, in real setup in small offices where i have pfsense. I got static IP WAN 1 and DHCP WAN 2.
I have IPsec, and IP Phones service that we pay for. When WAN 1 comes up, I would like Failover to switch to it lets say in 5 minutes. I am not script savvy.

Unfortunately, there is no GUI method to do what you're asking for automatically. I wish there was too. You don't have to be script savvy. Did you even look at the thread? They spell everything out for you. I'm not script savvy at all but I got someone else's script running on pfSense (different script) but it's the same idea. If I can do it, anyone can do it. I believe in you :)

johnpoz

Well in the default normal optimization for states.. Once a state is established it will stay open, even without any traffic for 24 hours

https://pfsense-docs.readthedocs.io/en/latest/config/advanced-setup.html

So unless the server/client involved in the conversation close the session/state with fin, or rst the state will stay open.. You could adjust the timeouts for established, but even in the aggressive mode your still looking at 5 hours.. For an established state without any traffic - and that counter would restart every time there is any traffic on that session.

If you want all traffic to switch back to wan 1 after it comes up by forcing it - you would really need to clear the states for anything using wan2

pfrickroll

@Raffi_ I skimmed through it but didnt find script there. I am just doing few things at a time at the moment.

johnpoz

That link he provided took you right to the post with the script

#!/bin/sh

# get active gateway and current time
CURRENT_TIME="$(date +"%c")"
CURRENT_GW="$(netstat -rn | grep default | awk '{print $4}')"

if [ $CURRENT_GW = "em2" ]; then
	#check if WAN1 is up or not
	WAN1_STATUS="$(pfSsh.php playback gatewaystatus brief | grep WANGW | awk '{print $2}')"
	if [ $WAN1_STATUS = "none" ]; then
		#WAN1 is back online, stop/start WAN2
		echo "$CURRENT_TIME: Bringing down WAN2"
		ifconfig em2 down
		echo "$CURRENT_TIME: Sleeping for 30s"
		sleep 30
		echo "$CURRENT_TIME: Bringing up WAN2"
		ifconfig em2 up
	else
		echo "$CURRENT_TIME: WAN1 is still down"
	fi
else
	echo "$CURRENT_TIME: Nothing to do!"
fi

And just below post was another with the cron info and slightly modified script :) so not sure what link you followed?

serbus

Hello!

There is also a built-in script /etc/rc.kill_states that can be modified.

https://forum.netgate.com/topic/135614/failback-from-primary-wan-after-failover-to-secondary-wan

John

Raffi_

@serbus said in Dual WAN Failover doesn't failover back to WAN 1:

Hello!

There is also a built-in script /etc/rc.kill_states that can be modified.

https://forum.netgate.com/topic/135614/failback-from-primary-wan-after-failover-to-secondary-wan

John

@serbus thank you. I knew I remember reading about another script/method to get the job done. Now that I'm looking at these again, I might take a little time to give one a try. The built-in script seems to make more sense since it's already there.

@pfrickroll said in Dual WAN Failover doesn't failover back to WAN 1:

I am just doing few things at a time at the moment.

Understood. You mentioned this being a new setup which you were beginning to test. This would be the best time to try any of these options out, especially if you haven't put this network into production yet. That is the most ideal scenario for testing anything you're not familiar with.

serbus

Hello!

There is a built-in system for packages that allows custom plugin code to be called on certain events.

It is in /etc/inc/pfsense-utils.inc in function pkg_call_plugins

It looks like several packages use the callback plugin (carp, certs). The gateway system also uses the callback in /etc/inc/gwlb.inc when there is a state change.

It is calling for a package that has setup a plugin called "plugin_gateway". I dont see any official packages that have a plugin named that, but it could be the one referenced in this post:

https://forum.netgate.com/topic/139455/list-of-hooks

https://github.com/jazzl0ver/pfSense-pkg-gatewayhook

I dont know why they would modify the official gwlb.inc code to call a plugin for a package that is not part of the official release...

John

Raffi_

FYI, I ended up using the first script in the original thread I linked (take wan2 down and back up when wan1 is back up). I didn't do that because it was a better solution, but I found it easier to modify. I only had to change the defined interface for WAN2.
https://forum.netgate.com/topic/84269/multi-wan-gateway-failover-not-switching-back-to-tier-1-gw-after-back-online/67?_=1601399952603

The second script below it seems fundamentally better (killing states), but I have a DHCP wan2 and didn't want to use that since I wasn't sure how to modify it for my scenario.

I haven't tested it yet since it's not big deal for me whether this works or not. I'm going to wait for a real event and see what happens.