PfSense - Auto reboot script when google is unreachable..

kilko · Jan 15, 2014, 2:30 PM

Senario:
My ISP suddenly drops the connection, even though all lights are "green" on the cable modem and it should be working..
Only solution I've found is to reboot the pfsense..
Read my other topic about this here: http://forum.pfsense.org/index.php/topic,69879.msg381954.html#msg381954

I found another one that had same problems here, with a more advanced script: http://volker.top.geek.nz/soft/script/pfsense-ifc-check)

My 10 steps - Howto:

1. Login to pfsense with ssh, select "8" for shell command

2. Go to: /usr/local/bin

3. To remount file systems as read-write, run: /etc/rc.conf_mount_rw

4. Create file: ping-check.sh (to create file, simple howto: vi ping-check.sh, then carefully click "i" and paste the code, click "esc", type ":wq!" - all in that order! )

Add this to file;


#!/bin/sh

# HOSTS can be either you ISP or google.com
HOSTS="google.com"
COUNT=2

echo "Pinging.."
echo "HOSTS: " $HOSTS
echo "COUNT: " $COUNT
######
for myHost in $HOSTS
do
  counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }' )
  echo "counting: " $counting

  if [ $counting -eq 2 ]; then
   echo "Ping OK"

  else
   # network down
   # Save RRD data
   /etc/rc.backup_rrd.sh
   #Reboot
   echo "Reboot!"
   reboot
fi
done

5. chmod 700 ping-check.sh

6. To mount as read-only again, run: /etc/rc.conf_mount_ro

7. exit

Now you need to add a cron job to automatically run this every 5 minutes..

8. Go into pfSense web interface - and select:

Packages (under System)
Cron (0.1.8 is what I found when writing this)
Select "+" and install Cron.

9. Then go into Cron (under Services)

10. Click "+" and add

minute: 5
hours: *
mday: *
month: *
wday: *
(who): root
command: /usr/local/bin/ping-check.sh

Click "Save"

Thats it!
Now the system will check if the pfSense box is able to ping every 5 minutes the host in the script, if not - it will reboot.
Testet on my 2.1-RELEASE (i386) and works perfectly well.

if I can just get curl into pfsense also, the pfsense box will be able to issue the command for rebooting the cable modem too.. but that is for later or next project ;)

Enjoy :-)

kilko · Jan 22, 2014, 7:48 PM

It there is heavy traffic, ping traffic may not get through.. therefore I simply modified script abit,
it now pings 10 times.. and if 2 or more pings are received okey - network is considered up and running:


#!/bin/bash
#
# put -xv after bash to debug
#
HOSTS="google.com"
COUNT=10

#debug
echo "HOSTS: " $HOSTS
echo "COUNT: " $COUNT
######
for myHost in $HOSTS
do
  counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

  #debug
  echo "counting: " $counting
  ######
  if [ $counting > 2 ]; then
   echo "Ping OK"

  else
   # network down
   # Save RRD data
   /etc/rc.backup_rrd.sh
   #Reboot
   echo "Reboot!"
   reboot
fi
done

TDJ211 · Jan 23, 2016, 4:08 AM

Nice, thanks for the tutorial!

TDJ211 · Jan 23, 2016, 8:25 PM

Question: Is there a way to track reboots other than the Uptime to make sure the script is working correctly? I dont want it going too crazy.

kilko · Jan 25, 2016, 1:06 PM

Nice to hear that is useful ;-)

Track reboot - you want to be notified when the system reboots ? if so, one would need to create two things;

1. when the ping check is run, if it fail we have to create a file with date/timestamp before it reboots.

2. create a startup scripts that checks if the file exists and mails this file to a gmail account. After mail successful, delete det file.

done.

I have no time to do this myself (now at least). I did not see the point of knowing when the reboot is done..
I just needed the system check if its online, and if not - try to get back online on its own.

Of course this also applies to the modem you have connected. And rebooting the modem would in most cases help re-connecting the devices. (refreshing IP/MAC/DNS from the ISP)
Over the years I have seen more problems with the modem, than pfSense box. The modem might go down…but pfSense is up. And when I reboot the modem and it gets back online.. while the pfSense box is still untouched, system is back working again.
Very rare that I need to reboot pfSense, but it happens.. when my IPS changes the IP/MAC locking againts my modem i think.

whitexp · Jan 26, 2016, 10:48 AM

Very good

Teddy · Dec 19, 2019, 11:58 AM

Hey everyone,
that script is still working properly. But how can i choose, by which interface the ping should be done?

I am having the normal WAN Interface and a VPN interface. If the VPN Connection is lost, PFSense doesn't realize that and is not resetting the connection / interface. So it would be an easy solution, to choose that i want to ping google.com by the VPN Interface.

I am really bad in writing scripts / programming, so does anyone have a solution, like "Ping google.com by interface ovpnc1" or whatever (in a proper programmed way, that commandline is obviously wrong)?

Brgds

Gertjan · Dec 19, 2019, 12:23 PM

@Teddy said in PfSense - Auto reboot script when google is unreachable..:

But how can i choose, by which interface the ping should be done

"ping" has many options.

Choose yours : man ping FreeBSD

Teddy · Dec 19, 2019, 12:48 PM

@Gertjan said in PfSense - Auto reboot script when google is unreachable..:

@Teddy said in PfSense - Auto reboot script when google is unreachable..:

But how can i choose, by which interface the ping should be done

"ping" has many options.

Choose yours : man ping FreeBSD

Thanks for that list. I suggest it is the -I option? I did now several tests, but all the time the ping fails with "ping -I OPT1".

Here is my complete script, that i am using (from another topic, which is closed):

#!/bin/sh

#=====================================================================
# pingtest.sh, v1.0.1
# Created 2009 by Bennett Lee
# Released to public domain
# https://forum.netgate.com/topic/16217/howto-ping-hosts-and-reset-reboot-on-failure/2
# (1) Attempts to ping several hosts to test connectivity.  After
#     first successful ping, script exits.
# (2) If all pings fail, resets interface and retries all pings.
# (3) If all pings fail again after reset, then reboots pfSense.
#
# History
# 1.0.1   Added delay to ensure interface resets (thx ktims).
# 1.0.0   Initial release.
#=====================================================================

#=====================================================================
# USER SETTINGS
#
# Set multiple ping targets separated by space.  Include numeric IPs
# (e.g., remote office, ISP gateway, etc.) for DNS issues which
# reboot will not correct.
ALLDEST="google.com yahoo.com 24.93.40.36 8.8.8.8"
# Interface to reset, usually your WAN
BOUNCE=OPT1

# Log file
LOGFILE=/root/pingtest.log
#=====================================================================

COUNT=1
while [ $COUNT -le 2 ]
do

	for DEST in $ALLDEST
	do
		echo `date +%Y%m%d.%H%M%S` "Pinging $DEST" >> $LOGFILE
		ping -c1 $DEST >/dev/null 2>/dev/null
		if [ $? -eq 0 ]
		then
			echo `date +%Y%m%d.%H%M%S` "Ping $DEST OK." >> $LOGFILE
			exit 0
		fi
	done

	if [ $COUNT -le 1 ]
	then
		echo `date +%Y%m%d.%H%M%S` "All pings failed. Resetting interface $BOUNCE." >> $LOGFILE
		/sbin/ifconfig $BOUNCE down
		# Give interface time to reset before bringing back up
		sleep 10
		/sbin/ifconfig $BOUNCE up
		# Give WAN time to establish connection
		sleep 60
	else
		echo `date +%Y%m%d.%H%M%S` "All pings failed twice. Rebooting..." >> $LOGFILE
		/sbin/shutdown -r now >> $LOGFILE
		exit 1
	fi

	COUNT=`expr $COUNT + 1`
done

I editet in line 34 "ping -c1 $DEST >/dev/null 2>/dev/null" to "ping -I OPT1 -c1 $DEST >/dev/null 2>/dev/null"

Gertjan · Dec 19, 2019, 1:45 PM

More like

ping -S a.b.c.d google.com

where a.b.c.d is the network address of your, for example, WAN.

Teddy · Dec 20, 2019, 2:06 AM

@Gertjan said in PfSense - Auto reboot script when google is unreachable..:

More like
ping -S a.b.c.d google.com
where a.b.c.d is the network address of your, for example, WAN.

Is there any way, to use the interface? Because my VPN IP is always dynamic.

I obviously have the WAN (but that is, because PFSense is connected to another Router) an internal IP (192.168.178.X).
And i have the LAN with 192.168.1.1, the firewall is blocking all traffic from LAN, if the VPN Connection get's lost.

Now, with your advice, i set "ping -S 192.168.1.1" (because: If the connection on VPN is lost, no traffic can pass due to a firewall-rule the LAN (192.168.1.1)...But anyhow the hosts (google.com, Yahoo etc.) can be pinged -> No restart of VPN Interface, but anyway no connection available (Chrome, Firefox show just, that there is no connection to the internet).

A ping in PFSense to all addresses, using every interface, is successfull. Weird behaviour, from which i can't find the problem now.

Shortly:
Ping google.com by LAN (192.168.1.1) is successful. But i have no access to the Internet. So, the VPN Interface must be reset to work properly again. Any ideas?

Gertjan · Dec 20, 2019, 11:33 AM

@Teddy said in PfSense - Auto reboot script when google is unreachable..:

Is there any way, to use the interface? Because my VPN IP is always dynamic.

Yep, easy.
You're scripting, right ?

Have a look at the diag_ping.php page / Diagnostics > Ping.
We can select an interface there, and the PHP gets the IP address.

@Teddy said in PfSense - Auto reboot script when google is unreachable..:

Any ideas?

Use the VPN interface ?

Teddy · Dec 20, 2019, 12:32 PM

@Gertjan
Yes, i am using a script from another user here in the board! ;)
And i can't really follow. That diag_ping.php i am using always for check, if a ping is possible. But how to connect it with my above mentioned script?

I need an IP-address, to add it in the script, right? But the VPN interface IP is not static, it is dynamic.
First it was 10.247.202.214, now after the reboot it is for example 10.246.201.107

Brig · Feb 20, 2021, 12:50 PM

I realise this is an old topic, but should this still work?

Teddy · Feb 20, 2021, 3:52 PM

@brig

Old but gold!
Still works really properly under 2.4.5!

Brig · Feb 20, 2021, 3:52 PM

@teddy Not working for me on 2.5 sadly

I have added one of my wan IPs in line

counting=$(ping -s M.Y.I.P -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

is that correct?

unfortunately I've no idea how to troubleshoot as I'm coming from the switching side of networking, so unless something just works when it comes to things like this it's a steep learning curve ha.

Teddy · Feb 20, 2021, 4:28 PM

@brig

Edit: My fault, I am using another script, here it is:
https://forum.netgate.com/topic/16217/howto-ping-hosts-and-reset-reboot-on-failure

This works for me, maybe try that?
It is pinging several times -> On success nothing happens
No succes -> Reboot interface
Still no success -> Reboot whole PFSense

Can end in a bootloop, if you really have no connection for several hours, but fixes itself, as soon as it has connection again and can ping.

Here is my config:

#!/bin/sh

#=====================================================================
# pingtest.sh, v1.0.1
# Created 2009 by Bennett Lee
# Released to public domain
# https://forum.netgate.com/topic/16217/howto-ping-hosts-and-reset-reboot-on-failure/2
# (1) Attempts to ping several hosts to test connectivity.  After
#     first successful ping, script exits.
# (2) If all pings fail, resets interface and retries all pings.
# (3) If all pings fail again after reset, then reboots pfSense.
#
# History
# 1.0.1   Added delay to ensure interface resets (thx ktims).
# 1.0.0   Initial release.
#=====================================================================

#=====================================================================
# USER SETTINGS
#
# Set multiple ping targets separated by space.  Include numeric IPs
# (e.g., remote office, ISP gateway, etc.) for DNS issues which
# reboot will not correct.
ALLDEST="google.com yahoo.com 24.93.40.36 8.8.8.8"
# Interface to reset, usually your WAN
BOUNCE=ovpnc1
BOUNCE=ovpnc3
BOUNCE=vmx3

# Log file
LOGFILE=/root/pingtest.log
#=====================================================================

COUNT=1
while [ $COUNT -le 2 ]
do

	for DEST in $ALLDEST
	do
		#echo `date +%Y%m%d.%H%M%S` "Pinging $DEST" >> $LOGFILE
		ping -c1 $DEST >/dev/null 2>/dev/null
		if [ $? -eq 0 ]
		then
			#echo `date +%Y%m%d.%H%M%S` "Ping $DEST OK." >> $LOGFILE
			exit 0
		fi
	done

	if [ $COUNT -le 1 ]
	then
		echo `date +%Y%m%d.%H%M%S` "All pings failed. Resetting interface $BOUNCE." >> $LOGFILE
		/sbin/ifconfig $BOUNCE down
		# Give interface time to reset before bringing back up
		sleep 10
		/sbin/ifconfig $BOUNCE up
		# Give WAN time to establish connection
		sleep 60
	else
		echo `date +%Y%m%d.%H%M%S` "All pings failed twice. Rebooting..." >> $LOGFILE
		/sbin/shutdown -r now >> $LOGFILE
		exit 1
	fi

	COUNT=`expr $COUNT + 1`
done

Better post your whole config-file you are using for it. And the names of your interfaces (Interfaces -> Assignments -> WAN vmx1 or onvp1 whatever...They have a special name, also depending on the used hardware.

In general, the only lines you have to edit are in this part:

#=====================================================================
# USER SETTINGS
#
# Set multiple ping targets separated by space.  Include numeric IPs
# (e.g., remote office, ISP gateway, etc.) for DNS issues which
# reboot will not correct.
ALLDEST="google.com yahoo.com 24.93.40.36 8.8.8.8"
# Interface to reset, usually your WAN
BOUNCE=ovpnc1
BOUNCE=ovpnc3
BOUNCE=vmx3

# Log file
LOGFILE=/root/pingtest.log
#=====================================================================

Alldest you can use google.com 8.8.8.8, all public IP addresses, which have a nearly 100% Uptime.
Behind bounce you need to put the interfaces name (which I mentioned above...so NOT WAN, you need to use the "technical" name, like int1 or whatever it is for your hardware.

If it still doesn't work, post your whole script-code you used for it and then we can check, what's going on.

Brig · Feb 20, 2021, 7:29 PM

@teddy The above works a treat! Thanks a lot for taking the time to reply.
For whatever reason my router drops the wan connection every couple of weeks, always whenever I'm out so can't manually reboot. This will solve that issue so thanks again.

Teddy · Feb 20, 2021, 11:34 PM

@brig

You're welcome! Works really properly.

Just don't forget the Cron-Job, switch off the test-log (otherwise your disk will run full) and when you ever have to restore from your config-file, you need to set it up again!

That script is not included in the backup-file! ;)

Enjoy it, the curve is steep, but PFSense is great!

linuxstyl · Jun 11, 2024, 5:45 AM

@Teddy thanks that's what I was looking for and it works in version 2.7.2