PfSense - Auto reboot script when google is unreachable..



  • Senario:
    My ISP suddenly drops the connection, even though all lights are "green" on the cable modem and it should be working..
    Only solution I've found is to reboot the pfsense..
    Read my other topic about this here: http://forum.pfsense.org/index.php/topic,69879.msg381954.html#msg381954

    My 10 steps - Howto:

    1. Login to pfsense with ssh, select "8" for shell command

    2. Go to: /usr/local/bin

    3. To remount file systems as read-write, run: /etc/rc.conf_mount_rw

    4. Create file: ping-check.sh  (to create file, simple howto: vi ping-check.sh, then carefully click "i" and paste the code, click "esc", type ":wq!" - all in that order! )

    Add this to file;

    
    #!/bin/sh
    
    # HOSTS can be either you ISP or google.com
    HOSTS="google.com"
    COUNT=2
    
    echo "Pinging.."
    echo "HOSTS: " $HOSTS
    echo "COUNT: " $COUNT
    ######
    for myHost in $HOSTS
    do
      counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }' )
      echo "counting: " $counting
    
      if [ $counting -eq 2 ]; then
       echo "Ping OK"
    
      else
       # network down
       # Save RRD data
       /etc/rc.backup_rrd.sh
       #Reboot
       echo "Reboot!"
       reboot
    fi
    done
    
    

    5. chmod 700 ping-check.sh

    6. To mount as read-only again, run: /etc/rc.conf_mount_ro

    7. exit

    Now you need to add a cron job to automatically run this every 5 minutes..

    8. Go into pfSense web interface - and select:

    • Packages (under System)
    • Cron (0.1.8 is what I found when writing this)
    • Select "+" and install Cron.

    9. Then go into Cron (under Services)

    10. Click "+" and add

    minute:  5
    hours:    *
    mday:    *
    month:  *
    wday:    *
    (who):  root
    command:  /usr/local/bin/ping-check.sh

    Click "Save"

    Thats it!
    Now the system will check if the pfSense box is able to ping every 5 minutes the host in the script, if not - it will reboot.
    Testet on my 2.1-RELEASE  (i386) and works perfectly well.

    if I can just get curl into pfsense also, the pfsense box will be able to issue the command for rebooting the cable modem too..  but that is for later or next project ;)

    Enjoy :-)



  • It there is heavy traffic, ping traffic may not get through.. therefore I simply modified script abit,
    it now pings 10 times.. and if 2 or more pings are received okey - network is considered up and running:

    
    #!/bin/bash
    #
    # put -xv after bash to debug
    #
    HOSTS="google.com"
    COUNT=10
    
    #debug
    echo "HOSTS: " $HOSTS
    echo "COUNT: " $COUNT
    ######
    for myHost in $HOSTS
    do
      counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
    
      #debug
      echo "counting: " $counting
      ######
      if [ $counting > 2 ]; then
       echo "Ping OK"
    
      else
       # network down
       # Save RRD data
       /etc/rc.backup_rrd.sh
       #Reboot
       echo "Reboot!"
       reboot
    fi
    done
    
    


  • Nice, thanks for the tutorial!



  • Question: Is there a way to track reboots other than the Uptime to make sure the script is working correctly? I dont want it going too crazy.



  • Nice to hear that is useful ;-)

    Track reboot - you want to be notified when the system reboots ? if so, one would need to create two things;

    1. when the ping check is run, if it fail we have to create a file with date/timestamp before it reboots.

    2. create a startup scripts that checks if the file exists and mails this file to a gmail account. After mail successful, delete det file.

    done.

    I have no  time to do this myself (now at least). I did not see the point of knowing when the reboot is done..
    I just needed the system check if its online, and if not - try to get back online on its own.

    Of course this also applies to the modem you have connected. And rebooting the modem would in most cases help re-connecting the devices. (refreshing IP/MAC/DNS from the ISP)
    Over the years I have seen more problems with the modem, than pfSense box. The modem might go down…but pfSense is up. And when I reboot the modem and it gets back online.. while the pfSense box is still untouched, system is back working again.
    Very rare that I need to reboot pfSense, but it happens.. when my IPS changes the IP/MAC locking againts my modem i think.



  • Very good



  • Hey everyone,
    that script is still working properly. But how can i choose, by which interface the ping should be done?

    I am having the normal WAN Interface and a VPN interface. If the VPN Connection is lost, PFSense doesn't realize that and is not resetting the connection / interface. So it would be an easy solution, to choose that i want to ping google.com by the VPN Interface.

    I am really bad in writing scripts / programming, so does anyone have a solution, like "Ping google.com by interface ovpnc1" or whatever (in a proper programmed way, that commandline is obviously wrong)?

    Brgds



  • @Teddy said in PfSense - Auto reboot script when google is unreachable..:

    But how can i choose, by which interface the ping should be done

    "ping" has many options.

    Choose yours : man ping FreeBSD



  • @Gertjan said in PfSense - Auto reboot script when google is unreachable..:

    @Teddy said in PfSense - Auto reboot script when google is unreachable..:

    But how can i choose, by which interface the ping should be done

    "ping" has many options.

    Choose yours : man ping FreeBSD

    Thanks for that list. I suggest it is the -I option? I did now several tests, but all the time the ping fails with "ping -I OPT1".

    Here is my complete script, that i am using (from another topic, which is closed):

    #!/bin/sh
    
    #=====================================================================
    # pingtest.sh, v1.0.1
    # Created 2009 by Bennett Lee
    # Released to public domain
    # https://forum.netgate.com/topic/16217/howto-ping-hosts-and-reset-reboot-on-failure/2
    # (1) Attempts to ping several hosts to test connectivity.  After
    #     first successful ping, script exits.
    # (2) If all pings fail, resets interface and retries all pings.
    # (3) If all pings fail again after reset, then reboots pfSense.
    #
    # History
    # 1.0.1   Added delay to ensure interface resets (thx ktims).
    # 1.0.0   Initial release.
    #=====================================================================
    
    #=====================================================================
    # USER SETTINGS
    #
    # Set multiple ping targets separated by space.  Include numeric IPs
    # (e.g., remote office, ISP gateway, etc.) for DNS issues which
    # reboot will not correct.
    ALLDEST="google.com yahoo.com 24.93.40.36 8.8.8.8"
    # Interface to reset, usually your WAN
    BOUNCE=OPT1
    
    # Log file
    LOGFILE=/root/pingtest.log
    #=====================================================================
    
    COUNT=1
    while [ $COUNT -le 2 ]
    do
    
    	for DEST in $ALLDEST
    	do
    		echo `date +%Y%m%d.%H%M%S` "Pinging $DEST" >> $LOGFILE
    		ping -c1 $DEST >/dev/null 2>/dev/null
    		if [ $? -eq 0 ]
    		then
    			echo `date +%Y%m%d.%H%M%S` "Ping $DEST OK." >> $LOGFILE
    			exit 0
    		fi
    	done
    
    	if [ $COUNT -le 1 ]
    	then
    		echo `date +%Y%m%d.%H%M%S` "All pings failed. Resetting interface $BOUNCE." >> $LOGFILE
    		/sbin/ifconfig $BOUNCE down
    		# Give interface time to reset before bringing back up
    		sleep 10
    		/sbin/ifconfig $BOUNCE up
    		# Give WAN time to establish connection
    		sleep 60
    	else
    		echo `date +%Y%m%d.%H%M%S` "All pings failed twice. Rebooting..." >> $LOGFILE
    		/sbin/shutdown -r now >> $LOGFILE
    		exit 1
    	fi
    
    	COUNT=`expr $COUNT + 1`
    done
    

    I editet in line 34 "ping -c1 $DEST >/dev/null 2>/dev/null" to "ping -I OPT1 -c1 $DEST >/dev/null 2>/dev/null"



  • More like

    ping -S a.b.c.d google.com
    

    where a.b.c.d is the network address of your, for example, WAN.



  • @Gertjan said in PfSense - Auto reboot script when google is unreachable..:

    More like

    ping -S a.b.c.d google.com
    

    where a.b.c.d is the network address of your, for example, WAN.

    Is there any way, to use the interface? Because my VPN IP is always dynamic.

    I obviously have the WAN (but that is, because PFSense is connected to another Router) an internal IP (192.168.178.X).
    And i have the LAN with 192.168.1.1, the firewall is blocking all traffic from LAN, if the VPN Connection get's lost.

    Now, with your advice, i set "ping -S 192.168.1.1" (because: If the connection on VPN is lost, no traffic can pass due to a firewall-rule the LAN (192.168.1.1)...But anyhow the hosts (google.com, Yahoo etc.) can be pinged -> No restart of VPN Interface, but anyway no connection available (Chrome, Firefox show just, that there is no connection to the internet).

    A ping in PFSense to all addresses, using every interface, is successfull. Weird behaviour, from which i can't find the problem now.

    Shortly:
    Ping google.com by LAN (192.168.1.1) is successful. But i have no access to the Internet. So, the VPN Interface must be reset to work properly again. Any ideas?



  • @Teddy said in PfSense - Auto reboot script when google is unreachable..:

    Is there any way, to use the interface? Because my VPN IP is always dynamic.

    Yep, easy.
    You're scripting, right ?

    Have a look at the diag_ping.php page / Diagnostics > Ping.
    We can select an interface there, and the PHP gets the IP address.

    @Teddy said in PfSense - Auto reboot script when google is unreachable..:

    Any ideas?

    Use the VPN interface ?



  • @Gertjan
    Yes, i am using a script from another user here in the board! ;)
    And i can't really follow. That diag_ping.php i am using always for check, if a ping is possible. But how to connect it with my above mentioned script?

    I need an IP-address, to add it in the script, right? But the VPN interface IP is not static, it is dynamic.
    First it was 10.247.202.214, now after the reboot it is for example 10.246.201.107


Log in to reply