PfSense - Auto reboot script when google is unreachable..



  • Senario:
    My ISP suddenly drops the connection, even though all lights are "green" on the cable modem and it should be working..
    Only solution I've found is to reboot the pfsense..
    Read my other topic about this here: http://forum.pfsense.org/index.php/topic,69879.msg381954.html#msg381954

    My 10 steps - Howto:

    1. Login to pfsense with ssh, select "8" for shell command

    2. Go to: /usr/local/bin

    3. To remount file systems as read-write, run: /etc/rc.conf_mount_rw

    4. Create file: ping-check.sh  (to create file, simple howto: vi ping-check.sh, then carefully click "i" and paste the code, click "esc", type ":wq!" - all in that order! )

    Add this to file;

    
    #!/bin/sh
    
    # HOSTS can be either you ISP or google.com
    HOSTS="google.com"
    COUNT=2
    
    echo "Pinging.."
    echo "HOSTS: " $HOSTS
    echo "COUNT: " $COUNT
    ######
    for myHost in $HOSTS
    do
      counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }' )
      echo "counting: " $counting
    
      if [ $counting -eq 2 ]; then
       echo "Ping OK"
    
      else
       # network down
       # Save RRD data
       /etc/rc.backup_rrd.sh
       #Reboot
       echo "Reboot!"
       reboot
    fi
    done
    
    

    5. chmod 700 ping-check.sh

    6. To mount as read-only again, run: /etc/rc.conf_mount_ro

    7. exit

    Now you need to add a cron job to automatically run this every 5 minutes..

    8. Go into pfSense web interface - and select:

    • Packages (under System)
    • Cron (0.1.8 is what I found when writing this)
    • Select "+" and install Cron.

    9. Then go into Cron (under Services)

    10. Click "+" and add

    minute:  5
    hours:    *
    mday:    *
    month:  *
    wday:    *
    (who):  root
    command:  /usr/local/bin/ping-check.sh

    Click "Save"

    Thats it!
    Now the system will check if the pfSense box is able to ping every 5 minutes the host in the script, if not - it will reboot.
    Testet on my 2.1-RELEASE  (i386) and works perfectly well.

    if I can just get curl into pfsense also, the pfsense box will be able to issue the command for rebooting the cable modem too..  but that is for later or next project ;)

    Enjoy :-)



  • It there is heavy traffic, ping traffic may not get through.. therefore I simply modified script abit,
    it now pings 10 times.. and if 2 or more pings are received okey - network is considered up and running:

    
    #!/bin/bash
    #
    # put -xv after bash to debug
    #
    HOSTS="google.com"
    COUNT=10
    
    #debug
    echo "HOSTS: " $HOSTS
    echo "COUNT: " $COUNT
    ######
    for myHost in $HOSTS
    do
      counting=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
    
      #debug
      echo "counting: " $counting
      ######
      if [ $counting > 2 ]; then
       echo "Ping OK"
    
      else
       # network down
       # Save RRD data
       /etc/rc.backup_rrd.sh
       #Reboot
       echo "Reboot!"
       reboot
    fi
    done
    
    


  • Nice, thanks for the tutorial!



  • Question: Is there a way to track reboots other than the Uptime to make sure the script is working correctly? I dont want it going too crazy.



  • Nice to hear that is useful ;-)

    Track reboot - you want to be notified when the system reboots ? if so, one would need to create two things;

    1. when the ping check is run, if it fail we have to create a file with date/timestamp before it reboots.

    2. create a startup scripts that checks if the file exists and mails this file to a gmail account. After mail successful, delete det file.

    done.

    I have no  time to do this myself (now at least). I did not see the point of knowing when the reboot is done..
    I just needed the system check if its online, and if not - try to get back online on its own.

    Of course this also applies to the modem you have connected. And rebooting the modem would in most cases help re-connecting the devices. (refreshing IP/MAC/DNS from the ISP)
    Over the years I have seen more problems with the modem, than pfSense box. The modem might go down…but pfSense is up. And when I reboot the modem and it gets back online.. while the pfSense box is still untouched, system is back working again.
    Very rare that I need to reboot pfSense, but it happens.. when my IPS changes the IP/MAC locking againts my modem i think.



  • Very good