Using NUT on pfSense to control 2 UPS attached to QNAP NAS and Ubuntu VM



  • Hi,

    I just wanted to share with you my experience over the last days where I set up 2 UPS with NUT. The whole monitoring and power outage handling is performed on the pfSense.

    NUT-based UPS setup with pfSense and QNAP NAS.png

    My setup:
    Room 1: technical room containing the 19" server rack with

    • cable entry point with the cable modem: those are indeed 2 devices with seperate power
    • pfSense firewall
    • 1 QNAP NAS as primary NAS. On the NAS also runs a no-graphical Debian VM which is handling the VoIP system including door comms (Asterisk) and hosting a Ubiquiti WiFi controller
    • 1 QNAP NAS as backup for the primary NAS.
    • 2 network switches
    • 1 APC Back-UPS 900 UPS connected via USB to the backup QNAP NAS

    Room 2: home cinema

    • HTPC (Intel i7-8700k, Win10) running KODI as media center. In addition the HTPC provides a Ubuntu VM with a full graphical desktop
    • 1 EATON 3S 550 UPS connected to the Ubuntu VM on the HTPC. The reason not connecting it directly to the Win10 client is simple. The HTPC has 2 NICs. One is used by Win10 (KODI) without internet access, the other is used only by the UbuntuVM which has internet access. In addition, NUT is very much better supported on linux than on a Windows client. The pass-through of the EATON USB connection works flawless with VMWare Workstation.

    Main goal:

    • safely shutdown NAS and depending file services on VMs during a power outage to prevent hdd damage or data inconsistencies (which I suffered already in the past)

    Lessons learnt:

    • use BASH instead of SHELL on pfSense for scripting. It took me hours to find out that the string comparison in the if-statements cannot be done by the SHELL version on pfSense. Once I installed BASH on the pfSense, the desired string operations were available.
    • Once the cable entry point lost power, the WAN interface went down. This caused the timer on the pfSense not to start. First, I had all important devices in the rack of room 1 connected to the APC UPS except the cable entry point. For my initial power tests I just unplugged the power plug of the rack. Once I switch off the power of the whole room by using the fuse box, the onbatt event was recognized correctly but the shutdown timer was not starting. I double checked if the correct event and correct action was triggered by upssched-cmd. All was ok. Then I saw that the VPN and WAN interface went down as soon as I switched off power in the fuse box. That was the point where I realized that the cable entry point was on a different power circuit. Once I connected the cable entry point to the APC UPS, switched off the power in the fuse box, the shutdown timer started as the VPN and WAN interface did not go down.
    • You can trigger your upssched-cmd events manually. Doing that I realized real quickly that I needed to execute the file by using the whole path in the shell. Meaning: in my case the upssched-cmd file is in /usr/share/etc/nut/upssched/. Even if I'm located in that directory already and execute ./upssched-cmd onbatt I got no result. I had to type the whole path in order to get the script running:
    /usr/share/etc/nut/upssched/upssched-cmd onbatt
    
    • Using a DEBUG case for testing the upssched-cmd script was very valuable to me
    • Shutting down pfSense worked for me only with
    yes | /etc/rc.initial.halt
    

    /sbin/shutdown -p +120s sometimes caused the pfSense not to cut power but to stop at the last stage (something with disconnecting the usb keyboard as last log entry)

    • to me, a quite big advantage of using the pfSense as central instance for monitoring and dealing with a power outage was, that mail notifications come out of the box. Here the notification settings are used that are specified under System / Advanced / Notifications.

    Where I did fail:

    • I didn't manage to connect from all my other clients to the APC UPS when it is directly connected to pfSense via USB. The UPS is of course correctly recognized by pfSense. I tried here NUT and APCUPSD. I read a lot about firewall rules that need to be implemented but didn't succeed here. This and the fact that both QNAP NAS recognized and connected to the UPS without problem, got me to the decision to connect the APC UPS to one of the QNAP NAS and not to the pfSense

    Here are the 2 script files on the pfSense:
    upssched.conf

    CMDSCRIPT /usr/local/etc/nut/upssched/upssched-cmd 
    PIPEFN /usr/local/etc/nut/upssched/upssched.pipe 
    LOCKFN /usr/local/etc/nut/upssched/upssched.lock
    # ============================================================================
    
    AT ONBATT * EXECUTE onbatt
    AT ONBATT * START-TIMER shutdown 15
    AT ONLINE * CANCEL-TIMER shutdown online
    
    AT COMMBAD eaton@ip EXECUTE commbad_eaton
    AT COMMBAD qnapups@ip EXECUTE commbad_apc
    AT COMMOK eaton@ip EXECUTE commok_eaton
    AT COMMOK qnapups@ip EXECUTE commok_apc
    
     AT LOWBATT * EXECUTE shutdown 
    

    upssched-cmd

    #!/usr/local/bin/bash
    
    DATE=$(date '+%d-%m-%Y %H:%M:%S')
    UPSLOG=$(cat /var/log/upslog | tail -20)
    FULL_OUTAGE=false
    
    UPS_APC=$(upsc qnapups@ip device.model)
    STAT_APC=$(upsc qnapups@ip ups.status)
    BATT_APC=$(upsc qnapups@ip battery.charge)
    
    UPS_EATON=$(upsc eaton@ip device.model)
    STAT_EATON=$(upsc eaton@ip ups.status)
    BATT_EATON=$(upsc eaton@ip battery.charge)
    
    
    	case $1 in
    	
    	# DEBUG
    	debug)
    			
    		;;
    			
    			
    	# Case: on battery		
    	onbatt)
    		echo "$DATE: DEBUG: # Case: onbatt" >> /var/log/upslog
    		echo "$DATE: DEBUG: battery status - APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog
    
    		#Full power outage			
    		if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]]
    		then
    			echo "$DATE: both UPSs are on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
    			
    		#Only EATON on battery
    		elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
    		then
    			echo "$DATE: Only EATON UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nOnly EATON UPS is on battery. Shutdown timer initiated for 15s. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: power outage @ $UPS_EATON, battery status: $BATT_EATON"
    		
    		#Only APC is on battery
    		elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
    		then
    			echo "$DATE: Only APC UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nOnly APC UPS is on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: power outage @ $UPS_APC, battery status: $BATT_APC"					
    		fi	
    		;;
    	
    
    	# Case: shutdown
    	shutdown)
    			
    		echo "$DATE: DEBUG: # Case: shutdown" >> /var/log/upslog
    		echo "$DATE: DEBUG: battery status: APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog
    				
    		# Check if a shutdown already has been initialized and prevent a duplicate shutdown sequence
    		# A file is used as SHUTDOWN flag: /tmp/SHUTDOWN
    		# The file is deleted at every reboot by a shell script in /usr/local/etc/rc.d/
    		
    		FILE=/tmp/SHUTDOWN
    		if [ -f "$FILE" ]; then
    			echo "$DATE: $FILE already exists. No furter shutdown sequence initiated." >> /var/log/upslog
    		else 
    			echo "$DATE: $FILE does not exist. Create file and continue shutdown sequence" >> /var/log/upslog
    			echo "SHUTDOWN INITIATED DUE TO POWERLOSS" > /tmp/SHUTDOWN
    			
    			
    			#All UPS on battery -> shutdown all
    			if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]] 			
    			then
    				echo "$DATE: both UPSs are on battery. Shutdown initiated."  >>  /var/log/upslog 
    				echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
    				
    				#Shutdown sequence
    				# (1)	HTPC:
    				# (1a)	VMs on HTPC: UbuntuVM
    				# (1b)  HTPC: shutting down from PrimaryNAS
    				# 
    				# (2)	PrimaryNAS: 
    				# (2a)	VMs on PrimaryNAS: DebianVM (Asterisk, UniFi)
    				# (2b)	PrimaryNAS & BackupNAS 
    				#
    				# (3)	pfSense
    				
    				# Seq. (1a)- shutting down UbuntuVM remotely. The remote script simply stops first some services manually and the starts the shutdown
    					echo "$DATE: Executing remote shutdown on UbuntuVM...."  >>  /var/log/upslog
    					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
    
    				# Seq. (1b) - initiated shutdown timer for HTPC
    					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
    					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
    		
    			
    				# Seq. (2a) - shutting down DebianVM (Asterisk, UniFi):
    					echo "$DATE: Executing remote shutdown on DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog					
    					ssh username@ip /sbin/shutdown now
    				
    				# Seq. (2b) - shutting down PrimaryNAS and BackupNAS
    					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
    					ssh username@ip /sbin/poweroff
    
    					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
    					ssh username@ip /sbin/poweroff
    				
    				
    				# Seq. (3) - pfSense
    					# Send logs before pfSense is shut down
    					UPSLOG=$(cat /var/log/upslog | tail -20)
    					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
    			
    					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog
    					sleep 180
    					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
    					yes | /etc/rc.initial.halt 
    								
    				
    			#Only EATON on battery
    			elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
    			then
    				echo "$DATE: EATON UPS is on battery. Shutdown initiated."  >>  /var/log/upslog 
    				echo -e "Issue time: $DATE \nEATON UPS is on battery. Shutdown initiated. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: shutdown initiated for EATON @ $UPS_EATON, battery status: $BATT_EATON"
    				
    				#Shutdown sequence
    				# (1)	VMs: 
    				# (1a)  UbuntuVM on HTPC
    				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
    								
    				# Seq. (1a)- shutting down UbuntuVM:
    					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
    					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
    													
    				
    				# Seq. (1b) - initiated shutdown timer for HTPC
    					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
    					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
    					
    			
    			#Only APC is on low battery
    			elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
    			then
    				echo "$DATE: only APC UPS is on low battery. Shutdown initiated."  >>  /var/log/upslog 
    				echo -e "Issue time: $DATE \nAPC UPS is on low battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: shutdown initiated for APC @ $UPS_APC, battery status: $BATT_APC"
    				
    				#Shutdown sequence
    				# (1)	VMs: 
    				# (1a)  UbuntuVM on HTPC
    				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
    				# (1c)  DebianVM (Asterisk, UniFi) on PrimaryNAS
    				#
    				# (2)	PrimaryNAS: PrimaryNAS & BackupNAS
    				#
    				# (3)	pfSense
    				
    				# Seq. (1a)- shutting down UbuntuVM:
    					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
    					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
    													
    				
    				# Seq. (1b) - initiated shutdown timer for HTPC
    					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
    					logger "Terminating shutdown for HTPC in 120s..."
    					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &				
    			
    				# Seq. (1c) - shutting down DebianVM (Asterisk, UniFi):
    					echo "$DATE: Shutting down host DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog
    					ssh username@ip /sbin/shutdown now &
    				
    								
    				# Seq. (2) - shutting down PrimaryNAS and BackupNAS
    					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
    					ssh username@ip /sbin/poweroff &
    
    					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
    					ssh username@ip /sbin/poweroff &
    
    
    				# Seq. (3) - pfSense
    					# Send logs before pfSense is shut down
    					UPSLOG=$(cat /var/log/upslog | tail -20)
    					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
    			
    					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog					
    					sleep 180
    					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
    					yes | /etc/rc.initial.halt
    			fi	
    			
    		fi
    		;;
    		
    	
    		
    	# Case: online
    	online)		
    		
    		#Full power 
    		if [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OL" ] 
    		then
    			echo "$DATE: Power fully restored."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nPower fully restored. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power fully restored @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
    			
    		#Only EATON online
    		elif [ "$STAT_APC" == "OB DISCHRG" ] && [ "$STAT_EATON" == "OL" ]
    		then
    			echo "$DATE: EATON UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nEATON UPS power restored.  APC is still on battery. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power restored @ $UPS_EATON, battery status: $BATT_EATON"
    		
    		#Only APC is online
    		elif [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OB" ]
    		then
    			echo "$DATE: APC UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
    			echo -e "Issue time: $DATE \nAPC UPS power restored.  APC is still on battery. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: power restored @ $UPS_APC, battery status: $BATT_APC"
    		fi	
    					
    		;;
    		
    				
    	# Case: comms to EATON interrupted
    	commbad_eaton)						
    		echo "$DATE: The communication with EATON UPS is lost." >> /var/log/upslog			
    		echo -e "Issue time: $DATE \nThe communication with EATON UPS is lost. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication lost @ $UPS_EATON, battery status: $BATT_EATON"
    		;;
    		
    	
    	# Case: comms to APC interrupted
    	commbad_apc)						
    		echo "$DATE: The communication with APC UPS is lost." >> /var/log/upslog			
    		echo -e "Issue time: $DATE \nThe communication with APC UPS is lost. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication lost @ $UPS_APC, battery status: $BATT_APC"
    		;;
    	
    		
    	# Case: comms to EATON restored
    	commok_eaton)
    		echo "$DATE: The communication with EATON UPS is re-established."  >> /var/log/upslog
    		echo -e "Issue time: $DATE \nThe communication with EATON UPS is re-established. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication restored @ $UPS_EATON, battery status: $BATT_EATON"
    		;;
    	
    	
    	# Case: comms to APC restored
    	commok_apc)
    		echo "$DATE: The communication with APC UPS is re-established."  >> /var/log/upslog
    		echo -e "Issue time: $DATE \nThe communication with APC UPS is re-established. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication restored @ $UPS_APC, battery status: $BATT_APC"
    		;;
    			
    		
    	
    	
    	# Case powerdown
    		powerdown)
    					
    		#/sbin/upssched fsd
    								
    		;;			
    	
    	 *)
    		logger -t upssched-cmd "Unrecognized command: $1"
    		;;
    	esac