Using NUT on pfSense to control 2 UPS attached to QNAP NAS and Ubuntu VM

LaUs3r

Hi,

I just wanted to share with you my experience over the last days where I set up 2 UPS with NUT. The whole monitoring and power outage handling is performed on the pfSense.

NUT-based UPS setup with pfSense and QNAP NAS.png

My setup:
Room 1: technical room containing the 19" server rack with

cable entry point with the cable modem: those are indeed 2 devices with seperate power
pfSense firewall
1 QNAP NAS as primary NAS. On the NAS also runs a no-graphical Debian VM which is handling the VoIP system including door comms (Asterisk) and hosting a Ubiquiti WiFi controller
1 QNAP NAS as backup for the primary NAS.
2 network switches
1 APC Back-UPS 900 UPS connected via USB to the backup QNAP NAS

Room 2: home cinema

HTPC (Intel i7-8700k, Win10) running KODI as media center. In addition the HTPC provides a Ubuntu VM with a full graphical desktop
1 EATON 3S 550 UPS connected to the Ubuntu VM on the HTPC. The reason not connecting it directly to the Win10 client is simple. The HTPC has 2 NICs. One is used by Win10 (KODI) without internet access, the other is used only by the UbuntuVM which has internet access. In addition, NUT is very much better supported on linux than on a Windows client. The pass-through of the EATON USB connection works flawless with VMWare Workstation.

Main goal:

safely shutdown NAS and depending file services on VMs during a power outage to prevent hdd damage or data inconsistencies (which I suffered already in the past)

Lessons learnt:

use BASH instead of SHELL on pfSense for scripting. It took me hours to find out that the string comparison in the if-statements cannot be done by the SHELL version on pfSense. Once I installed BASH on the pfSense, the desired string operations were available.
Once the cable entry point lost power, the WAN interface went down. This caused the timer on the pfSense not to start. First, I had all important devices in the rack of room 1 connected to the APC UPS except the cable entry point. For my initial power tests I just unplugged the power plug of the rack. Once I switch off the power of the whole room by using the fuse box, the onbatt event was recognized correctly but the shutdown timer was not starting. I double checked if the correct event and correct action was triggered by upssched-cmd. All was ok. Then I saw that the VPN and WAN interface went down as soon as I switched off power in the fuse box. That was the point where I realized that the cable entry point was on a different power circuit. Once I connected the cable entry point to the APC UPS, switched off the power in the fuse box, the shutdown timer started as the VPN and WAN interface did not go down.
You can trigger your upssched-cmd events manually. Doing that I realized real quickly that I needed to execute the file by using the whole path in the shell. Meaning: in my case the upssched-cmd file is in /usr/share/etc/nut/upssched/. Even if I'm located in that directory already and execute ./upssched-cmd onbatt I got no result. I had to type the whole path in order to get the script running:

/usr/share/etc/nut/upssched/upssched-cmd onbatt

Using a DEBUG case for testing the upssched-cmd script was very valuable to me
Shutting down pfSense worked for me only with

yes | /etc/rc.initial.halt

/sbin/shutdown -p +120s sometimes caused the pfSense not to cut power but to stop at the last stage (something with disconnecting the usb keyboard as last log entry)

to me, a quite big advantage of using the pfSense as central instance for monitoring and dealing with a power outage was, that mail notifications come out of the box. Here the notification settings are used that are specified under System / Advanced / Notifications.

Where I did fail:

I didn't manage to connect from all my other clients to the APC UPS when it is directly connected to pfSense via USB. The UPS is of course correctly recognized by pfSense. I tried here NUT and APCUPSD. I read a lot about firewall rules that need to be implemented but didn't succeed here. This and the fact that both QNAP NAS recognized and connected to the UPS without problem, got me to the decision to connect the APC UPS to one of the QNAP NAS and not to the pfSense

Here are the 2 script files on the pfSense:
upssched.conf

CMDSCRIPT /usr/local/etc/nut/upssched/upssched-cmd 
PIPEFN /usr/local/etc/nut/upssched/upssched.pipe 
LOCKFN /usr/local/etc/nut/upssched/upssched.lock
# ============================================================================

AT ONBATT * EXECUTE onbatt
AT ONBATT * START-TIMER shutdown 15
AT ONLINE * CANCEL-TIMER shutdown online

AT COMMBAD eaton@ip EXECUTE commbad_eaton
AT COMMBAD qnapups@ip EXECUTE commbad_apc
AT COMMOK eaton@ip EXECUTE commok_eaton
AT COMMOK qnapups@ip EXECUTE commok_apc

 AT LOWBATT * EXECUTE shutdown

upssched-cmd

#!/usr/local/bin/bash

DATE=$(date '+%d-%m-%Y %H:%M:%S')
UPSLOG=$(cat /var/log/upslog | tail -20)
FULL_OUTAGE=false

UPS_APC=$(upsc qnapups@ip device.model)
STAT_APC=$(upsc qnapups@ip ups.status)
BATT_APC=$(upsc qnapups@ip battery.charge)

UPS_EATON=$(upsc eaton@ip device.model)
STAT_EATON=$(upsc eaton@ip ups.status)
BATT_EATON=$(upsc eaton@ip battery.charge)


	case $1 in
	
	# DEBUG
	debug)
			
		;;
			
			
	# Case: on battery		
	onbatt)
		echo "$DATE: DEBUG: # Case: onbatt" >> /var/log/upslog
		echo "$DATE: DEBUG: battery status - APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog

		#Full power outage			
		if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]]
		then
			echo "$DATE: both UPSs are on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
			
		#Only EATON on battery
		elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
		then
			echo "$DATE: Only EATON UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nOnly EATON UPS is on battery. Shutdown timer initiated for 15s. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: power outage @ $UPS_EATON, battery status: $BATT_EATON"
		
		#Only APC is on battery
		elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
		then
			echo "$DATE: Only APC UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nOnly APC UPS is on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: power outage @ $UPS_APC, battery status: $BATT_APC"					
		fi	
		;;
	

	# Case: shutdown
	shutdown)
			
		echo "$DATE: DEBUG: # Case: shutdown" >> /var/log/upslog
		echo "$DATE: DEBUG: battery status: APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog
				
		# Check if a shutdown already has been initialized and prevent a duplicate shutdown sequence
		# A file is used as SHUTDOWN flag: /tmp/SHUTDOWN
		# The file is deleted at every reboot by a shell script in /usr/local/etc/rc.d/
		
		FILE=/tmp/SHUTDOWN
		if [ -f "$FILE" ]; then
			echo "$DATE: $FILE already exists. No furter shutdown sequence initiated." >> /var/log/upslog
		else 
			echo "$DATE: $FILE does not exist. Create file and continue shutdown sequence" >> /var/log/upslog
			echo "SHUTDOWN INITIATED DUE TO POWERLOSS" > /tmp/SHUTDOWN
			
			
			#All UPS on battery -> shutdown all
			if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]] 			
			then
				echo "$DATE: both UPSs are on battery. Shutdown initiated."  >>  /var/log/upslog 
				echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
				
				#Shutdown sequence
				# (1)	HTPC:
				# (1a)	VMs on HTPC: UbuntuVM
				# (1b)  HTPC: shutting down from PrimaryNAS
				# 
				# (2)	PrimaryNAS: 
				# (2a)	VMs on PrimaryNAS: DebianVM (Asterisk, UniFi)
				# (2b)	PrimaryNAS & BackupNAS 
				#
				# (3)	pfSense
				
				# Seq. (1a)- shutting down UbuntuVM remotely. The remote script simply stops first some services manually and the starts the shutdown
					echo "$DATE: Executing remote shutdown on UbuntuVM...."  >>  /var/log/upslog
					ssh username@ip sudo /usr/local/bin/shutdownRemotely &

				# Seq. (1b) - initiated shutdown timer for HTPC
					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
		
			
				# Seq. (2a) - shutting down DebianVM (Asterisk, UniFi):
					echo "$DATE: Executing remote shutdown on DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog					
					ssh username@ip /sbin/shutdown now
				
				# Seq. (2b) - shutting down PrimaryNAS and BackupNAS
					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
					ssh username@ip /sbin/poweroff

					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
					ssh username@ip /sbin/poweroff
				
				
				# Seq. (3) - pfSense
					# Send logs before pfSense is shut down
					UPSLOG=$(cat /var/log/upslog | tail -20)
					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
			
					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog
					sleep 180
					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
					yes | /etc/rc.initial.halt 
								
				
			#Only EATON on battery
			elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
			then
				echo "$DATE: EATON UPS is on battery. Shutdown initiated."  >>  /var/log/upslog 
				echo -e "Issue time: $DATE \nEATON UPS is on battery. Shutdown initiated. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: shutdown initiated for EATON @ $UPS_EATON, battery status: $BATT_EATON"
				
				#Shutdown sequence
				# (1)	VMs: 
				# (1a)  UbuntuVM on HTPC
				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
								
				# Seq. (1a)- shutting down UbuntuVM:
					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
													
				
				# Seq. (1b) - initiated shutdown timer for HTPC
					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
					
			
			#Only APC is on low battery
			elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
			then
				echo "$DATE: only APC UPS is on low battery. Shutdown initiated."  >>  /var/log/upslog 
				echo -e "Issue time: $DATE \nAPC UPS is on low battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: shutdown initiated for APC @ $UPS_APC, battery status: $BATT_APC"
				
				#Shutdown sequence
				# (1)	VMs: 
				# (1a)  UbuntuVM on HTPC
				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
				# (1c)  DebianVM (Asterisk, UniFi) on PrimaryNAS
				#
				# (2)	PrimaryNAS: PrimaryNAS & BackupNAS
				#
				# (3)	pfSense
				
				# Seq. (1a)- shutting down UbuntuVM:
					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
													
				
				# Seq. (1b) - initiated shutdown timer for HTPC
					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
					logger "Terminating shutdown for HTPC in 120s..."
					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &				
			
				# Seq. (1c) - shutting down DebianVM (Asterisk, UniFi):
					echo "$DATE: Shutting down host DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog
					ssh username@ip /sbin/shutdown now &
				
								
				# Seq. (2) - shutting down PrimaryNAS and BackupNAS
					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
					ssh username@ip /sbin/poweroff &

					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
					ssh username@ip /sbin/poweroff &


				# Seq. (3) - pfSense
					# Send logs before pfSense is shut down
					UPSLOG=$(cat /var/log/upslog | tail -20)
					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
			
					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog					
					sleep 180
					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
					yes | /etc/rc.initial.halt
			fi	
			
		fi
		;;
		
	
		
	# Case: online
	online)		
		
		#Full power 
		if [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OL" ] 
		then
			echo "$DATE: Power fully restored."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nPower fully restored. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power fully restored @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
			
		#Only EATON online
		elif [ "$STAT_APC" == "OB DISCHRG" ] && [ "$STAT_EATON" == "OL" ]
		then
			echo "$DATE: EATON UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nEATON UPS power restored.  APC is still on battery. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power restored @ $UPS_EATON, battery status: $BATT_EATON"
		
		#Only APC is online
		elif [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OB" ]
		then
			echo "$DATE: APC UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
			echo -e "Issue time: $DATE \nAPC UPS power restored.  APC is still on battery. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: power restored @ $UPS_APC, battery status: $BATT_APC"
		fi	
					
		;;
		
				
	# Case: comms to EATON interrupted
	commbad_eaton)						
		echo "$DATE: The communication with EATON UPS is lost." >> /var/log/upslog			
		echo -e "Issue time: $DATE \nThe communication with EATON UPS is lost. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication lost @ $UPS_EATON, battery status: $BATT_EATON"
		;;
		
	
	# Case: comms to APC interrupted
	commbad_apc)						
		echo "$DATE: The communication with APC UPS is lost." >> /var/log/upslog			
		echo -e "Issue time: $DATE \nThe communication with APC UPS is lost. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication lost @ $UPS_APC, battery status: $BATT_APC"
		;;
	
		
	# Case: comms to EATON restored
	commok_eaton)
		echo "$DATE: The communication with EATON UPS is re-established."  >> /var/log/upslog
		echo -e "Issue time: $DATE \nThe communication with EATON UPS is re-established. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication restored @ $UPS_EATON, battery status: $BATT_EATON"
		;;
	
	
	# Case: comms to APC restored
	commok_apc)
		echo "$DATE: The communication with APC UPS is re-established."  >> /var/log/upslog
		echo -e "Issue time: $DATE \nThe communication with APC UPS is re-established. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication restored @ $UPS_APC, battery status: $BATT_APC"
		;;
			
		
	
	
	# Case powerdown
		powerdown)
					
		#/sbin/upssched fsd
								
		;;			
	
	 *)
		logger -t upssched-cmd "Unrecognized command: $1"
		;;
	esac