Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Using NUT on pfSense to control 2 UPS attached to QNAP NAS and Ubuntu VM

    Scheduled Pinned Locked Moved General pfSense Questions
    pfsensenutqnapnas
    1 Posts 1 Posters 1.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      LaUs3r
      last edited by LaUs3r

      Hi,

      I just wanted to share with you my experience over the last days where I set up 2 UPS with NUT. The whole monitoring and power outage handling is performed on the pfSense.

      NUT-based UPS setup with pfSense and QNAP NAS.png

      My setup:
      Room 1: technical room containing the 19" server rack with

      • cable entry point with the cable modem: those are indeed 2 devices with seperate power
      • pfSense firewall
      • 1 QNAP NAS as primary NAS. On the NAS also runs a no-graphical Debian VM which is handling the VoIP system including door comms (Asterisk) and hosting a Ubiquiti WiFi controller
      • 1 QNAP NAS as backup for the primary NAS.
      • 2 network switches
      • 1 APC Back-UPS 900 UPS connected via USB to the backup QNAP NAS

      Room 2: home cinema

      • HTPC (Intel i7-8700k, Win10) running KODI as media center. In addition the HTPC provides a Ubuntu VM with a full graphical desktop
      • 1 EATON 3S 550 UPS connected to the Ubuntu VM on the HTPC. The reason not connecting it directly to the Win10 client is simple. The HTPC has 2 NICs. One is used by Win10 (KODI) without internet access, the other is used only by the UbuntuVM which has internet access. In addition, NUT is very much better supported on linux than on a Windows client. The pass-through of the EATON USB connection works flawless with VMWare Workstation.

      Main goal:

      • safely shutdown NAS and depending file services on VMs during a power outage to prevent hdd damage or data inconsistencies (which I suffered already in the past)

      Lessons learnt:

      • use BASH instead of SHELL on pfSense for scripting. It took me hours to find out that the string comparison in the if-statements cannot be done by the SHELL version on pfSense. Once I installed BASH on the pfSense, the desired string operations were available.
      • Once the cable entry point lost power, the WAN interface went down. This caused the timer on the pfSense not to start. First, I had all important devices in the rack of room 1 connected to the APC UPS except the cable entry point. For my initial power tests I just unplugged the power plug of the rack. Once I switch off the power of the whole room by using the fuse box, the onbatt event was recognized correctly but the shutdown timer was not starting. I double checked if the correct event and correct action was triggered by upssched-cmd. All was ok. Then I saw that the VPN and WAN interface went down as soon as I switched off power in the fuse box. That was the point where I realized that the cable entry point was on a different power circuit. Once I connected the cable entry point to the APC UPS, switched off the power in the fuse box, the shutdown timer started as the VPN and WAN interface did not go down.
      • You can trigger your upssched-cmd events manually. Doing that I realized real quickly that I needed to execute the file by using the whole path in the shell. Meaning: in my case the upssched-cmd file is in /usr/share/etc/nut/upssched/. Even if I'm located in that directory already and execute ./upssched-cmd onbatt I got no result. I had to type the whole path in order to get the script running:
      /usr/share/etc/nut/upssched/upssched-cmd onbatt
      
      • Using a DEBUG case for testing the upssched-cmd script was very valuable to me
      • Shutting down pfSense worked for me only with
      yes | /etc/rc.initial.halt
      

      /sbin/shutdown -p +120s sometimes caused the pfSense not to cut power but to stop at the last stage (something with disconnecting the usb keyboard as last log entry)

      • to me, a quite big advantage of using the pfSense as central instance for monitoring and dealing with a power outage was, that mail notifications come out of the box. Here the notification settings are used that are specified under System / Advanced / Notifications.

      Where I did fail:

      • I didn't manage to connect from all my other clients to the APC UPS when it is directly connected to pfSense via USB. The UPS is of course correctly recognized by pfSense. I tried here NUT and APCUPSD. I read a lot about firewall rules that need to be implemented but didn't succeed here. This and the fact that both QNAP NAS recognized and connected to the UPS without problem, got me to the decision to connect the APC UPS to one of the QNAP NAS and not to the pfSense

      Here are the 2 script files on the pfSense:
      upssched.conf

      CMDSCRIPT /usr/local/etc/nut/upssched/upssched-cmd 
      PIPEFN /usr/local/etc/nut/upssched/upssched.pipe 
      LOCKFN /usr/local/etc/nut/upssched/upssched.lock
      # ============================================================================
      
      AT ONBATT * EXECUTE onbatt
      AT ONBATT * START-TIMER shutdown 15
      AT ONLINE * CANCEL-TIMER shutdown online
      
      AT COMMBAD eaton@ip EXECUTE commbad_eaton
      AT COMMBAD qnapups@ip EXECUTE commbad_apc
      AT COMMOK eaton@ip EXECUTE commok_eaton
      AT COMMOK qnapups@ip EXECUTE commok_apc
      
       AT LOWBATT * EXECUTE shutdown 
      

      upssched-cmd

      #!/usr/local/bin/bash
      
      DATE=$(date '+%d-%m-%Y %H:%M:%S')
      UPSLOG=$(cat /var/log/upslog | tail -20)
      FULL_OUTAGE=false
      
      UPS_APC=$(upsc qnapups@ip device.model)
      STAT_APC=$(upsc qnapups@ip ups.status)
      BATT_APC=$(upsc qnapups@ip battery.charge)
      
      UPS_EATON=$(upsc eaton@ip device.model)
      STAT_EATON=$(upsc eaton@ip ups.status)
      BATT_EATON=$(upsc eaton@ip battery.charge)
      
      
      	case $1 in
      	
      	# DEBUG
      	debug)
      			
      		;;
      			
      			
      	# Case: on battery		
      	onbatt)
      		echo "$DATE: DEBUG: # Case: onbatt" >> /var/log/upslog
      		echo "$DATE: DEBUG: battery status - APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog
      
      		#Full power outage			
      		if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]]
      		then
      			echo "$DATE: both UPSs are on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
      			
      		#Only EATON on battery
      		elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
      		then
      			echo "$DATE: Only EATON UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nOnly EATON UPS is on battery. Shutdown timer initiated for 15s. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: power outage @ $UPS_EATON, battery status: $BATT_EATON"
      		
      		#Only APC is on battery
      		elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
      		then
      			echo "$DATE: Only APC UPS is on battery. Shutdown timer initiated for 15s."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nOnly APC UPS is on battery. Shutdown timer initiated for 15s. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: power outage @ $UPS_APC, battery status: $BATT_APC"					
      		fi	
      		;;
      	
      
      	# Case: shutdown
      	shutdown)
      			
      		echo "$DATE: DEBUG: # Case: shutdown" >> /var/log/upslog
      		echo "$DATE: DEBUG: battery status: APC: $STAT_APC, EATON: $STAT_EATON" >> /var/log/upslog
      				
      		# Check if a shutdown already has been initialized and prevent a duplicate shutdown sequence
      		# A file is used as SHUTDOWN flag: /tmp/SHUTDOWN
      		# The file is deleted at every reboot by a shell script in /usr/local/etc/rc.d/
      		
      		FILE=/tmp/SHUTDOWN
      		if [ -f "$FILE" ]; then
      			echo "$DATE: $FILE already exists. No furter shutdown sequence initiated." >> /var/log/upslog
      		else 
      			echo "$DATE: $FILE does not exist. Create file and continue shutdown sequence" >> /var/log/upslog
      			echo "SHUTDOWN INITIATED DUE TO POWERLOSS" > /tmp/SHUTDOWN
      			
      			
      			#All UPS on battery -> shutdown all
      			if [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON = *OB* ]] 			
      			then
      				echo "$DATE: both UPSs are on battery. Shutdown initiated."  >>  /var/log/upslog 
      				echo -e "Issue time: $DATE \nBoth UPSs are on battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: Full power outage @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
      				
      				#Shutdown sequence
      				# (1)	HTPC:
      				# (1a)	VMs on HTPC: UbuntuVM
      				# (1b)  HTPC: shutting down from PrimaryNAS
      				# 
      				# (2)	PrimaryNAS: 
      				# (2a)	VMs on PrimaryNAS: DebianVM (Asterisk, UniFi)
      				# (2b)	PrimaryNAS & BackupNAS 
      				#
      				# (3)	pfSense
      				
      				# Seq. (1a)- shutting down UbuntuVM remotely. The remote script simply stops first some services manually and the starts the shutdown
      					echo "$DATE: Executing remote shutdown on UbuntuVM...."  >>  /var/log/upslog
      					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
      
      				# Seq. (1b) - initiated shutdown timer for HTPC
      					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
      					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
      		
      			
      				# Seq. (2a) - shutting down DebianVM (Asterisk, UniFi):
      					echo "$DATE: Executing remote shutdown on DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog					
      					ssh username@ip /sbin/shutdown now
      				
      				# Seq. (2b) - shutting down PrimaryNAS and BackupNAS
      					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
      					ssh username@ip /sbin/poweroff
      
      					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
      					ssh username@ip /sbin/poweroff
      				
      				
      				# Seq. (3) - pfSense
      					# Send logs before pfSense is shut down
      					UPSLOG=$(cat /var/log/upslog | tail -20)
      					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
      			
      					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog
      					sleep 180
      					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
      					yes | /etc/rc.initial.halt 
      								
      				
      			#Only EATON on battery
      			elif [[ $STAT_APC != *"DISCHRG"* ]] && [[ $STAT_EATON = *OB* ]]
      			then
      				echo "$DATE: EATON UPS is on battery. Shutdown initiated."  >>  /var/log/upslog 
      				echo -e "Issue time: $DATE \nEATON UPS is on battery. Shutdown initiated. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATON Battery status: $BATT_EATON" | mail.php -s="UPS ALERT: shutdown initiated for EATON @ $UPS_EATON, battery status: $BATT_EATON"
      				
      				#Shutdown sequence
      				# (1)	VMs: 
      				# (1a)  UbuntuVM on HTPC
      				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
      								
      				# Seq. (1a)- shutting down UbuntuVM:
      					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
      					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
      													
      				
      				# Seq. (1b) - initiated shutdown timer for HTPC
      					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
      					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &
      					
      			
      			#Only APC is on low battery
      			elif [[ $STAT_APC = *DISCHRG* ]] && [[ $STAT_EATON != *"OB"* ]]
      			then
      				echo "$DATE: only APC UPS is on low battery. Shutdown initiated."  >>  /var/log/upslog 
      				echo -e "Issue time: $DATE \nAPC UPS is on low battery. Shutdown initiated. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC" | mail.php -s="UPS ALERT: shutdown initiated for APC @ $UPS_APC, battery status: $BATT_APC"
      				
      				#Shutdown sequence
      				# (1)	VMs: 
      				# (1a)  UbuntuVM on HTPC
      				# (1b)  HTPC: Initiate shutdown timer on HTPC from PrimaryNAS
      				# (1c)  DebianVM (Asterisk, UniFi) on PrimaryNAS
      				#
      				# (2)	PrimaryNAS: PrimaryNAS & BackupNAS
      				#
      				# (3)	pfSense
      				
      				# Seq. (1a)- shutting down UbuntuVM:
      					echo "$DATE: Executing remoteShutdown on UbuntuVM...."  >>  /var/log/upslog
      					ssh username@ip sudo /usr/local/bin/shutdownRemotely &
      													
      				
      				# Seq. (1b) - initiated shutdown timer for HTPC
      					echo "$DATE: Terminating shutdown for HTPC in 120s..."  >>  /var/log/upslog
      					logger "Terminating shutdown for HTPC in 120s..."
      					ssh username@ip /mnt/ext/opt/samba/bin/net rpc shutdown -f -t 120 -C "Shutting down HTPC in 120s" -U username%password -I ip &				
      			
      				# Seq. (1c) - shutting down DebianVM (Asterisk, UniFi):
      					echo "$DATE: Shutting down host DebianVM (Asterisk, UniFi)...."  >>  /var/log/upslog
      					ssh username@ip /sbin/shutdown now &
      				
      								
      				# Seq. (2) - shutting down PrimaryNAS and BackupNAS
      					echo "$DATE: Shutting down PrimaryNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
      					ssh username@ip /sbin/poweroff &
      
      					echo "$DATE: Shutting down BackupNAS: ssh username@ip /share/Backup/Scripts/shutdownRemotely..." >> /var/log/upslog
      					ssh username@ip /sbin/poweroff &
      
      
      				# Seq. (3) - pfSense
      					# Send logs before pfSense is shut down
      					UPSLOG=$(cat /var/log/upslog | tail -20)
      					echo -e "Issue time: $DATE \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON \nUPSLOG: \n$UPSLOG" | mail.php -s="UPS ALERT: All UPSs are down - last e-mail before shutdown. Logs attached"
      			
      					echo "$DATE: Shutting down host pfSense in 180s...."  >>  /var/log/upslog					
      					sleep 180
      					echo "$DATE: Done. Commencing shutdown"  >>  /var/log/upslog					
      					yes | /etc/rc.initial.halt
      			fi	
      			
      		fi
      		;;
      		
      	
      		
      	# Case: online
      	online)		
      		
      		#Full power 
      		if [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OL" ] 
      		then
      			echo "$DATE: Power fully restored."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nPower fully restored. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPC Battery status: $BATT_APC \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power fully restored @ $UPS_APC, battery status: $BATT_APC, $UPS_EATON, battery status: $BATT_EATON"
      			
      		#Only EATON online
      		elif [ "$STAT_APC" == "OB DISCHRG" ] && [ "$STAT_EATON" == "OL" ]
      		then
      			echo "$DATE: EATON UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nEATON UPS power restored.  APC is still on battery. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: power restored @ $UPS_EATON, battery status: $BATT_EATON"
      		
      		#Only APC is online
      		elif [ "$STAT_APC" == "OL CHRG" ] || [ "$STAT_APC" == "OL" ] && [ "$STAT_EATON" == "OB" ]
      		then
      			echo "$DATE: APC UPS power is restored. APC is still on battery."  >>  /var/log/upslog 
      			echo -e "Issue time: $DATE \nAPC UPS power restored.  APC is still on battery. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: power restored @ $UPS_APC, battery status: $BATT_APC"
      		fi	
      					
      		;;
      		
      				
      	# Case: comms to EATON interrupted
      	commbad_eaton)						
      		echo "$DATE: The communication with EATON UPS is lost." >> /var/log/upslog			
      		echo -e "Issue time: $DATE \nThe communication with EATON UPS is lost. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication lost @ $UPS_EATON, battery status: $BATT_EATON"
      		;;
      		
      	
      	# Case: comms to APC interrupted
      	commbad_apc)						
      		echo "$DATE: The communication with APC UPS is lost." >> /var/log/upslog			
      		echo -e "Issue time: $DATE \nThe communication with APC UPS is lost. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication lost @ $UPS_APC, battery status: $BATT_APC"
      		;;
      	
      		
      	# Case: comms to EATON restored
      	commok_eaton)
      		echo "$DATE: The communication with EATON UPS is re-established."  >> /var/log/upslog
      		echo -e "Issue time: $DATE \nThe communication with EATON UPS is re-established. \nEATON: $UPS_EATON \nEATON status: $STAT_EATON \nEATonline status: $BATT_EATON" | mail.php -s="UPS ALERT: Communication restored @ $UPS_EATON, battery status: $BATT_EATON"
      		;;
      	
      	
      	# Case: comms to APC restored
      	commok_apc)
      		echo "$DATE: The communication with APC UPS is re-established."  >> /var/log/upslog
      		echo -e "Issue time: $DATE \nThe communication with APC UPS is re-established. \nAPC: $UPS_APC \nAPC status: $STAT_APC \nAPCline status: $BATT_APC" | mail.php -s="UPS ALERT: Communication restored @ $UPS_APC, battery status: $BATT_APC"
      		;;
      			
      		
      	
      	
      	# Case powerdown
      		powerdown)
      					
      		#/sbin/upssched fsd
      								
      		;;			
      	
      	 *)
      		logger -t upssched-cmd "Unrecognized command: $1"
      		;;
      	esac
      
      
      1 Reply Last reply Reply Quote 0
      • First post
        Last post
      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.