NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason

DomV

I'm using NUT 2.8.2_6 package in pfsense 25.11.1 and it successfully shut down's devices and after that finally the sockets on UPS. The problem comes when the system gets it's power back and the firewall start's to boot up... after some time (~3 min) it for some reason shut down's sockets on the ups without warning... This is really undesirable because it defeats the whole point of having power backup.

I've been able to verify it's pfsense who is doing that by leaving the firewall unplugged after UPS reconnect's power... Everything works until pfsense boot's up again... I've also been able to reproduce it without any modifications in advanced section of NUT package.

My UPS is Back-UPS BGM2200-MSX and it's connected using USB cable with usbhid-ups driver.

PS: I was doing the test in quick succession after shutdown... the UPS seems to ignore my ondelay=130 parametr...

dennypage

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

it successfully shut down's devices and after that finally the sockets on UPS

Can you clarify what this means please? Specifically "successfully shut down's devices" and "the sockets on UPS"? A quick look at that UPS does not show any switched sockets...

The phrase "I've been able to verify it's pfsense who is doing that by leaving the firewall unplugged" is a strong of statement. I don't think enough is known yet to make that blanket statement, particularly since pfSense itself doesn't actually do anything with the UPS. It could be the NUT driver running on pfSense, or the UPS itself.

DomV

@dennypage After it send's shutdown commands to servers and they turn off it than proceeds with the shutdown process for pfsense itself. After that it sends killpower command to the UPS and it's cut's power after predefined 120 seconds.

Broadcast Message from root@pfsense.REDACTED.arpa                                
        (no tty) at 1:43 CET...                                                
                                                                               
UPS apc on battery                                                             
                                                                               
                                                                               
Broadcast Message from root@pfsense.REDACTED.arpa                                
        (no tty) at 1:43 CET...                                                
                                                                               
Executing automatic power-fail shutdown                                        

                                                                               
                                                                               
Broadcast Message from root@pfsense.REDACTED.arpa                                
        (no tty) at 1:43 CET...                                                
                                                                               
Auto logout and shutdown proceeding                                            
                                                                               
                                                                               
*** FINAL System shutdown message from root@pfsense.REDACTED.arpa ***          

System going down IMMEDIATELY                                                  

                                                                               
                                                                               
Broadcast Message from root@pfsense.REDACTED.arpa                                
        (/dev/console) at 1:45 CET...                                          
                                                                               
NUT killing power...                                                           
                                                                               
Connection to 192.168.1.1 closed by remote host.
Connection to 192.168.1.1 closed.

dennypage

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

After it send's shutdown commands to servers and they turn off it than proceeds with the shutdown process for pfsense itself. After that it sends killpower command to the UPS and its cut's power after predefined 120 seconds.

FWIW, NUT doesn't actually send a shutdown command to remote clients -- Remote clients actually make their own shutdown decisions based on reported status (such as low battery indication), run time remaining, and battery percentage remaining.

We can remove the remote clients from consideration. You can confirm by disconnecting the remote clients from the NUT upsd running on the pfSense system. In fact, I recommend that you do this to simplify your testing environment.

Also, the command to kill the power is send by nut before instructing the host OS (pfSense in this case) to shut down. The shutdown delay implemented by the UPS is to allow the OS to complete its shutdown. The shutdown delay is usually 20 seconds. Have you changed this value in the UPS? A shutdown delay of 120 seconds is quite large...

If you have changed the shutdown delay, have you also configured the startup delay? Most UPSs do not do this automatically, so if you change one you have to change both. And sometimes other related parameters.

My recommendation for how to proceed is to remove any advanced configuration you may have for NUT, restore the UPS's default values, and begin testing from there.

Gertjan

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

UPS apc on battery

You see this message on the console access while this very same UPS is connected to the main electricity, 120 or 230 V ?
Before cycling the batterie, or even the entire UPS, you two should have a talk.
Get the name of your UPS - get the name you've used here :

[25.11.1-RELEASE][root@pfSense.bhf.tld]/root: cat /usr/local/etc/nut/upsmon.conf
MONITOR UPS 1 local-monitor 9b661f17bf95d56f892e master
.....

This makes sense as :

You need more :

[25.11.1-RELEASE][root@pfSense.bhf.tld]/root: cat /usr/local/etc/nut/upsd.users
[admin]
password=e8f2009d2a5ea51adf82
actions=set
instcmds=all
[local-monitor]
password=9b661f17bf95d56f892e
upsmon master
.....

Here you see who is the 'admin user : it's admin, and also the password of this admin user.

To know what command your UPS accepts, easy : no authentifcaition is needed : just ask :

[25.11.1-RELEASE][root@pfSense.bhf.tld]/root: upscmd -l UPS
Instant commands supported on UPS [UPS]:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
driver.killpower - Tell the driver daemon to initiate UPS shutdown; should be unlocked with driver.flag.allow_killpower option or variable setting
driver.reload - Reload running driver configuration from the file system (only works for changes in some options)
driver.reload-or-error - Reload running driver configuration from the file system (only works for changes in some options); return an error if something changed and could not be applied live (so the caller can restart it with new options)
driver.reload-or-exit - Reload running driver configuration from the file system (only works for changes in some options); exit the running driver if something changed and could not be applied live (so service management framework can restart it with new options)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.stop - Stop a shutdown in progress
test.battery.start.deep - Start a deep battery test
test.battery.start.quick - Start a quick battery test
test.battery.stop - Stop the battery test

To actually execute a command, you need the user and password.
Here I execute a quick battery test :

[25.11.1-RELEASE][root@pfSense.brit-hotel-fumel.net]/root: upscmd -u admin -p e8f2009d2a5ea51adf82 UPS test.battery.start.quick
OK

and I detected relay activity in my UPS (an APC also, with just one big green status led).

But there is more :

[25.11.1-RELEASE][root@pfSense.bhf.tld]/root: upsrw -l -u admin -p e8f2009d2a5ea51adf82 ups
[battery.charge.low]
Remaining battery level when UPS switches to LB (percent)
Type: STRING
Maximum length: 10
Value: 10

[battery.mfr.date]
Battery manufacturing date
Type: STRING
Maximum length: 10
Value: 2025/06/06

[battery.runtime.low]
Remaining battery runtime when UPS switches to LB (seconds)
Type: STRING
Maximum length: 10
Value: 120

[driver.debug]
Current debug verbosity level of the driver program
Type: NUMBER
Value: 0

[driver.flag.allow_killpower]
Safety flip-switch to allow the driver daemon to send UPS shutdown command (accessible via driver.killpower)
Type: NUMBER
Value: 0

[input.sensitivity]
Input power sensitivity
Type: STRING
Maximum length: 10
Value: low

[input.transfer.high]
High voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 295

[input.transfer.low]
Low voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 145

[ups.delay.shutdown]
Interval to wait after shutdown with delay command (seconds)
Type: STRING
Maximum length: 10
Value: 20

I'm just showing the UPS variables here.
You can also 'set' them, like the "[battery.mfr.date]" which is set to "2025/06/06" and that's the date I changed the batterie in the UPS.

This is the command to show the 'status' of your UPS :

[25.11.1-RELEASE][root@pfSense.bhf.tld]/root: upsc UPS
battery.charge: 98
battery.charge.low: 10
battery.mfr.date: 2025/06/06
battery.runtime: 1124
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.6
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS BX950MI
device.serial: 9B2506A09546
device.type: ups
driver.debug: 0
driver.flag.allow_killpower: 0
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: auto
driver.state: quiet
driver.version: 2.8.2
driver.version.data: APC HID 0.100
driver.version.internal: 0.53
driver.version.usb: libusb-1.0.0 (API: 0x1000102)
input.sensitivity: low
input.transfer.high: 295
input.transfer.low: 145
input.voltage: 232.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 295202G -302202G
ups.load: 27
ups.mfr: American Power Conversion
ups.mfr.date: 2025/02/21
ups.model: Back-UPS BX950MI
ups.productid: 0002
ups.realpower.nominal: 520
ups.serial: 9B2506A09546
ups.status: OL CHRG
ups.test.result: Done and passed
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

Where "ups.status: OL CHRG" means : UPS is OnLine (so powered) and CHaRGing.
"input.voltage.nominal: 230" means my UPS is powered.
The battery state is at 98 %, as I just executed a test, a % or two are missing.

Why all this info ?
Now you know how to question your UPS, and check in what state it is.
An UPS that signals 'UPS apc on battery' means that "NUT", the software that runs on pfSense actually communicated with the UPS, and it detected that it wasn't powered.
If it was powered, it's time to check the battery fuse, if there is one - my APC has one.
Check the battery tension. Etc.
And always : even if the UPS doesn't have a big load hooked up to it, change the battery after 36 months max. Normally, when I buy my APC's, I replace the batterie ones. After 6 years I recycle them to places where UPS has less priority. After 9 years => Land fill.

DomV

@dennypage UPS shows these values as usable:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
driver.killpower - Tell the driver daemon to initiate UPS shutdown; should be unlocked with driver.flag.allow_killpower option or variable setting
driver.reload - Reload running driver configuration from the file system (only works for changes in some options)
driver.reload-or-error - Reload running driver configuration from the file system (only works for changes in some options); return an error if something changed and could not be applied live (so the caller can restart it with new options)
driver.reload-or-exit - Reload running driver configuration from the file system (only works for changes in some options); exit the running driver if something changed and could not be applied live (so service management framework can restart it with new options)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.stop - Stop a shutdown in progress
test.battery.start.deep - Start a deep battery test
test.battery.start.quick - Start a quick battery test
test.battery.stop - Stop the battery test

Output of upsrw -l:

[battery.charge.low]
Remaining battery level when UPS switches to LB (percent)
Type: STRING
Maximum length: 10
Value: 97

[battery.mfr.date]
Battery manufacturing date
Type: STRING
Maximum length: 10
Value: 1980/01/01

[battery.runtime.low]
Remaining battery runtime when UPS switches to LB (seconds)
Type: STRING
Maximum length: 10
Value: 120

[driver.debug]
Current debug verbosity level of the driver program
Type: NUMBER
Value: 0

[driver.flag.allow_killpower]
Safety flip-switch to allow the driver daemon to send UPS shutdown command (accessible via driver.killpower)
Type: NUMBER
Value: 0

[input.sensitivity]
Input power sensitivity
Type: STRING
Maximum length: 10
Value: medium

[input.transfer.high]
High voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 298

[input.transfer.low]
Low voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 176

[ups.delay.shutdown]
Interval to wait after shutdown with delay command (seconds)
Type: STRING
Maximum length: 10
Value: 20

I tried for example changing the battery mfr date:

upsrw -s "battery.date=2025November17" -u admin "apc@192.168.1.1"
Password: 
Unexpected response from upsd: ERR ACCESS-DENIED

admin should have all the rights but somehow getting this...

DomV

@dennypage I removed all my settings an re-ran my test and it was behaving the same... shutdown was alright (default settings tho) but when it than boot up it cut the power after 4 minutes from boot for no reason. Have to mention that at that point it was in this mode: OL CHRG LB

marcg

@DomV battery.charge.low is set to 97%. Try changing it to something like 15% via upsrw or the APC PowerChute utility.

NUT triggers a shutdown when the runtime or charge drop below battery.runtime.low or battery.charge.low, respectively. With the current settings, if NUT is started with anything less than 97% charge, NUT may be shutting down the host.

For the UPS outlets turning off, there's also configuration on the UPS (websearch for "apc backups minimum battery level for restart"). You may be able to check/set those with upsrw or via the APC PowerChute utility.

DomV

I've been able to install nut on raspberry pi and with default config it works like it should... I doesn't turn the sockets off after boot... that's wierd...

marcg

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

I've been able to install nut on raspberry pi and with default config it works like it should... I doesn't turn the sockets off after boot... that's wierd...

What are the outputs of upsc and upsrw -l on the Pi?

DomV

@marcg

This is upsc on the pi:

battery.charge: 91
battery.charge.low: 97
battery.mfr.date: 1980/01/01
battery.runtime: 9836
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 27.7
battery.voltage.nominal: 24.0
device.mfr: American Power Conversion
device.model: Back-UPS BGM2200-MSX
device.serial: REDACTED
device.type: ups
driver.debug: 0
driver.flag.allow_killpower: 0
driver.name: usbhid-ups
driver.parameter.bus: 001
driver.parameter.busport: 002
driver.parameter.device: 003
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.product: Back-UPS BGM2200-MSX FW:481600G5-481700G2
driver.parameter.productid: 0002
driver.parameter.serial: REDACTED
driver.parameter.synchronous: auto
driver.parameter.vendor: American Power Conversion
driver.parameter.vendorid: 051D
driver.state: quiet
driver.version: 2.8.1
driver.version.data: APC HID 0.100
driver.version.internal: 0.52
driver.version.usb: libusb-1.0.28 (API: 0x100010a)
input.sensitivity: medium
input.transfer.high: 298
input.transfer.low: 176
input.transfer.reason: input voltage out of range
input.voltage: 241.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 481600G5-481700G2
ups.load: 0
ups.mfr: American Power Conversion
ups.mfr.date: 2025/11/17
ups.model: Back-UPS BGM2200-MSX
ups.productid: 0002
ups.realpower.nominal: 1320
ups.serial: REDACTED
ups.status: OL CHRG LB
ups.test.result: Done and passed
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

This is upsrw -l on the pi:

[battery.charge.low]
Remaining battery level when UPS switches to LB (percent)
Type: STRING
Maximum length: 10
Value: 97

[battery.mfr.date]
Battery manufacturing date
Type: STRING
Maximum length: 10
Value: 1980/01/01

[battery.runtime.low]
Remaining battery runtime when UPS switches to LB (seconds)
Type: STRING
Maximum length: 10
Value: 120

[driver.debug]
Current debug verbosity level of the driver program
Type: NUMBER
Value: 0

[driver.flag.allow_killpower]
Safety flip-switch to allow the driver daemon to send UPS shutdown command (accessible via driver.killpower)
Type: NUMBER
Value: 0

[input.sensitivity]
Input power sensitivity
Type: STRING
Maximum length: 10
Value: medium

[input.transfer.high]
High voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 298

[input.transfer.low]
Low voltage transfer point (V)
Type: STRING
Maximum length: 10
Value: 176

[ups.delay.shutdown]
Interval to wait after shutdown with delay command (seconds)
Type: STRING
Maximum length: 10
Value: 20

dennypage

@marcg said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

battery.charge.low is set to 97%. Try changing it to something like 15% via upsrw or the APC PowerChute utility.

Good eye!

This would certainly cause the behavior mentioned.

@DomV, you should reset your UPS variables to default values. Generally, you should leave the variables at default values unless you have a very good reason to change them.

You can try restoring the default values via upsrw, but if that doesn't work then check with the manual or manufacturer on how to reset your UPS.

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

I tried for example changing the battery mfr date:

upsrw -s "battery.date=2025November17" -u admin "apc@192.168.1.1"
Password:
Unexpected response from upsd: ERR ACCESS-DENIED

You should not expect this to work as battery.date is not an available variable listed by upsrw. Try battery.mfr.date.

marcg

@DomV battery.charge.low is showing as 97% on the Pi, too. You may want to lower that to prevent future issues.

DomV

@marcg Because I think that this is the default value for this UPS. And I still really don't understand why with the pi it works correctly even with this setting.

@dennypage Also, this worked upsrw -s "battery.mfr.date=2025/11/17" -u admin "apc@192.168.1.100" of course it did... ahh

dennypage

@DomV said in NUT server - After power is turned back on, pfsense shut's down sockets without warning for no reason:

Because I think that this is the default value for this UPS. And I still really don't understand why with the pi it works correctly even with this setting.

The default for low battery on APC UPSs is 10%. 97% would not make sense to use as a default setting on any UPS.

As to the Pi, there are any number of configuration reasons that would result in a change of behavior. If you want to explore that further, I recommend the NUT Users mailing list.