NUT package (2.8.0 and below)

[kevindd992002]

@dennypage

There was a blackout in my house a few days ago. I've noticed a very weird behavior with my Eaton UPS while I was actually monitoring the shutdown process. Here are the logs:

Oct 11 11:49:14 	kernel 		Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1
Oct 11 11:49:14 	kernel 		CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU)
Oct 11 11:49:14 	kernel 		VT(vga): resolution 640x480
Oct 11 11:49:14 	kernel 		FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Oct 11 11:49:14 	kernel 		FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense amd64
Oct 11 11:49:14 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Oct 11 11:49:14 	kernel 		The Regents of the University of California. All rights reserved.
Oct 11 11:49:14 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Oct 11 11:49:14 	kernel 		Copyright (c) 1992-2020 The FreeBSD Project.
Oct 11 11:49:14 	kernel 		---<<BOOT>>---
Oct 11 11:49:14 	syslogd 		kernel boot file is /boot/kernel/kernel
Oct 11 11:42:10 	kernel 		done. 
Oct 11 09:48:27 	shutdown 	45407 	power-down by root:
Oct 11 09:48:25 	upsd 	61950 	User monuser@192.168.10.10 logged into UPS [ups]
Oct 11 09:48:22 	upsmon 	18273 	Auto logout and shutdown proceeding
Oct 11 09:48:22 	upsmon 	18273 	Executing automatic power-fail shutdown
Oct 11 09:48:21 	upsd 	61950 	User monuser@192.168.10.10 logged out from UPS [ups]
Oct 11 09:47:18 	php 	27924 	nut_email.php: Message sent to kevindd992002@yahoo.com OK
Oct 11 09:47:04 	upsd 	61950 	Client local-monitor@::1 set FSD on UPS [ups]
Oct 11 09:47:04 	upsmon 	18273 	UPS ups battery is low 

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Also, at 9:48:21 I see a logout from one of the remote monitors, but why is there another login event at 9:48:25 right before the shutdown event by root at 09:48:27?

[Gertjan]

@kevindd992002 said in NUT package:

Any ideas what's causing this?

The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.

Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS.

[kevindd992002]

@gertjan said in NUT package:

@kevindd992002 said in NUT package:

Any ideas what's causing this?

The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.

Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS.

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet. So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

[dennypage]

@andreymoiseev Can't tell anything without log info. Have you tried restarting the service? (first icon on the Status line)

[Gertjan]

@kevindd992002 said in NUT package:

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.

That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.

@kevindd992002 said in NUT package:

So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.

I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".

Example : my UPS is called 'ups' in my pfSense.

[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2019/06/25
battery.runtime: 828
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.5
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 700U
device.serial: 3B1926X61525
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 300
input.transfer.low: 150
input.voltage: 232.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 924.Z5 .I
ups.firmware.aux: Z5
ups.load: 35
ups.mfr: American Power Conversion
ups.mfr.date: 2019/06/25
ups.model: Back-UPS XS 700U
ups.productid: 0002
ups.realpower.nominal: 390
ups.serial: 3B1926X61525
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

I can read these parameters but the thing is : can we set (some of) them ?

@kevindd992002 said in NUT package:

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on

That's seems the correct behaviour to me.

@kevindd992002 said in NUT package:

and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.

Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.

[kevindd992002]

@gertjan said in NUT package:

@kevindd992002 said in NUT package:

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.

That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.

Are you saying that when a blackout occurs, only the remote and local (pfsense) monitors shut down and the UPS continues to supply power to the outlets until the battery gets drained?

I have this arguments set and I'm not sure if they make a difference in the event when all monitors are already in shutdown state:

override.battery.charge.warning = 80
override.battery.charge.low = 70
override.battery.runtime.low = 420

@kevindd992002 said in NUT package:

So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.

I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".

I thought it does both. @dennypage can probably confirm but I thought NUT is able to shut down the UPS outlets. What you're saying makes sense though.

Example : my UPS is called 'ups' in my pfSense.

[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2019/06/25
battery.runtime: 828
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.5
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 700U
device.serial: 3B1926X61525
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 300
input.transfer.low: 150
input.voltage: 232.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 924.Z5 .I
ups.firmware.aux: Z5
ups.load: 35
ups.mfr: American Power Conversion
ups.mfr.date: 2019/06/25
ups.model: Back-UPS XS 700U
ups.productid: 0002
ups.realpower.nominal: 390
ups.serial: 3B1926X61525
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

I can read these parameters but the thing is : can we set (some of) them ?

I know in the past @dennypage confirmed that some of the parameters can be modified.

@kevindd992002 said in NUT package:

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on

That's seems the correct behaviour to me.

@kevindd992002 said in NUT package:

and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.

Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.

I see. So since the UPS tells NUT it is using mains power, NUT doesn't do any of its power shutdown sequences, correct?

[dennypage]

@kevindd992002 said in NUT package:

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Issues such as these seem to be common with low end Eatons...

Research the "ondelay" parameter and these variables (if your UPS has them):

• battery.charge.restart
• battery.runtime.restart
• ups.delay.start
• ups.start.battery
• ups.timer.start

Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.

In the end, you may not be able to fix it.

[kevindd992002]

@dennypage said in NUT package:

@kevindd992002 said in NUT package:

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Issues such as these seem to be common with low end Eatons...

Research the "ondelay" parameter and these variables (if your UPS has them):

• battery.charge.restart
• battery.runtime.restart
• ups.delay.start
• ups.start.battery
• ups.timer.start

Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.

In the end, you may not be able to fix it.

If I remember correctly, this is an Eaton that has 1KVA+ capacity. Are all consumer UPS Eatons considered low end?

Can you confirm what NUT does based on the discussion above? Does it send a command to the UPS to shut down its outlets during the shutdown process?

[dennypage]

@kevindd992002 Please see advice above.

You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.

You should research the variables I gave you. Particularly those with battery in their name.

If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.

[kevindd992002]

@dennypage said in NUT package:

@kevindd992002 Please see advice above.

You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.

You should research the variables I gave you. Particularly those with battery in their name.

If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.

Yes, I did see your advice above. I just wanted to confirm the confusions we had in the discussion above where we are not sure if NUT can send commands to the UPS or just read data from it. I assume it's the former but I wantes to confirm with you. I guess my question is: when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?

Aha! You're probably right. I did recently change the thresholds to a higher level and the UPS probably doesn't support that. If I remove the thresholds, I should also remove "ignorelb" so that it uses the default thresholds of the UPS, correct?

[dennypage]

@kevindd992002 I believe the only time you should use ignorelb is if you have a ups that incorrectly asserts low battery immediately when mains fail.

Edit: This is incorrect. Using ignorelb is required in order for battery.charge.low and battery.runtime.low to be checked. Code reference here.

[AndreyMoiseev]

@dennypage said in NUT package:

Can't tell anything without log info. Have you tried restarting the service? (first icon on the Status line)
reboot didn't help , unfortunately now there is no time for experiments, I replaced ippon with Apc, everything worked without problems

[kevindd992002]

@dennypage said in NUT package:

@kevindd992002 I believe the only time you should use ignorelb is if you have a ups that incorrectly asserts low battery immediately when mains fail.

If I remember correctly, when we were discussing this a few years ago, we came to a conclusion that ignorelb is needed for the warning/low battery thresholds override to work. Is this not the case anymore?

And my pending question again: "when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?"

[Gertjan]

@kevindd992002 said in NUT package:

when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?

NUT, the server, the one that is connected to one or multiple 'local' UPS, polls the status of the UPS. Somthing like every 5 seconds : the UPS doesn't send anything.

The NUT server doesn't send to it's clients. The clients do the same thing : they poll the NUT server every xx seconds.

So, NUT as a process can have a 'local UPS' or a remote, other NUT process as a souce. This means that the code for NUT as a server or as a client is the same. Only the source changes.

A local UPS isn't necessary a local serial or USB device, it can even be a IP based SNMP or any other remotely access type of thing.

I've read a lot of the manual yesterday ( the source code, as it is available for everybody ) and i'm pretty sure NUT server and client are only polling the 'upstream' info. This can be another NUT installation, or an UPS.
( still not 100 % convinced, though )

Btw : NUT won't shut down the UPS.
The UPS exposes variables in it's status record that indicate what the battery situation is ( a percentage) and a value where it goes "down". That will be the moment that the outlets go down.
Up to NUT to to it's thing : shutting down locally.
So, NUT itself will never 'see' that the UPS goes down. As it shuts down the device it's running ion => it shuts down itself ;)

Also : NUT is not a program, it a collections of device and interface dependant drivers, server & client programs, tools etc.

And again : I'm not a NUT specialist at all, but I have a couple of UPSs around me for ages now. @work and @home.

[kevindd992002]

@gertjan said in NUT package:

@kevindd992002 said in NUT package:

when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?

NUT, the server, the one that is connected to one or multiple 'local' UPS, polls the status of the UPS. Somthing like every 5 seconds : the UPS doesn't send anything.

The NUT server doesn't send to it's clients. The clients do the same thing : they poll the NUT server every xx seconds.

So, NUT as a process can have a 'local UPS' or a remote, other NUT process as a souce. This means that the code for NUT as a server or as a client is the same. Only the source changes.

A local UPS isn't necessary a local serial or USB device, it can even be a IP based SNMP or any other remotely access type of thing.

I've read a lot of the manual yesterday ( the source code, as it is available for everybody ) and i'm pretty sure NUT server and client are only polling the 'upstream' info. This can be another NUT installation, or an UPS.
( still not 100 % convinced, though )

Btw : NUT won't shut down the UPS.
The UPS exposes variables in it's status record that indicate what the battery situation is ( a percentage) and a value where it goes "down". That will be the moment that the outlets go down.
Up to NUT to to it's thing : shutting down locally.
So, NUT itself will never 'see' that the UPS goes down. As it shuts down the device it's running ion => it shuts down itself ;)

Also : NUT is not a program, it a collections of device and interface dependant drivers, server & client programs, tools etc.

And again : I'm not a NUT specialist at all, but I have a couple of UPSs around me for ages now. @work and @home.

I see, thanks for the detailed explanation. So as soon as pfsense (my NUT server), shuts down, it's all up to the UPS when to shut off power from its outlets? When does this usually happen? If I remember correctly, the UPS does not wait for its battery to be depleted before doing this. If that's the case, how does the UPS know when to do this? At what point, after a blackout, does it happen?

[Gertjan]

@kevindd992002 said in NUT package:

it's all up to the UPS when to shut off power from its outlets?

Yep.
Mostly to protect battery, as discharging it to much will affect the life span.
I guess there isn't even a relay or switch on the outlets of an UPS, the DC to AC convert, just before the transfo, it just stops.

@kevindd992002 said in NUT package:

When does this usually happen?

The internal logic of the UPS maintains a table of battery tension and current over time. This tables helps to predict the battery behaviour over time. Everything is done to get a maximum of load unload cycles out of a battery.
Classic Lead accid batteries have far less then 1000 cycles.

@kevindd992002 said in NUT package:

not wait for its battery to be depleted before doing this

Noop. The UPS protects it's own battery for doing that. Depending on the type of battery used, depleting it will kill it.

[dennypage]

@gertjan At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).

[Gertjan]

@dennypage said in NUT package:

At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).

Aha : I hope there is some known delay then, because when NUT sends that command, the host system is still running.
System can take their time to shut down (to really power or cold mode).

Thanks, because you said so, my brain started to help me to finding this : https://networkupstools.org/docs/user-manual.pdf page 35 and further on "NUT outlets management".

[dennypage]

@kevindd992002 said in NUT package:

If I remember correctly, when we were discussing this a few years ago, we came to a conclusion that ignorelb is needed for the warning/low battery thresholds override to work.

You are correct. Thank you for reminding me.

Ignorelb code reference here.

[kevindd992002]

@gertjan said in NUT package:

@dennypage said in NUT package:

At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).

Aha : I hope there is some known delay then, because when NUT sends that command, the host system is still running.
System can take their time to shut down (to really power or cold mode).

Thanks, because you said so, my brain started to help me to finding this : https://networkupstools.org/docs/user-manual.pdf page 35 and further on "NUT outlets management".

Yes, that's the ups.shutdown.delay that you see in your post above. It's also what @dennypage mentioned to me in the past.

Your APC and both my APC and EATON UPS'es have a default value of 20 seconds.