-
There was a blackout in my house a few days ago. I've noticed a very weird behavior with my Eaton UPS while I was actually monitoring the shutdown process. Here are the logs:
Oct 11 11:49:14 kernel Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 Oct 11 11:49:14 kernel CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU) Oct 11 11:49:14 kernel VT(vga): resolution 640x480 Oct 11 11:49:14 kernel FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) Oct 11 11:49:14 kernel FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense amd64 Oct 11 11:49:14 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Oct 11 11:49:14 kernel The Regents of the University of California. All rights reserved. Oct 11 11:49:14 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Oct 11 11:49:14 kernel Copyright (c) 1992-2020 The FreeBSD Project. Oct 11 11:49:14 kernel ---<<BOOT>>--- Oct 11 11:49:14 syslogd kernel boot file is /boot/kernel/kernel Oct 11 11:42:10 kernel done. Oct 11 09:48:27 shutdown 45407 power-down by root: Oct 11 09:48:25 upsd 61950 User monuser@192.168.10.10 logged into UPS [ups] Oct 11 09:48:22 upsmon 18273 Auto logout and shutdown proceeding Oct 11 09:48:22 upsmon 18273 Executing automatic power-fail shutdown Oct 11 09:48:21 upsd 61950 User monuser@192.168.10.10 logged out from UPS [ups] Oct 11 09:47:18 php 27924 nut_email.php: Message sent to kevindd992002@yahoo.com OK Oct 11 09:47:04 upsd 61950 Client local-monitor@::1 set FSD on UPS [ups] Oct 11 09:47:04 upsmon 18273 UPS ups battery is lowThe weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?
Also, at 9:48:21 I see a logout from one of the remote monitors, but why is there another login event at 9:48:25 right before the shutdown event by root at 09:48:27?
-
@kevindd992002 said in NUT package:
Any ideas what's causing this?
The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS. -
@gertjan said in NUT package:
@kevindd992002 said in NUT package:
Any ideas what's causing this?
The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS.Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet. So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?
I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.
-
@andreymoiseev Can't tell anything without log info. Have you tried restarting the service? (first icon on the Status line)
-
@kevindd992002 said in NUT package:
Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.
That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.@kevindd992002 said in NUT package:
So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?
I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".
Example : my UPS is called 'ups' in my pfSense.
[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost battery.charge: 100 battery.charge.low: 10 battery.charge.warning: 50 battery.date: 2001/09/25 battery.mfr.date: 2019/06/25 battery.runtime: 828 battery.runtime.low: 120 battery.type: PbAc battery.voltage: 13.5 battery.voltage.nominal: 12.0 device.mfr: American Power Conversion device.model: Back-UPS XS 700U device.serial: 3B1926X61525 device.type: ups driver.name: usbhid-ups driver.parameter.pollfreq: 30 driver.parameter.pollinterval: 2 driver.parameter.port: auto driver.parameter.synchronous: no driver.version: 2.7.4 driver.version.data: APC HID 0.96 driver.version.internal: 0.41 input.sensitivity: medium input.transfer.high: 300 input.transfer.low: 150 input.voltage: 232.0 input.voltage.nominal: 230 ups.beeper.status: disabled ups.delay.shutdown: 20 ups.firmware: 924.Z5 .I ups.firmware.aux: Z5 ups.load: 35 ups.mfr: American Power Conversion ups.mfr.date: 2019/06/25 ups.model: Back-UPS XS 700U ups.productid: 0002 ups.realpower.nominal: 390 ups.serial: 3B1926X61525 ups.status: OL ups.test.result: No test initiated ups.timer.reboot: 0 ups.timer.shutdown: -1 ups.vendorid: 051dI can read these parameters but the thing is : can we set (some of) them ?
@kevindd992002 said in NUT package:
I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on
That's seems the correct behaviour to me.
@kevindd992002 said in NUT package:
and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.
The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.
-
@gertjan said in NUT package:
@kevindd992002 said in NUT package:
Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.
That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.Are you saying that when a blackout occurs, only the remote and local (pfsense) monitors shut down and the UPS continues to supply power to the outlets until the battery gets drained?
I have this arguments set and I'm not sure if they make a difference in the event when all monitors are already in shutdown state:
override.battery.charge.warning = 80
override.battery.charge.low = 70
override.battery.runtime.low = 420@kevindd992002 said in NUT package:
So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?
I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".
I thought it does both. @dennypage can probably confirm but I thought NUT is able to shut down the UPS outlets. What you're saying makes sense though.
Example : my UPS is called 'ups' in my pfSense.
[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost battery.charge: 100 battery.charge.low: 10 battery.charge.warning: 50 battery.date: 2001/09/25 battery.mfr.date: 2019/06/25 battery.runtime: 828 battery.runtime.low: 120 battery.type: PbAc battery.voltage: 13.5 battery.voltage.nominal: 12.0 device.mfr: American Power Conversion device.model: Back-UPS XS 700U device.serial: 3B1926X61525 device.type: ups driver.name: usbhid-ups driver.parameter.pollfreq: 30 driver.parameter.pollinterval: 2 driver.parameter.port: auto driver.parameter.synchronous: no driver.version: 2.7.4 driver.version.data: APC HID 0.96 driver.version.internal: 0.41 input.sensitivity: medium input.transfer.high: 300 input.transfer.low: 150 input.voltage: 232.0 input.voltage.nominal: 230 ups.beeper.status: disabled ups.delay.shutdown: 20 ups.firmware: 924.Z5 .I ups.firmware.aux: Z5 ups.load: 35 ups.mfr: American Power Conversion ups.mfr.date: 2019/06/25 ups.model: Back-UPS XS 700U ups.productid: 0002 ups.realpower.nominal: 390 ups.serial: 3B1926X61525 ups.status: OL ups.test.result: No test initiated ups.timer.reboot: 0 ups.timer.shutdown: -1 ups.vendorid: 051dI can read these parameters but the thing is : can we set (some of) them ?
I know in the past @dennypage confirmed that some of the parameters can be modified.
@kevindd992002 said in NUT package:
I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on
That's seems the correct behaviour to me.
@kevindd992002 said in NUT package:
and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.
The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.
I see. So since the UPS tells NUT it is using mains power, NUT doesn't do any of its power shutdown sequences, correct?
-
@kevindd992002 said in NUT package:
The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?
Issues such as these seem to be common with low end Eatons...
Research the "ondelay" parameter and these variables (if your UPS has them):
- battery.charge.restart
- battery.runtime.restart
- ups.delay.start
- ups.start.battery
- ups.timer.start
Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.
In the end, you may not be able to fix it.
-
@dennypage said in NUT package:
@kevindd992002 said in NUT package:
The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?
Issues such as these seem to be common with low end Eatons...
Research the "ondelay" parameter and these variables (if your UPS has them):
- battery.charge.restart
- battery.runtime.restart
- ups.delay.start
- ups.start.battery
- ups.timer.start
Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.
In the end, you may not be able to fix it.
If I remember correctly, this is an Eaton that has 1KVA+ capacity. Are all consumer UPS Eatons considered low end?
Can you confirm what NUT does based on the discussion above? Does it send a command to the UPS to shut down its outlets during the shutdown process?
-
@kevindd992002 Please see advice above.
You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.
You should research the variables I gave you. Particularly those with battery in their name.
If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.
-
@dennypage said in NUT package:
@kevindd992002 Please see advice above.
You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.
You should research the variables I gave you. Particularly those with battery in their name.
If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.
Yes, I did see your advice above. I just wanted to confirm the confusions we had in the discussion above where we are not sure if NUT can send commands to the UPS or just read data from it. I assume it's the former but I wantes to confirm with you. I guess my question is: when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?
Aha! You're probably right. I did recently change the thresholds to a higher level and the UPS probably doesn't support that. If I remove the thresholds, I should also remove "ignorelb" so that it uses the default thresholds of the UPS, correct?
-
@kevindd992002 I believe the only time you should use ignorelb is if you have a ups that incorrectly asserts low battery immediately when mains fail.
Edit: This is incorrect. Using ignorelb is required in order for battery.charge.low and battery.runtime.low to be checked. Code reference here.
-
@dennypage said in NUT package:
Can't tell anything without log info. Have you tried restarting the service? (first icon on the Status line)
reboot didn't help , unfortunately now there is no time for experiments, I replaced ippon with Apc, everything worked without problems -
@dennypage said in NUT package:
@kevindd992002 I believe the only time you should use ignorelb is if you have a ups that incorrectly asserts low battery immediately when mains fail.
If I remember correctly, when we were discussing this a few years ago, we came to a conclusion that ignorelb is needed for the warning/low battery thresholds override to work. Is this not the case anymore?
And my pending question again: "when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?"
-
@kevindd992002 said in NUT package:
when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?
NUT, the server, the one that is connected to one or multiple 'local' UPS, polls the status of the UPS. Somthing like every 5 seconds : the UPS doesn't send anything.
The NUT server doesn't send to it's clients. The clients do the same thing : they poll the NUT server every xx seconds.
So, NUT as a process can have a 'local UPS' or a remote, other NUT process as a souce. This means that the code for NUT as a server or as a client is the same. Only the source changes.
A local UPS isn't necessary a local serial or USB device, it can even be a IP based SNMP or any other remotely access type of thing.
I've read a lot of the manual yesterday ( the source code, as it is available for everybody ) and i'm pretty sure NUT server and client are only polling the 'upstream' info. This can be another NUT installation, or an UPS.
( still not 100 % convinced, though )Btw : NUT won't shut down the UPS.
The UPS exposes variables in it's status record that indicate what the battery situation is ( a percentage) and a value where it goes "down". That will be the moment that the outlets go down.
Up to NUT to to it's thing : shutting down locally.
So, NUT itself will never 'see' that the UPS goes down. As it shuts down the device it's running ion => it shuts down itself ;)Also : NUT is not a program, it a collections of device and interface dependant drivers, server & client programs, tools etc.
And again : I'm not a NUT specialist at all, but I have a couple of UPSs around me for ages now. @work and @home.
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@gertjan said in NUT package:
@kevindd992002 said in NUT package:
when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?
NUT, the server, the one that is connected to one or multiple 'local' UPS, polls the status of the UPS. Somthing like every 5 seconds : the UPS doesn't send anything.
The NUT server doesn't send to it's clients. The clients do the same thing : they poll the NUT server every xx seconds.
So, NUT as a process can have a 'local UPS' or a remote, other NUT process as a souce. This means that the code for NUT as a server or as a client is the same. Only the source changes.
A local UPS isn't necessary a local serial or USB device, it can even be a IP based SNMP or any other remotely access type of thing.
I've read a lot of the manual yesterday ( the source code, as it is available for everybody ) and i'm pretty sure NUT server and client are only polling the 'upstream' info. This can be another NUT installation, or an UPS.
( still not 100 % convinced, though )Btw : NUT won't shut down the UPS.
The UPS exposes variables in it's status record that indicate what the battery situation is ( a percentage) and a value where it goes "down". That will be the moment that the outlets go down.
Up to NUT to to it's thing : shutting down locally.
So, NUT itself will never 'see' that the UPS goes down. As it shuts down the device it's running ion => it shuts down itself ;)Also : NUT is not a program, it a collections of device and interface dependant drivers, server & client programs, tools etc.
And again : I'm not a NUT specialist at all, but I have a couple of UPSs around me for ages now. @work and @home.
I see, thanks for the detailed explanation. So as soon as pfsense (my NUT server), shuts down, it's all up to the UPS when to shut off power from its outlets? When does this usually happen? If I remember correctly, the UPS does not wait for its battery to be depleted before doing this. If that's the case, how does the UPS know when to do this? At what point, after a blackout, does it happen?
-
@kevindd992002 said in NUT package:
it's all up to the UPS when to shut off power from its outlets?
Yep.
Mostly to protect battery, as discharging it to much will affect the life span.
I guess there isn't even a relay or switch on the outlets of an UPS, the DC to AC convert, just before the transfo, it just stops.@kevindd992002 said in NUT package:
When does this usually happen?
The internal logic of the UPS maintains a table of battery tension and current over time. This tables helps to predict the battery behaviour over time. Everything is done to get a maximum of load unload cycles out of a battery.
Classic Lead accid batteries have far less then 1000 cycles.@kevindd992002 said in NUT package:
not wait for its battery to be depleted before doing this
Noop. The UPS protects it's own battery for doing that. Depending on the type of battery used, depleting it will kill it.
-
@gertjan At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).
-
@dennypage said in NUT package:
At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).
Aha : I hope there is some known delay then, because when NUT sends that command, the host system is still running.
System can take their time to shut down (to really power or cold mode).Thanks, because you said so, my brain started to help me to finding this : https://networkupstools.org/docs/user-manual.pdf page 35 and further on "NUT outlets management".
-
@kevindd992002 said in NUT package:
If I remember correctly, when we were discussing this a few years ago, we came to a conclusion that ignorelb is needed for the warning/low battery thresholds override to work.
You are correct. Thank you for reminding me.
Ignorelb code reference here.
-
@gertjan said in NUT package:
@dennypage said in NUT package:
At the end of the shutdown process, the NUT master does send a command to the UPS to kill the power (after a delay).
Aha : I hope there is some known delay then, because when NUT sends that command, the host system is still running.
System can take their time to shut down (to really power or cold mode).Thanks, because you said so, my brain started to help me to finding this : https://networkupstools.org/docs/user-manual.pdf page 35 and further on "NUT outlets management".
Yes, that's the ups.shutdown.delay that you see in your post above. It's also what @dennypage mentioned to me in the past.
Your APC and both my APC and EATON UPS'es have a default value of 20 seconds.
