NUT Package (2.8.1 and above)

GeneralGresi

Hi guys,

I have a very similar issue, however, not during self tests or something like that, but during "normal" powerfailures too.
I get about the same error messages though, and I can confirm that the UPS is in good state, and the batteries are well too, I also tried with a different OS, with an older NUT Version, where the problem does not occur.
Think is, once power from the wall gets lost, I get the following messages, before the pfsense and all slaves immediately shut down +- 30 Seconds (with >2h of runtime left according to the UPS!)

Jan 25 18:33:08 snmp-ups 53155 Requesting UPS [Eaton_5PX2200] to power off, as/if handled by its driver by default (may exit), due to socket protocol request
Jan 25 18:33:03 upsmon 89442 Auto logout and shutdown proceeding
Jan 25 18:33:03 upsmon 89442 Executing automatic power-fail shutdown
Jan 25 18:33:03 upsd 22669 Client local-monitor@127.0.0.1 set FSD on UPS [Eaton_5PX2200]
Jan 25 18:33:03 upsmon 89442 UPS [Eaton_5PX2200] is reported as (administratively) OFF
Jan 25 18:33:03 upsmon 89442 UPS Eaton_5PX2200: administratively OFF or asleep
Jan 25 18:33:03 upsmon 89442 UPS Eaton_5PX2200 on battery

The UPS Logs are as follows:
2024/01/25 18:32:52 Normal AC NOK
2024/01/25 18:32:52 ABM state resting
2024/01/25 18:32:52 UPS on battery
2024/01/25 18:32:53 System shutdown in 2 h 17 mn 16 s
2024/01/25 18:32:53 Outlet group 1 shutdown in 2 h 49 mn 35 s
2024/01/25 18:32:53 Outlet group 2 shutdown in 2 h 49 mn 35 s
2024/01/25 18:32:53 ABM state Off
2024/01/25 18:32:53 Normal AC voltage out of tolerance

I have replaced the two files mentioned in the above posts, however, I don't think that solved the issue, cause this is not during self test mode nor calibration mode, this is a quite normal condition.

In addition: I updated my pfsense box from 2.7.0 to 2.7.2 just one or two weeks ago, about 3 weeks before I had a poweroutage of about 2 hours, which was handled as usual, no unexpected shutdowns, everything ran through.
Now with that version, it takes under a minute and everything wents black.

I ran upsc on a different machine, and it seems like during a power Outage, the Eaton 5PX Reports the Status "OFF".
ups.status: OB OFF OB

Any ideas?

dennypage

@GeneralGresi said in NUT Package (2.8.1 and above):

Jan 25 18:33:08 snmp-ups 53155 Requesting UPS [Eaton_5PX2200] to power off, as/if handled by its driver by default (may exit), due to socket protocol request
Jan 25 18:33:03 upsmon 89442 Auto logout and shutdown proceeding
Jan 25 18:33:03 upsmon 89442 Executing automatic power-fail shutdown
Jan 25 18:33:03 upsd 22669 Client local-monitor@127.0.0.1 set FSD on UPS [Eaton_5PX2200]
Jan 25 18:33:03 upsmon 89442 UPS [Eaton_5PX2200] is reported as (administratively) OFF
Jan 25 18:33:03 upsmon 89442 UPS Eaton_5PX2200: administratively OFF or asleep
Jan 25 18:33:03 upsmon 89442 UPS Eaton_5PX2200 on battery

This appears to be the failsafe when loosing connection to the UPS while on battery. This could either be a communication issue between the host and the UPS, or a bug in the snmp-ups driver.

I haven't seen the bug with the snmp-ups driver, but of course that doesn't mean it isn't there. There is a pending update to the NUT package. If you want to try an updated file in the interim, I've attached a newer snmp-ups to this post.

snmp-ups.gz

GeneralGresi

@dennypage

Thank's for the fast reply.
Replaced the new snmp-ups file, and restarted the service, however still the same issue.
I unplug the UPS for testing purposes, wait about 30-60 Seconds and the machine shuts down.
I was able to gather a screenshot from the status page:

This looks nearly the same as with a nut 2.8.0 Installation on a Bebian machine during a power loss situation, however, without the forced shutdown Flag.
(nut 2.8.0: ups.status: OB OFF OB)
I would say as this happens concurrently for 5 testruns now, I would rule out the communication loss theory, especially because everything else, including the Debian boxes stay alive - even if I let a longer time pass (as it should be).

I don't know much about the underlyings of nut, however I'd say it's the OFF Flag coming from the UPS via SNMP in a power loss situation?
As the UPS doesn't report any error in it's webinterface, I would not see the UPS itself as the reason.

dennypage

@GeneralGresi said in NUT Package (2.8.1 and above):

I would say as this happens concurrently for 5 testruns now, I would rule out the communication loss theory, especially because everything else, including the Debian boxes stay alive - even if I let a longer time pass (as it should be).

I don't know much about the underlyings of nut, however I'd say it's the OFF Flag coming from the UPS via SNMP in a power loss situation?

You cannot put too much into the Debian boxes not noticing or caring. They may still be on NUT 2.7.4. NUT 2.8.0 and above are much more attentive and sensitive to some UPS notifications.

Only other thing I can think of is to explore the new OFFDURATION configuration option. This would go into the Additional configuration lines for upsmon.conf section.

hbastasic

@dennypage said in NUT Package (2.8.1 and above):

For those of you that are on amd64 based systems (Intel or AMD), and are severely affected by the shutdown on calibration/self-test issue, attached are replacement versions of upsmon and usbhid-ups that you can use until the update is published.

Note that these files are for amd64 systems only. I do not have a build/test system for arm. Sorry!

upsmon.gz

usbhid-ups.gz

Having the shutdown bc. calibration issue, i guess, on APC Back-UPS ES 850G2

Event log:
Jan 27 19:40:41 localhost usbhid-ups[580]: ups_status_set: seems that UPS [ups] is in OL+DISCHRG state now. Is it calibrating or do you perhaps want to set 'onlinedischarge' option? Some UPS models (e.g. CyberPower UT series) emit OL+DISCHRG when offline.
Jan 27 19:40:43 localhost usbhid-ups[580]: ups_status_set: seems that UPS [ups] is in OL+DISCHRG state now. Is it calibrating or do you perhaps want to set 'onlinedischarge' option? Some UPS models (e.g. CyberPower UT series) emit OL+DISCHRG when offline.
...
Jan 27 19:40:46 localhost upsmon[95193]: Host sync timer expired, forcing shutdown
Jan 27 19:40:46 localhost upsmon[95193]: Executing automatic power-fail shutdown
Jan 27 19:40:46 localhost upsmon[95193]: Auto logout and shutdown proceeding
...
Jan 27 19:40:51 localhost shutdown[20307]: power-down by root:
Jan 27 19:40:51 localhost usbhid-ups[580]: sock_connect: enabling asynchronous mode (auto)
Jan 27 19:40:51 localhost kernel:
Jan 27 19:40:51 localhost kernel: Netgate pfSense Plus is now shutting down ...

@dennypage
Should i use the replacement files?

dennypage

@hbastasic said in NUT Package (2.8.1 and above):

Should i use the replacement files?

Up to you. If it were my system, I would.

pfguy2018

@NinthWave said in NUT Package (2.8.1 and above):

Move the files in appropriate directories and make sure to have appropriate file permission or CHMOD 755
@dennypage said in NUT Package (2.8.1 and above):

[23.09.1-RELEASE][root@fw]/root: ls -l /usr/local/sbin/upsmon* /usr/local/libexec/nut/usbhid-ups*
-rwxr-xr-x 1 root wheel 333728 Dec 27 10:14 /usr/local/libexec/nut/usbhid-ups
-rwxr-xr-x 1 root wheel 287088 Nov 1 00:57 /usr/local/libexec/nut/usbhid-ups.org
-rwxr-xr-x 1 root wheel 87760 Dec 27 10:13 /usr/local/sbin/upsmon
-rwxr-xr-x 1 root wheel 68904 Nov 1 00:57 /usr/local/sbin/upsmon.org
[23.09.1-RELEASE][root@fw]/root:

Restart service and voilà

I have been waiting for the "official" patch to be released, but am experiencing frequent shutdowns as a result of this issue, so I would like to update manually. But I am not at all versed in how to accomplish the 2 steps outlined above ("Move the files in appropriate directories and make sure to have appropriate file permission or CHMOD 755") (I have been able to download and extract the archives). Would it be possible for someone to provide the specific commands needed, and where exactly to enter them in pfSense, to accomplish these steps?

Thanks in advance!

hbastasic

@dennypage said in NUT Package (2.8.1 and above):

@hbastasic said in NUT Package (2.8.1 and above):

Should i use the replacement files?

Up to you. If it were my system, I would.

Thanks @dennypage for prompt reply, replaced the files, NUT started and everything works ok. Now just have to wait for the APC's next battery test.

btw, is there a way to invoke a battery test from pfSense, shell command or something?

pfpv

@hbastasic said in NUT Package (2.8.1 and above):

is there a way to invoke a battery test from pfSense, shell command or something?

Usually you can initiate it by pressing the physical power button on the UPS for 10 seconds or until the UPS switches to battery mode indicating the test has started. This is kind of dangerous. If you release the button earlier the UPS will turn off. This can happen unintentionally - if the button is wearing off or if you flinch. Some info: https://www.apc.com/us/en/faqs/FA405317/

I also found this from 2019, I don't know if it still works:

@dennypage said in NUT package (2.8.0 and below):

You can initiate a battery test via nut, but you will have to use the command line. Log into the system and use
upscmd -l ups
to see what commands are available. Look for commands that begin "test.battery..." Start with a quick test if available, then proceed to a deep test.

WARNING if the battery is defective, running these tests can cause the ups to cut power to the load (your pfSense box). Use at your own risk!

hbastasic

@pfpv said in NUT Package (2.8.1 and above):

@hbastasic said in NUT Package (2.8.1 and above):

is there a way to invoke a battery test from pfSense, shell command or something?

Usually you can initiate it by pressing the physical power button on the UPS for 10 seconds or until the UPS switches to battery mode indicating the test has started. This is kind of dangerous. If you release the button earlier the UPS will turn off. This can happen unintentionally - if the button is wearing off or if you flinch. I also found this from 2019, I don't know if it still works:

well, thanks, forgot to check the manual, APC FAQ, although they mention only SmartUPS line nothing on BackUPS

@dennypage said in NUT package (2.8.0 and below):
You can initiate a battery test via nut, but you will have to use the command line. Log into the system and use
upscmd -l ups
to see what commands are available. Look for commands that begin "test.battery..." Start with a quick test if available, then proceed to a deep test.

WARNING if the battery is defective, running these tests can cause the ups to cut power to the load (your pfSense box). Use at your own risk!

command line seems more convenient, thanks for this find, waiting for wife's all clear to start screwing with internet ...

pfpv

@hbastasic, find a manual for your particular model. I found this for two random APC Back-UPS's:

Press and hold the POWER ON/OFF button for 4 to 8 seconds to initiate the UPS Self Test.
During On Line Mode, a longer press of the POWER button until three beeps are heard will perform a manual battery self-test. Then, the LED will flash and UPS will enter self-test mode. Note: This will happen only when battery is fully charged in On Line Mode.

pfguy2018

@pfguy2018 said in NUT Package (2.8.1 and above):

@NinthWave said in NUT Package (2.8.1 and above):

Move the files in appropriate directories and make sure to have appropriate file permission or CHMOD 755
@dennypage said in NUT Package (2.8.1 and above):

[23.09.1-RELEASE][root@fw]/root: ls -l /usr/local/sbin/upsmon* /usr/local/libexec/nut/usbhid-ups*
-rwxr-xr-x 1 root wheel 333728 Dec 27 10:14 /usr/local/libexec/nut/usbhid-ups
-rwxr-xr-x 1 root wheel 287088 Nov 1 00:57 /usr/local/libexec/nut/usbhid-ups.org
-rwxr-xr-x 1 root wheel 87760 Dec 27 10:13 /usr/local/sbin/upsmon
-rwxr-xr-x 1 root wheel 68904 Nov 1 00:57 /usr/local/sbin/upsmon.org
[23.09.1-RELEASE][root@fw]/root:

Restart service and voilà

I have been waiting for the "official" patch to be released, but am experiencing frequent shutdowns as a result of this issue, so I would like to update manually. But I am not at all versed in how to accomplish the 2 steps outlined above ("Move the files in appropriate directories and make sure to have appropriate file permission or CHMOD 755") (I have been able to download and extract the archives). Would it be possible for someone to provide the specific commands needed, and where exactly to enter them in pfSense, to accomplish these steps?

Thanks in advance!

I have tried to figure out how to do this. I think I have moved the files into the correc directories, but I have no idea how to run "CHMOD 755" and now NUT won't start. Can someone please walk me through this?

hbastasic

@pfpv said in NUT Package (2.8.1 and above):

@hbastasic, find a manual for your particular model. I found this for two random APC Back-UPS's:

Press and hold the POWER ON/OFF button for 4 to 8 seconds to initiate the UPS Self Test.

During On Line Mode, a longer press of the POWER button until three beeps are heard will perform a manual battery self-test. Then, the LED will flash and UPS will enter self-test mode. Note: This will happen only when battery is fully charged in On Line Mode.

yep, found it, thanks
RTFM is still current, and somehow comforting or ... shaming

will report after self-test, just for reference

endy66

Short question about client / slave logins.
I have two different client machines I wanna read the UPS values. UPS is ofc connected to pfSense and works.
So for a longer time I already had a client entry withing the "Additional configuration lines for upsd.users" field:

[client-xx1]
password = mypass
upsmon slave

Now for better log reading, I've added a second entry, for my second client

[client-xx2]
password = mypass
upsmon slave

I'm able to connect, however I can't see that this second client has connected from within the system logs in my pfSense. Does someone know why that is?

dennypage

@endy66 said in NUT Package (2.8.1 and above):

Now for better log reading, I've added a second entry, for my second client
...
I'm able to connect, however I can't see that this second client has connected from within the system logs in my pfSense. Does someone know why that is?

No idea. Are you sure it is connecting with the username you expect? FWIW, you can distinguish client by IP address. Do you see a connection from the IP address you expect?

endy66

@dennypage said in NUT Package (2.8.1 and above):

@endy66 said in NUT Package (2.8.1 and above):

Now for better log reading, I've added a second entry, for my second client
...
I'm able to connect, however I can't see that this second client has connected from within the system logs in my pfSense. Does someone know why that is?

No idea. Are you sure it is connecting with the username you expect? FWIW, you can distinguish client by IP address. Do you see a connection from the IP address you expect?

After a few restarts of the NUT service on my pfSense it's there in the log. However the second login doesn't have an explicit IP address.

upsd	81154	User client-xx1@192.168.1.xx logged into UPS [CyberPower_USV]
upsd	81154	User client-xx2@::1 logged into UPS [CyberPower_USV]

dennypage

@endy66 said in NUT Package (2.8.1 and above):

After a few restarts of the NUT service on my pfSense it's there in the log. However the second login doesn't have an explicit IP address.

upsd 81154 User client-xx1@192.168.1.xx logged into UPS [CyberPower_USV]
upsd 81154 User client-xx2@::1 logged into UPS [CyberPower_USV]

"::1" is an IP address. It's the IPv6 version of localhost, equivalent to 127.0.0.1 in IPv4.

endy66

@dennypage said in NUT Package (2.8.1 and above):

@endy66 said in NUT Package (2.8.1 and above):

After a few restarts of the NUT service on my pfSense it's there in the log. However the second login doesn't have an explicit IP address.

upsd 81154 User client-xx1@192.168.1.xx logged into UPS [CyberPower_USV]
upsd 81154 User client-xx2@::1 logged into UPS [CyberPower_USV]

"::1" is an IP address. It's the IPv6 version of localhost, equivalent to 127.0.0.1 in IPv4.

Hm this is weird because this user is connecting from a different host, so shouldn't it be anything other than a "localhost" address?

dennypage

@endy66 said in NUT Package (2.8.1 and above):

Hm this is weird because this user is connecting from a different host, so shouldn't it be anything other than a "localhost" address?

Referring to post #2 in this thread, how are you allowing remote connections? Are you using option 1 (NAT/Port forward)? Or are you using option 2 (LISTEN)?

endy66

@dennypage said in NUT Package (2.8.1 and above):

@endy66 said in NUT Package (2.8.1 and above):

Hm this is weird because this user is connecting from a different host, so shouldn't it be anything other than a "localhost" address?

Referring to post #2 in this thread, how are you allowing remote connections? Are you using option 1 (NAT/Port forward)? Or are you using option 2 (LISTEN)?

I'm using option 2 (LISTEN).