NUT Package (2.8.1 and above)

dennypage

@pfpv said in NUT Package (2.8.1 and above):

I've had this setup for a couple of years without issues. I recently updated pfSense Plus to 23.09.1 and I guess this updated the nut package to 2.8.2.

Today my pfSense suddenly shut down. I got a message in Pushover "Auto logout and shutdown proceeding". I didn't understand what was going on as there was no power failure. I ran to the basement and everything was on except pfSense. The UPS's fan was on, so I guess the UPS did a self-test. I noticed before that my Back-UPS RS 1500G and other similar units do self-tests from time to time and those that have fans turn them on for some time. This never caused a problem.

A shutdown being initiated during a self test usually indicates that the batteries are failing. How old are the batteries? UPS batteries need to be replaced every three years or so.

pfpv

@dennypage You misunderstood the issue. Please re-read the posts and the link I provided. There was no shutdown initiated. The batteries are only about 1 year old. The Synology didn't even log anything.

Have you seen this message from a UPS before "administratively OFF or asleep"? In the link I posted it is explained that sometimes APC UPS's send an OFF message during self tests and at the same time they send NOTOFF. Before NUT 2.8.2 the OFF message was ignored. Now it causes a shutdown.

Interestingly, NUT is only at 2.8.1 version itself: https://github.com/networkupstools/nut/releases Where did the 2.8.2 come from?

We need to roll back to avoid APC UPS unexpected shutdowns. Is there a way to roll back?

dennypage

@pfpv I you would like my recommendation on a solution, the first thing I would do is connect the UPS to a system other than a Synology as Synology has a rather messed up NUT implementation, and only support Synology systems as clients.

In my opinion, it would be much better to use pfSense or a linux system as the server and have the Synology be a client. FWIW, after doing this I would also recommend a run-time calibration, regardless of how old your batteries are.

In answer to your questions about versions: The version of NUT is actually FreeBSD's nut-devel-2023.10.07_1, which is based on git just prior to the 2.8.1 release. You can see this on the Installed Packages screen. The upper level package, pfSense-pkg-nut-2.8.2, is labeled as 2.8.2 because of a mistake being made at the moment of package publication. This is mentioned earlier in this thread. The published package should have been pfSense-pkg-nut-2.8.1_1, but once it was published as 2.8.2 the version number could not be downgraded. It doesn't affect anything other than human perception. If you really want to know more, you can read the discussions in the Redmine and Git PRs.

hspindel

Have read through all the config recommendations above, and still not working.

pfSense+ 23.09-RELEASE running on Netgate 8200

Trying to connect Home Assistant to NUT. Eaton UPS already connected to NUT running locally on pfSense with no issues. NUT package 2.8.0_2. Quirk installed to make NUT work properly, but don't remember now what I did.

Changed nut.conf MODE=netserver
upsd.users: added a new user and password, upsmon slave
Changed upsd.conf. Added LISTEN <ipOfPFsense> 3493

Attempt to login to NUT from Home Assistant with:
Host: <ip of pFsense>
Port: 3493
Username: new user added above to upsd.users
Password: password for new user in upsd.users

Home Assistant always reports Failed to Connect to NUT Server.

Have tried many variations on the above, but no success. Restarted NUT after each attempted config change.

Home Assistant and pfSense are on the same network, so I don't think this is a firewall issue. Default allow LAN to any rule is present.

dennypage

@hspindel said in NUT Package (2.8.1 and above):

Changed nut.conf MODE=netserver
upsd.users: added a new user and password, upsmon slave
Changed upsd.conf. Added LISTEN <ipOfPFsense> 3493

nut.conf is not used on pfSense. You are not hand editing the ups config files are you? If you are, then that's not the right way to configure NUT on pfSense--every time you restart they will be overwritten.

Please post the configuration you have in Services / UPS / UPS Settings.

hspindel

@dennypage said in NUT Package (2.8.1 and above):

nut.conf is not used on pfSense. You are not hand editing the ups config files are you? If you are, then that's not the right way to configure NUT on pfSense--every time you restart they will be overwritten.

Please post the configuration you have in Services / UPS / UPS Settings.

nut.conf is the only file I hand edited because I couldn't see how to do it in the GUI. The rest of the files I used the GUI to modify.

What is the right way to specify MODE=netserver?

My config page looks like this:

UPS Type: Local USB
UPS Name: EatonUPS
Enable notifications: yes
Driver: usbhid
No extra arguments to driver
No extra arguments to upsmon.conf
ups.conf: pollinterval=10
upsd.conf: LISTEN pfSenseIP 3493 (also tried LISTEN 0.0.0.0 3493)
upsd.users:
[howard]
password=xxxxxxx
upsmon slave

Thank you!

dennypage

@hspindel said in NUT Package (2.8.1 and above):

What is the right way to specify MODE=netserver?

There isn't one. The file nut.conf is unused in pfSense, and most other places. It is intended to be used by generic OS distributions, but I haven't really seen much uptake on it for the systems I work with.

Returning to the attempt to connect remotely... Have you checked the firewall log? Are you able to successfully telnet to port 3493 from the LAN?

hspindel

@dennypage said in NUT Package (2.8.1 and above):

There isn't one. The file nut.conf is unused in pfSense, and most other places. It is intended to be used by generic OS distributions, but I haven't really seen much uptake on it for the systems I work with.

Okay, then NUT on pfSense is automatically configured as remotely accessible? I haven't found a setting to enable/disable it.

Returning to the attempt to connect remotely... Have you checked the firewall log? Are you able to successfully telnet to port 3493 from the LAN?

Nothing in the firewall logs. Yes, I can telnet in. There is no banner message, but if I type Hello I get a response ERR UNKNOWN-COMMAND. Tried to guess at a valid command. "LIST UPS" returned a good looking response. Couldn't guess any other commands.

So perhaps this is more a problem in Home Assistant.

dennypage

@hspindel said in NUT Package (2.8.1 and above):

Okay, then NUT on pfSense is automatically configured as remotely accessible? I haven't found a setting to enable/disable it.

It is not remotely accessible by default. You are making it remotely accessible by addition of the LISTEN directive in the Extra arguments to upsd.conf.

Yes, I can telnet in. There is no banner message, but if I type Hello I get a response ERR UNKNOWN-COMMAND. Tried to guess at a valid command. "LIST UPS" returned a good looking response. Couldn't guess any other commands.

Try these commands:

USERNAME howard
PASSWORD xxxxxxx
LOGIN EatonUPS

If you receive "OK" in response to the commands it will confirm your setup.

Then you can then move on to beating your head against HomeAssistant.

Never done it myself, but...

If you are using the hass add-on, I assume you have all of these set, yes?

mode: netclient 
remote_ups_host: pfSense_IP_Addr
remote_ups_name: EatonUPS
remote_ups_user: howard
remote_ups_password: xxxxxxx

No leading or trailing spaces, yes? No '#' or other special characters in the password, yes?

If it is still failing, you will have to go into the HomeAssistant container for NUT and look at the logs there to see what's happening.

hspindel

@dennypage said in NUT Package (2.8.1 and above):

Try these commands:

USERNAME howard
PASSWORD xxxxxxx
LOGIN EatonUPS

All of these responded with OK.

Then you can then move on to beating your head against HomeAssistant.

Time to do that!

If you are using the hass add-on, I assume you have all of these set, yes?
mode: netclient 
remote_ups_host: pfSense_IP_Addr
remote_ups_name: EatonUPS
remote_ups_user: howard
remote_ups_password: xxxxxxx

I'm using a Synology DS1522 container for Home Assistant. It doesn't appear to have any of these settings - it just set things up automagically when I installed it.

If it is still failing, you will have to go into the HomeAssistant container for NUT and look at the logs there to see what's happening.

The log isn't very helpful (at least to me). It says:
ERROR (SyncWorker-2) [homeassistant.components.nut] Failure getting NUT ups alias, socket error.[0m

I will pursue this with the HomeAssistant people. My gut-level feeling right now is that this is some kind network problem with a container connecting to pfSense.

Thank you for your help.

simon_hp

I am using the NUT package to connect to an Eaton UPS via SNMP v3.

Unfortunately I found I was not able to get the configuration as needed by using the pfSense UI to add additional items to the various configuration files, so having saved the configuration with no extra settings in the UI, I edited the configuration files directly in /usr/local/etc/nut to contain the entries required. Once this had been done, NUT started correctly and communicated with the Eaton UPS successfully for about 12 days. After that I started receiving emails that the UPS had disconnected. On checking, I found that the configuration files had become reset and lost all the settings I had added. I restored a backup copy of the configuration files, restarted the NUT service and it connected immediately.

Is there something I am missing here - do the configuration files get overwritten on a periodic basis, or how can I prevent a re-occurrance.

dennypage

@simon_hp said in NUT Package (2.8.1 and above):

I found I was not able to get the configuration as needed by using the pfSense UI to add additional items to the various configuration files, so having saved the configuration with no extra settings in the UI, I edited the configuration files directly in /usr/local/etc/nut to contain the entries required.

What were you not able to configure in the UI?

pfpv

@dennypage said in NUT Package (2.8.1 and above):

@pfpv I you would like my recommendation on a solution, the first thing I would do is connect the UPS to a system other than a Synology as Synology has a rather messed up NUT implementation, and only support Synology systems as clients.

In my opinion, it would be much better to use pfSense or a linux system as the server and have the Synology be a client. FWIW, after doing this I would also recommend a run-time calibration, regardless of how old your batteries are.

In answer to your questions about versions: The version of NUT is actually FreeBSD's nut-devel-2023.10.07_1, which is based on git just prior to the 2.8.1 release. You can see this on the Installed Packages screen. The upper level package, pfSense-pkg-nut-2.8.2, is labeled as 2.8.2 because of a mistake being made at the moment of package publication. This is mentioned earlier in this thread. The published package should have been pfSense-pkg-nut-2.8.1_1, but once it was published as 2.8.2 the version number could not be downgraded. It doesn't affect anything other than human perception. If you really want to know more, you can read the discussions in the Redmine and Git PRs.

Hi @dennypage, I respectfully disagree about Synology. It's just a server that passes UPS messages. It's very stable, does it properly, has worked for years and I haven't seen any complaints. I chose to connect my UPS to Synology because in my opinion it is the most critical piece of equipment to be properly shut down, and it provides a NUT server for other equipment. The messages are up to NUT on the client system to interpret. Synology itself didn't bork unlike pfSense.

I can assure you that equipment is not at fault. And here is the proof:
https://github.com/networkupstools/nut/issues/2104
https://github.com/networkupstools/nut/pull/2108

The NUT 2.8.1 release notes have the following now: "Also upsmon should now recognize OFF and BYPASS flags in ups.status and report that these states begin or end. The OFF state usually means than an administrative action happened to power off the load, but the UPS device is still alive and communicating (USB, SNMP, etc.); corresponding MONITOR'ed amount of power sources are considered not being "fed" for the power value calculation purposes. The BYPASS state is now treated similarly to ONBATT: currently this UPS "feeds" its load, but if later communications fail, it is considered dead. This may have unintended consequences for devices (or NUT drivers) that do not report these modes correctly (e.g. an APC calibration routine seems to start with a few seconds of "OFF" state), so the reported status is only considered as a loss of feed if it persists for more than OFFDURATION seconds. [#2044, #2104]"
Link: https://networkupstools.org/historic/v2.8.1/docs/release-notes.chunked/NUT_Release_Notes.html#_release_notes_for_nut_2_8_1_what_8217_s_new_since_2_8_0

I read your notes about the release of your 2.8.2 pfSense NUT package. You used nut-devel as there was no indication the a new NUT release was coming. But NUT 2.8.1 came out shortly after and it included a fix for the described faulty behavior.

Since APC UPS's have bi-monthly self-test routines I predict that forums will soon be flooded with messages like mine about "administratively OFF or asleep" and pfSense equipment suddenly shutting down for no reason.

Please, please, please update the pfSense NUT package to 2.8.1 NUT release. It will mitigate the problem.

dennypage

@pfpv said in NUT Package (2.8.1 and above):

I respectfully disagree about Synology. It's just a server that passes UPS messages. It's very stable, does it properly, has worked for years and I haven't seen any complaints. I chose to connect my UPS to Synology because in my opinion it is the most critical piece of equipment to be properly shut down, and it provides a NUT server for other equipment.

I don't disagree that there is a bug in NUT, and I will be looking at that shortly. That said...

I'm in my third generation of Synology equipment. Fifteen plus years. I have handled a number of support issues arising from Synology's NUT implementation, both mine and others. Their NUT implementation started out very straight forward, but over time it has evolved, becoming more and more specialized and complex. With DSM 7.2, it's gotten to the point that I don't consider it to be completely reliable, and view it as a primary of last resort.

I will point out one thing that you said that you may wish to reconsider. You indicate that the NAS is the most important thing to have a proper shutdown. I agree with this general sentiment. However, by running the NAS as the NUT primary, you are actually incurring higher risk for the NAS rather than less.

The NUT primary does not initiate a shutdown until all the associated secondaries have logged out of the primary. Assuming a default polling interval of 5 seconds, a pfSense or Linux system will take something on the order of 10-15 seconds before they log out, and another 30-90 seconds to complete a shutdown. This means that the NAS will not begin its shutdown until 10-15 seconds after the UPS declares a low battery. Depending upon configuration and current activity, a Synology usually takes over 2 minutes to complete a shutdown. If the UPS is off in calculating remaining runtime, you run the risk of exhausting the battery before the NAS has completed its shutdown.

If you reverse this situation and use pfSense or a Linux system as the primary, then the NAS will begin its shutdown within 5 seconds. Not only does this give a wider margin of safety for the NAS, it can give an increased margin of safety for the other systems as well. When the NAS shuts down, there is suddenly a lot less load on the UPS, which gives more time for the other systems to complete their shutdown even if the estimated remaining runtime was incorrect.

The relevant passage from upsmon.conf:

# Also, since the primary system stays up the longest, it suffers higher risks
# of ungraceful shutdown if the estimation of remaining runtime (or of the
# time it takes to shut down this system) was guessed wrong. By consequence,
# the "secondary" systems typically monitor the power environment state
# through the 'upsd' processes running on the remote (often "primary") systems
# and do not directly interact with an UPS (no local NUT drivers are running
# on the secondary systems). As such, secondaries typically shut down as
# soon as there is a sufficiently long power outage, or a low-battery alert
# from the UPS, or a loss of connection to the primary while the power was
# last known to be missing.

As a general rule, you want to have systems that represent the highest UPS load and/or longest shutdown time as secondaries, and a system that represents lower load and is fast to shut down as the primary.

dennypage

@hspindel said in NUT Package (2.8.1 and above):

I'm using a Synology DS1522 container for Home Assistant. It doesn't appear to have any of these settings - it just set things up automagically when I installed it.

If you don't have those settings, then I assume you are not using the hass add-on. What then are you using? And how are you configuring it?

As an aside, if you have HomeAssistant running in a container on your Synology, why are you running NUT inside the container? I assume it is not for shutting down the container which would be handled by the Synology. Are you running HA automation based on the UPS values?

hspindel

@dennypage said in NUT Package (2.8.1 and above):

@hspindel said in NUT Package (2.8.1 and above):

I'm using a Synology DS1522 container for Home Assistant. It doesn't appear to have any of these settings - it just set things up automagically when I installed it.

If you don't have those settings, then I assume you are not using the hass add-on. What then are you using? And how are you configuring it?

Within Home Assistant, one has the option to select Settings/Devices and Services/Add Integration. From the Add Integration page, one can search for NUT. Upon selecting NUT, Home Assistant pops up a dialog box to "Connect to the NUT server" which wants host, poirt, username, and password. There are no configuration options offered, and nothing I've tried in the dialog box gets a response other than "Failed to Connect."

As an aside, if you have HomeAssistant running in a container on your Synology, why are you running NUT inside the container? I assume it is not for shutting down the container which would be handled by the Synology. Are you running HA automation based on the UPS values?

AFAIK (and I could be confused) I am not trying to run NUT inside the container, just the Home Assistant NUT client.

I have using Home Assistant to shut down two other Synologies (APC units). The UPSes for those Synologies are connected to a Windows machine and a Linux machine. Windows and Linux are running apcupsd, which is not understood by Synology. So Home Assistant is configured to watch apcupsd and send a shutdown message to the relevant Synology.

Another UPS (an Eaton) is connected to pfSense. So I was trying to configure Home Assistant to pay attention to the NUT server on pfSense and operate similarly to the way the APC/Synologies work with Home Assistant.

If I can't get that to work, my next try would be to get the Synology as a NUT client to pay direct attention to the NUT server on pfSense. I have read that Synology's non-standard implementation of NUT makes this problematic, but I can search for a known working configuration.

hspindel

@dennypage

I guess the whole Home Assistant thing is a fuggediboutit now.

I was able to get the Synology to directly connect to the NUT server on pfSense.

Since all I care about is powering down the Synology correctly in the event of a power failure, I'm good now.

Thank you for your help.

simon_hp

@dennypage Thank you for your quick response, much appreciated.

When I installed the UPS some weeks ago, I did struggle when I tried to use the additional settings entries in the pfSense Nut UI. As I mentioned, I ended up saving only the UPS Type, UPS name, notifications, and Remote IP address or hostname. After that I manually edited the conf files and the service started successfully.

Fast forward 12 days and I stated getting notifications of comms failure to the UPS. The conf files seemed to have reverted to default,, so I restored the files and all started correctly and I made a post above that you responded to.

In order to answer your question correctly, I went back in and reconfigured the UPS through the UI - which is a very much preferred way of doing it. I am not sure what I did differently - but I was successful getting the UPS connected via the UI settings this time (YAY!). The entries I made below were (details changed to protect the innocent)PostUpload1.txt

I am also unsure of the difference between the entry in upsmon.conf which nominates "primary", and the upsd.users file that nominates "master".

I am not using pfSense to relay UPS status to any other machine, as my Windows and Linux servers directly connect to the UPS themselves, and my monitoring system (Zabbix) also directly connects using snmp.

Appreciate any suggestions or explanations,
Regards
Simon

dennypage

@hspindel said in NUT Package (2.8.1 and above):

I have using Home Assistant to shut down two other Synologies (APC units). The UPSes for those Synologies are connected to a Windows machine and a Linux machine. Windows and Linux are running apcupsd, which is not understood by Synology. So Home Assistant is configured to watch apcupsd and send a shutdown message to the relevant Synology.

This is kinda convoluted, and sounds fragile to me. I think using NUT directly on all the systems would be a better choice. NUT has a shutdown interlock for secondaries that will be lost by using HA. FWIW, NUT does support being a client of apcupsd, but there is no way to configure this on Synology.

@hspindel said in NUT Package (2.8.1 and above):

If I can't get that to work, my next try would be to get the Synology as a NUT client to pay direct attention to the NUT server on pfSense. I have read that Synology's non-standard implementation of NUT makes this problematic, but I can search for a known working configuration.

Synology does use a very non-standard configuration for NUT, but what you are talking about is easily accomplished. The only restriction is that you must use default values for ups name, username and password.

See post #2 in this thread, Notes on remote access to NUT for details on setting up remote access, and see Synology setup details in the Notes on Synology section at the end of the post.

dennypage

@simon_hp said in NUT Package (2.8.1 and above):

I am also unsure of the difference between the entry in upsmon.conf which nominates "primary", and the upsd.users file that nominates "master".

In 2.7 and below, NUT used the terms "master" and "slave". In NUT 2.8+, the terms "primary" and "secondary" were added, but are fully interchangeable with the prior terms.