-
NUT frequently loses connection to my Eaton SMART1500LCD with a message "Communications with UPS lost".
Did a ,lot of research to try and find a cure. Found recommendations to increase pollinterval. This seems to help but does not cure the problem.
Currently, I am running a cron job every five minutes to run nut.sh restart.
Anybody have a better solution?
-
@hspindel Look in the system log to see what is going on? Then look in the NUT support thread for advice?
Quick guess is that you should look in the support thread beginning around February of 2023.
-
I have the exact UPS model mentioned in this article and I am running version 2.7. This is a workaround that I am using:
Temporarily stop NUT Service with Watchguard
Temporarily stop the NUT service
Verify that it does not still restart on its own (as I've seen this happen). If the service is restarted again, repeat the above 2 steps and wait a few minutes and verify again
Enable SSH
Download an older USBHID-UPS driver found here: https://forum.netgate.com/assets/uploads/files/1678659799995-usbhid-ups.gz
For reference, the shasum and sha256sum checksums of the unzipped file are:49ce9131502bfb8b789ee97b7fb3fc81fc9f8fff usbhid-ups
999a2653559dbc50ecc8ba592a67587b1e307a1495f6e8ebbd3d8e90e3967133 usbhid-upsUsing PUTTY or WinSCP client, connect to your PFSense box using SSH
Locate /usr/local/libexec/nut
Rename the current file USBHID-UPS to USBHID-UPS.old so that you have a backup file that you can reverse to if needed
Upload the new downloaded USBHID-UPS driver that you have downloaded earlier and place it in this directory
Set the file permission of USBHID-UPS to rwxr-xr-x (0755)
Exit from your current Shell session if using PUTTY or if you using a third-party remote file explorer like WinSCP.
Disable SSH functionality in PFSense
In PFSense GUI, go to the NUT Advanced Options and put these options in the text boxes then save the changes made:
General Settings:USB Type: Local USB
UPS Name: "Your Friendly Name for your UPS" In my case, it's TRIPPLITEUPS
Notifications: Enable notificationsFor Driver Settings:
Driver: usbhid
Extra Arguments to driver (optional):user=root
Advanced Settings:
Additional configuration lines for upsmon.conf
RUN_AS_USER root
NOTIFYFLAG NOCOMM SYSLOG
NOTIFYFLAG COMMOK SYSLOG
NOTIFYFLAG COMMBAD SYSLOG
NOTIFYFLAG ONLINE SYSLOG+EXEC
NOTIFYFLAG ONBATT SYSLOG+EXEC
NOTIFYFLAG LOWBATT SYSLOG+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+EXEC
NOTIFYFLAG REPLBATT SYSLOG+EXEC
NOTIFYFLAG FSD SYSLOG+EXEC
NOTIFYFLAG NOPARENT SYSLOG+EXEC
POLLFREQ 60
POLLFREQALERT 60
DEADTIME 180Additional configuration lines for ups.conf
user=root
pollinterval=5
port = /dev/ugen0.3NOTE:
To check which port your UPS is using on your hardware, go to Diagnostics -- Command Prompt and execute the following Shell command:
usbconfig
If the port field is different than what it shows in this example, change it to match the one for your UPS unit
If you go to the UPS Status, you'll see that your UPS is online now.Restart the NUT service
Enable notifications in Service Watchdog for NUT ServiceKindly note the following, you will see three alerts that appears in the system logs for the UPS logs which is okay to ignore due to known bug that online sources are aware of with using PFSense 2.7 Nut package (drivers)
For a test, I have unplugged the UPS from the power outlet and as expected, it went on battery, added the event into the event log and sent me an E-Mail notification. Then I plugged the UPS back into the power outlet again and again as expected, it added an event to the system log and sent me an E-Mail alert saying its back online
Also note, if you uninstall the NUT package and reinstall the NUT package again, it will update the driver file that we uploaded in the NUT directory earlier in PFSense with the latest version which is unstable for some UPS devices like Tripplite.
On a side note, the previous version of PFSense 2.6 did not have this issue with the NUT package so I am wondering if someone can come up with a permenant solution to address this issue.
-
Watching with interest as I have never managed to get pfSense NUT to talk to my Eaton 1 GbE NIC (Eaton 5P 650) or direct via USB. Synology DSM's NUT has zero issues with the Eaton (USB or network) and same for a macOS NUT app (network).
Having NUT on my firewall would be ideal as I is that last item on my ideal power-off sequence.
️ -
You can try the solution I mentioned above using a USB cable from your UPS device to your PFsense box and see if that works for you. Worse scenario, you can undo those changes and use the previous NUT driver file in that file directory.
-
@pfsense4ever
Yep, seems like a plan.
️ -
@pfsense4ever said in NUT disconnects: Eaton SMART1500LCD:
Download an older USBHID-UPS driver found here: https://forum.netgate.com/assets/uploads/files/1678659799995-usbhid-ups.gz
I was kinda hoping that OP would learn how to fish here. All the information is to be found in the NUT support thread. FYI, the executable you are linking to is actually a newer version of usbhid-ups rather than an older version. It's a version I built from the nut-devel package rather than the normal nut package.
@pfsense4ever, The configuration you have is way over done. The only pertinent line in the upsmon.conf section is the RUN_AS_USER line. All other lines should be removed.
For ups.conf, all the lines should be removed except for the "user=root" line. There are two reasons that you might want that (temporarily). First, if you have a permissions issue with /var/db/nut. If you do, you can fix it by changing the ownership of the directory to user "nut". The second issue is email notifications. Non root email notifications are broken in 23.05.1, so if you need email notifications you currently need to run uspd as root. Non root email notifications are fixed in 23.09, so you should remove the line after installing.
There is really no need to slow the poll interval for a USB device. SNMP perhaps, but not USB.
In general, specifying an explicit port for USB is a bad idea because changes in the USB configuration can cause the port to move. Also, unless OP has the exact same pfSense hardware with the same USB connections in the same ports, you should not expect their UPS to be at the same address. The only reason you would need to use an explicit USB port is in a situation where you have multiple USB UPSs attached to the same system, and you need to explicitly address one of them.
FWIW, 23.09 will move to nut-devel, which is based on current nut git. This will remove the need for replacing the usb and snmp executables. I haven't checked yet, but it may also have the quirk updates, which would remove the need for RUN_AS_USER in upsmon.conf.
-
@RobbieTT said in NUT disconnects: Eaton SMART1500LCD:
Synology DSM's NUT has zero issues with the Eaton (USB or network) and same for a macOS NUT app (network).
Several reasons for the situation with Synology:
- Synology's NUT implementation is still based on 2.7.4.
- All the USB screwups were introduced in 2.8.0.
- Synology is Linux based rather than FreeBSD based.
The USB screwups that were introduced in 2.8.0 are specifically disabled for Linux, so even if Synology had upgraded to 2.8.0 they would not experience the same issues. Honestly, the fact that everything works fine on Linux is probably why there hasn't been a huge outcry over NUT 2.8.0.
NUT network (master/slave) connections are unrelated to what UPS driver is in use.
-
@dennypage
System logs are pretty useless. This is all they say:Oct 27 23:10:00 upsmon 5576 Communications with UPS EatonUPS lost
Oct 27 23:10:00 upsmon 5576 UPS [EatonUPS]: connect failed: Connection failure: Connection refusedThen after restarting nut.sh:
Oct 27 23:30:05 upsmon 43287 Communications with UPS EatonUPS established
Oct 27 23:30:05 upsd 44370 User local-monitor@::1 logged into UPS [EatonUPS]
Oct 27 23:30:01 upsd 44370 Startup successful
Oct 27 23:30:01 upsd 44133 Connected to UPS [EatonUPS]: usbhid-ups-EatonUPS
Oct 27 23:30:01 upsd 44133 listening on 127.0.0.1 port 3493
Oct 27 23:30:01 upsd 44133 listening on ::1 port 3493
Oct 27 23:30:00 usbhid-ups 43949 Startup successfulI did read through the posts you suggested. There were several suggestions that did not work - among them: tried a new USB cable, tried the port directive, tried the pollinterval directive.
In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes. With the interruptonly directive active, I pulled the plug on the UPS and the loss of line power was detected and reactivation of line power was also detected.
I am using the NUT package 2.8.0_2, which according to the package manager is the latest available. I saw some posts indicating the problems I'm seeing could be a failure in libusb, but it doesn't look like an update is available.
Configuration is LocalUSB with driver USBHid.
Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf. Finally, after the above fixes, email notification did not work. This was fixed by adding RUN_AS_USER root to Additional configuration lines for upsmon.conf
Thank you for your help. It did steer me to a solution.
-
@hspindel said in NUT disconnects: Eaton SMART1500LCD:
In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes.
Unfortunately, with the default version of the usb driver, interruptonly will eventually fail. There is a memory corruption issue in the USB reconnect code (libusb). It don't always hit the issue every reconnect, but eventually it does. You can probably test this by unplugging and re-plugging the cable a few times.
Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf.
If NUT is unable to start without root, this means that you either have a directory permission issue for /var/db/nut, or you have a missing quirk which results in a USB permission issue. I'd have to know which process (upsd or usbhid-ups) doesn't start in order to know which.
If it's a permission issue for /var/db/nut, upsd will fail to start. You can (easily) address the permission problem by doing a chown on the directory.
If it's a USB permission issue, usbhid-ups will fail to start. This issue is controlled by quirks registered in the kernel. If there is no quirk for your specific manufacturer/model of UPS, then the kernel will attach a default HID driver to the device on its own. Removing this driver requires root privileges, which is why usbhid-ups fails to start. Until the kernel is updated with the quirk, you can address the issue by adding the missing quirk to /boot/loader.conf.local to address this issue without using root.
-
@dennypage said in NUT disconnects: Eaton SMART1500LCD:
@hspindel said in NUT disconnects: Eaton SMART1500LCD:
In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes.
Unfortunately, with the default version of the usb driver, interruptonly will eventually fail. There is a memory corruption issue in the USB reconnect code (libusb). It don't always hit the issue every reconnect, but eventually it does. You can probably test this by unplugging and re-plugging the cable a few times.
With the interruptonly flag, nut has been connected without problem for 48 hours now.
I did read that there are problems in Eaton's internal implementation of USB, so perhaps there is nothing more that can be done within nut?
What change am I missing that could replace interrupt only? No other change I tried kept nut connected for more than about five minutes.
What would be the symptom if interruptonly fails? I tried unplugging/replugging the UPS from line power several times and the broadcast messages say the expected "on battery" and "on line power" and it looks like it is behaving normally.
The only anomaly I see is if I click on the Refresh icon in Services/UPS/Status I get a console message that communication with the UPS was lost followed immediately by a communication established message. (Perhaps the refresh icon is causing a nut.sh restart instead of just updating status as I'd expect?)
Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf.
If NUT is unable to start without root, this means that you either have a directory permission issue for /var/db/nut, or you have a missing quirk which results in a USB permission issue. I'd have to know which process (upsd or usbhid-ups) doesn't start in order to know which.
If it's a permission issue for /var/db/nut, upsd will fail to start. You can (easily) address the permission problem by doing a chown on the directory.
Owner of /var/db/nut is nut. Owner of all the files in /var/db/nut is root, with group nut. Should the owner of those files be nut?
-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsd.pid
-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsmon.pid
srw-rw---- 1 root nut 0 Oct 28 17:38 usbhid-ups-EatonUPS
-rw-r--r-- 1 root nut 6 Oct 28 17:38 usbhid-ups-EatonUPS.pidIf it's a USB permission issue, usbhid-ups will fail to start. This issue is controlled by quirks registered in the kernel. If there is no quirk for your specific manufacturer/model of UPS, then the kernel will attach a default HID driver to the device on its own. Removing this driver requires root privileges, which is why usbhid-ups fails to start. Until the kernel is updated with the quirk, you can address the issue by adding the missing quirk to /boot/loader.conf.local to address this issue without using root.
I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?
Thank you very much for your help!
-
@hspindel said in NUT disconnects: Eaton SMART1500LCD:
I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?
It's not there by default but you can create one. The key thing with the loader.conf.local file is it is read last and effectively overrides the other configuration already set by pfSense.
I have all sorts of stuff in mine, 99% of which is commented out as it is a handy place to leave all my config notes so I know 'how I got here' or why a particular 'default' setting is actually the best setting.
️ -
@hspindel said in NUT disconnects: Eaton SMART1500LCD:
I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it?
Yes, you would create it.
Here is a sample entry:
hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE"That is a quirk for the gen 3 series of UPSs from APC, and it tells the OS to ignore the device (do not attach a default driver). This allows NUT to freely attach to the device without requiring root privilege.
Your entry will be different. What you would need to put in there would be determined by examining the output of "usbconfig".
Usbconfig documentation can be found here and quirk documentation can be found here.
-
@dennypage
I read the docs for usbconfig and quirk. Still don't know what I would change in your example /boot/loader.conf.local for my config.Pretty clear I want UQ_HID_IGNORE, but the other fields are mysterious to me.
Here is the output from usbconfig:
usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)Any further guidance on the interruptonly issue mentioned a couple posts ago?
Thank you.
-
@hspindel Try the '-v' option...
-
@dennypage For comparison, this is the usbconfig -v output for the UPS corresponding to the quirk I mentioned above:
ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.1 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x0000 <Probed by interface class> bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0040 idVendor = 0x051d idProduct = 0x0003 bcdDevice = 0x0001 iManufacturer = 0x0001 <American Power Conversion > iProduct = 0x0002 <Smart-UPS_1000 FW:UPS 16.1 / ID=1047> iSerialNumber = 0x0003 <ASXXXXXXXXX > bNumConfigurations = 0x0001 Configuration index 0 bLength = 0x0009 bDescriptorType = 0x0002 wTotalLength = 0x0029 bNumInterfaces = 0x0001 bConfigurationValue = 0x0001 iConfiguration = 0x0000 <no string> bmAttributes = 0x00e0 bMaxPower = 0x0005 Interface 0 bLength = 0x0009 bDescriptorType = 0x0004 bInterfaceNumber = 0x0000 bAlternateSetting = 0x0000 bNumEndpoints = 0x0002 bInterfaceClass = 0x0003 <HID device> bInterfaceSubClass = 0x0000 bInterfaceProtocol = 0x0000 iInterface = 0x0000 <no string> Additional Descriptor bLength = 0x09 bDescriptorType = 0x21 bDescriptorSubType = 0x11 RAW dump: 0x00 | 0x09, 0x21, 0x11, 0x01, 0x21, 0x01, 0x22, 0xc4, 0x08 | 0x01 Endpoint 0 bLength = 0x0007 bDescriptorType = 0x0005 bEndpointAddress = 0x0081 <IN> -
Does anyone know what pfsense version will address the nut package issue for the Smart1500 LCD UPS? What driver is pfsense looking for in version 2.7 as it says "driver not connected" and how this can be corrected?
-
Thank you for the pointers. The output of usbconfig -v is:
ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)
bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0110
bDeviceClass = 0x0000 <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0008
idVendor = 0x09ae
idProduct = 0x2012
bcdDevice = 0x0009
iManufacturer = 0x0001 <Tripp Lite >
iProduct = 0x0002 <Tripp Lite UPS >
iSerialNumber = 0x0000 <no string>
bNumConfigurations = 0x0001My best guess at the line to install the quirk is:
hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"
But I am not at all confident I got that right. How does it look?
Thank you.
-
@pfsense4ever
Please read through all the discussion above.I got the UPS to connect by adding RUN_AS_USER root to Additional configuration lines for upsmon.conf. However, @dennypage says in this discussion that adding a USB quirk is a better solution. Still working on that.
-
[Wrong thread, sorry]
