NUT disconnects: Eaton SMART1500LCD

dennypage

@RobbieTT said in NUT disconnects: Eaton SMART1500LCD:

Synology DSM's NUT has zero issues with the Eaton (USB or network) and same for a macOS NUT app (network).

Several reasons for the situation with Synology:

Synology's NUT implementation is still based on 2.7.4.
All the USB screwups were introduced in 2.8.0.
Synology is Linux based rather than FreeBSD based.

The USB screwups that were introduced in 2.8.0 are specifically disabled for Linux, so even if Synology had upgraded to 2.8.0 they would not experience the same issues. Honestly, the fact that everything works fine on Linux is probably why there hasn't been a huge outcry over NUT 2.8.0.

NUT network (master/slave) connections are unrelated to what UPS driver is in use.

hspindel

@dennypage
System logs are pretty useless. This is all they say:

Oct 27 23:10:00 upsmon 5576 Communications with UPS EatonUPS lost
Oct 27 23:10:00 upsmon 5576 UPS [EatonUPS]: connect failed: Connection failure: Connection refused

Then after restarting nut.sh:

Oct 27 23:30:05 upsmon 43287 Communications with UPS EatonUPS established
Oct 27 23:30:05 upsd 44370 User local-monitor@::1 logged into UPS [EatonUPS]
Oct 27 23:30:01 upsd 44370 Startup successful
Oct 27 23:30:01 upsd 44133 Connected to UPS [EatonUPS]: usbhid-ups-EatonUPS
Oct 27 23:30:01 upsd 44133 listening on 127.0.0.1 port 3493
Oct 27 23:30:01 upsd 44133 listening on ::1 port 3493
Oct 27 23:30:00 usbhid-ups 43949 Startup successful

I did read through the posts you suggested. There were several suggestions that did not work - among them: tried a new USB cable, tried the port directive, tried the pollinterval directive.

In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes. With the interruptonly directive active, I pulled the plug on the UPS and the loss of line power was detected and reactivation of line power was also detected.

I am using the NUT package 2.8.0_2, which according to the package manager is the latest available. I saw some posts indicating the problems I'm seeing could be a failure in libusb, but it doesn't look like an update is available.

Configuration is LocalUSB with driver USBHid.

Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf. Finally, after the above fixes, email notification did not work. This was fixed by adding RUN_AS_USER root to Additional configuration lines for upsmon.conf

Thank you for your help. It did steer me to a solution.

dennypage

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes.

Unfortunately, with the default version of the usb driver, interruptonly will eventually fail. There is a memory corruption issue in the USB reconnect code (libusb). It don't always hit the issue every reconnect, but eventually it does. You can probably test this by unplugging and re-plugging the cable a few times.

Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf.

If NUT is unable to start without root, this means that you either have a directory permission issue for /var/db/nut, or you have a missing quirk which results in a USB permission issue. I'd have to know which process (upsd or usbhid-ups) doesn't start in order to know which.

If it's a permission issue for /var/db/nut, upsd will fail to start. You can (easily) address the permission problem by doing a chown on the directory.

If it's a USB permission issue, usbhid-ups will fail to start. This issue is controlled by quirks registered in the kernel. If there is no quirk for your specific manufacturer/model of UPS, then the kernel will attach a default HID driver to the device on its own. Removing this driver requires root privileges, which is why usbhid-ups fails to start. Until the kernel is updated with the quirk, you can address the issue by adding the missing quirk to /boot/loader.conf.local to address this issue without using root.

hspindel

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes.

Unfortunately, with the default version of the usb driver, interruptonly will eventually fail. There is a memory corruption issue in the USB reconnect code (libusb). It don't always hit the issue every reconnect, but eventually it does. You can probably test this by unplugging and re-plugging the cable a few times.

With the interruptonly flag, nut has been connected without problem for 48 hours now.

I did read that there are problems in Eaton's internal implementation of USB, so perhaps there is nothing more that can be done within nut?

What change am I missing that could replace interrupt only? No other change I tried kept nut connected for more than about five minutes.

What would be the symptom if interruptonly fails? I tried unplugging/replugging the UPS from line power several times and the broadcast messages say the expected "on battery" and "on line power" and it looks like it is behaving normally.

The only anomaly I see is if I click on the Refresh icon in Services/UPS/Status I get a console message that communication with the UPS was lost followed immediately by a communication established message. (Perhaps the refresh icon is causing a nut.sh restart instead of just updating status as I'd expect?)

Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf.

If NUT is unable to start without root, this means that you either have a directory permission issue for /var/db/nut, or you have a missing quirk which results in a USB permission issue. I'd have to know which process (upsd or usbhid-ups) doesn't start in order to know which.

If it's a permission issue for /var/db/nut, upsd will fail to start. You can (easily) address the permission problem by doing a chown on the directory.

Owner of /var/db/nut is nut. Owner of all the files in /var/db/nut is root, with group nut. Should the owner of those files be nut?

-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsd.pid
-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsmon.pid
srw-rw---- 1 root nut 0 Oct 28 17:38 usbhid-ups-EatonUPS
-rw-r--r-- 1 root nut 6 Oct 28 17:38 usbhid-ups-EatonUPS.pid

If it's a USB permission issue, usbhid-ups will fail to start. This issue is controlled by quirks registered in the kernel. If there is no quirk for your specific manufacturer/model of UPS, then the kernel will attach a default HID driver to the device on its own. Removing this driver requires root privileges, which is why usbhid-ups fails to start. Until the kernel is updated with the quirk, you can address the issue by adding the missing quirk to /boot/loader.conf.local to address this issue without using root.

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?

Thank you very much for your help!

RobbieTT

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?

It's not there by default but you can create one. The key thing with the loader.conf.local file is it is read last and effectively overrides the other configuration already set by pfSense.

I have all sorts of stuff in mine, 99% of which is commented out as it is a handy place to leave all my config notes so I know 'how I got here' or why a particular 'default' setting is actually the best setting.

️

dennypage

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it?

Yes, you would create it.

Here is a sample entry:

hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE"

That is a quirk for the gen 3 series of UPSs from APC, and it tells the OS to ignore the device (do not attach a default driver). This allows NUT to freely attach to the device without requiring root privilege.

Your entry will be different. What you would need to put in there would be determined by examining the output of "usbconfig".

Usbconfig documentation can be found here and quirk documentation can be found here.

hspindel

@dennypage
I read the docs for usbconfig and quirk. Still don't know what I would change in your example /boot/loader.conf.local for my config.

Pretty clear I want UQ_HID_IGNORE, but the other fields are mysterious to me.

Here is the output from usbconfig:

usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)

Any further guidance on the interruptonly issue mentioned a couple posts ago?

Thank you.

dennypage

@hspindel Try the '-v' option...

dennypage

@dennypage For comparison, this is the usbconfig -v output for the UPS corresponding to the quirk I mentioned above:

ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.1 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x051d
  idProduct = 0x0003
  bcdDevice = 0x0001
  iManufacturer = 0x0001  <American Power Conversion >
  iProduct = 0x0002  <Smart-UPS_1000 FW:UPS 16.1 / ID=1047>
  iSerialNumber = 0x0003  <ASXXXXXXXXX    >
  bNumConfigurations = 0x0001


 Configuration index 0

    bLength = 0x0009
    bDescriptorType = 0x0002
    wTotalLength = 0x0029
    bNumInterfaces = 0x0001
    bConfigurationValue = 0x0001
    iConfiguration = 0x0000  <no string>
    bmAttributes = 0x00e0
    bMaxPower = 0x0005

    Interface 0
      bLength = 0x0009
      bDescriptorType = 0x0004
      bInterfaceNumber = 0x0000
      bAlternateSetting = 0x0000
      bNumEndpoints = 0x0002
      bInterfaceClass = 0x0003  <HID device>
      bInterfaceSubClass = 0x0000
      bInterfaceProtocol = 0x0000
      iInterface = 0x0000  <no string>

      Additional Descriptor

      bLength = 0x09
      bDescriptorType = 0x21
      bDescriptorSubType = 0x11
       RAW dump:
       0x00 | 0x09, 0x21, 0x11, 0x01, 0x21, 0x01, 0x22, 0xc4,
       0x08 | 0x01

     Endpoint 0
        bLength = 0x0007
        bDescriptorType = 0x0005
        bEndpointAddress = 0x0081  <IN>

pfsense4ever

Does anyone know what pfsense version will address the nut package issue for the Smart1500 LCD UPS? What driver is pfsense looking for in version 2.7 as it says "driver not connected" and how this can be corrected?

hspindel

@dennypage

Thank you for the pointers. The output of usbconfig -v is:

ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)

bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0110
bDeviceClass = 0x0000 <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0008
idVendor = 0x09ae
idProduct = 0x2012
bcdDevice = 0x0009
iManufacturer = 0x0001 <Tripp Lite >
iProduct = 0x0002 <Tripp Lite UPS >
iSerialNumber = 0x0000 <no string>
bNumConfigurations = 0x0001

My best guess at the line to install the quirk is:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

But I am not at all confident I got that right. How does it look?

Thank you.

hspindel

@pfsense4ever
Please read through all the discussion above.

I got the UPS to connect by adding RUN_AS_USER root to Additional configuration lines for upsmon.conf. However, @dennypage says in this discussion that adding a USB quirk is a better solution. Still working on that.

dennypage

[Wrong thread, sorry]

dennypage

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

My best guess at the line to install the quirk is:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

But I am not at all confident I got that right. How does it look?

Looks good to me.

You can test without having to reboot by using this command:

usbconfig add_dev_quirk_vplh 0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE

followed by disconnect and reconnect of the USB cable.

There is a usbconfig option to show the attached driver if any. IIRC, it's show_ifdrv. Run it before and after the disconnect/reconnect and you should see that after there is no attached driver. Alternatively, you can simply try to start NUT (without root) to see if it works.

Note that using usbconfig to install the quirk is temporary and will not survive a reboot. So assuming it works, you will want to install it properly in /boot/loader.conf.local.

dennypage

Please see here regarding testing of the new pfSense NUT package. I would appreciate any help you can offer. Thanks!

[Note that the new package addresses the crash issue after successful start, but does not have anything regarding the quirk issue that prevents initial start]

dennypage

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

hspindel

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

When I have a chance to test it, I will be happy to post the results. Been rather busy. :-)

hspindel

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

Mostly good results. Thank you for all the help.

The changes I had previously made to the NUT configuration were:

Extra Arguments to driver (optional)
interruptonly

Additional configuration lines for upsmon.conf
RUN_AS_USER root

Additional configuration lines for ups.conf
user=root
pollinterval=10

I deleted all of the above changes except pollinterval. Rebooted the pfSense router and verified that UPS connection did not work (as expected, wanted to make sure I was testing from a baseline).
Created the file /boot/loader.conf.local with contents:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

Rebooted the pfSense router and was able to verify that the UPS was correctly monitored by NUT. Pulled the UPS power cord a couple times to verify that NUT detected that and it was handled correctly.
Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly
With the interruptonly flag present, the UPS connection is once again stable without need for setting root access. However,@dennypage mentioned above that this is not a good solution due to a memory corruption issue in the USB reconnect code (libusb). I am not seeing any problems and at present don't have a another solution that works.
Verified that with interruptonly present, NUT still responds correctly to unplugging and replugging the UPS.

dennypage

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

Excellent job creating the quirk!

Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly

Is this with the 2.8.0_2 package, or the new 2.8.1 package?

hspindel

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

Excellent job creating the quirk!

Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly

Is this with the 2.8.0_2 package, or the new 2.8.1 package?

I am running 2.8.0_2. The package manager for pfSense+ doesn't list a 2.8.1 upgrade available.

This is on a Netgate 8200.