NUT disconnects: Eaton SMART1500LCD

hspindel · Oct 30, 2023, 5:17 AM

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

In case anyone else reads this, what finally did seem to work is to add the interruptonly directive (Extra Arguments to driver). I have been running about 15 hours now without issue, when previously it wouldn't last 15 minutes.

Unfortunately, with the default version of the usb driver, interruptonly will eventually fail. There is a memory corruption issue in the USB reconnect code (libusb). It don't always hit the issue every reconnect, but eventually it does. You can probably test this by unplugging and re-plugging the cable a few times.

With the interruptonly flag, nut has been connected without problem for 48 hours now.

I did read that there are problems in Eaton's internal implementation of USB, so perhaps there is nothing more that can be done within nut?

What change am I missing that could replace interrupt only? No other change I tried kept nut connected for more than about five minutes.

What would be the symptom if interruptonly fails? I tried unplugging/replugging the UPS from line power several times and the broadcast messages say the expected "on battery" and "on line power" and it looks like it is behaving normally.

The only anomaly I see is if I click on the Refresh icon in Services/UPS/Status I get a console message that communication with the UPS was lost followed immediately by a communication established message. (Perhaps the refresh icon is causing a nut.sh restart instead of just updating status as I'd expect?)

Also, in case anyone else is reading, a different problem with NUT was that it wouldn't connect with the UPS at all. This was solved by adding user=root to Additional configuration lines for ups.conf.

If NUT is unable to start without root, this means that you either have a directory permission issue for /var/db/nut, or you have a missing quirk which results in a USB permission issue. I'd have to know which process (upsd or usbhid-ups) doesn't start in order to know which.

If it's a permission issue for /var/db/nut, upsd will fail to start. You can (easily) address the permission problem by doing a chown on the directory.

Owner of /var/db/nut is nut. Owner of all the files in /var/db/nut is root, with group nut. Should the owner of those files be nut?

-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsd.pid
-rw-r--r-- 1 root nut 6 Oct 28 17:38 upsmon.pid
srw-rw---- 1 root nut 0 Oct 28 17:38 usbhid-ups-EatonUPS
-rw-r--r-- 1 root nut 6 Oct 28 17:38 usbhid-ups-EatonUPS.pid

If it's a USB permission issue, usbhid-ups will fail to start. This issue is controlled by quirks registered in the kernel. If there is no quirk for your specific manufacturer/model of UPS, then the kernel will attach a default HID driver to the device on its own. Removing this driver requires root privileges, which is why usbhid-ups fails to start. Until the kernel is updated with the quirk, you can address the issue by adding the missing quirk to /boot/loader.conf.local to address this issue without using root.

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?

Thank you very much for your help!

RobbieTT · Oct 30, 2023, 12:30 PM

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it? Instead of creating/editing a file, is this something that is controlled from pfSense System/Advanced/Tunables?

It's not there by default but you can create one. The key thing with the loader.conf.local file is it is read last and effectively overrides the other configuration already set by pfSense.

I have all sorts of stuff in mine, 99% of which is commented out as it is a handy place to leave all my config notes so I know 'how I got here' or why a particular 'default' setting is actually the best setting.

️

dennypage · Oct 30, 2023, 3:55 PM

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I don't have a file named /boot/loader.conf.local. Am I supposed to create one? What goes in it?

Yes, you would create it.

Here is a sample entry:

hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE"

That is a quirk for the gen 3 series of UPSs from APC, and it tells the OS to ignore the device (do not attach a default driver). This allows NUT to freely attach to the device without requiring root privilege.

Your entry will be different. What you would need to put in there would be determined by examining the output of "usbconfig".

Usbconfig documentation can be found here and quirk documentation can be found here.

hspindel · Oct 30, 2023, 9:36 PM

@dennypage
I read the docs for usbconfig and quirk. Still don't know what I would change in your example /boot/loader.conf.local for my config.

Pretty clear I want UQ_HID_IGNORE, but the other fields are mysterious to me.

Here is the output from usbconfig:

usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)

Any further guidance on the interruptonly issue mentioned a couple posts ago?

Thank you.

dennypage · Oct 31, 2023, 2:11 AM

@hspindel Try the '-v' option...

dennypage · Nov 8, 2023, 5:49 AM

@dennypage For comparison, this is the usbconfig -v output for the UPS corresponding to the quirk I mentioned above:

ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.1 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x051d
  idProduct = 0x0003
  bcdDevice = 0x0001
  iManufacturer = 0x0001  <American Power Conversion >
  iProduct = 0x0002  <Smart-UPS_1000 FW:UPS 16.1 / ID=1047>
  iSerialNumber = 0x0003  <ASXXXXXXXXX    >
  bNumConfigurations = 0x0001


 Configuration index 0

    bLength = 0x0009
    bDescriptorType = 0x0002
    wTotalLength = 0x0029
    bNumInterfaces = 0x0001
    bConfigurationValue = 0x0001
    iConfiguration = 0x0000  <no string>
    bmAttributes = 0x00e0
    bMaxPower = 0x0005

    Interface 0
      bLength = 0x0009
      bDescriptorType = 0x0004
      bInterfaceNumber = 0x0000
      bAlternateSetting = 0x0000
      bNumEndpoints = 0x0002
      bInterfaceClass = 0x0003  <HID device>
      bInterfaceSubClass = 0x0000
      bInterfaceProtocol = 0x0000
      iInterface = 0x0000  <no string>

      Additional Descriptor

      bLength = 0x09
      bDescriptorType = 0x21
      bDescriptorSubType = 0x11
       RAW dump:
       0x00 | 0x09, 0x21, 0x11, 0x01, 0x21, 0x01, 0x22, 0xc4,
       0x08 | 0x01

     Endpoint 0
        bLength = 0x0007
        bDescriptorType = 0x0005
        bEndpointAddress = 0x0081  <IN>

pfsense4ever · Nov 8, 2023, 4:33 AM

Does anyone know what pfsense version will address the nut package issue for the Smart1500 LCD UPS? What driver is pfsense looking for in version 2.7 as it says "driver not connected" and how this can be corrected?

hspindel · Nov 8, 2023, 5:49 AM

@dennypage

Thank you for the pointers. The output of usbconfig -v is:

ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)

bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0110
bDeviceClass = 0x0000 <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0008
idVendor = 0x09ae
idProduct = 0x2012
bcdDevice = 0x0009
iManufacturer = 0x0001 <Tripp Lite >
iProduct = 0x0002 <Tripp Lite UPS >
iSerialNumber = 0x0000 <no string>
bNumConfigurations = 0x0001

My best guess at the line to install the quirk is:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

But I am not at all confident I got that right. How does it look?

Thank you.

hspindel · Nov 8, 2023, 5:52 AM

@pfsense4ever
Please read through all the discussion above.

I got the UPS to connect by adding RUN_AS_USER root to Additional configuration lines for upsmon.conf. However, @dennypage says in this discussion that adding a USB quirk is a better solution. Still working on that.

dennypage · Nov 8, 2023, 6:21 AM

[Wrong thread, sorry]

dennypage · Nov 8, 2023, 6:35 AM

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

My best guess at the line to install the quirk is:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

But I am not at all confident I got that right. How does it look?

Looks good to me.

You can test without having to reboot by using this command:

usbconfig add_dev_quirk_vplh 0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE

followed by disconnect and reconnect of the USB cable.

There is a usbconfig option to show the attached driver if any. IIRC, it's show_ifdrv. Run it before and after the disconnect/reconnect and you should see that after there is no attached driver. Alternatively, you can simply try to start NUT (without root) to see if it works.

Note that using usbconfig to install the quirk is temporary and will not survive a reboot. So assuming it works, you will want to install it properly in /boot/loader.conf.local.

dennypage · Nov 9, 2023, 5:06 PM

Please see here regarding testing of the new pfSense NUT package. I would appreciate any help you can offer. Thanks!

[Note that the new package addresses the crash issue after successful start, but does not have anything regarding the quirk issue that prevents initial start]

dennypage · Nov 9, 2023, 5:36 PM

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

hspindel · Nov 10, 2023, 12:26 AM

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

When I have a chance to test it, I will be happy to post the results. Been rather busy. :-)

hspindel · Nov 10, 2023, 7:25 AM

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel Can you confirm that the quirk did in fact work? I'd like to use this as an example in a post I am doing explaining creating quirks. Thanks.

Mostly good results. Thank you for all the help.

The changes I had previously made to the NUT configuration were:

Extra Arguments to driver (optional)
interruptonly

Additional configuration lines for upsmon.conf
RUN_AS_USER root

Additional configuration lines for ups.conf
user=root
pollinterval=10

I deleted all of the above changes except pollinterval. Rebooted the pfSense router and verified that UPS connection did not work (as expected, wanted to make sure I was testing from a baseline).
Created the file /boot/loader.conf.local with contents:

hw.usb.quirk.0="0x09ae 0x2012 0x0000 0xffff UQ_HID_IGNORE"

Rebooted the pfSense router and was able to verify that the UPS was correctly monitored by NUT. Pulled the UPS power cord a couple times to verify that NUT detected that and it was handled correctly.
Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly
With the interruptonly flag present, the UPS connection is once again stable without need for setting root access. However,@dennypage mentioned above that this is not a good solution due to a memory corruption issue in the USB reconnect code (libusb). I am not seeing any problems and at present don't have a another solution that works.
Verified that with interruptonly present, NUT still responds correctly to unplugging and replugging the UPS.

dennypage · Nov 10, 2023, 3:51 PM

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

Excellent job creating the quirk!

Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly

Is this with the 2.8.0_2 package, or the new 2.8.1 package?

hspindel · Nov 11, 2023, 12:00 AM

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

Excellent job creating the quirk!

Left the configuration running. After about five minutes, NUT loses connection to the UPS. Only thing that fixed that was:

Extra Arguments to driver (optional): interruptonly

Is this with the 2.8.0_2 package, or the new 2.8.1 package?

I am running 2.8.0_2. The package manager for pfSense+ doesn't list a 2.8.1 upgrade available.

This is on a Netgate 8200.

dennypage · Nov 11, 2023, 6:52 AM

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I am running 2.8.0_2. The package manager for pfSense+ doesn't list a 2.8.1 upgrade available.

Please see the Call for testing here. I would appreciate it if you could give it a spin. Thanks.

hspindel · Nov 11, 2023, 6:53 AM

@dennypage said in NUT disconnects: Eaton SMART1500LCD:

@hspindel said in NUT disconnects: Eaton SMART1500LCD:

I am running 2.8.0_2. The package manager for pfSense+ doesn't list a 2.8.1 upgrade available.

Please see the Call for testing here. I would appreciate it if you could give it a spin. Thanks.

TLDR: v2.8.1 so far is working well.

Documenting everything I encountered in case it helps anyone else.

Had a heckuva time finding the nut-devel-2023.10.07_1 package. Could not find it on Redmine. Unfortunately, didn't write down where I eventually found it. For future writeups, a direct link to the package would likely be helpful. Also, note that the full package name I found is amd64_nut_devel-2023.10.07_1.pkg. (Of course, this is hardware-specific - I am running on a Netgate 8200).

I downloaded the package to my PC. Had another heckuva time transferring the package to the pfSense router. Could not get ftp to work - ftp said "file not found." This was with both a Windows ftp server and a Linux ftp server tried with multiple login credentials. Eventually solved this problem by installing a tftp server on pfSense and a tftp client on Windows. File successfully transferred with tftp.
Using the pfSense GUI, I deleted the interruptonly flag.
Using the pfSense GUI, I stopped the UPS service.
SSH into pfSense+. Uninstalled NUT with pkg remove -fy nut. No problems seen.
Installed new NUT with pkg install -fy /tmp/amd64_nut-devel-2023.10.07_1.pkg. Only problem seen was I initially forgot the "amd64" in front of the name. Once I got the name right, the package installed without issue.
Back to the pfSense GUI, I started the UPS service. After a short time, the UPS service came up fine.
Pulled power and restored power to the UPS. Verified that the UPS service handled this correctly - no problems seen.
Noticed that the pfSense+ GUI still reports the nut package version as 2.8.0_2. Not particularly worried about that - just reporting it.
Without the interruptonly flag, v2.8.0_2 would disconnect and reconnect approximately every five minutes. v.2.8.1 has now been running for about 90 minutes with no issues seen. Will leave this running at least overnight, but it seems to be that v2.8.1 (along with the previously installed quirk) has fixed the issues I've seen.

Thank you very much to @dennypage for the excellent support in getting this working.

hspindel · Nov 11, 2023, 10:28 PM

@dennypage
To further follow up: v.2.8.1 worked overnight with no issues! Just pulled power and reconnected as a final test, and NUT responded as expected.