NUT package (2.8.0 and below)

dennypage

For those of you affected by the APC issue (and those that will be in the future):

First, I believe that the issue only affects recent APC UPS designs. Those in the last couple of years or so. The core issue is that a new device identification number has been introduced by APC, which FreeBSD is not aware of. Since the kernel does not recognize the device as a UPS that should be ignored, the kernel is attaching an internal driver to it. The presence of this internal driver on the device precludes any non root user from attaching to the device.

This cannot be reasonably fixed in the pfSense nut package or the underlying FreeBSD nut package. The permanent solution requires an adjustment to the usb quirk table in the kernel itself. A patch has been established for the kernel, but I don't know how long it will take to flow through.

In the interim, there are two solutions available:

The first solution, as noted previously, is to add the line

user=root

to the "Additional configuration lines for ups.conf" section under Advanced settings. This will cause the usbhid-ups command to be run as root. The disadvantage of this approach is that you are running the driver as root, which may be of concern to some.

The alternative solution is to add the line

hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE"

to file "/boot/loader.conf.local". Note that you may have to create this file. The disadvantage of this approach is that you will have to reboot pfSense before it will work.
If you previously implemented the user=root approach, you can remove the user=root line following the reboot.

dennypage

For those of you affected by the CyberPower issue (and those that will be in the future):

First, I want to thank all of you for your help in tracking this down. Particularly @shaffergr, who was kind enough, and trusting enough, to run test builds for me against his CyberPower.

I tracked this down to a double free by the usb code in nut. FWIW, I don’t believe that this issue is completely limited to CyberPower UPSs, but it is probably pretty difficult to encounter with other units as hitting the issue requires a reconnect of the UPS on the bus. CyberPower units are well known to randomly disconnect and reconnect seemingly at random.

For those wondering why this issue did not happen in prior to 23.01: 22.05 and below used nut version 2.7.4. In 2.7.4, nut did not actually close the device when a disconnect happened. Presumably, this may have resulted in a memory leak at shutdown, but I didn’t explore enough to confirm. In 2.8.0, when a device disconnects nut actually closes the device when it disconnects. In fact, it closes it twice on all systems other than Linux. There is even a comment in the code noting how the double close would cause corruption on Linux systems. Unfortunately contrary to the code comments it also causes corruption in FreeBSD.

Having traced this down in the 2.8.0 release code, when I went to the current development version I surprised to find that someone had beat me to it, and fixed the issue back in August. It's just not been release yet. :)

To resolve this will require a new version of nut. Either a new release from the nut team, or for the pfSense team to move from the 2.8.0 release version to the current development version of nut. The nut team is looking toward 2.8.1, but it appears that they have a few things they still want to address before putting that out. I will explore the concept of moving to nut-devel with the pfSense team as time permits (I think they are pretty busy right now). But no matter which way it goes, it’s going to take some time.

In the interim, the only known work-around using the release code (discovered by @tman222) is to add the line

interruptonly

to the Extra Arguments to driver section. This will cut down some of the information you can see about your UPS, but the important stuff needed to monitor and shutdown should still be there.

Alternatively, if brave souls are interested, I have a build of usbhid-ups made from the FreeBSD nut-devel package. I do not have a CyberPower available so I haven't been able to directly test it, but I expect that it will work.

If you you decide you would like a copy, reach out to me and I will see about getting it to you. For reference, the shasum and sha256sum checksums are:

49ce9131502bfb8b789ee97b7fb3fc81fc9f8fff  usbhid-ups
999a2653559dbc50ecc8ba592a67587b1e307a1495f6e8ebbd3d8e90e3967133  usbhid-ups

dennypage

I'm going to sleep now...

tnowak

Hi!
Last year I was struggling to connect to Ever ECO Pro 1200 AVR CDS UPS.
This was mostly because nut was in version 2.7 and support for that UPS has been added in 2.8.

Lately I tried it again. This time nut package is already in version 2.8 (2.8.0_2).
I've installed it and configured with usbhid driver and some basic extra arguments:

port=auto
vendorid=2e51
productid=0000

Unfortunately, it still doesn't work. Why?

Log:

Feb 22 20:42:29	upsmon	71579	Poll UPS [ever] failed - Driver not connected

tnowak

dennypage

@tnowak First, I would recommend removing everything from the extra arguments section. Following that, test again. If it doesn't work please post the output from usbhid-ups, either from the system log or from the command line. The command line would be this:

/usr/local/libexec/nut/usbhid-ups -a ever

FYI, the port=auto is handled by the package and should never be added to the extra arguments section for usbhid. The vendorid/productid can be added back later if necessary.

tnowak

This post is deleted!

tnowak

@dennypage Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

dennypage

@tnowak said in NUT package:

Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

I expect that you are actually in the same situation as the new gen APC listed above: No quirk covering your UPS device (I actually don't see anything from Ever in the table at all).

To confirm, use these steps:

disable nut (Services / UPS / Settings) and save the config
unplug the usb connection to the ups and wait 5 seconds
re-plug the usb connection to the ups
run "usbconfig -d ugen0.2 show_ifdrv"

and post the result. My expectation is that you will see two lines, similar to this:

ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)
ugen0.2.0: uhid0: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047, class 0/0, rev 2.00/0.01, addr 1>

tman222

Hi @dennypage - thanks again for all your help looking into the signal 10/11 issue with CyberPower UPS units. I'm fine to continue running mine with the interruptonly flag workaround for now even if fewer variables are monitored. After a couple days of running this way, things appear to be stable. If it this setup ends up crashing at some point, I'll probably give the updated usbhid-ups driver a try. Also, if you do end up releasing a nut-devel package at some point that includes fixes post 2.8.0, I'd be happy to try that out as well.

JonathanLee

@dennypage You are amazing I appreciate all you do. Again, thanks for taking the time to look into this issue reported within this discussion. It seems to be a problem with many other users now and you already have a solid solution for it.

tnowak

@dennypage said in NUT package:

@tnowak said in NUT package:

Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

I expect that you are actually in the same situation as the new gen APC listed above: No quirk covering your UPS device (I actually don't see anything from Ever in the table at all).

To confirm, use these steps:

disable nut (Services / UPS / Settings) and save the config

unplug the usb connection to the ups and wait 5 seconds

re-plug the usb connection to the ups

run "usbconfig -d ugen0.2 show_ifdrv"

and post the result. My expectation is that you will see two lines, similar to this:
ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)
ugen0.2.0: uhid0: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047, class 0/0, rev 2.00/0.01, addr 1>

Result:

ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.2.0: uhid0: <EVER ECO PRO AVR CDS, class 0/0, rev 2.00/1.00, addr 2>

PS. I've also noticed a problem with nut loosing connection to this UPS even with user=root after some time. Then when I restart nut it shows up again.

dennypage

@tnowak said in NUT package:

Result:
ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.2.0: uhid0: <EVER ECO PRO AVR CDS, class 0/0, rev 2.00/1.00, addr 2>

PS. I've also noticed a problem with nut loosing connection to this UPS even with user=root after some time. Then when I restart nut it shows up again.

Yep, that shows a kernel driver attached to the device. Same situation as the new series APC devices. Not too surprising, because I don't find any Ever devices defined in the usb table.

You can either use the user=root approach, or you can develop a quirk setting for /boot/loader.conf.local. Based on your prior post, I believe that the correct value would be this:

hw.usb.quirk.0="0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE"

You can test this in advance by running this:

usbconfig add_dev_quirk_vplh 0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE

followed by unplugging and replugging the usb cable to your ups. If the values are correct, when you run "usbconfig -d ugen0.2 show_ifdrv" again, you should only see one line of output like so:

ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

The ugen0.2.0 should be gone.

If this test works then you can add the line to your /boot/loader.conf.local file.

As to loosing communication after a time, I would still need to see the output from usbhid-ups, either from the system log or from the command line. There seem to be a few issues that do not produce entries in the system log, so I would recommend using the command line as previously discussed.

tnowak

@dennypage said in NUT package:

hw.usb.quirk.0="0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE"
You can test this in advance by running this:
usbconfig add_dev_quirk_vlph 0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE
followed by unplugging and replugging the usb cable to your ups. If the values are correct, when you run "usbconfig -d ugen0.2 show_ifdrv" again, you should only see one line of output like so:
ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
The ugen0.2.0 should be gone.

Thanks, this was very helpfull. I had to modifiy command a bit, as I've noticed its add_dev_quirk_vplh not vlph and I changed pid (product id) to 0x0000. Now the second line is gone:

Now NUT starts without user=root just fine:

Feb 23 22:27:05	upsmon	20539	Communications with UPS ever established
Feb 23 22:27:05	upsd	23193	User local-monitor@127.0.0.1 logged into UPS [ever]
Feb 23 22:27:01	php	16832	/usr/local/sbin/acbupload.php: End of configuration backup to https://acb.netgate.com/save (success).
Feb 23 22:27:01	upsd	23193	Startup successful
Feb 23 22:27:01	upsd	23193	Connected to UPS [ever]: usbhid-ups-ever
Feb 23 22:27:01	upsd	23193	listening on 127.0.0.1 port 3493
Feb 23 22:27:01	upsd	23193	listening on ::1 port 3493
Feb 23 22:27:00	usbhid-ups	21611	Startup successful

But soon after one minute or so:

Feb 23 22:28:15	upsmon	20539	Poll UPS [ever] failed - Driver not connected
Feb 23 22:28:10	upsmon	20539	Communications with UPS ever lost
Feb 23 22:28:10	upsmon	20539	Poll UPS [ever] failed - Driver not connected
Feb 23 22:28:08	kernel		pid 21611 (usbhid-ups), jid 0, uid 66: exited on signal 10
Feb 23 22:28:08	upsd	23193	Can't connect to UPS [ever] (usbhid-ups-ever): Connection refused

dennypage

@tnowak said in NUT package:

I had to modifiy command a bit, as I've noticed its add_dev_quirk_vplh not vlph

Sorry, typo. I corrected the original post.

But soon after one minute or so:
Feb 23 22:28:08 kernel pid 21611 (usbhid-ups), jid 0, uid 66: exited on signal 10

Congratulations, you are a double winner.
The post above regarding the CyberPower UPS units applies to you as well.

tnowak

@dennypage said in NUT package:

Congratulations, you are a double winner.

Wow, amazing! I deployed that workaround for the time being and it works reliably now! Looking forward for future nut package releases that solves this issue.

You're the man @dennypage! Thank you VERY much for your support that is extremely competent and helpful.

shaffergr

@dennypage

the "usbhid-ups" binary built from the FreeBSD nut-devel port you provided earlier this week to test looks to have solved the problem for me (or at least for those that have CyberPower UPSs). As of this morning eastern time, it's been running for over 72 hours with no more "exit on signal 10" errors in my system log file. Thanks for all your help in identifying the issue and that its already been fixed in the newer version of nut.

JonathanLee

@shaffergr is this available form package manager now?

shaffergr

@jonathanlee

No. Denny provided me a build from nut-devel branch so that we could validate if the signal 10 issue was fix or not.

JPP 0

@dennypage I've been seeing logs like this:

Feb 25 19:37:13 gatekeeper kernel: pid 29800 (usbhid-ups), jid 0, uid 0: exited on signal 10
Feb 25 19:37:15 gatekeeper upsmon[28298]: Poll UPS [tripplite] failed - Driver not connected
Feb 25 19:37:15 gatekeeper upsmon[28298]: Communications with UPS tripplite lost
Feb 25 19:37:20 gatekeeper upsmon[28298]: Poll UPS [tripplite] failed - Driver not connected
Feb 25 19:37:25 gatekeeper upsmon[28298]: Poll UPS [tripplite] failed - Driver not connected
Feb 25 19:37:30 gatekeeper upsmon[28298]: Poll UPS [tripplite] failed - Driver not connected
Feb 25 19:37:35 gatekeeper upsmon[28298]: Poll UPS [tripplite] failed - Driver not connected

<goes on the same until manually restarted.>

setting interruptonly appears to have mitigated it. UPS is a Tripp Lite SMART1500LCD rack mount unit. It seemed to work fine prior to the last update.