NUT package (2.8.0 and below)

keyser

I have realized that all I did was re-compile the problematic v2.8.0 instead of the devel version. Oops. Here's the ARM verson of usbhid-ups from nut-devel-2023.06.06

usbhid-ups.gz

sha256sum of uncompressed file:
cdeb8d4400e0c721c878c0af084f48356323c29b7f9ef4fc526b4d9a3ff339a5 usbhid-ups

Hi Maltz. Thank you for your work. I have been running your compiled ARM64 usbhid.ups on my SG-2100 for about a week now, and I can confirm it at least solves the problem of loosing the UPS status at the random disconnects. It is now able to reconnect without addtional config on 23.05 (no run as root fx).
Here is a extract from my log when the disconnect happes, and it then reconnects successfully:

Jul 5 01:01:02 php-cgi 61460 rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Jul 4 23:36:29 upsmon 4478 Communications with UPS eaton5sc established
Jul 4 23:36:29 upsd 6047 UPS [eaton5sc] data is no longer stale
Jul 4 23:36:24 upsmon 4478 Communications with UPS eaton5sc lost
Jul 4 23:36:24 upsmon 4478 Poll UPS [eaton5sc] failed - Data stale
Jul 4 23:36:24 kernel ugen1.2: <EATON 5S> at usbus1
Jul 4 23:36:22 usbhid-ups 13495 libusb1: Could not open any HID devices: no USB buses found
Jul 4 23:36:21 kernel ugen1.2: <EATON 5S> at usbus1 (disconnected)
Jul 4 23:36:21 upsd 6047 Data for UPS [eaton5sc] is stale - check driver
Jul 4 23:36:21 usbhid-ups 13495 libusb1: Could not open any HID devices: insufficient permissions on everything
Jul 4 12:30:26 php-cgi 78812 rc.update_urltables: /etc/rc.update_urltables: pfB_Blocked_Countries_v4 does not need updating.

Any idea if the disconnect issue could be a voltage problem or some USB settings thing i pfSense? It seems strange driver says there is no USB buses found at disconnect time. I have never had issues on this SG-2100 with another UPS I have, so it could be a driver issue when something unforseen happens. The same UPS that shows issues here never disconnects on a Windows machine I have (or my Raspberry Pi).

dennypage

@keyser said in NUT package:

Any idea if the disconnect issue could be a voltage problem or some USB settings thing i pfSense? It seems strange driver says there is no USB buses found at disconnect time.

Some UPS brands, Notably Cyberpower, are well known to randomly disconnect on USB. I don't know why, but it's a fact of life with some manufactures.

I wouldn't read anything into the "no USB buses found" message. That's just a generic hard coded error that NUT reports when it has exhausted its attempts to open a USB HID device.

keyser

@dennypage I guess it’s something I have to live with. I just find it curious that the very same UPS with the same cable shows no such behaviour on Windows Installs or my Rasberry Pi (Rasberry Pi OS). So my logic says that indicates it something with the USB/USB Driver on pfSense.

dennypage

@keyser said in NUT package:

I just find it curious that the very same UPS with the same cable shows no such behaviour on Windows Installs or my Rasberry Pi (Rasberry Pi OS). The same UPS that shows issues here never disconnects on a Windows machine I have (or my Raspberry Pi).

There are a lot of things different among the three systems you list. USB chips, hub implementations, base drivers for both the USB chip and the hub, drivers for HID, etc. Too many differences to draw conclusions.

It could also be that it's happening on those systems as well but is just not reported. You would have to put a USB analyzer on it to know for sure. As this behavior is well known for some UPSs, I tend to put that as the highest likelihood.

Maltz

@keyser Maybe try googling the UPS model number and "NUT" or "Network UPS Tools", and/or the error from your logs and see what pops up. It's much more likely that the root cause is in upstream FreeBSD or NUT, like the other two recent issues have been, than in pfSense or this NUT wrapper package. (Assuming the UPS itself isn't being weird, a la CyberPower)

But dennypage is right - there are too many variables to do much troubleshooting without being hands-on with the specific gear.

lcbbcl

@dennypage i come back after a 2.7 clean install with the same error like before.
upsmon 85035 UPS [APC]: connect failed: Connection failure: Connection refused
I add user=root in ups.conf and also hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE" on loader.conf.local and it is not working anymore

dennypage

@lcbbcl You didn't give detail as to what "not working" means (permission failure, bus error, etc.), but the usual caveats about disconnect/reconnect or reboot following install still apply. The quirk definitely requires a reboot.

I would suggest that you install the updated usbhid-ups and reboot. If that doesn't work, then please post some detail about what you are seeing.

lcbbcl

@dennypage Done is fixed , i deleted and redo the loader.cond.local

Nuggets 0

@dennypage

Unfortunately this did not work for me. Even worse it seems to have broken it more. I would be able to restart the NUT service and it would work for a while and then crap out, but with the dev usbhid-ups, it won't connect at all.

pfSense+ 23.05.1-RELEASE (amd64)
NUT 2.8.0_2. UPS Type: Local USB. Driver: usbhid
Cyperpower CP1500AVRLCD connected over USB.
Was working fine prior to upgrading from pfsense CE 2.6.0. Already tried reinstalling the NUT package.

Tried the below to resolve:

interruptonly by itself
user=root by itself
Use the dev usbhid-ups by itself
Didn't try any combination of the above together.

Steps taken to use the dev usbhid_ups

Stopped NUT
Backed up original usbhid-ups from /usr/local/libexec/nut/
Removed and uploaded the dev usbhid-ups you provided.
chmod 755 (the permissions didn't match)
Ran "logger beforerestart" to have a clear spot in the logs to reference (you'll see that in the log snip below)
Started NUT. Failed to connect
Rebooted. Still failed to connect.
Stopped NUT, removed dev usbhid-ups, and restored original. Fixed permissions
Started NUT. Successfully connects on startup but disconnects after some time.

Logs from using the dev usbhid-ups

Sep  7 16:01:02 pfbox root[96703]: beforerestart
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Configuration Change: admin@10.10.10.123 (Local Database): Updated UPS settings
Sep  7 16:01:02 pfbox check_reload_status[435]: Syncing firewall
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Stopping service nut
Sep  7 16:01:02 pfbox upsmon[21097]: Signal 15: exiting
Sep  7 16:01:02 pfbox upsd[22142]: User local-monitor@::1 logged out from UPS [CP1500AVRLCD]
Sep  7 16:01:02 pfbox upsd[22142]: mainloop: Interrupted system call
Sep  7 16:01:02 pfbox upsd[22142]: Signal 15: exiting
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Starting service nut
Sep  7 16:01:02 pfbox upsmon[1506]: Startup successful
Sep  7 16:01:02 pfbox upsmon[2025]: UPS [CP1500AVRLCD]: connect failed: Connection failure: Connection refused
Sep  7 16:01:02 pfbox upsmon[2025]: Communications with UPS CP1500AVRLCD lost
Sep  7 16:01:03 pfbox upsd[14606]: listening on ::1 port 3493
Sep  7 16:01:03 pfbox upsd[14606]: listening on 127.0.0.1 port 3493
Sep  7 16:01:03 pfbox upsd[14606]: Can't connect to UPS [CP1500AVRLCD] (usbhid-ups-CP1500AVRLCD): Connection refused
Sep  7 16:01:03 pfbox upsd[14784]: Startup successful

On the original usbhid-ups, it starts up fine, and then later starts to have the same issue. This is with no additional options configured.

Sep  7 16:05:26 pfbox upsmon[86229]: Startup successful
Sep  7 16:05:26 pfbox upsmon[86705]: UPS [CP1500AVRLCD]: connect failed: Connection failure: Connection refused
Sep  7 16:05:26 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD lost
Sep  7 16:05:26 pfbox usbhid-ups[87908]: Startup successful
Sep  7 16:05:27 pfbox upsd[88031]: listening on ::1 port 3493
Sep  7 16:05:27 pfbox upsd[88031]: listening on 127.0.0.1 port 3493
Sep  7 16:05:27 pfbox upsd[88031]: Connected to UPS [CP1500AVRLCD]: usbhid-ups-CP1500AVRLCD
Sep  7 16:05:27 pfbox upsd[88080]: Startup successful
Sep  7 16:05:31 pfbox upsd[88080]: User local-monitor@::1 logged into UPS [CP1500AVRLCD]
Sep  7 16:05:31 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD established
###Gap in time where it was working###
Sep  7 17:52:08 pfbox upsd[88080]: Can't connect to UPS [CP1500AVRLCD] (usbhid-ups-CP1500AVRLCD): Connection refused
**Sep  7 17:52:08 pfbox kernel: pid 87908 (usbhid-ups), jid 0, uid 316: exited on signal 10**
Sep  7 17:52:13 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected
Sep  7 17:52:13 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD lost
Sep  7 17:52:18 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected
Sep  7 17:52:23 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected

I tried enabling debug in the additional options for NUT based on the NUT documentation but didn't see any debug messaging being output anywhere.

Any wisdom or guidance for me? My telegram bot is annoying the hell out of me

dennypage

@Nuggets-0 said in NUT package:

kernel: pid 87908 (usbhid-ups), jid 0, uid 316: exited on signal 10

I would suggest double checking the file checksums against the alternate usbhid-ups checksums posted back on March 12th.

Nuggets 0

@dennypage

Thanks for the quick response (and for all that you've done). I did do that when I originally did it but I did it again so I could get you the command output below. I use openhashtab which does it for you in the properties tab and confirms if it's a match if you paste in a value to compare it to. I copied the sha1 and sha256 values you provided and confirmed them when I downloaded and extracted. Then when I copied them to the system, I ran sha1 and sha256 on the file and compared it to the confirmed downloaded copy which also match the values you provided.

/usr/local/libexec/nut: sha1 usbhid-ups; sha256 usbhid-ups
SHA1 (usbhid-ups) = 49ce9131502bfb8b789ee97b7fb3fc81fc9f8fff
SHA256 (usbhid-ups) = 999a2653559dbc50ecc8ba592a67587b1e307a1495f6e8ebbd3d8e90e3967133

I'm using the usbhid-ups you provided on that post from ~~July 7th~~ March 12th (the one I replied to). Is that the right one or is there an updated one?

EDIT: ~~Just realized you said there was one on March 12th. I'll try that one and report back.~~
Not sure where I got July 7th, but the March 12th one is indeed the one I tried. SHA1 and SHA256 above match it and I even have it in my notes that it's the one I tried. Just mixed up a date when typing this post.

dennypage

@Nuggets-0 Sorry, I mixed up which was dev and which was original. Looking at your logs, the original version is the one that is core dumping (as you would expect).

The logs with the dev version do not show any output from usbhid-ups. You are on an x86_64 system, yes? If so, can you try starting the executable by hand please?

If you are on a ARM instead of an x86_64, you need to use a different executable. I believe that someone else posted an ARM executable. [Edit: On Jul 7, 2023]

Nuggets 0

@dennypage said in NUT package:

@Nuggets-0 Sorry, I mixed up which was dev and which was original. Looking at your logs, the original version is the one that is core dumping (as you would expect).

The logs with the dev version do not show any output from usbhid-ups. You are on an x86_64 system, yes? If so, can you try starting the executable by hand please?

If you are on a ARM instead of an x86_64, you need to use a different executable. I believe that someone else posted an ARM executable. [Edit: On Jul 7, 2023]

Yes, I'm on x86_64. I tried to perform your instructions with the dev usbhid-ups

I tested it both in the CLI and from the Execute Shell Command in pfSense. They both had the same issue

Command:

/usr/local/libexec/nut/usbhid-ups -a CP1500AVRLCD

Response:

Can't chdir to /var/db/nut: Permission denied
Network UPS Tools - Generic HID driver 0.49 (11-eol-49673-g687a1b3d4995)
USB communication driver (libusb 1.0) 0.45

So in my SSH session, I see that /var/db/nut was set to 750. I set it to 777 (for testing) and ran again. This is the response:

Network UPS Tools - Generic HID driver 0.49 (11-eol-49673-g687a1b3d4995)
USB communication driver (libusb 1.0) 0.45
libusb1: Could not open any HID devices: no USB buses found
No matching HID UPS found

I ran the usbconfig -v command you recommended before and I see the UPS. If I put in the old usbhid-ups without any reboots, it connects just fine (and eventually cores as you mentioned)

ugen0.2: <CPS CP1500AVRLCDa> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (2mA)

In all of this I can't figure out where the issue is or if I'm running the command incorrect to launch it manually. I've also moved it to multiple USB ports. There are no logs in the syslog as the usbhid-ups never ran successfully. I also never see the pid file being generated in /var/db/nut/ for usbhid-ups. I set /var/db/nut back to 750 as it was originally until I hear back. The original usbhid-ups is functioning again until it cores so I haven't broken anything yet in my troubleshooting

Any suggestions?

Edit: If you need more output from usbconfig -v, let me know. The post was being flagged as spam with the more verbose information I had tried to put in.

dennypage

@Nuggets-0

Do

chown nut /var/db/nut

and restart the service.

Nuggets 0

@dennypage
Didn't want to leave you hanging.

So I had accidentally started the regular usbhid-ups with the 777 permissions on /var/db/nut. It ran for over 12 hours without a disconnect. I decided to see if I could reproduce with 750 on /var/db/nut. So far it's been running for 5 hours without disconnect. Most of the time it would only last 2-3 hour at max before a disconnect. I didn't do the chown because nut was already owner for both user and group.

I looked at the /var/db/nut/ files beforehand and after changing the permissions to see if a file wasn't getting updated, but all of them were listing pids correctly. The only file I couldn't check the contents of was the socket file. But all files were removed and created with the start and stop of the service with both 750 and 777 on /var/db/nut. So I have no idea if this fixed it or what. I'll come back if I run into any issues.

Thanks again for all your assistant and quick responses.

dennypage

@Nuggets-0 The permissions will be a problem again when you reboot. It's only my list to fix as soon as I can.

Nuggets 0

@dennypage said in NUT package:

@Nuggets-0

Do
chown nut /var/db/nut
and restart the service.

So naturally, my previous steps didn't fix it.

Running chown as requested didn't fix it as /var/db/nut was already owned by nut.

However, with the dev usbhid-ups, I set user=nut in ups.conf and it came up. For the record, user=root worked too but I wanted to get it working with the nut user instead of root. Not sure why user=nut works. The daemon is supposed to start as the user nut and nut has all the right properties (owner and 750). But at least it's working on the dev usbhid-ups now.

I'll monitor for a bit and come back.

incith

@Nuggets-0 it's a local daemon just run it as root why complicate things

Nuggets 0

@incith said in NUT package:

@Nuggets-0 it's a local daemon just run it as root why complicate things

Both user=root and user=nut worked both the same. I merely tested using user=nut on a whim and it worked.

I have my upsd hosting the NUT server so it's available to other servers running on the same UPS. I want pfsense to be the last server running should everything else need to go down, so it makes sense to host it on pfsense. I have rules in place as well as a user/password to prevent from unfettered access. So having it run as root is a potential risk to me, not so much others.

What I don't understand is WHY it works when using user=nut. usbhid-ups is supposed to start as the nut user anyways. Here usbhid-ups running on the original file:
79502 nut 1 20 0 13M 3420K select 0 0:00 0.01% usbhid-ups

Using user=nut The nut user is the owner and has full access to associated files. user=nut is telling the nut user to act like itself. Of course, this could all be my confusion with what is getting started with nut vs root.

Edit: Forgot to mention, it had been running fine under user=nut for at least 12 hours. Telegram bots had an outage yesterday so it did some funky stuff in pfsense with my notifications, so I restarted the NUT service so it would initiate monitoring again. Will update probably tomorrow to give it more time to run into any issues.

dennypage

@Nuggets-0 The old version wanted to be "uucp" and the new version wants to be "nut", so now there is a disagreement between the NUT executable and the pfSense NUT package.