NUT package (2.8.0 and below)

dennypage

I just find it curious that the very same UPS with the same cable shows no such behaviour on Windows Installs or my Rasberry Pi (Rasberry Pi OS). The same UPS that shows issues here never disconnects on a Windows machine I have (or my Raspberry Pi).

There are a lot of things different among the three systems you list. USB chips, hub implementations, base drivers for both the USB chip and the hub, drivers for HID, etc. Too many differences to draw conclusions.

It could also be that it's happening on those systems as well but is just not reported. You would have to put a USB analyzer on it to know for sure. As this behavior is well known for some UPSs, I tend to put that as the highest likelihood.

Maltz

@keyser Maybe try googling the UPS model number and "NUT" or "Network UPS Tools", and/or the error from your logs and see what pops up. It's much more likely that the root cause is in upstream FreeBSD or NUT, like the other two recent issues have been, than in pfSense or this NUT wrapper package. (Assuming the UPS itself isn't being weird, a la CyberPower)

But dennypage is right - there are too many variables to do much troubleshooting without being hands-on with the specific gear.

lcbbcl

@dennypage i come back after a 2.7 clean install with the same error like before.
upsmon 85035 UPS [APC]: connect failed: Connection failure: Connection refused
I add user=root in ups.conf and also hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE" on loader.conf.local and it is not working anymore

dennypage

@lcbbcl You didn't give detail as to what "not working" means (permission failure, bus error, etc.), but the usual caveats about disconnect/reconnect or reboot following install still apply. The quirk definitely requires a reboot.

I would suggest that you install the updated usbhid-ups and reboot. If that doesn't work, then please post some detail about what you are seeing.

lcbbcl

@dennypage Done is fixed , i deleted and redo the loader.cond.local

Nuggets 0

@dennypage

Unfortunately this did not work for me. Even worse it seems to have broken it more. I would be able to restart the NUT service and it would work for a while and then crap out, but with the dev usbhid-ups, it won't connect at all.

pfSense+ 23.05.1-RELEASE (amd64)
NUT 2.8.0_2. UPS Type: Local USB. Driver: usbhid
Cyperpower CP1500AVRLCD connected over USB.
Was working fine prior to upgrading from pfsense CE 2.6.0. Already tried reinstalling the NUT package.

Tried the below to resolve:

interruptonly by itself
user=root by itself
Use the dev usbhid-ups by itself
Didn't try any combination of the above together.

Steps taken to use the dev usbhid_ups

Stopped NUT
Backed up original usbhid-ups from /usr/local/libexec/nut/
Removed and uploaded the dev usbhid-ups you provided.
chmod 755 (the permissions didn't match)
Ran "logger beforerestart" to have a clear spot in the logs to reference (you'll see that in the log snip below)
Started NUT. Failed to connect
Rebooted. Still failed to connect.
Stopped NUT, removed dev usbhid-ups, and restored original. Fixed permissions
Started NUT. Successfully connects on startup but disconnects after some time.

Logs from using the dev usbhid-ups

Sep  7 16:01:02 pfbox root[96703]: beforerestart
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Configuration Change: admin@10.10.10.123 (Local Database): Updated UPS settings
Sep  7 16:01:02 pfbox check_reload_status[435]: Syncing firewall
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Stopping service nut
Sep  7 16:01:02 pfbox upsmon[21097]: Signal 15: exiting
Sep  7 16:01:02 pfbox upsd[22142]: User local-monitor@::1 logged out from UPS [CP1500AVRLCD]
Sep  7 16:01:02 pfbox upsd[22142]: mainloop: Interrupted system call
Sep  7 16:01:02 pfbox upsd[22142]: Signal 15: exiting
Sep  7 16:01:02 pfbox php-fpm[396]: /nut_settings.php: Starting service nut
Sep  7 16:01:02 pfbox upsmon[1506]: Startup successful
Sep  7 16:01:02 pfbox upsmon[2025]: UPS [CP1500AVRLCD]: connect failed: Connection failure: Connection refused
Sep  7 16:01:02 pfbox upsmon[2025]: Communications with UPS CP1500AVRLCD lost
Sep  7 16:01:03 pfbox upsd[14606]: listening on ::1 port 3493
Sep  7 16:01:03 pfbox upsd[14606]: listening on 127.0.0.1 port 3493
Sep  7 16:01:03 pfbox upsd[14606]: Can't connect to UPS [CP1500AVRLCD] (usbhid-ups-CP1500AVRLCD): Connection refused
Sep  7 16:01:03 pfbox upsd[14784]: Startup successful

On the original usbhid-ups, it starts up fine, and then later starts to have the same issue. This is with no additional options configured.

Sep  7 16:05:26 pfbox upsmon[86229]: Startup successful
Sep  7 16:05:26 pfbox upsmon[86705]: UPS [CP1500AVRLCD]: connect failed: Connection failure: Connection refused
Sep  7 16:05:26 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD lost
Sep  7 16:05:26 pfbox usbhid-ups[87908]: Startup successful
Sep  7 16:05:27 pfbox upsd[88031]: listening on ::1 port 3493
Sep  7 16:05:27 pfbox upsd[88031]: listening on 127.0.0.1 port 3493
Sep  7 16:05:27 pfbox upsd[88031]: Connected to UPS [CP1500AVRLCD]: usbhid-ups-CP1500AVRLCD
Sep  7 16:05:27 pfbox upsd[88080]: Startup successful
Sep  7 16:05:31 pfbox upsd[88080]: User local-monitor@::1 logged into UPS [CP1500AVRLCD]
Sep  7 16:05:31 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD established
###Gap in time where it was working###
Sep  7 17:52:08 pfbox upsd[88080]: Can't connect to UPS [CP1500AVRLCD] (usbhid-ups-CP1500AVRLCD): Connection refused
**Sep  7 17:52:08 pfbox kernel: pid 87908 (usbhid-ups), jid 0, uid 316: exited on signal 10**
Sep  7 17:52:13 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected
Sep  7 17:52:13 pfbox upsmon[86705]: Communications with UPS CP1500AVRLCD lost
Sep  7 17:52:18 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected
Sep  7 17:52:23 pfbox upsmon[86705]: Poll UPS [CP1500AVRLCD] failed - Driver not connected

I tried enabling debug in the additional options for NUT based on the NUT documentation but didn't see any debug messaging being output anywhere.

Any wisdom or guidance for me? My telegram bot is annoying the hell out of me

dennypage

@Nuggets-0 said in NUT package:

kernel: pid 87908 (usbhid-ups), jid 0, uid 316: exited on signal 10

I would suggest double checking the file checksums against the alternate usbhid-ups checksums posted back on March 12th.

Nuggets 0

@dennypage

Thanks for the quick response (and for all that you've done). I did do that when I originally did it but I did it again so I could get you the command output below. I use openhashtab which does it for you in the properties tab and confirms if it's a match if you paste in a value to compare it to. I copied the sha1 and sha256 values you provided and confirmed them when I downloaded and extracted. Then when I copied them to the system, I ran sha1 and sha256 on the file and compared it to the confirmed downloaded copy which also match the values you provided.

/usr/local/libexec/nut: sha1 usbhid-ups; sha256 usbhid-ups
SHA1 (usbhid-ups) = 49ce9131502bfb8b789ee97b7fb3fc81fc9f8fff
SHA256 (usbhid-ups) = 999a2653559dbc50ecc8ba592a67587b1e307a1495f6e8ebbd3d8e90e3967133

I'm using the usbhid-ups you provided on that post from ~~July 7th~~ March 12th (the one I replied to). Is that the right one or is there an updated one?

EDIT: ~~Just realized you said there was one on March 12th. I'll try that one and report back.~~
Not sure where I got July 7th, but the March 12th one is indeed the one I tried. SHA1 and SHA256 above match it and I even have it in my notes that it's the one I tried. Just mixed up a date when typing this post.

dennypage

@Nuggets-0 Sorry, I mixed up which was dev and which was original. Looking at your logs, the original version is the one that is core dumping (as you would expect).

The logs with the dev version do not show any output from usbhid-ups. You are on an x86_64 system, yes? If so, can you try starting the executable by hand please?

If you are on a ARM instead of an x86_64, you need to use a different executable. I believe that someone else posted an ARM executable. [Edit: On Jul 7, 2023]

Nuggets 0

@dennypage said in NUT package:

@Nuggets-0 Sorry, I mixed up which was dev and which was original. Looking at your logs, the original version is the one that is core dumping (as you would expect).

The logs with the dev version do not show any output from usbhid-ups. You are on an x86_64 system, yes? If so, can you try starting the executable by hand please?

If you are on a ARM instead of an x86_64, you need to use a different executable. I believe that someone else posted an ARM executable. [Edit: On Jul 7, 2023]

Yes, I'm on x86_64. I tried to perform your instructions with the dev usbhid-ups

I tested it both in the CLI and from the Execute Shell Command in pfSense. They both had the same issue

Command:

/usr/local/libexec/nut/usbhid-ups -a CP1500AVRLCD

Response:

Can't chdir to /var/db/nut: Permission denied
Network UPS Tools - Generic HID driver 0.49 (11-eol-49673-g687a1b3d4995)
USB communication driver (libusb 1.0) 0.45

So in my SSH session, I see that /var/db/nut was set to 750. I set it to 777 (for testing) and ran again. This is the response:

Network UPS Tools - Generic HID driver 0.49 (11-eol-49673-g687a1b3d4995)
USB communication driver (libusb 1.0) 0.45
libusb1: Could not open any HID devices: no USB buses found
No matching HID UPS found

I ran the usbconfig -v command you recommended before and I see the UPS. If I put in the old usbhid-ups without any reboots, it connects just fine (and eventually cores as you mentioned)

ugen0.2: <CPS CP1500AVRLCDa> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (2mA)

In all of this I can't figure out where the issue is or if I'm running the command incorrect to launch it manually. I've also moved it to multiple USB ports. There are no logs in the syslog as the usbhid-ups never ran successfully. I also never see the pid file being generated in /var/db/nut/ for usbhid-ups. I set /var/db/nut back to 750 as it was originally until I hear back. The original usbhid-ups is functioning again until it cores so I haven't broken anything yet in my troubleshooting

Any suggestions?

Edit: If you need more output from usbconfig -v, let me know. The post was being flagged as spam with the more verbose information I had tried to put in.

dennypage

@Nuggets-0

Do

chown nut /var/db/nut

and restart the service.

Nuggets 0

@dennypage
Didn't want to leave you hanging.

So I had accidentally started the regular usbhid-ups with the 777 permissions on /var/db/nut. It ran for over 12 hours without a disconnect. I decided to see if I could reproduce with 750 on /var/db/nut. So far it's been running for 5 hours without disconnect. Most of the time it would only last 2-3 hour at max before a disconnect. I didn't do the chown because nut was already owner for both user and group.

I looked at the /var/db/nut/ files beforehand and after changing the permissions to see if a file wasn't getting updated, but all of them were listing pids correctly. The only file I couldn't check the contents of was the socket file. But all files were removed and created with the start and stop of the service with both 750 and 777 on /var/db/nut. So I have no idea if this fixed it or what. I'll come back if I run into any issues.

Thanks again for all your assistant and quick responses.

dennypage

@Nuggets-0 The permissions will be a problem again when you reboot. It's only my list to fix as soon as I can.

Nuggets 0

@dennypage said in NUT package:

@Nuggets-0

Do
chown nut /var/db/nut
and restart the service.

So naturally, my previous steps didn't fix it.

Running chown as requested didn't fix it as /var/db/nut was already owned by nut.

However, with the dev usbhid-ups, I set user=nut in ups.conf and it came up. For the record, user=root worked too but I wanted to get it working with the nut user instead of root. Not sure why user=nut works. The daemon is supposed to start as the user nut and nut has all the right properties (owner and 750). But at least it's working on the dev usbhid-ups now.

I'll monitor for a bit and come back.

incith

@Nuggets-0 it's a local daemon just run it as root why complicate things

Nuggets 0

@incith said in NUT package:

@Nuggets-0 it's a local daemon just run it as root why complicate things

Both user=root and user=nut worked both the same. I merely tested using user=nut on a whim and it worked.

I have my upsd hosting the NUT server so it's available to other servers running on the same UPS. I want pfsense to be the last server running should everything else need to go down, so it makes sense to host it on pfsense. I have rules in place as well as a user/password to prevent from unfettered access. So having it run as root is a potential risk to me, not so much others.

What I don't understand is WHY it works when using user=nut. usbhid-ups is supposed to start as the nut user anyways. Here usbhid-ups running on the original file:
79502 nut 1 20 0 13M 3420K select 0 0:00 0.01% usbhid-ups

Using user=nut The nut user is the owner and has full access to associated files. user=nut is telling the nut user to act like itself. Of course, this could all be my confusion with what is getting started with nut vs root.

Edit: Forgot to mention, it had been running fine under user=nut for at least 12 hours. Telegram bots had an outage yesterday so it did some funky stuff in pfsense with my notifications, so I restarted the NUT service so it would initiate monitoring again. Will update probably tomorrow to give it more time to run into any issues.

dennypage

@Nuggets-0 The old version wanted to be "uucp" and the new version wants to be "nut", so now there is a disagreement between the NUT executable and the pfSense NUT package.

Samlink

Hello, I recently re-activated the SMTP notifications after 1 year of using only telegram (because of https://redmine.pfsense.org/issues/14031), but I no longer receive NUT notifications on email, only telegram.

On nut settings, the Enable notifications is checked.
I'm running 2.8.0_2.

Am I missing something?

dennypage

@Samlink Unfortunately, the fix for the Redmine issue you referred to broke the smtp notification mechanism for non root processes. I am hoping this gets fixed in the next release set.

In the interim, you should be able to work around the issue by adding

RUN_AS_USER root

in the "Additional configuration lines for upsmon.conf" section.

daplumber

So here's another data point:

Finally got around to buying a new UPS for my pfsense box (A 4 core i5-6500 Dell Optiplex)
pfsense 23.05.1-RELEASE and nut 2.8.0_2
CyberPower CST135UC2 $120 from Costco - I really should have known better, but I'd used CPS UPS' with pfsense NUT before with no issues. Sigh.

After much futzing and the dreaded CPS disconnect yo-yo and much searching I finally found this thread.

So I now have a stable UPS, yay.

All it took was adding the "user=root" to ups.conf
AND
"interruptonly" to driver arguments.

I see far fewer parameters, but I can live with the workarounds. For grins and giggles they are:

Variable	Value
battery.charge	100
battery.runtime	16525
battery.runtime.low	300
device.mfr	CPS
device.model	CST135UC2
device.serial	CDZNP7001732
device.type	ups
driver.flag.interruptonly	enabled
driver.name	usbhid-ups
driver.parameter.pollfreq	30
driver.parameter.pollinterval	2
driver.parameter.port	auto
driver.parameter.synchronous	auto
driver.version	2.8.0
driver.version.data	CyberPower HID 0.6
driver.version.internal	0.47
driver.version.usb	libusb-1.0.0 (API: 0x1000102)
ups.beeper.status	disabled
ups.mfr	CPS
ups.model	CST135UC2
ups.productid	0601
ups.serial	CDZNP7001732
ups.status	OL
ups.vendorid	0764

and here's the "usbconfig -v -d ugen0.5 show_ifdrv" output:

ugen0.5: <CPS CST135UC2> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

  bLength = 0x0012 
  bDescriptorType = 0x0001 
  bcdUSB = 0x0200 
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000 
  bDeviceProtocol = 0x0000 
  bMaxPacketSize0 = 0x0040 
  idVendor = 0x0764 
  idProduct = 0x0601 
  bcdDevice = 0x0200 
  iManufacturer = 0x0003  <retrieving string failed>
  iProduct = 0x0001  <CST135UC2>
  iSerialNumber = 0x0002  <CDZNP7001732>
  bNumConfigurations = 0x0001 


 Configuration index 0

    bLength = 0x0009 
    bDescriptorType = 0x0002 
    wTotalLength = 0x0029 
    bNumInterfaces = 0x0001 
    bConfigurationValue = 0x0001 
    iConfiguration = 0x0000  <no string>
    bmAttributes = 0x00c0 
    bMaxPower = 0x0032 

    Interface 0
      bLength = 0x0009 
      bDescriptorType = 0x0004 
      bInterfaceNumber = 0x0000 
      bAlternateSetting = 0x0000 
      bNumEndpoints = 0x0002 
      bInterfaceClass = 0x0003  <HID device>
      bInterfaceSubClass = 0x0000 
      bInterfaceProtocol = 0x0000 
      iInterface = 0x0000  <no string>

      Additional Descriptor

      bLength = 0x09
      bDescriptorType = 0x21
      bDescriptorSubType = 0x11
       RAW dump: 
       0x00 | 0x09, 0x21, 0x11, 0x01, 0x21, 0x01, 0x22, 0xd6, 
       0x08 | 0x02

     Endpoint 0
        bLength = 0x0007 
        bDescriptorType = 0x0005 
        bEndpointAddress = 0x0081  <IN>
        bmAttributes = 0x0003  <INTERRUPT>
        wMaxPacketSize = 0x0040 
        bInterval = 0x0020 
        bRefresh = 0x0000 
        bSynchAddress = 0x0000 

     Endpoint 1
        bLength = 0x0007 
        bDescriptorType = 0x0005 
        bEndpointAddress = 0x0002  <OUT>
        bmAttributes = 0x0003  <INTERRUPT>
        wMaxPacketSize = 0x0040 
        bInterval = 0x0020 
        bRefresh = 0x0000 
        bSynchAddress = 0x0000

The really weird part is the id wasn't recognized on some of the ports, which considering they're all ugen0 was a little freaky.

So that's what I get for being lazy and cheap. Sigh again.

Now I just have to remember to remove everything when the nut package eventually gets updated.

Thank you for all the hard work here, and I volunteer as guinea pig for testing patched code if needed. Since the Optiplex is Intel i5, 4 cores, 8GB of RAM and about half a TB of SSD free it's not exactly a constrained test platform. Happy to set up a user for ssh or VPN remote access during test periods My home fiber is 1G/1G uncapped so slinging large files around isn't much of an issue either.