NUT package (2.8.0 and below)

keyser

@lcbbcl I believe I have found what is causing the issue you are experiencing. To correct the issue will require an update to the pfSense nut package as well as updating the underlying FreeBSD nut package used by pfSense. It will take some time.

In the interim, you can address the issue by adding the line "user=root" to "Additional configuration lines for ups.conf" under Advanced settings.

Your hard and dedicated work on this package is an inspiration. Thank you for the effort :-)

I assume it was an access permissions issue since running it as root solves the issue? Can you comment on why it only happens for some UPS models?

dennypage

@keyser said in NUT package:

I assume it was an access permissions issue since running it as root solves the issue? Can you comment on why it only happens for some UPS models?

Yes, it's a permissions issue. I am hoping it can be addressed in the FreeBSD nut package (or perhaps in kernel config?), but if not I will have to add the user=root to the config for all usb devices.

What models it happens on are dependent upon the quirks defined for that device. If you want to jump down the rabbit hole, run "usbconfig dump_quirk_names" and "usbconfig dump_device_quirks".

NB: This is not related to the signal 10/11 issue being experienced with CyberPower units.

4o4rh

I see the github packages have been moved to

https://github.com/pfsense/FreeBSD-ports

But i don't see the nut package in there.

Where is the code for nut, or is it closed source now?

In the upsmon configuration, can we please add two config boxes for the below files.

/usr/local/etc/nut/upssched.conf with permission 644
/usr/local/bin/upssched-cmd with permission 755

To use the scheduler to trigger events from upsmon. these files need to be configurable.
Also, there should be a note for using them, the notification box has to be disabled.
here is a sample of working files

upsmon

# upsmon.conf - Configuration for network UPS Tools upsmon
# set the path to the notify command processor !!! the UPS notifications option must be unchecked !!! 
NOTIFYCMD /usr/local/sbin/upssched

# Poll the UPS every 60 seconds.
POLLFREQ 60

# Wait this many seconds for "Notify Shutdown" messages from slaves
POLLFREQALERT 30

# time UPS may go missing before being declared dead
DEADTIME 600

# Warn every 12 hours if battery needs to be replaced
RBWARNTIME 43200

# Warn every 5 minutes, if UPS is unreachable
NOCOMMWARNTIME 300

# Messages for the events %s will be replaced with the UPS system identifier in question
NOTIFYMSG ONLINE   "UPS %s on line power"
NOTIFYMSG ONBATT   "UPS %s on battery"
NOTIFYMSG LOWBATT  "UPS %s battery is low"
NOTIFYMSG FSD      "UPS %s forced shutdown in progress"
NOTIFYMSG COMMOK   "UPS %s communications (re-)established"
NOTIFYMSG COMMBAD  "UPS %s communications lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs to be replaced"
NOTIFYMSG NOCOMM   "UPS %s is not reachable"
NOTIFYMSG NOPARENT "upsmon parent process died, shutdown impossible"

# Change behavior of upsmon on certain events.
# Possible values for the flags:
# SYSLOG - Write the message in the syslog
# WALL - Write the message to all users on the system
# EXEC - Execute NOTIFYCMD (see above, in our case see upssched.conf) with the message
# IGNORE - Don't do anything
# If you use IGNORE, don't use any other flags on the same line.
# NOTIFYFLAG <notify type> <flag>[+<flag>][+<flag>] ...
NOTIFYFLAG ONLINE   SYSLOG
NOTIFYFLAG ONBATT   SYSLOG
NOTIFYFLAG LOWBATT  SYSLOG
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG
NOTIFYFLAG REPLBATT SYSLOG
NOTIFYFLAG NOCOMM   SYSLOG
NOTIFYFLAG NOPARENT SYSLOG

upssched

# Network UPS Tools - upssched.conf file

#Command script to run
CMDSCRIPT /usr/local/bin/upssched-cmd

# command for pipe and lock files
PIPEFN /var/db/nut/upssched.pipe
LOCKFN /var/db/nut/upssched.lock

# ============================================================================
# AT <notifytype> <upsname> <command>
# Define a handler for a specific event <notifytype> on UPS <upsname>.
# <upsname> can be the special value * to apply this handler to every
# possible value of <upsname>.
# Run the command <command> via your CMDSCRIPT when it happens.
# Note that any AT that matches both the <notifytype> and the <upsname>
# for the current event will be used.
# ============================================================================
# Possible AT commands
# - START-TIMER <timername> <interval>
#   Start a timer called <timername> that will trigger after <interval>
#   seconds, calling your CMDSCRIPT with <timername> as the first
#   argument.
#   Example:
#   Start a timer that'll execute when any UPS (*) has been gone 10 seconds
# AT COMMBAD * START-TIMER COMMBAD 1800
#   -----------------------------------------------------------------------
# - CANCEL-TIMER <timername> [cmd]
#   Cancel a running timer called <timername>, if possible. If the timer
#   has passed then pass the optional argument <cmd> to CMDSCRIPT.
#   Example:
#   If a specific UPS (myups@localhost) comes back online, then stop the
#   timer before it triggers
# AT COMMOK * CANCEL-TIMER COMMBAD COMMOK
#   -----------------------------------------------------------------------
# - EXECUTE <command>
#   Immediately pass <command> as an argument to CMDSCRIPT.
#
#   Example:
#   If any UPS (*) reverts to utility power, then execute
#   'ups-back-on-line' via CMDSCRIPT.

# UPS is online in syslog. we don't need action when no UPS connected
# AT ONLINE * CANCEL-TIMER onbatt 

# UPS on battery in syslog. badcomms will shutdown for us
# AT ONBATT * START-TIMER onbatt 300

# UPS low battery is syslog. we should badcomms to shutdown
# AT LOWBATT * START-TIMER lowbatt 300

# UPS forced shutdown by master or upsmon
AT FSD * EXECUTE powerdown

# UPS communications restored
AT COMMOK * CANCEL-TIMER badcomms

# UPS communications failed. 
AT COMMBAD * START-TIMER badcomms 600

# UPS shutdown recorded in syslog
# AT SHUTDOWN * EXECUTE powerdown

# UPS needs replacement battery. Info in syslog
# AT REPLBATT * EXECUTE replbatt

# UPS is not reachable. Relying on deadtime for shutdown
AT NOCOMM * EXECUTE commbad

# UPSMON parent process has died - shutdown not possible.
# AT NOPARENT * EXECUTE noparent

upssched-cmd

#! /bin/sh
#
# This script should be called by upssched via the CMDSCRIPT directive.
#
# Here is a quick example to show how to handle a bunch of possible
# timer names with the help of the case structure.
#
# This script may be replaced with another program without harm.
#
# The first argument passed to your CMDSCRIPT is the name of the timer
# from your AT lines.
#
# case $1 in
#	upsgone)
#		logger -t upssched-cmd "The UPS has been gone for awhile"
#		;;
#	*)
#		logger -t upssched-cmd "Unrecognized command: $1"
#		;;
# esac

case $1 in
   onbatt)
      # function not needed as we only use syslog
      logger -t upssched-cmd "UPS is running on battery."
   ;;
   
   online)
      # function not needed as we only use syslog
      logger -t upssched-cmd "UPS back online."
   ;;

   lowbatt)
      # function not needed as we only use syslog
      logger -t upssched-cmd "UPS has low battery"
      # we rely on badcomms when no UPS connected
      # /usr/local/sbin/upsmon -c fsd
   ;;

   badcomms)
      logger -t upssched-cmd "UPS offline too long, forced shutdown."
      /usr/local/sbin/upsmon -c fsd
   ;;

   commok)
      # function not needed as we only use syslog
      logger -t upssched-cmd "UPS communications restored."
   ;;
   
   powerdown)
      logger -t upssched-cmd "UPS forced shutdown started"
      /usr/local/sbin/upsmon -c fsd
   ;;

   replbatt)
      # function not needed as we only use syslog
      logger -t upssched-cmd "UPS battery needs replacing"
   ;;

   noparent)
      # function not needed as we only use syslog
      logger -t upssched-cmd "upsmon parent process died. Shutdown not possible"
   ;;

   *)
      logger -t upssched-cmd "No code defined for command: $1"
   ;;
esac

dennypage

@gwaitsi It hasn't moved since I wrote it. sysutils/pfSense-pkg-nut.

xavier1

This solved my issue. Everything is working great. Thank you :)

@dennypage said in NUT package:

@lcbbcl I believe I have found what is causing the issue you are experiencing. To correct the issue will require an update to the pfSense nut package as well as updating the underlying FreeBSD nut package used by pfSense. It will take some time.

In the interim, you can address the issue by adding the line "user=root" to "Additional configuration lines for ups.conf" under Advanced settings.

dennypage

@lcbbcl, @xavier1 Would you do me a favor and post the result of

usbconfig -v

from your systems please? Thanks.

lcbbcl

@dennypage
Here it is

usbconfig -v
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.1.0: uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1>

 bLength = 0x0012 
 bDescriptorType = 0x0001 
 bcdUSB = 0x0300 
 bDeviceClass = 0x0009  <HUB>
 bDeviceSubClass = 0x0000 
 bDeviceProtocol = 0x0003 
 bMaxPacketSize0 = 0x0009 
 idVendor = 0x0000 
 idProduct = 0x0000 
 bcdDevice = 0x0100 
 iManufacturer = 0x0001  <Intel>
 iProduct = 0x0002  <XHCI root HUB>
 iSerialNumber = 0x0000  <no string>
 bNumConfigurations = 0x0001 


Configuration index 0

   bLength = 0x0009 
   bDescriptorType = 0x0002 
   wTotalLength = 0x001f 
   bNumInterfaces = 0x0001 
   bConfigurationValue = 0x0001 
   iConfiguration = 0x0000  <no string>
   bmAttributes = 0x0040 
   bMaxPower = 0x0000 

   Interface 0
     bLength = 0x0009 
     bDescriptorType = 0x0004 
     bInterfaceNumber = 0x0000 
     bAlternateSetting = 0x0000 
     bNumEndpoints = 0x0001 
     bInterfaceClass = 0x0009  <HUB>
     bInterfaceSubClass = 0x0000 
     bInterfaceProtocol = 0x0000 
     iInterface = 0x0000  <no string>

    Endpoint 0
       bLength = 0x0007 
       bDescriptorType = 0x0005 
       bEndpointAddress = 0x0081  <IN>
       bmAttributes = 0x0003  <INTERRUPT>
       wMaxPacketSize = 0x0002 
       bInterval = 0x00ff 
       bRefresh = 0x0000 
       bSynchAddress = 0x0000 

     Additional Descriptor

     bLength = 0x06
     bDescriptorType = 0x30
     bDescriptorSubType = 0x00
      RAW dump: 
      0x00 | 0x06, 0x30, 0x00, 0x00, 0x00, 0x00




ugen0.2: <American Power Conversion Smart-UPS C 1000 FW:UPS 12.0 / ID1005> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (2mA)

 bLength = 0x0012 
 bDescriptorType = 0x0001 
 bcdUSB = 0x0200 
 bDeviceClass = 0x0000  <Probed by interface class>
 bDeviceSubClass = 0x0000 
 bDeviceProtocol = 0x0000 
 bMaxPacketSize0 = 0x0040 
 idVendor = 0x051d 
 idProduct = 0x0003 
 bcdDevice = 0x0106 
 iManufacturer = 0x0001  <American Power Conversion >
 iProduct = 0x0002  <Smart-UPS C 1000 FW:UPS 12.0 / ID=1005>
 iSerialNumber = 0x0003  <AS1506222288  >
 bNumConfigurations = 0x0001 


Configuration index 0

   bLength = 0x0009 
   bDescriptorType = 0x0002 
   wTotalLength = 0x0029 
   bNumInterfaces = 0x0001 
   bConfigurationValue = 0x0001 
   iConfiguration = 0x0000  <no string>
   bmAttributes = 0x00e0 
   bMaxPower = 0x0001 

   Interface 0
     bLength = 0x0009 
     bDescriptorType = 0x0004 
     bInterfaceNumber = 0x0000 
     bAlternateSetting = 0x0000 
     bNumEndpoints = 0x0002 
     bInterfaceClass = 0x0003  <HID device>
     bInterfaceSubClass = 0x0000 
     bInterfaceProtocol = 0x0000 
     iInterface = 0x0000  <no string>

     Additional Descriptor

     bLength = 0x09
     bDescriptorType = 0x21
     bDescriptorSubType = 0x00
      RAW dump: 
      0x00 | 0x09, 0x21, 0x00, 0x01, 0x21, 0x01, 0x22, 0x03, 
      0x08 | 0x02

    Endpoint 0
       bLength = 0x0007 
       bDescriptorType = 0x0005 
       bEndpointAddress = 0x0081  <IN>
       bmAttributes = 0x0003  <INTERRUPT>
       wMaxPacketSize = 0x0040 
       bInterval = 0x0014 
       bRefresh = 0x0000 
       bSynchAddress = 0x0000 

    Endpoint 1
       bLength = 0x0007 
       bDescriptorType = 0x0005 
       bEndpointAddress = 0x0001  <OUT>
       bmAttributes = 0x0003  <INTERRUPT>
       wMaxPacketSize = 0x0040 
       bInterval = 0x000a 
       bRefresh = 0x0000 
       bSynchAddress = 0x0000

dennypage

For those of you affected by the APC issue (and those that will be in the future):

First, I believe that the issue only affects recent APC UPS designs. Those in the last couple of years or so. The core issue is that a new device identification number has been introduced by APC, which FreeBSD is not aware of. Since the kernel does not recognize the device as a UPS that should be ignored, the kernel is attaching an internal driver to it. The presence of this internal driver on the device precludes any non root user from attaching to the device.

This cannot be reasonably fixed in the pfSense nut package or the underlying FreeBSD nut package. The permanent solution requires an adjustment to the usb quirk table in the kernel itself. A patch has been established for the kernel, but I don't know how long it will take to flow through.

In the interim, there are two solutions available:

The first solution, as noted previously, is to add the line

user=root

to the "Additional configuration lines for ups.conf" section under Advanced settings. This will cause the usbhid-ups command to be run as root. The disadvantage of this approach is that you are running the driver as root, which may be of concern to some.

The alternative solution is to add the line

hw.usb.quirk.0="0x051d 0x0003 0x0000 0xffff UQ_HID_IGNORE"

to file "/boot/loader.conf.local". Note that you may have to create this file. The disadvantage of this approach is that you will have to reboot pfSense before it will work.
If you previously implemented the user=root approach, you can remove the user=root line following the reboot.

dennypage

For those of you affected by the CyberPower issue (and those that will be in the future):

First, I want to thank all of you for your help in tracking this down. Particularly @shaffergr, who was kind enough, and trusting enough, to run test builds for me against his CyberPower.

I tracked this down to a double free by the usb code in nut. FWIW, I don’t believe that this issue is completely limited to CyberPower UPSs, but it is probably pretty difficult to encounter with other units as hitting the issue requires a reconnect of the UPS on the bus. CyberPower units are well known to randomly disconnect and reconnect seemingly at random.

For those wondering why this issue did not happen in prior to 23.01: 22.05 and below used nut version 2.7.4. In 2.7.4, nut did not actually close the device when a disconnect happened. Presumably, this may have resulted in a memory leak at shutdown, but I didn’t explore enough to confirm. In 2.8.0, when a device disconnects nut actually closes the device when it disconnects. In fact, it closes it twice on all systems other than Linux. There is even a comment in the code noting how the double close would cause corruption on Linux systems. Unfortunately contrary to the code comments it also causes corruption in FreeBSD.

Having traced this down in the 2.8.0 release code, when I went to the current development version I surprised to find that someone had beat me to it, and fixed the issue back in August. It's just not been release yet. :)

To resolve this will require a new version of nut. Either a new release from the nut team, or for the pfSense team to move from the 2.8.0 release version to the current development version of nut. The nut team is looking toward 2.8.1, but it appears that they have a few things they still want to address before putting that out. I will explore the concept of moving to nut-devel with the pfSense team as time permits (I think they are pretty busy right now). But no matter which way it goes, it’s going to take some time.

In the interim, the only known work-around using the release code (discovered by @tman222) is to add the line

interruptonly

to the Extra Arguments to driver section. This will cut down some of the information you can see about your UPS, but the important stuff needed to monitor and shutdown should still be there.

Alternatively, if brave souls are interested, I have a build of usbhid-ups made from the FreeBSD nut-devel package. I do not have a CyberPower available so I haven't been able to directly test it, but I expect that it will work.

If you you decide you would like a copy, reach out to me and I will see about getting it to you. For reference, the shasum and sha256sum checksums are:

49ce9131502bfb8b789ee97b7fb3fc81fc9f8fff  usbhid-ups
999a2653559dbc50ecc8ba592a67587b1e307a1495f6e8ebbd3d8e90e3967133  usbhid-ups

dennypage

I'm going to sleep now...

tnowak

Hi!
Last year I was struggling to connect to Ever ECO Pro 1200 AVR CDS UPS.
This was mostly because nut was in version 2.7 and support for that UPS has been added in 2.8.

Lately I tried it again. This time nut package is already in version 2.8 (2.8.0_2).
I've installed it and configured with usbhid driver and some basic extra arguments:

port=auto
vendorid=2e51
productid=0000

Unfortunately, it still doesn't work. Why?

Log:

Feb 22 20:42:29	upsmon	71579	Poll UPS [ever] failed - Driver not connected

tnowak

dennypage

@tnowak First, I would recommend removing everything from the extra arguments section. Following that, test again. If it doesn't work please post the output from usbhid-ups, either from the system log or from the command line. The command line would be this:

/usr/local/libexec/nut/usbhid-ups -a ever

FYI, the port=auto is handled by the package and should never be added to the extra arguments section for usbhid. The vendorid/productid can be added back later if necessary.

tnowak

This post is deleted!

tnowak

@dennypage Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

dennypage

@tnowak said in NUT package:

Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

I expect that you are actually in the same situation as the new gen APC listed above: No quirk covering your UPS device (I actually don't see anything from Ever in the table at all).

To confirm, use these steps:

disable nut (Services / UPS / Settings) and save the config
unplug the usb connection to the ups and wait 5 seconds
re-plug the usb connection to the ups
run "usbconfig -d ugen0.2 show_ifdrv"

and post the result. My expectation is that you will see two lines, similar to this:

ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)
ugen0.2.0: uhid0: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047, class 0/0, rev 2.00/0.01, addr 1>

tman222

Hi @dennypage - thanks again for all your help looking into the signal 10/11 issue with CyberPower UPS units. I'm fine to continue running mine with the interruptonly flag workaround for now even if fewer variables are monitored. After a couple days of running this way, things appear to be stable. If it this setup ends up crashing at some point, I'll probably give the updated usbhid-ups driver a try. Also, if you do end up releasing a nut-devel package at some point that includes fixes post 2.8.0, I'd be happy to try that out as well.

JonathanLee

@dennypage You are amazing I appreciate all you do. Again, thanks for taking the time to look into this issue reported within this discussion. It seems to be a problem with many other users now and you already have a solid solution for it.

tnowak

@dennypage said in NUT package:

@tnowak said in NUT package:

Solved it by adding user=root in ups.conf section. But this is rather a workaround than a solution. Anyway, this seems to be a problem of nut / file / dev permissions.

I expect that you are actually in the same situation as the new gen APC listed above: No quirk covering your UPS device (I actually don't see anything from Ever in the table at all).

To confirm, use these steps:

disable nut (Services / UPS / Settings) and save the config

unplug the usb connection to the ups and wait 5 seconds

re-plug the usb connection to the ups

run "usbconfig -d ugen0.2 show_ifdrv"

and post the result. My expectation is that you will see two lines, similar to this:
ugen0.2: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (10mA)
ugen0.2.0: uhid0: <American Power Conversion Smart-UPS1000 FW:UPS 16.0 / ID1047, class 0/0, rev 2.00/0.01, addr 1>

Result:

ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.2.0: uhid0: <EVER ECO PRO AVR CDS, class 0/0, rev 2.00/1.00, addr 2>

PS. I've also noticed a problem with nut loosing connection to this UPS even with user=root after some time. Then when I restart nut it shows up again.

dennypage

@tnowak said in NUT package:

Result:
ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.2.0: uhid0: <EVER ECO PRO AVR CDS, class 0/0, rev 2.00/1.00, addr 2>

PS. I've also noticed a problem with nut loosing connection to this UPS even with user=root after some time. Then when I restart nut it shows up again.

Yep, that shows a kernel driver attached to the device. Same situation as the new series APC devices. Not too surprising, because I don't find any Ever devices defined in the usb table.

You can either use the user=root approach, or you can develop a quirk setting for /boot/loader.conf.local. Based on your prior post, I believe that the correct value would be this:

hw.usb.quirk.0="0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE"

You can test this in advance by running this:

usbconfig add_dev_quirk_vplh 0x2e51 0x0002 0x0000 0xffff UQ_HID_IGNORE

followed by unplugging and replugging the usb cable to your ups. If the values are correct, when you run "usbconfig -d ugen0.2 show_ifdrv" again, you should only see one line of output like so:

ugen0.2: <EVER ECO PRO AVR CDS> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

The ugen0.2.0 should be gone.

If this test works then you can add the line to your /boot/loader.conf.local file.

As to loosing communication after a time, I would still need to see the output from usbhid-ups, either from the system log or from the command line. There seem to be a few issues that do not produce entries in the system log, so I would recommend using the command line as previously discussed.