-
Hello everyone,
I have noticed that when there are multiple power losses over a long period of time, sooner or later, there will be a safe shutdown sent by my pfsense running NUT to all the devices listening in. Power comes back up and devices start again. Often this repeat 2-3 times. Finally UPS comes back up, devices start, but then the power goes out again while I suppose the UPS is almost dead, UPS gives out and turns off and thus I get an improper shutdown.
Is there a parameter setting to avoid automatic restart after safe shutdown? i.e. if NUT says to safe-shutdown, then a person will have to come and turn on the devices again?
Or is there an alternative approach?
Thanks!
-
@joesl said in NUT package:
Is there a parameter setting to avoid automatic restart after safe shutdown? i.e. if NUT says to safe-shutdown, then a person will have to come and turn on the devices again?
A couple of approaches...
You can configure the BIOS to not automatically start when power is restored. This will mean that human intervention will always be required for startup.
A much better option, if available, is to configure the UPS to not restore load power until there is sufficient battery available. Use upsrw and see if you have a variable called "battery.charge.restart".
NUT variable documentation is here.
-
@dennypage Thanks a lot!
I ran
upsrw upsnameand I don't see that variable. Am I safe to assume it's not available?
Upon further reading and thanks to your leads, I think the issue is on my devices after all due to a poor software handling. It seems they keep going on/off until pfsense goes into FSD, then they don't receive anymore info from the UPS and just die when the UPS dies. -
@joesl said in NUT package:
Or is there an alternative approach?
Configure controlled devices to both
- Not automatically starting on power supplied
- But instead wake on LAN
-
@joesl said in NUT package:
I ran upsrw upsname and I don't see that variable. Am I safe to assume it's not available?
Yes, it's safe to assume that the UPS does not support configuring the minimum battery level before start. Your UPS probably uses a fixed value like 3%.
Upon further reading and thanks to your leads, I think the issue is on my devices after all due to a poor software handling. It seems they keep going on/off until pfsense goes into FSD, then they don't receive anymore info from the UPS and just die when the UPS dies.
Based on your description, the UPS is starting the load before there is enough battery for all attached systems to complete a full startup, including NUT connections, before entering a low battery condition. So if mains power goes out before the NUT server fully starts and the NUT clients all reconnect to the server, the server will initiate shutdown and the clients will never know. This is not a software problem per se.
The only way to address the issue is to ensure that the UPS has enough battery to fully bring the load system(s) up to a fully functional state before reentering a low battery condition. Since the UPS has no idea what's connected to it, the configuration needs to be either minimum percentage of battery charge before start, or wait X number of minutes before start.
If I were in your situation, I would consider these options:
- Replace the batteries in the UPS if they are more than 24 months old.
- Run a full battery calibration test to ensure that the UPS has a correct view of battery %.
- Get a larger UPS that has more battery capacity.
- Move some of the client systems off the UPS to reduce the load (which will increase the effective reserve).
You could also look at getting a network (SNMP) based UPS so that the individual hosts all speak directly to the UPS. While this feature alone may not address your issue, network UPSs usually have larger capacities and more configurability, so your issue would likely be solved one way or another.
-
@patch said in NUT package:
Configure controlled devices to both
- Not automatically starting on power supplied
- But instead wake on LAN
Hmm... I don't personally know of any system were a WOL packet will trigger the BIOS to boot an operating system.
-
@dennypage Thanks a lot for your whole writeup!
I think you are probably correct on what happened (though I'm still a little confused). This has happened 2-3 times in the past couple of years. I actually have a substantial UPS for my load which was able to go on for almost 6 hours of these on/off conditions (always happens during the night and eventually gives up in the early morning). I use BackUPS Pro with the external battery pack and only have a 150W load on the main UPS. All other devices on secondary UPS.
I captured some logs if you are interested in this post. I can't understand the logic for the devices (I have multiple and all behave in the same way).. 7:16 devices go to a "kind of shutdown" state, 7:29 pfsense sends a "we are doomed" signal and turns off, 7:34 the devices decide to turn on again (probably coz power went back to UPS briefly) and at this stage there is no more NUT (pfsense is off) then UPS dies and so do the devices. I would expect a permanent shutdown on the devices like pfsense does when we reach the LOWBATT/FSD state sent by pfsense but they just keep doing the "safe state" instead. Moreover the devices have a "restart automatically when power supply issue is fixed" flag set to OFF, but still they turned on.. Opened a ticket with vendor.
As of now I did an override on NUT to consider the low battery state to be 30% and disabled the "safe state" on the devices with a different logic. The hope is that they go in shutdown/safestate together (pfsense and devices) and the issue is just the out-of-sync dance.
Thanks again for your help!
-
Im having a problem with a Tripplite Internet550U. I am using the USB cable that came with the UPS. The UPS is brand new. I installed the NUT package on my Netgate SG2100 and then connected my Tripplite to the SG2100 USB port. Then I go to the Services>UPS and then the UPS settings. I select Local USB, then I give the USB a name "INTERNET550U" and then I select the driver USBHID (which is the driver that it says should be compatible with that UPS on the NUT compatibility website). I click Save and then wait and eventually get the "Status Alert: The UPS requires attention." Then in the System log I get "Can't connect to UPS [INTERNET550U] (usbhid-ups-INTERNET550U): No such file or directory" then "Startup successful" then "User local-monitor@::1 logged into UPS [INTERNET550U]" then "UPS INTERNET550U is unavailable" and finally "Poll UPS [INTERNET550U] failed - Driver not connected" every 5 seconds forever.
Things I have already tried:- rebooting UPS by holding down the power button and waiting until it turns off, then waiting 10 seconds and turning the UPS back on.
- Unplugged and replugged the USB cable multiple times at both the UPS side and the Netgate side
- rebooted the SG2100 multiple times
- uninstalled NUT package, rebooted SG2100, reinstalled NUT package, rebooted SG2100
- Tried Tripplite driver instead of USBHID driver
- Attempted to add "user=root" in the ups.conf advanced settings (per https://forum.netgate.com/topic/117510/pfsense-2-3-4-with-nut-2-7-4_4-and-tripp-lite-smx500rt1u-usb-no-matching-hid-ups/8)
Any assistance with this issue would be greatly appreciated! Thank you!
-
@missionit-tech said in NUT package:
Then in the System log I get "Can't connect to UPS [INTERNET550U] (usbhid-ups-INTERNET550U): No such file or directory" then "Startup successful" then "User local-monitor@::1 logged into UPS [INTERNET550U]" then "UPS INTERNET550U is unavailable" and finally "Poll UPS [INTERNET550U] failed - Driver not connected" every 5 seconds forever.
Please
- reboot pfSense with the UPS attached
- post a screen capture of your UPS settings (Services -> UPS -> UPS Settings)
- post system log entries matching 'usb' (Status -> System Logs -> System -> General, click on wrench icon and type 'usb' in the Message field, click Apply Filter).
-
USB logs:
Apr 15 14:02:39 upsd 75535 Can't connect to UPS [INTERNET550U] (usbhid-ups-INTERNET550U): No such file or directory
Apr 15 14:06:25 kernel xhci0: <Marvell Integrated USB 3.0 controller> mem 0x58000-0x5bfff irq 16 on simplebus1
Apr 15 14:06:25 kernel usbus0 on xhci0
Apr 15 14:06:25 kernel ehci0: <Marvell Integrated USB 2.0 controller> mem 0x5e000-0x5ffff irq 17 on simplebus1
Apr 15 14:06:25 kernel usbus1: EHCI version 1.0
Apr 15 14:06:25 kernel usbus1 on ehci0
Apr 15 14:06:25 kernel usbus0: 5.0Gbps Super Speed USB v3.0
Apr 15 14:06:25 kernel usbus1: 480Mbps High Speed USB v2.0
Apr 15 14:06:25 kernel ugen0.1: <Marvell XHCI root HUB> at usbus0
Apr 15 14:06:25 kernel uhub0: <Marvell XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Apr 15 14:06:25 kernel ugen1.1: <Marvell EHCI root HUB> at usbus1
Apr 15 14:06:25 kernel uhub1: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Apr 15 14:06:25 kernel ugen1.2: <Tripp Lite INTERNET550U> at usbus1
Apr 15 14:06:25 kernel uhid0: <Tripp Lite INTERNET550U, class 0/0, rev 2.00/0.02, addr 2> on usbus1
Apr 15 14:06:53 upsd 42898 Can't connect to UPS [INTERNET550U] (usbhid-ups-INTERNET550U): No such file or directory
Apr 15 14:06:57 upsd 57232 Can't connect to UPS [INTERNET550U] (usbhid-ups-INTERNET550U): No such file or directoryThank you so much for your assistance! Looking forward to hearing back from you!
-
Try different USB ports on the host. Looking to try different hubs on the machine. You are currently connected to usbus1 and looking to try connecting to usbus0.
One other (bizarre) thought comes to mind. No rational reason other than I've never seen anyone do uppercase before. Change UPS Name to simply lowercase 'myups' or some such.
-
@dennypage Thanks so much for your assistance with this! Unfortunately, the Netgate SG-2100 only has 1 USB port on it so I dont have any options to try a different USB port. I tried renaming to "myups" and its still the same issue.
-
Can you run the following please?
- disable the UPS service
- unplug and re-plug the usb connection
- run the usbconfig command
- re-enable the UPS service
- run the usbconfig command again
Looking to see something similar to this:
[22.01-RELEASE][root@fw.cococafe.com]/root: usbconfig ugen0.1: <0x8086 XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) ugen0.2: <Tripp Lite Tripp Lite UPS> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA) [22.01-RELEASE][root@fw.cococafe.com]/root: -
Hi All.
I have been unable to locate any details on whether I can monitor the UPS state via pfSenses built-in SNMP service, if I have NUT installed and working as aspected?
I Guess the question is: Does NUT have any integration/MIBs that needs to be installed, so the built-in SNMP Service can see state and UPS info from NUT?
Love the no fuss of using the official appliances :-)
-
@missionit-tech Were you able to get your UPS working?
-
@keyser said in NUT package:
Does NUT have any integration/MIBs that needs to be installed, so the built-in SNMP Service can see state and UPS info from NUT?
If I understand your question correctly, you are asking if you can monitor the state of a non SNMP based UPS connected to the pfSense system via the SNMP server on pfSense. Assuming so...
This is more an SNMP question than a NUT question. The SNMP server (bsnmp or net-snmp) determines what/how/where something is exposed via SNMP. There is nothing built-in to either SNMP package to support NUT status, however both support extensions that will allow you to do what you want. You will have to configure the integration yourself. Search for "bsnmp-ucd" (bsnmp) or "net-snmp-extend-mib" (net-snmp) for details.
Here is an example integration using net-snmp.
One additional note: The configuration UI for bsnmp does not have any allowance for extensions, so you would have to go out to the filesystem and modify the config file. It might be easier to use the net-snmp package, which does have allowance for extensions in the UI. See the Extended Commands section at the bottom of the Host Information tab.
HTH
-
@dennypage Thank you for the detailed answer.
This is a little above my allowed time to invest in geekery :-)
I was just wondering if perhaps someone had already done that integration :-)
Love the no fuss of using the official appliances :-)
-
@dennypage
I have not made it back to the site yet to test the last troubleshooting steps that you suggested. Sorry for the delay getting back to you on it. Ill keep you posted when I get back onsite. Thanks for checking in. -
-
@dennypage said in NUT package:
@keyser Using librenms by chance? If so, it's pretty easy. The integration script is here. All you have to do is put the script on the local filesystem and add the script in the Extended Commands section for net-snmp...
Unfortunately not. I’m Using Zabbix, and have recently moved away from a Zabbix Agent on my pfSense and instead started using the “pfSense SNMP” template Zabbix has made available.
It works well, allthough a few counters/details are missing compared to rbecelli’s pfSense Active agent template.
