NUT Package (2.8.1 and above)

dennypage

@rarlup Sorry for the delayed response -- I've been traveling.

Couple of things I noticed looking at your logs. First, you have a lot of pogo. I assume that this is from your WAN connection, but I can't tell without more logs. I would definitely check to see if there are other sources of restart occurring, such as a local client plugged directly into a port on the firewall.

Secondly, I note that it is taking quite a long time for your UPS driver to start. As an experiment, you might remove the ampersand from the start of upsdrvctl in /usr/local/etc/rc.d/nut.sh and see if that helps the startup.

As to the shutdown, I'm guessing that you may have had a brief power outage just before your WAN reset. If NUT looses connection with the UPS, and the last known state was on battery, the default behavior of NUT is to initiate a shutdown.

rarlup

@dennypage said in NUT Package (2.8.1 and above):

Sorry for the delayed response -- I've been traveling

No worries - hope it was for pleasure :)
And please forgive me for not thanking you sooner for your help, and especially on a Saturday - greatly appreciated.

@dennypage said in NUT Package (2.8.1 and above):

you have a lot of pogo

what do you mean? bouncing back and forth / up and down?

@dennypage said in NUT Package (2.8.1 and above):

Secondly, I note that it is taking quite a long time for your UPS driver to start. As an experiment, you might remove the ampersand from the start of upsdrvctl in /usr/local/etc/rc.d/nut.sh and see if that helps the startup.

Will try later today/tmr. Currently under business hours.

@dennypage said in NUT Package (2.8.1 and above):

As to the shutdown

Devices (mostly Asustor NASes, as I mentioned before) are scheduled to gracefully shutdown 5min after the On Battery event (without a corresponding On A/C event). The thing is it didn't lose the connection with the UPS it just showed On Battery after WAN down event and stayed with that status. I'm 100% convinced there was no power outage since there was no relay click (comes instantly, talking few milliseconds) and as I was standing right by it.
Why was I standing right by it?
Because I was the one that cut(literally) the wan cable. This may have caused a few ms short - I do agree it may have caused pfSense to do weird things.
I cut it since the wiring was damaged and I wanted to redo the connection. Previously, just breathing near the cable could cause a WAN down event so this may be the actual cause - faulty wiring on one of the pfSense interfaces can/will cause weird UPS status?

rarlup

@dennypage said in NUT Package (2.8.1 and above):

Secondly, I note that it is taking quite a long time for your UPS driver to start. As an experiment, you might remove the ampersand from the start of upsdrvctl in /usr/local/etc/rc.d/nut.sh and see if that helps the startup.

Removed the &

 	/usr/local/sbin/upsdrvctl start &

 	/usr/local/sbin/upsdrvctl start

And restarted the service.
Seems like it helped - no more poll ups failures and connections being refused. Not sure if this is related to edited file.

Feb 19 03:07:45 	upsmon 	17987 	Signal 15: exiting
Feb 19 03:07:45 	upsd 	17183 	User local-monitor@127.0.0.1 logged out from UPS [asustor]
Feb 19 03:07:45 	upsd 	17183 	mainloop: Interrupted system call
Feb 19 03:07:45 	upsd 	17183 	Signal 15: exiting
Feb 19 03:07:45 	usbhid-ups 	19139 	Signal 15: exiting
Feb 19 03:07:52 	usbhid-ups 	26744 	Startup successful
Feb 19 03:07:52 	upsd 	26874 	listening on 127.0.0.1 port 3493
Feb 19 03:07:52 	upsd 	26874 	listening on ::1 port 3493
Feb 19 03:07:52 	upsd 	26874 	listening on 192.168.x.1 port 3493
Feb 19 03:07:52 	upsd 	26874 	listening on 10.x.x.1 port 3493
Feb 19 03:07:52 	upsd 	26874 	Connected to UPS [asustor]: usbhid-ups-asustor
Feb 19 03:07:52 	upsd 	26874 	Found 1 UPS defined in ups.conf
Feb 19 03:07:52 	upsd 	27039 	Startup successful
Feb 19 03:07:53 	upsmon 	27550 	Startup successful
Feb 19 03:07:53 	upsd 	27039 	User local-monitor@127.0.0.1 logged into UPS [asustor]

Gertjan

@rarlup said in NUT Package (2.8.1 and above):

faulty wiring on one of the pfSense interfaces can/will cause weird UPS status?

A faulty network cable, WAN, LAN, whatever, will cause system network event. One of the effects will be that that interface is taken down (and/or up if the connections returns).
Software that is bound (uses) to a (any !) interface will get restarted, even if that interface wasn't the one triggering the event.
Instead of a weird behavior, look at the logs again : it gets stopped, and restarted again.
The monitoring process will re open the USB port, if that's your UPS connection, re contact the UPS, signal the obtained stated to the local NUT process, etc.
Even the localhost or 127.0.0.1 is no exception, it's probably not the interface that can be taken down by a bad cable, as it has no cable ^^ but processes using it will also get restarted.

dennypage

@rarlup said in NUT Package (2.8.1 and above):

you have a lot of pogo
what do you mean? bouncing back and forth / up and down?

Yes.

@rarlup said in NUT Package (2.8.1 and above):

The thing is it didn't lose the connection with the UPS it just showed On Battery after WAN down event and stayed with that status.

There was a persistent connection loss in the logs you posted. Feb 14 12:35:26.

@rarlup said in NUT Package (2.8.1 and above):

Previously, just breathing near the cable could cause a WAN down event so this may be the actual cause - faulty wiring on one of the pfSense interfaces can/will cause weird UPS status?

Yes, this could be the cause of the pogo.

dennypage

A common mistake people make is to directly connect a PC or Mac to a port on pfSense rather than through a switch. The frequent sleep/wake cycles of these systems, which bring link down/up on their ports, wreak havoc on the pfSense.

Gertjan

@dennypage said in NUT Package (2.8.1 and above):

A common mistake people make is to directly connect a PC or Mac to a port on pfSense rather than through a switch. The frequent sleep/wake cycles of these systems, which bring link down/up on their ports, wreak havoc on the pfSense.

Yeah ... big +1 that one.
No excuse if they have an UPS : the upstream ISP device (router, modem, powered ONT, whatever, pfSense itself, and all the LAN attached switch(es) should be powered by the same - or linked - UPS.
This solves a load of potential pitfalls.
(admin) Live is so easy ones you reached this conclusion / setup ^^

rarlup

@dennypage
Not the case - that pfsense box has 9 interfaces (5 rj45 and 4 sfp+)
0 - wan pppoe
1 - another pfsense box
2 - not used yet
3 - not used
4 - switch A

0 - lacp to switch B
1 - lacp to switch B
2 - not used
3 - not used

Checked the UPS logs and no new events. Guess that & fixed it.

rarlup

@dennypage said in NUT Package (2.8.1 and above):

There was a persistent connection loss in the logs you posted. Feb 14 12:35:26.

So WAN glitched, triggered package restart and somehow it detected the UPS as on battery before losing connection to it?

Gertjan

@rarlup said in NUT Package (2.8.1 and above):

detected the UPS as on battery

If that state was known at the moment, then it is because NUT got that state from the UPS.
An UPS on battery, yeah, something has glitched - and the WAN agrees ^^

rarlup

I see and it makes sense. Thank you both!
Looking to purchase a few more UPSes - any particular model that is 100% compatible with NUT and you can truly recommend?
min 700VA, cheap (<$200) and rack mountable would be ideal

dennypage

@rarlup said in NUT Package (2.8.1 and above):

Looking to purchase a few more UPSes - any particular model that is 100% compatible with NUT and you can truly recommend?

NUT is compatible with a wide variety of UPSs. That said, some UPSs have more quirks than others.

Specific recommendations are difficult, but I would generally stay with usb-hid capable units from one of the big manufacturers such as APC, Cyber Power or Tripp Lite.

My personal preference is for APC, but they tend to be a bit more expensive.

j.koopmann

Hi @dennypage

thanks so much for your tremendous effort on this package. For months now after upgrading pfSense my previously working (most likely pre nut 2.8.x) Eaton 9SX 3000i keeps loosing the USB connection. A restart of nut fixes it immediately. I tried to work my way through the thread but seem to be at a loss. Everything is running as root already. I have not applied quirks. I am on nut 2.8.2. I have not set things to interrupt. Using usbhid-ups with no additional parameters.

Any (!) hint on how to further debug this, where to look in the logs, make adjustments whatever would be greatly appreciated!

Gertjan

@j-koopmann said in NUT Package (2.8.1 and above):

Any (!) hint on how to further debug this

Check also NUT suddenly stops working every app. 6 minutes.
It's also a "The UPS is a usb connected Eaton 9SX 3000i.".

j.koopmann

@Gertjan turns out it is my thread and still an unresolved problem. I just had forgotten I was already working on it here in the forum. :-)

No quirks installed. Problem persisting.

Gertjan

@j-koopmann

Do you see the disconnect in the system log ?

Normally, like a mouse, or a printer, ones the USB link is made, it stays up until 'something' happens'.
Btw : I said 'link', be ware a simple USB connection it a lot more more then an simple electrical link. USB connectors, the wire (the cable) etc can be in a bad shape. USB concentrator (the USB chip) used on both side can be 'military grade' or worse then 'alibaba' grade.
"APC", as mentioned above, delivers the close to set it and forget it (and their battery still dies at a worst moment ... )

To test your UPS : forget about the genric NUT = pfSense for now. Hook it up using a desktop top PC, with the software that came long with it.
Check for some time if you see 'disconnect issues'.

Have look at this page : https://networkupstools.org/docs/man/ and locate (couldn't find it myself right now) the schema that lays out how all this stuff is interconnected.
You'll find details about upsd - the ups deamon, the connection method, which is 'USB' in this case, the driver (usbhid) used, "upsmon".
There are command line tools that can give you a lot of details :

[25.03-BETA][root@pfSense.bhf.tld]/root: upsc UPS
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2024/06/19
battery.runtime: 500
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.3
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
...

but a USB that 'disconnects' is, imho, more a hardware issue, like a cable or connector, or a bad power issue.
UPSs have often internally bad power ( ) as they do ugly things with power - it's their job.

j.koopmann

@Gertjan is it fair to assume that I would be seeing and usb disconnects the kernel is aware of in dmesg?

I just intentionally pulled the USB:

ugen0.2: <EATON Eaton 9SX> at usbus0 (disconnected)
ugen0.2: <EATON Eaton 9SX> at usbus0

However when nut stops working there is NOTHING in dmsg indicating an issue. Moreover: Once it stops working a simple restart of the nut service makes it work again immediately. If the USB stops working due to a hardware issue I would suspect that the problem would persist. Vice versa: When I pulled the USB and reinserted it, NUT was immediately happy again and did NOT loose its connection. Indeed it did not even throw an error in syslog...

So if this was a problem on the Eaton or a physical problem I would assume that I should be seeing usb related dmesg/syslog output. I do not.

Moreover the problem started pretty much when I upgraded from 23.x to 24.x. What was the nut version with 23.x? I read somewhere that nut 2.8 has introduced some usb related regressions. Agreed it could be unrelated but the timing is very suspicious.