NUT suddenly stops working every app. 6 minutes

Gertjan

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

I am not happy buying a new UPS if the old one is working and has been doing so flawlessly for the past months.

Not forcing you to get out and buy something just for testing.

An UPS is a security device, and I was presuming you have more to protect a just pfSense.
So : you have at least 2 UPS, so you can do : the exchange trick, which is perfect to rules things out.

Btw : UPS batteries, even when they are not solicited often, do die - give them 3 years, and they switched roles : instead of being a security device in your house, they will set your house in fire (batteries overheating : even the fire squads run away from it).

Again : USB connection (electrical) are not perfect.
UPSMon (NUT) : as we use the same version, identical at a bit level, only our settings differ. For example : my USB driver is probably not the same as yours. You use the same USB driver as I do :

see here : https://networkupstools.org/stable-hcl.html

The USB chip used on the UPS side .... we'll never know,

And maybe the internal power of your UPS is 'flaky'. (Capacitors go bad also ...)

Normally, I change my UPSs after the second battery change, if they didn't go bad before.

j.koopmann

@Gertjan

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

Not forcing you to get out and buy something just for testing.

I know. :-)

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

An UPS is a security device, and I was presuming you have more to protect a just pfSense.

Yes one server and a switch and other equipment but for a home installation only hence only one UPS. Moreover the entire house is on a big PV driven battery so the UPS only will have to cover for a few minutes. Not worth buying two...

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

Btw : UPS batteries, even when they are not solicited often, do die - give them 3 years,

I am aware. But the old UPS is only appr. 12 months old. :-) So I am not going to buy y new one just to rule this one out. Worst case: It is not working or I have to restart nut every x hours. --> Home installation.

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

Normally, I change my UPSs after the second battery change, if they didn't go bad before.

Agreed. Will do that in x years. :-)

Thanks for your help. Let me change cables and USB port and see if something changes.

dennypage

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

This setup worked flawlessly for months so my main suspicion is that the update triggered a bug.

Unplug and re-plug the USB cable to confirm you are not experiencing a simple permissions issue.

Also, had you installed a quirk by chance?

j.koopmann

@dennypage following up here from the other thread:

No quirks installed. How would this be a permissions issue if it is working after a restart of NUT and sometimes simply stops working out of the blue? I am happy to reattach the USB when it next happens.

Gertjan

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

and sometimes simply stops working out of the blue?

As said in the other thread : when you connect, for example, a RJ45 network cable from the pfSense WAN to the the upstream LAN port of your ISP router, you presume it stays up until one of the two devices go down / are powered down.
If it doesn't, start ditching cables, NICs (drivers ?) or even devices until your connection is stable.

USB connection are often less mission critical, but, look at their physical connection method : just some metal plates against other metal plates, so the smallest pollution or whatever can make the connection 'bad'.
Swap cables.
Swap USB ports.
If the connection stays bad, swap the UPS.

Go back to the https://networkupstools.org/ web site, and this time check with the user forum first with the obvious question : "what is the best one ?"

Btw : I presume your USB connection going down isn't a NUT issue. The USB hardware signals the usb-hid driver that the USB connection went down. This gets signaled upwards, up until you see a line in the log : 'lost UPS'.
I can create this log line also with my pfSense NUT install - the very same as yours : I have to rip out the USB cable to see it.
I'm using a Netgate classic "4100" hooked up to an APC UPS. It's pretty rock solid.

j.koopmann

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

and sometimes simply stops working out of the blue?

As said in the other thread : when you connect, for example, a RJ45 network cable from the pfSense WAN to the the upstream LAN port of your ISP router, you presume it stays up until one of the two devices go down / are powered down.
If it doesn't, start ditching cables, NICs (drivers ?) or even devices until your connection is stable.

If it does not stay up however I would see up/down messages in dmesg or alike and/or an increase in error counters.

USB connection are often less mission critical, but, look at their physical connection method : just some metal plates against other metal plates, so the smallest pollution or whatever can make the connection 'bad'.
Swap cables.
Swap USB ports.
If the connection stays bad, swap the UPS.

All understood. But if the USB connection is physically causing problems and disconnects, the kernel would throw disconnect errors on dmesg would it not?

Go back to the https://networkupstools.org/ web site, and this time check with the user forum first with the obvious question : "what is the best one ?"

So your answer is (again): Buy a better UPS. I am glad to at one point in time switch from Eaton to APC or if I am 100% sure this is the problem do so earlier. As stated before: This worked flawlessly for months up until the very point I upgraded from pfsense 23.x to 24.x. On that day (which also changed nut packages) the trouble started. This is one hell of a coincidence is it not?

Btw : I presume your USB connection going down isn't a NUT issue.

Assuming it is the USB connection that is going down. Other than the nut daemon showing "a problem" there is not indication that the USB connection is going down. No error message pointing to this.

The USB hardware signals the usb-hid driver that the USB connection went down.

which it would also signal in dmesg. Which it does not not meaning that the usb hardware is not going down or in a way the kernel does not recognize it.

dennypage

You are using 2.8.2_5, yes?

Do you have anything in the additional arguments to driver section?

Regarding USB physical issues, in addition to swapping USB ports and cables, you can also try adding a powered USB hub in-between.

j.koopmann

@dennypage

pkg info | grep nut
nut-2.8.2                      Network UPS Tools
pfSense-pkg-nut-2.8.2_4        Network UPS Tools

And GUI saying 2.8.2_4 as well. pfSense 24.11-RELEASE (amd64).

How can I upgrade to 2.8.2_5?

Will look for a powered USB hub in my archive. Same question to you: Would a USB physical error that is so severe that the driver stops working not also result in some sort of disconnect message in dmesg?

Is there a way to increase the debugging level in usb-hid to get more info the next time it stops working?

j.koopmann

Do you have anything in the additional arguments to driver section?

[Keller]
driver=usbhid-ups
port=auto

No. Not really other than port....

dennypage

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

pkg info | grep nut
nut-2.8.2                      Network UPS Tools
pfSense-pkg-nut-2.8.2_4        Network UPS Tools

Those versions are fine with 2.7.2. You should see 2.8.2_5 when you upgrade to pfSense 2.8.0.

FWIW, you should consider upgrading to 2.8.0, with its newer kernel and USB core, before spending a lot of time on this.

Will look for a powered USB hub in my archive. Same question to you: Would a USB physical error that is so severe that the driver stops working not also result in some sort of disconnect message in dmesg?

"Severe" isn't the right way to think of it. The behavior in question would be a momentary disconnect, or rejection of a basic USB communication. It may or may not be logged.

Is there a way to increase the debugging level in usb-hid to get more info the next time it stops working?

First start the service from the UI, and then kill the driver:

killall usbhid-ups

Then, run the driver with debug:

/usr/local/libexec/nut/usbhid-ups -DDD -a YOUR_UPS_NAME_HERE

See what error it shows. If it doesn't show enough you can add more 'D's until it does up to "-DDDDDD".

Think about installing pfSense 2.8.0.

[FYI, I am traveling with very limited availability for the next several weeks...]

Gertjan

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

But if the USB connection is physically causing problems and disconnects, the kernel would throw disconnect errors on dmesg would it not?

No timestamp in 'dmesg' but I can see these :

[25.03-BETA][root@pfSense.bhf.tld]/root: dmesg | grep '700U'
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0 (disconnected)
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0 (disconnected)
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0 (disconnected)
ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0

(We had some minor power issues last week)

If the USB connection itself is ok, but the other side goes bad logically, then I can image dmesg wouldn't show a hardware event.

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

So your answer is (again): Buy a better UPS

Because I can't ask you to open up the UPS, so you can check what happens at the other side.
Is the UPS operated by a dumb controller ? Or something with a CPU ? Does it have an serial console interface ? Can you connect to it ? If possible, this would help clarify the situation.

The only thing I'm pretty sure about, as you and I use the same binaries on the pfSense side, that the issue isn't called 'NUT'.

What differs :
My NUT settings versus your NUT settings (might be identical !).
Me : a 4100 - you use what ? What USB chipset set etc ?
Me : APC : you : something else.
Our USB cables are not identical neither ^^

If it's only you using pfSense + NUT with a "Eaton 9SX 3000i." then there is little hope : you have to dive into it.

Btw : from what I know, NUT, on a low level, opens a USB connection. NUT then sends over what really look like a serial or console connection, "status request commands" to the UPS.
The UPS answers with ... what I've shown above with the upsc command.
If any of the UPS replies gets miss interpreted, because on character is off (misspelled, whatever) then the UPS driver might get stalled .... and times out.
This would explain what you see - and this is, afaik, what the quirks solution can do for you (denny talked about these).

To see this happening, you have to use the NUT doc, and start the NUT processes in debug mode so you can see way more details. These could show you 'why' the communications stops.

And that's the real issue : you have to investigate, and this might take "some time".
It's that ... or, sorry, you opt for the 'change' option.
I prefer of course that you find out what is really happening.
Of course I hate the 'change it' solution ;)

edit : lol, denny just posted how to start the driver in debug mode
Now I see a lot of comm going on.
I didn't know there was that much of a 'chat' bewteen NUT and the UPS.

dennypage

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

If any of the UPS replies gets miss interpreted, because on character is off (misspelled, whatever) then the UPS driver might get stalled .... and times out.
This would explain what you see - and this is, afaik, what the quirks solution can do for you (denny talked about these).

Quirks are used in FreeBSD to identify the device as a UPS to the kernel, which will prevent the kernel from attaching a default kernel driver to the USB device when it is discovered.

Removing an already attached kernel driver requires root privileges, which is why you see some people use "user=root". They do this because their system is missing the appropriate quirk for the UPS.

IMO, it's a better choice to install the missing quirk than to run the NUT driver as root.

j.koopmann

@Gertjan

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

If the USB connection itself is ok, but the other side goes bad logically, then I can image dmesg wouldn't show a hardware event.

Agreed. Possible of course. If it would not coincide with my upgrade months ago this would be my conclusion as well. :-)

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

Is the UPS operated by a dumb controller ? Or something with a CPU ? Does it have an serial console interface ? Can you connect to it ? If possible, this would help clarify the situation.

I really cannot say what exactly the Eaton 9SX 3000i is using but I would suspect some sort of controller. It also has a RS232 and apparently the mge-shut driver should work. I will have to check if my firewall has RS232 and can try to switch. Otherwise I would need a USB to serial interface.

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

If it's only you using pfSense + NUT with a "Eaton 9SX 3000i." then there is little hope : you have to dive into it.

Yep. Trying to do so!

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

If any of the UPS replies gets miss interpreted, because on character is off (misspelled, whatever) then the UPS driver might get stalled .... and times out.
This would explain what you see - and this is, afaik, what the quirks solution can do for you (denny talked about these).

To see this happening, you have to use the NUT doc, and start the NUT processes in debug mode so you can see way more details. These could show you 'why' the communications stops.

Exactly. Will try to do that next thing (might have to install screen first. :-) )

@Gertjan said in NUT suddenly stops working every app. 6 minutes:

And that's the real issue : you have to investigate, and this might take "some time".

Oh that is not an issue. It is more an issue to get better information on where to look and how to look and denny just provided that. THANKS! And yes there is quite a but of chatter.

j.koopmann

This post is deleted!

j.koopmann

@dennypage regarding quirks: I get that and I am running this with user=root at the moment. Do you suspect that a missing quirk is the problem? It would not make sense to me because then it should not work at all without a quirk. :-)

I can't really go to 2.8.0 since I am on the 24.11. I could upgrade to 25.03 beta but have not seen anything in the changelog that would indicate usb or kernel related changes. 24.11 already is on

15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162

dennypage

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

I get that and I am running this with user=root at the moment. Do you suspect that a missing quirk is the problem? It would not make sense to me because then it should not work at all without a quirk. :-)

If the UPS is momentarily disconnecting on the USB, it would be re-discovered by the kernel when it comes back and the default kernel driver would be installed. I don't remember if removing the kernel driver is handled by the NUT USB driver on reconnect, or only on initial connect. I am not able to look through the code at the current time.

I would certainly put the effort in to do a quirk and get rid of user=root.

See information on quirks here.

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

I can't really go to 2.8.0 since I am on the 24.11.

Sorry, for some reason I thought you were on 2.7.2. My bad.

j.koopmann

@dennypage said in NUT suddenly stops working every app. 6 minutes:

If the UPS is momentarily disconnecting on the USB, it would be re-discovered by the kernel when it comes back and the default kernel driver would be installed. I don't remember if removing the kernel driver is handled by the NUT USB driver on reconnect, or only on initial connect. I am not able to look through the code at the current time.

Will take a look in a moment and do my best.

IF however this is caused by a disconnect and the kernel would discover this, that would surely (99%) create the appropriate messages in dmesg.

I have the driver running with -DDD and of course it is not showing any errors whatsoever. Will try to have it log to a logfile and run it in screen and then wait for the next failure to happen hoping to find anything that will help us (or rather me).

j.koopmann

@dennypage I removed user = root and restarted. All is working fine which means I do not need a quirk does it not. However:

root    76470   0.1  0.1  18600   7828  -  Ss   18:08        0:00.00 /usr/local/sbin/upsmon
nut     76794   0.1  0.1  18736   8268  -  S    18:08        0:00.02 /usr/local/sbin/upsmon
nut     43869   0.0  0.0  13888   3740  -  Ss   18:08        0:00.00 /usr/local/libexec/nut/usbhid-ups -a Keller
root    67123   0.0  0.2  26228  14152  -  Ss   18:08        0:00.00 /usr/local/sbin/upsd -u root

Is this the normal output? Two upsmon one as nut one as root? And upsd -u root? I checked /usr/local/etc/nut and there is no reference to root in the *.conf files anymore...

dennypage

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

@dennypage I removed user = root and restarted. All is working fine which means I do not need a quirk does it not. However:
root    76470   0.1  0.1  18600   7828  -  Ss   18:08        0:00.00 /usr/local/sbin/upsmon
nut     76794   0.1  0.1  18736   8268  -  S    18:08        0:00.02 /usr/local/sbin/upsmon
nut     43869   0.0  0.0  13888   3740  -  Ss   18:08        0:00.00 /usr/local/libexec/nut/usbhid-ups -a Keller
root    67123   0.0  0.2  26228  14152  -  Ss   18:08        0:00.00 /usr/local/sbin/upsd -u root
Is this the normal output? Two upsmon one as nut one as root? And upsd -u root? I checked /usr/local/etc/nut and there is no reference to root in the *.conf files anymore...

Yes, that is correct. The NUT driver, /usr/local/libexec/nut/usbhid-ups in this case, is the process that would need root privs to override the kernel driver.

dennypage

@dennypage said in [NUT suddenly stops working every app.

@j-koopmann said in NUT suddenly stops working every app. 6 minutes:

I can't really go to 2.8.0 since I am on the 24.11.

Sorry, for some reason I thought you were on 2.7.2. My bad.

FWIW, you have boot environments, so testing with 25.03 Beta is pretty easy. Or you could wait for GA. But 24.11 -> 25.03 is not as big a step as 2.7.2 -> 2.8.0.