Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    NUT suddenly stops working every app. 6 minutes

    Scheduled Pinned Locked Moved UPS Tools
    46 Posts 3 Posters 2.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • dennypageD
      dennypage @Gertjan
      last edited by

      @Gertjan said in NUT suddenly stops working every app. 6 minutes:

      If any of the UPS replies gets miss interpreted, because on character is off (misspelled, whatever) then the UPS driver might get stalled .... and times out.
      This would explain what you see - and this is, afaik, what the quirks solution can do for you (denny talked about these).

      Quirks are used in FreeBSD to identify the device as a UPS to the kernel, which will prevent the kernel from attaching a default kernel driver to the USB device when it is discovered.

      Removing an already attached kernel driver requires root privileges, which is why you see some people use "user=root". They do this because their system is missing the appropriate quirk for the UPS.

      IMO, it's a better choice to install the missing quirk than to run the NUT driver as root.

      J 2 Replies Last reply Reply Quote 0
      • J
        j.koopmann @Gertjan
        last edited by

        @Gertjan

        @Gertjan said in NUT suddenly stops working every app. 6 minutes:

        If the USB connection itself is ok, but the other side goes bad logically, then I can image dmesg wouldn't show a hardware event.

        Agreed. Possible of course. If it would not coincide with my upgrade months ago this would be my conclusion as well. :-)

        @Gertjan said in NUT suddenly stops working every app. 6 minutes:

        Is the UPS operated by a dumb controller ? Or something with a CPU ? Does it have an serial console interface ? Can you connect to it ? If possible, this would help clarify the situation.

        I really cannot say what exactly the Eaton 9SX 3000i is using but I would suspect some sort of controller. It also has a RS232 and apparently the mge-shut driver should work. I will have to check if my firewall has RS232 and can try to switch. Otherwise I would need a USB to serial interface.

        @Gertjan said in NUT suddenly stops working every app. 6 minutes:

        If it's only you using pfSense + NUT with a "Eaton 9SX 3000i." then there is little hope : you have to dive into it.

        Yep. Trying to do so!

        @Gertjan said in NUT suddenly stops working every app. 6 minutes:

        If any of the UPS replies gets miss interpreted, because on character is off (misspelled, whatever) then the UPS driver might get stalled .... and times out.
        This would explain what you see - and this is, afaik, what the quirks solution can do for you (denny talked about these).

        To see this happening, you have to use the NUT doc, and start the NUT processes in debug mode so you can see way more details. These could show you 'why' the communications stops.

        Exactly. Will try to do that next thing (might have to install screen first. :-) )

        @Gertjan said in NUT suddenly stops working every app. 6 minutes:

        And that's the real issue : you have to investigate, and this might take "some time".

        Oh that is not an issue. It is more an issue to get better information on where to look and how to look and denny just provided that. THANKS! And yes there is quite a but of chatter.

        GertjanG 1 Reply Last reply Reply Quote 0
        • J
          j.koopmann @dennypage
          last edited by

          This post is deleted!
          dennypageD 1 Reply Last reply Reply Quote 0
          • J
            j.koopmann @dennypage
            last edited by

            @dennypage regarding quirks: I get that and I am running this with user=root at the moment. Do you suspect that a missing quirk is the problem? It would not make sense to me because then it should not work at all without a quirk. :-)

            I can't really go to 2.8.0 since I am on the 24.11. I could upgrade to 25.03 beta but have not seen anything in the changelog that would indicate usb or kernel related changes. 24.11 already is on

            15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162

            1 Reply Last reply Reply Quote 0
            • dennypageD
              dennypage @j.koopmann
              last edited by

              @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

              I get that and I am running this with user=root at the moment. Do you suspect that a missing quirk is the problem? It would not make sense to me because then it should not work at all without a quirk. :-)

              If the UPS is momentarily disconnecting on the USB, it would be re-discovered by the kernel when it comes back and the default kernel driver would be installed. I don't remember if removing the kernel driver is handled by the NUT USB driver on reconnect, or only on initial connect. I am not able to look through the code at the current time.

              I would certainly put the effort in to do a quirk and get rid of user=root.

              See information on quirks here.

              @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

              I can't really go to 2.8.0 since I am on the 24.11.

              Sorry, for some reason I thought you were on 2.7.2. My bad.

              J dennypageD 3 Replies Last reply Reply Quote 0
              • J
                j.koopmann @dennypage
                last edited by

                @dennypage said in NUT suddenly stops working every app. 6 minutes:

                If the UPS is momentarily disconnecting on the USB, it would be re-discovered by the kernel when it comes back and the default kernel driver would be installed. I don't remember if removing the kernel driver is handled by the NUT USB driver on reconnect, or only on initial connect. I am not able to look through the code at the current time.

                Will take a look in a moment and do my best.

                IF however this is caused by a disconnect and the kernel would discover this, that would surely (99%) create the appropriate messages in dmesg.

                I have the driver running with -DDD and of course it is not showing any errors whatsoever. Will try to have it log to a logfile and run it in screen and then wait for the next failure to happen hoping to find anything that will help us (or rather me).

                1 Reply Last reply Reply Quote 0
                • J
                  j.koopmann @dennypage
                  last edited by

                  @dennypage I removed user = root and restarted. All is working fine which means I do not need a quirk does it not. However:

                  root    76470   0.1  0.1  18600   7828  -  Ss   18:08        0:00.00 /usr/local/sbin/upsmon
nut     76794   0.1  0.1  18736   8268  -  S    18:08        0:00.02 /usr/local/sbin/upsmon
nut     43869   0.0  0.0  13888   3740  -  Ss   18:08        0:00.00 /usr/local/libexec/nut/usbhid-ups -a Keller
root    67123   0.0  0.2  26228  14152  -  Ss   18:08        0:00.00 /usr/local/sbin/upsd -u root

                  Is this the normal output? Two upsmon one as nut one as root? And upsd -u root? I checked /usr/local/etc/nut and there is no reference to root in the *.conf files anymore...

                  dennypageD 1 Reply Last reply Reply Quote 0
                  • dennypageD
                    dennypage @j.koopmann
                    last edited by

                    @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                    @dennypage I removed user = root and restarted. All is working fine which means I do not need a quirk does it not. However:

                    root    76470   0.1  0.1  18600   7828  -  Ss   18:08        0:00.00 /usr/local/sbin/upsmon
nut     76794   0.1  0.1  18736   8268  -  S    18:08        0:00.02 /usr/local/sbin/upsmon
nut     43869   0.0  0.0  13888   3740  -  Ss   18:08        0:00.00 /usr/local/libexec/nut/usbhid-ups -a Keller
root    67123   0.0  0.2  26228  14152  -  Ss   18:08        0:00.00 /usr/local/sbin/upsd -u root

                    Is this the normal output? Two upsmon one as nut one as root? And upsd -u root? I checked /usr/local/etc/nut and there is no reference to root in the *.conf files anymore...

                    Yes, that is correct. The NUT driver, /usr/local/libexec/nut/usbhid-ups in this case, is the process that would need root privs to override the kernel driver.

                    1 Reply Last reply Reply Quote 0
                    • dennypageD
                      dennypage @dennypage
                      last edited by

                      @dennypage said in [NUT suddenly stops working every app.

                      @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                      I can't really go to 2.8.0 since I am on the 24.11.

                      Sorry, for some reason I thought you were on 2.7.2. My bad.

                      FWIW, you have boot environments, so testing with 25.03 Beta is pretty easy. Or you could wait for GA. But 24.11 -> 25.03 is not as big a step as 2.7.2 -> 2.8.0.

                      1 Reply Last reply Reply Quote 0
                      • GertjanG
                        Gertjan @j.koopmann
                        last edited by

                        @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                        It also has a RS232

                        That might be a good 'plan B'.
                        Cables that convert a serial port to an USB port exist. NUT will allow you to use a serial port. That is, it's still a USB cable that connects, and that connect will enable a virtual 'serial' interface, avaible for pfSense thus NUT.

                        No "help me" PM's please. Use the forum, the community will thank you.
                        Edit : and where are the logs ??

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          j.koopmann @Gertjan
                          last edited by

                          @Gertjan good theory. However.....

                          The eaton does 9600 baud and the serial driver is fix 2400 baud...

                          So today 6:30 Uhr the usbhid-ups stopped working. It was signal 15 SIGTERM and after debugging for a while it was due to a VPN connection falling down which triggered a suspected IP change

                          php-fpm 66039 - - /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - x.x.x.x > y.y.y.y - Restarting packages.

                          The entire stack was killed and came back... So back to the drawing board and restart the driver with -DDD to see if the real error resurfaces. However: It was ok for a long time now...

                          GertjanG 1 Reply Last reply Reply Quote 0
                          • GertjanG
                            Gertjan @j.koopmann
                            last edited by

                            @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                            stopped working. It was signal 15 SIGTERM and after

                            SIGTERM is send to a process - NUT in this case - by the system - pfSense or FreeBSD - because some system event happened.
                            Examples : an interface was set up, or vanished.
                            ( still strange, as localhost - 127.0.0.1 - actually always stays up during the run time of the OS )
                            Anyway, "VPN" did something with the available interface and all interface bound process like nginx, unbound, DHCP etc etc will receive a 'terminate' and will be restarted.
                            The dreaded :

                            Restarting packages.

                            @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                            The entire stack was killed and came back... So back to the drawing board and restart the driver with -DDD to see if the real error resurfaces. However: It was ok for a long time now...

                            So, all the NUT processes were restarted, but a working USB connection wasn't created ?

                            No "help me" PM's please. Use the forum, the community will thank you.
                            Edit : and where are the logs ??

                            J 1 Reply Last reply Reply Quote 0
                            • J
                              j.koopmann @Gertjan
                              last edited by

                              @Gertjan said in NUT suddenly stops working every app. 6 minutes:

                              SIGTERM is send to a process - NUT in this case - by the system - pfSense or FreeBSD - because some system event happened.

                              I might have hidden it too well due to the way I communicate: I am in the business for > 30 years... I am well aware what a SIGTERM is. :-)

                              @Gertjan said in NUT suddenly stops working every app. 6 minutes:

                              Anyway, "VPN" did something with the available interface and all interface bound process like nginx, unbound, DHCP etc etc will receive a 'terminate' and will be restarted.

                              I know. That was the conclusion today.

                              @Gertjan said in NUT suddenly stops working every app. 6 minutes:

                              So, all the NUT processes were restarted, but a working USB connection wasn't created ?

                              Whenever usbhid-ups is restarted it ALWAYS establishes a working USB connection. Everything was working fine this morning. I was inspecting my -DDD log and just saw that it stopped recording around 6:30 which led me to believe the original problem had resurfaced. But it was the VPN connection triggering the service restart.

                              So I reenabled debugging and wait for a "real" failure.

                              J 1 Reply Last reply Reply Quote 1
                              • J
                                j.koopmann @j.koopmann
                                last edited by

                                So far everything is stable without me changing anything else. Who knows why....

                                J 1 Reply Last reply Reply Quote 0
                                • J
                                  j.koopmann @j.koopmann
                                  last edited by

                                  Started happening again (not that frequently but still...)

                                  Today I noticed it stopped around 6:30.

                                  upsmon 89561 - - Poll UPS [Keller] failed - Protocol error

                                  The usbhid-ups was still running. I killed it and restarted it with -DDD. As always it comes up flawlessly and communicates with the UPS. BUT upsmon continues to show Protocol errors. That is new (to me). Once usbhid-ups is back again (started with -a Keller of course) usbmon should be able to reestablish connection to it. In this case it failes!

                                  After restarting usbmon and usbhid-ups things come back. Why is usbmon showing a protocol error when usbhid-ups clearly is running and showing no errors and has a clearly established communication with the UPS?

                                  GertjanG dennypageD 2 Replies Last reply Reply Quote 0
                                  • GertjanG
                                    Gertjan @j.koopmann
                                    last edited by

                                    @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                                    upsmon 89561 - - Poll UPS [Keller] failed - Protocol error

                                    I've tried to generate such a message myself.

                                    I srated this on the console :

                                    tail -f /var/log/system.log

                                    and then disconnected the USB cable that connects pfSense to the UPS.

                                    This was the result :

                                    <38>1 2025-06-26T14:26:40.471014+02:00 bhf.tld sshd 32452 - - Accepted publickey for root from 192.168.1.6 port 51478 ssh2: RSA SHA256:t6AMtOQbd+vBU56dvmXq3xE+lWsMMoOUn9njUtgMwTQ
<2>1 2025-06-26T14:27:09.833170+02:00 bhf.tld kernel - - - ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0 (disconnected)
<28>1 2025-06-26T14:27:12.075594+02:00 bhf.tld usbhid-ups 88307 - - libusb1: Could not open any HID devices: no USB buses found
<29>1 2025-06-26T14:27:12.075778+02:00 bhf.tld upsd 87637 - - Data for UPS [UPS] is stale - check driver
<2>1 2025-06-26T14:27:13.652138+02:00 bhf.tld kernel - - - ugen0.2: <American Power Conversion Back-UPS XS 700U   FW:924.Z5 .I USB FW:Z5> at usbus0
<29>1 2025-06-26T14:27:14.212641+02:00 bhf.tld upsd 87637 - - UPS [UPS] data is no longer stale

                                    and when checking the UPS :

                                    d578732d-008c-4b3b-bfda-aa404941b6d4-image.png

                                    and the settings page : all was fine.

                                    I guess its time to dive in the upsmon source file to see what "Poll UPS [xxxx] failed - Protocol error" really means, what the connect is where this line is logged.

                                    No "help me" PM's please. Use the forum, the community will thank you.
                                    Edit : and where are the logs ??

                                    J 1 Reply Last reply Reply Quote 0
                                    • J
                                      j.koopmann @Gertjan
                                      last edited by

                                      @Gertjan I agree. I suppose you disconnected and then reconnected?

                                      When I restart via pfsense and then kill the driver and manually restart it with -DDD upsmon either does not notice the problem or recovers automatically. Which is why I am puzzled that exactly this did not happen. And yes I suspect an error in upsmon. But I am not experienced enough to dig into the source code.

                                      GertjanG 1 Reply Last reply Reply Quote 0
                                      • GertjanG
                                        Gertjan @j.koopmann
                                        last edited by Gertjan

                                        @j-koopmann said in NUT suddenly stops working every app. 6 minutes:

                                        then kill the driver

                                        That can/will not happen in reality ?! right ?

                                        It probably happens when processes get restarted because there was an Interface event somewhere.
                                        For me, that's against my "church rules". It says : as the admin, do everything, and more, so that interfaces always say up. No exceptions allowed.
                                        And I'm not allowed to believe.
                                        I have to fact check myself being my own big-brother - click and see for yourself.
                                        That's why I use an UPS btw ^^

                                        edit : the stats have a granularity (?) of '300 seconds'. That elmans that smaller outages are possible a can go unnoticed in stats.
                                        But I'm a "good admin", I rarely look at the dashboard page. I look at the logs, as that is place where the real fun is being showed.

                                        No "help me" PM's please. Use the forum, the community will thank you.
                                        Edit : and where are the logs ??

                                        J 1 Reply Last reply Reply Quote 0
                                        • J
                                          j.koopmann @Gertjan
                                          last edited by

                                          @Gertjan said in NUT suddenly stops working every app. 6 minutes:

                                          That can/will not happen in reality ?! right ?

                                          I hope not. Only reason for me doing it currently is to have more debug output and see if the underlying problem is in the USB communication between the driver and the UPS (which was speculated but I believe not to be the case so far) or somewhere else.

                                          I just wanted to second your conclusion (at least I believe). If the USB connection goes away and comes back upsmon regularly is fine. Even if I kill the driver and get it back online, upsmon is fine. Not so this morning however.

                                          GertjanG 1 Reply Last reply Reply Quote 0
                                          • GertjanG
                                            Gertjan @j.koopmann
                                            last edited by

                                            @j-koopmann

                                            Normally, I don't checkup with my 4100 all the time : as long as PC applications that need a connection, work fine, I can work (which doesnt' include pfSense baby sitting).

                                            I do have this on :

                                            3f4b7338-38eb-4336-b310-d87672b91b99-image.png

                                            = WinNut 2.0 client on my desktop.

                                            And it does happen that is shows this :

                                            82169195-37e7-42f5-9c4d-c6f5e62b5771-image.png so I know I have to go to pfSense and start UPS again as it failed 'for some reason'.
                                            Now I'm thinking about it, this happens more often then the wished for 'never' ^^

                                            No "help me" PM's please. Use the forum, the community will thank you.
                                            Edit : and where are the logs ??

                                            J 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.