Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfsense 2.7 menu stuck with NUT crash

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 5 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      darkcorner
      last edited by darkcorner

      After upgrading to 2.7, I have a crash report with NUT, but the main problem is that the whole menu is blocked and I can't enter anywhere.
      I can try to log in from the terminal (by remote from a PC in the LAN), assuming port 22 is active, but how do I delete the package?

      === Update
      With a test, port 22 is closed 😠

      dennypageD 1 Reply Last reply Reply Quote 0
      • dennypageD
        dennypage @darkcorner
        last edited by

        @darkcorner Very, very hard to imagine how a NUT crash could be a cause for the UI being blocked.

        Is the unit actually powered up? Are you able to connect via HTTPS? If so, what do you see?

        1 Reply Last reply Reply Quote 1
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          You mean the console menu appears unresponsive?

          Is the machine responding at all? How are you connected?

          Steve

          1 Reply Last reply Reply Quote 0
          • D
            darkcorner
            last edited by

            I connect remotely to a PC, so both the Internet and the firewall are active.
            From the PC I log in to pfSense.
            The dashport appears, the first column on the left is composed, the last element is for NUT and reports errors.
            Immediately below the menu there is a yellow line indicating the crash report. I can go in and see it.
            But I can't close the bar, nor does any click on the menu line work.

            This firewall is the first firewall I updated in August that took days to reboot.

            Side note: I remotely updated a third firewall tonight. 15 minutes have already passed since the final message announcing its restart, but it still doesn't respond nor can I connect to other PCs on the network.
            If it doesn't unlock by tomorrow I will have to go to this customer to restart it by hand.

            dennypageD 1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              It can take more than 15mins of the storage and/or connection is slow.

              Are you able to get the crash report?

              Are LAN side hosts able to connect?

              D 1 Reply Last reply Reply Quote 0
              • D
                darkcorner @stephenw10
                last edited by darkcorner

                @stephenw10
                The crash report is related to NUT (see below), but the problem is not being able to use the GUI menu and, if SSH is not enabled, not even from the terminal.

                If you are referring to the pfSense that crashed tonight, I'm remote from home, if pfSense doesn't start I can't test the LAN.
                If, however, you are referring to pfSense with the menu blocked, as I wrote initially, yes, I can connect remotely to the PC and from there to the pfSense GUI.

                about pfSense crashing tonight, in the virtual machine the update took 5 minutes.
                This pfSense is instead installed on a quadcore i5 PC, but now it hasn't been connected for more than an hour.
                I'll try tomorrow morning, again remotely.
                If it doesn't connect, I take the car and go to the customer to see what happens.

                Crash report(s)

                Crash report begins.  Anonymous machine information:
                
                amd64
                14.0-CURRENT
                FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL
                
                Crash report details:
                
                PHP Errors:
                [04-Aug-2023 14:33:26 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
                Stack trace:
                #0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
                #1 /etc/inc/pkg-utils.inc(709): eval()
                #2 /etc/rc.start_packages(66): sync_package('nut')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 133
                [04-Aug-2023 14:35:25 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
                Stack trace:
                #0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
                #1 /etc/inc/pkg-utils.inc(709): eval()
                #2 /etc/rc.start_packages(66): sync_package('nut')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 133
                [08-Aug-2023 13:43:48 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
                Stack trace:
                #0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
                #1 /etc/inc/pkg-utils.inc(709): eval()
                #2 /etc/rc.start_packages(66): sync_package('nut')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 133
                [08-Aug-2023 13:47:51 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
                Stack trace:
                #0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
                #1 /etc/inc/pkg-utils.inc(709): eval()
                #2 /etc/rc.start_packages(66): sync_package('nut')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 133
                [08-Aug-2023 13:53:53 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
                Stack trace:
                #0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
                #1 /etc/inc/pkg-utils.inc(709): eval()
                #2 /etc/rc.start_packages(66): sync_package('nut')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 133
                [12-Aug-2023 11:20:39 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
                Stack trace:
                #0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
                #1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
                #2 /usr/local/www/index.php(428): include('/usr/local/www/...')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 306
                
                
                
                No FreeBSD crash data found.
                
                
                ====================
                Crash report begins.  Anonymous machine information:
                
                amd64
                14.0-CURRENT
                FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL
                
                Crash report details:
                
                PHP Errors:
                [18-Sep-2023 16:50:31 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
                Stack trace:
                #0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
                #1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
                #2 /usr/local/www/index.php(428): include('/usr/local/www/...')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 306
                [18-Sep-2023 17:00:43 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
                Stack trace:
                #0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
                #1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
                #2 /usr/local/www/index.php(428): include('/usr/local/www/...')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 306
                [28-Sep-2023 12:00:41 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
                Stack trace:
                #0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
                #1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
                #2 /usr/local/www/index.php(428): include('/usr/local/www/...')
                #3 {main}
                  thrown in /usr/local/pkg/nut/nut.inc on line 306
                
                No FreeBSD crash data found.
                
                1 Reply Last reply Reply Quote 0
                • dennypageD
                  dennypage @darkcorner
                  last edited by

                  @darkcorner said in pfsense 2.7 menu stuck with NUT crash:

                  This firewall is the first firewall I updated in August that took days to reboot.

                  Days?!? Something sounds really wrong here.

                  Was it days to reboot, or days until you were able to get in? Does the firewall use a static WAN address, or is it a dynamic address?

                  D 1 Reply Last reply Reply Quote 0
                  • D
                    darkcorner @dennypage
                    last edited by

                    @dennypage
                    The update was successful, the message that it was restarting appeared and I lost the connection.
                    I found it again days later.
                    The office was closed for the holidays and there was no one there to start it manually.
                    The line is on optical fiber with static IP.

                    I'm afraid that with pfsense blocked, tonight's update will happen again.
                    And it would be the third firewall (on three different PCs) that crashed with the update.

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      SteveITS Galactic Empire @darkcorner
                      last edited by

                      @darkcorner I’m not following what isn’t working. You can’t connect to the pfSense web GUI from LAN? But internet is working since you have control of a PC?

                      It doesn’t help much now but the upgrade guide recommends removing packages when upgrading. Otherwise the upgrade will uninstall/reinstall packages for you, even if PHP is being upgraded during the operation. I’ve never used NUT so I have no insight there, re upgrades.

                      Without console access, logs, etc., it’s going to be difficult to diagnose what happened.

                      10-15m upgrade time is common on eMMC storage.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote šŸ‘ helpful posts!

                      D 1 Reply Last reply Reply Quote 0
                      • D
                        darkcorner @SteveITS
                        last edited by

                        @SteveITS
                        I solved the problem with NUT with an intuition.
                        I thought the yellow line containing the error message was blocking the menu, but this too could not be closed with a click on (x).
                        However, this line contained a link that allowed me to open the explanation page.
                        So I tried to see if at least from this page the menu worked and it actually worked.
                        I immediately removed the NUT package from the package manager and now the entire menu works and the entire dashboard has also reappeared.

                        In the other firewall, the one that crashed this evening, I had removed all the packages, but evidently it didn't help much.
                        After 5 hours I still don't have a remote connection.
                        We'll see if there's any news tomorrow morning.

                        NollipfSenseN S 2 Replies Last reply Reply Quote 0
                        • NollipfSenseN
                          NollipfSense @darkcorner
                          last edited by NollipfSense

                          @darkcorner said in pfsense 2.7 menu stuck with NUT crash:

                          solved the problem with NUT with an intuition.

                          Hmm...That statement is very confusing, indeed...even baffling. I run the NUT package and found it really simple rarely with a hiccup. I know it doesn't like upgraded and prefers to be removed first and re-installed on a clean pfSense.

                          Something you're doing is not right.

                          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                          1 Reply Last reply Reply Quote 0
                          • S
                            SteveITS Galactic Empire @darkcorner
                            last edited by

                            @darkcorner Ah, so the issue was a menu was covered up? Next time a picture would help get that across. :)

                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                            Upvote šŸ‘ helpful posts!

                            1 Reply Last reply Reply Quote 0
                            • D
                              darkcorner
                              last edited by

                              @NollipfSense
                              All three firewalls have always worked fine without errors before the update to 2.7 and have not required corrections in their configuration for more than a year.
                              And all three crashed while rebooting after installation.
                              Now I take the car and go to the third office to see what's happening because the customer also works on Sunday.
                              From the console maybe I see some messages that I don't see remotely since everything is blocked.

                              I don't know what's wrong with what I'm doing.
                              This certainly wasn't how the update was supposed to be done.
                              If they could not be updated immediately, the packages also had to be deactivated.
                              Then, after the reboot, you could decide what was best to do, but with pfSense active and the GUI working.
                              PCs and *nix servers have been updated for 40 years without all these problems.
                              In any case, the latest pfSense had no packages because I removed them all.
                              Out of 5 attempts (3 PCs and 2 two VMs), 4 crashed and only one worked on the first try.

                              @SteveITS
                              No, the menu was visible, but it was locked.
                              I wrote it down: there was the menu, the yellow error bar and only the first column of the dashboard.
                              Everything blocked, even the (x) to close the yellow bar.
                              The only thing that worked was the link in this error bar which took you to the report page. On this page the menu worked.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Of those that failed and did not have NUT installed have you been able to connect to any directly to see where it's failing?

                                1 Reply Last reply Reply Quote 0
                                • D
                                  darkcorner
                                  last edited by darkcorner

                                  Apologies for the photo, which isn't the best, but it explains what happened (probably to all three firewalls).

                                  This photo is of the third pfsense firewall that crashed after upgrading from 2.6 to 2.7.
                                  Updates that were done remotely by connecting in all three cases to a PC with AnyDesk and from this PC by accessing pfSense on the LAN.
                                  All three successfully completed the update in the GUI, then disappeared because obviously if something happens to pfSense not even the remote connection via the Internet will work anymore.

                                  The first pfSense was stuck for days and I always thought that was how long it took to update.
                                  The second one crashed, but I was able to go in front of the PC and restart it by hand.
                                  The third one crashed last Saturday and yesterday I went to see what happened.

                                  I don't know what happened after the update in the GUI, but in the console you can clearly see that there is the "Success" message and hidden there is the "Login" request.
                                  With a normal reboot the list of NICs and the menu should have appeared instead.

                                  It is clear that if it is waiting for login, the system is stopped and must be started manually instead.

                                  This situation was the same in the second PC, I didn't notice "login".

                                  At this point I think that the first PC also froze when logging in and remained that way for days because there was no one to restart it. Then some problem with the electricity network shut down the PC and it then started up normally.

                                  This post was created due to a problem on NUT which, as I explained in the last replies, I then resolved.
                                  All three had NUTs, and only one has it now because the UPS was removed while waiting for a new one to arrive.

                                  PCs are different. The first two have different old motherboards with Celeron. The third one has i3-4150 CPU 3.50GHz quadcore.
                                  All three offices have a maximum of 3 active users, on average 2.

                                  WhatsApp Image 2023-10-02 at 01.57.06.jpeg

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, that's odd. That's what you see if you have set the console to be password protected but I assume that was not set on any of the affected systems?

                                    If you logged in at that prompt did it allow the upgrade to complete? Or then present the menu as expected?

                                    D 1 Reply Last reply Reply Quote 0
                                    • D
                                      darkcorner @stephenw10
                                      last edited by darkcorner

                                      @stephenw10
                                      I'm used to leaving configurations as standard as possible. I wouldn't even know where to configure console protection with a password.
                                      So the answer is, no, there shouldn't be.

                                      If anything, I'd better check now.

                                      dennypageD 1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Hmm, might need to see more of the failing boot process to know more there. It's not booting to the amnesiac prompt which I might have otherwise expected.

                                        I assume that it boots correctly on following reboots?

                                        1 Reply Last reply Reply Quote 0
                                        • dennypageD
                                          dennypage @darkcorner
                                          last edited by

                                          @darkcorner said in pfsense 2.7 menu stuck with NUT crash:

                                          I wouldn't even know where to configure console protection with a password. So the answer is, no, there shouldn't be.

                                          It's a checkbox in System / Advanced / Admin Access.

                                          Screenshot 2023-10-03 at 07.56.17.png

                                          It's quite likely something you set and didn't think too much about at the time. It's in a bunch of pfSense HowTos.

                                          1 Reply Last reply Reply Quote 1
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.