pfsense 2.7 menu stuck with NUT crash

darkcorner

After upgrading to 2.7, I have a crash report with NUT, but the main problem is that the whole menu is blocked and I can't enter anywhere.
I can try to log in from the terminal (by remote from a PC in the LAN), assuming port 22 is active, but how do I delete the package?

=== Update
With a test, port 22 is closed

dennypage

@darkcorner Very, very hard to imagine how a NUT crash could be a cause for the UI being blocked.

Is the unit actually powered up? Are you able to connect via HTTPS? If so, what do you see?

stephenw10

You mean the console menu appears unresponsive?

Is the machine responding at all? How are you connected?

Steve

darkcorner

I connect remotely to a PC, so both the Internet and the firewall are active.
From the PC I log in to pfSense.
The dashport appears, the first column on the left is composed, the last element is for NUT and reports errors.
Immediately below the menu there is a yellow line indicating the crash report. I can go in and see it.
But I can't close the bar, nor does any click on the menu line work.

This firewall is the first firewall I updated in August that took days to reboot.

Side note: I remotely updated a third firewall tonight. 15 minutes have already passed since the final message announcing its restart, but it still doesn't respond nor can I connect to other PCs on the network.
If it doesn't unlock by tomorrow I will have to go to this customer to restart it by hand.

stephenw10

It can take more than 15mins of the storage and/or connection is slow.

Are you able to get the crash report?

Are LAN side hosts able to connect?

darkcorner

@stephenw10
The crash report is related to NUT (see below), but the problem is not being able to use the GUI menu and, if SSH is not enabled, not even from the terminal.

If you are referring to the pfSense that crashed tonight, I'm remote from home, if pfSense doesn't start I can't test the LAN.
If, however, you are referring to pfSense with the menu blocked, as I wrote initially, yes, I can connect remotely to the PC and from there to the pfSense GUI.

about pfSense crashing tonight, in the virtual machine the update took 5 minutes.
This pfSense is instead installed on a quadcore i5 PC, but now it hasn't been connected for more than an hour.
I'll try tomorrow morning, again remotely.
If it doesn't connect, I take the car and go to the customer to see what happens.

Crash report(s)

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[04-Aug-2023 14:33:26 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
Stack trace:
#0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
#1 /etc/inc/pkg-utils.inc(709): eval()
#2 /etc/rc.start_packages(66): sync_package('nut')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 133
[04-Aug-2023 14:35:25 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
Stack trace:
#0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
#1 /etc/inc/pkg-utils.inc(709): eval()
#2 /etc/rc.start_packages(66): sync_package('nut')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 133
[08-Aug-2023 13:43:48 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
Stack trace:
#0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
#1 /etc/inc/pkg-utils.inc(709): eval()
#2 /etc/rc.start_packages(66): sync_package('nut')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 133
[08-Aug-2023 13:47:51 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
Stack trace:
#0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
#1 /etc/inc/pkg-utils.inc(709): eval()
#2 /etc/rc.start_packages(66): sync_package('nut')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 133
[08-Aug-2023 13:53:53 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:133
Stack trace:
#0 /etc/inc/pkg-utils.inc(709) : eval()'d code(1): nut_sync_config()
#1 /etc/inc/pkg-utils.inc(709): eval()
#2 /etc/rc.start_packages(66): sync_package('nut')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 133
[12-Aug-2023 11:20:39 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
Stack trace:
#0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
#1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
#2 /usr/local/www/index.php(428): include('/usr/local/www/...')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 306



No FreeBSD crash data found.


====================
Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[18-Sep-2023 16:50:31 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
Stack trace:
#0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
#1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
#2 /usr/local/www/index.php(428): include('/usr/local/www/...')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 306
[18-Sep-2023 17:00:43 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
Stack trace:
#0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
#1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
#2 /usr/local/www/index.php(428): include('/usr/local/www/...')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 306
[28-Sep-2023 12:00:41 Europe/Rome] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/nut/nut.inc:306
Stack trace:
#0 /usr/local/www/widgets/widgets/nut_status.widget.php(55): nut_ups_status()
#1 /usr/local/www/widgets/widgets/nut_status.widget.php(84): print_table()
#2 /usr/local/www/index.php(428): include('/usr/local/www/...')
#3 {main}
  thrown in /usr/local/pkg/nut/nut.inc on line 306

No FreeBSD crash data found.

dennypage

@darkcorner said in pfsense 2.7 menu stuck with NUT crash:

This firewall is the first firewall I updated in August that took days to reboot.

Days?!? Something sounds really wrong here.

Was it days to reboot, or days until you were able to get in? Does the firewall use a static WAN address, or is it a dynamic address?

darkcorner

@dennypage
The update was successful, the message that it was restarting appeared and I lost the connection.
I found it again days later.
The office was closed for the holidays and there was no one there to start it manually.
The line is on optical fiber with static IP.

I'm afraid that with pfsense blocked, tonight's update will happen again.
And it would be the third firewall (on three different PCs) that crashed with the update.

SteveITS

@darkcorner I’m not following what isn’t working. You can’t connect to the pfSense web GUI from LAN? But internet is working since you have control of a PC?

It doesn’t help much now but the upgrade guide recommends removing packages when upgrading. Otherwise the upgrade will uninstall/reinstall packages for you, even if PHP is being upgraded during the operation. I’ve never used NUT so I have no insight there, re upgrades.

Without console access, logs, etc., it’s going to be difficult to diagnose what happened.

10-15m upgrade time is common on eMMC storage.

darkcorner

@SteveITS
I solved the problem with NUT with an intuition.
I thought the yellow line containing the error message was blocking the menu, but this too could not be closed with a click on (x).
However, this line contained a link that allowed me to open the explanation page.
So I tried to see if at least from this page the menu worked and it actually worked.
I immediately removed the NUT package from the package manager and now the entire menu works and the entire dashboard has also reappeared.

In the other firewall, the one that crashed this evening, I had removed all the packages, but evidently it didn't help much.
After 5 hours I still don't have a remote connection.
We'll see if there's any news tomorrow morning.

NollipfSense

@darkcorner said in pfsense 2.7 menu stuck with NUT crash:

solved the problem with NUT with an intuition.

Hmm...That statement is very confusing, indeed...even baffling. I run the NUT package and found it really simple rarely with a hiccup. I know it doesn't like upgraded and prefers to be removed first and re-installed on a clean pfSense.

Something you're doing is not right.

SteveITS

@darkcorner Ah, so the issue was a menu was covered up? Next time a picture would help get that across. :)

darkcorner

@NollipfSense
All three firewalls have always worked fine without errors before the update to 2.7 and have not required corrections in their configuration for more than a year.
And all three crashed while rebooting after installation.
Now I take the car and go to the third office to see what's happening because the customer also works on Sunday.
From the console maybe I see some messages that I don't see remotely since everything is blocked.

I don't know what's wrong with what I'm doing.
This certainly wasn't how the update was supposed to be done.
If they could not be updated immediately, the packages also had to be deactivated.
Then, after the reboot, you could decide what was best to do, but with pfSense active and the GUI working.
PCs and *nix servers have been updated for 40 years without all these problems.
In any case, the latest pfSense had no packages because I removed them all.
Out of 5 attempts (3 PCs and 2 two VMs), 4 crashed and only one worked on the first try.

@SteveITS
No, the menu was visible, but it was locked.
I wrote it down: there was the menu, the yellow error bar and only the first column of the dashboard.
Everything blocked, even the (x) to close the yellow bar.
The only thing that worked was the link in this error bar which took you to the report page. On this page the menu worked.

stephenw10

Of those that failed and did not have NUT installed have you been able to connect to any directly to see where it's failing?

darkcorner

Apologies for the photo, which isn't the best, but it explains what happened (probably to all three firewalls).

This photo is of the third pfsense firewall that crashed after upgrading from 2.6 to 2.7.
Updates that were done remotely by connecting in all three cases to a PC with AnyDesk and from this PC by accessing pfSense on the LAN.
All three successfully completed the update in the GUI, then disappeared because obviously if something happens to pfSense not even the remote connection via the Internet will work anymore.

The first pfSense was stuck for days and I always thought that was how long it took to update.
The second one crashed, but I was able to go in front of the PC and restart it by hand.
The third one crashed last Saturday and yesterday I went to see what happened.

I don't know what happened after the update in the GUI, but in the console you can clearly see that there is the "Success" message and hidden there is the "Login" request.
With a normal reboot the list of NICs and the menu should have appeared instead.

It is clear that if it is waiting for login, the system is stopped and must be started manually instead.

This situation was the same in the second PC, I didn't notice "login".

At this point I think that the first PC also froze when logging in and remained that way for days because there was no one to restart it. Then some problem with the electricity network shut down the PC and it then started up normally.

This post was created due to a problem on NUT which, as I explained in the last replies, I then resolved.
All three had NUTs, and only one has it now because the UPS was removed while waiting for a new one to arrive.

PCs are different. The first two have different old motherboards with Celeron. The third one has i3-4150 CPU 3.50GHz quadcore.
All three offices have a maximum of 3 active users, on average 2.

stephenw10

Hmm, that's odd. That's what you see if you have set the console to be password protected but I assume that was not set on any of the affected systems?

If you logged in at that prompt did it allow the upgrade to complete? Or then present the menu as expected?

darkcorner

@stephenw10
I'm used to leaving configurations as standard as possible. I wouldn't even know where to configure console protection with a password.
So the answer is, no, there shouldn't be.

If anything, I'd better check now.

stephenw10

Hmm, might need to see more of the failing boot process to know more there. It's not booting to the amnesiac prompt which I might have otherwise expected.

I assume that it boots correctly on following reboots?

dennypage

@darkcorner said in pfsense 2.7 menu stuck with NUT crash:

I wouldn't even know where to configure console protection with a password. So the answer is, no, there shouldn't be.

It's a checkbox in System / Advanced / Admin Access.

Screenshot 2023-10-03 at 07.56.17.png

It's quite likely something you set and didn't think too much about at the time. It's in a bunch of pfSense HowTos.