Issue with SG-3100 and 22.01? [Solved]
-
Hi,
I am getting a bit nervous today. I updated my SG-3100 on February 19th, 2022 to 22.01.
Previously it runs with 21.05.2 for months without any issue.
At March 14th I noticed that my internet connection was gone. SG-3100 answers to ping, but was not responsive to GUI or SSH. Only plugging of power restarts the box.Today, April 7th, the same issue occurs again. Ping was answered, but no Web GUI access or SSH was possible.
Because I need to back online fast (sitting in home office) I restarted the box again by power plug.But now I have some concerns that this is an periodic issue. Until yesterday there was nothing unusal with ressouces or any other value as far as I could see.
Next I will connect an older laptop to the serial console and run a terminal program to catch anything (if any).
Anyhow, this issue did not occur with 21.05.2, so anyone else who had such an issue?
After reboot a crash report contains only a single line without any valuable (my guess) information:
[07-Apr-2022 06:53:00 Europe/Berlin] PHP Fatal error: Unable to start pfSense module in Unknown on line 0I noticed the issue at 08:10, so it seems that the crash was about 1.5 hours ago.
Thanks for any idea how to pinpoint the issue.
Regards
-
Nothing else logged?
Yes, I would connect to the serial console and see what shows there. Try restarting PHP from the console menu.
Steve
-
S stephenw10 moved this topic from General pfSense Questions on
-
Yes, nothing else, this was all.
And now I know for sure that system crashes about 06:53am local time.
I am running periodic updates to a weather site and this stopped at 06:53 and continues after my reboot at 08:15am.
At this time no one accessed the pfSense, so I am wondering why php is crashed.Regards
-
With weeks in between incidents like that it could be something like RAM or disk exhaustion.
Check the graphs in Status > Monitoring. Do they show continuously increasing resource use anywhere?
-
I will keep an eye in the next time at this page.
Until yet I always checkech in the dashboard the usage, but this was always low:
and did not increase unusal.
Regards
-
The monitoring graphs log that over time so should show anything like that still if it happened.
-
Upps, just seen that it stores over time. Clearly seen, the gap from this morning when services stops:
But usage is constant every time!?
Regards
Edit: And here is the first drop in March:
-
So no big CPU usage before it stopped logging. You can check other resources by changing the graph settings. I would definitely check memory.
Steve
-
Yes, seems that memory is slowly, but steadily decreasing over the days!
But if I see it correct, there was more than 65% free this morning!?
From today:
And in total since running on 22.01:
Regards
-
Hmm, still mostly free though. That wouldn't stop it responding.
-
Is this the Unit you Update Wireguard and got into the wrong branch issue?
If so, go for a Backup and clean reinstall.
-
Hi, I have the exact same problem with a 2100. I did a clean install of 2.6 about a month ago, and this problem has happened twice since then. I had no issues with the device before.
The GUI becomes inaccessible, SSH unreachable, cron jobs stop running etc., but routing still works to some extent (I can access my VPN server behind pfSense remotely).
I haven't been able to find anything relevant in the logs, so any pointers as to where to look would be appreciated.
I use pfSense at an SMB so this is an absolute deal breaker for me unfortunately. -
There is no 2.6 image for the 2100, which is aarch64, I assume you mean 22.01?
Some services remaining up whilst others fail is typical of something like RAM exhaustion though so check the same things. It can also be caused by a failing drive which then prevents any errors being logged.
Steve
-
@stephenw10: Yes, I meant 22.01, sorry. I don't see anything unusual with RAM usage. Is there anything in S.M.A.R.T. data that would indicate that the M.2 SSD is failing? Also, is there an M.2 SATA SSD (2242) that you'd "officially" recommend?
-
Not other than the one that was already fitted if it has one.
I would expect some errors in the SMART data if it is failing.
I might expect to see other errors logged also.
-
@stephenw10: It didn't come with an M.2 SSD, I installed one. Do you know what errors/indicators I should be looking for, specifically?
-
Not anything specific. Any errors are bad!
Try running the short test and check the results.
Logging the console output is usually the best way to diagnose a drive failure of you can since the system will often dump error output there that cannot be written to the system log.
Steve
-
@stephenw10: I've run a short test, but it doesn't show any errors. I'll reinstall to eMMC. I really hope it's the SSD and not a software issue.
-
My device has no SSD, only eMMC. Hopefully its not the hardware.
Next days I will setup my standby device from scratch and replace this unit when issue occurrs next time.Regards
-
I would also consider re-installing 22.01 clean and restoring your config to rule out any issues during the upgrade.
Steve