pfsense is shutting down or hang randomly
-
Hi community
First of all, I didn't found any way to post this in "Support" for english. Choices are just for any other language. So I'm sorry if I use the wrong place for this post.
Since 3 Weeks pfsense is shutting down or crashing at free will / randomly. Uptime can be from days to Minutes before it's happening. It can last from days to Minutes until it does it again.
I searched deep down in the web, but did not find any solution for this problem.
Here I provide some information of my configuration and hardware:
This is the hardware I use for pfsense:
https://www.amazon.de/-/en/gp/product/B08BXF1TWP/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&th=1
This is what I find in pfsense Dashboard
-
BIOS Vendor: American Megatrends Inc.
-
Version: 5.6.5
-
Release Date: Tue Oct 22 2019
-
2.6.0-RELEASE (amd64)
-
built on Mon Jan 31 19:57:53 UTC 2022
-
FreeBSD 12.3-STABLE
-
CPU Type Intel(R) Celeron(R) CPU J1900 @ 1.99GHz
-
4 CPUs: 1 package(s) x 4 core(s)
-
AES-NI CPU Crypto: No
-
QAT Crypto: No
-
State table size 0% (371/756000) Show states
-
MBUF Usage 1% (9626/1000000)
-
Temperature 38.0°C
-
Load average 0.25, 0.24, 0.18
-
CPU usage 3%
-
Memory usage 8% of 7570 MiB
-
SWAP usage 0% of 4095 MiB
I hope anyone can help me with this.
I'm not very firm with the CLI / FBSD.
I am able to SSH in and execute commands. Alltough, some of the are "not allowed", even when I signed in with the most powerful admin user.How can I check the hardware logs when SSH in pfsense?
What would I need to look for?
Do you have any hint what the source of the problem could be?Thanks for any help and have a wonderful Christmas time.
-
-
So when it has 'crashed' you're still able to SSH into it? Not console?
You can view the system logs at /var/log/system.log
So:
cat /var/log/system.logto view the whole log. -
Hi @stephenw10
Thanks for your feedback.
No, SSH was not working then. I tried to troubleshoot the system after a reboot. I will check the log you mentioned asap and come back.
Cheers
-
Hi @stephenw10
I was able to get the output of the system log scp'ing to my Linux machine:
Notice: I obfuscated username and IP's for security reasons, but left anything else like it was written to the file.
After screening it again, I only found one potential hint:
Dec23 / Dec23, after pfblocker cron started, pfsense was down (I remember starting it in the morning on the 24th.), as it seems in the log. Because nothing else was logged after that, until I restarted, if my thinking is correct:
Dec 23 23:00:00 mydomain php[64293]: [pfBlockerNG] Starting cron process.
Dec 24 10:50:57 mydomain syslogd: kernel boot file is /boot/kernel/kernelNo log entries in between...
That's a very good learning experience for me, so I'l appreciate your help very much.
(...in between I'l run a) a filesystem test on reboot and b) pumping some GB through the LAN Ports to stress test.)
-
Can you still access it at the console when it stops responding?
When it does fail are you manually rebooting it?
Nothing in those logs looks particularly suspect.
-
Nope, I can't.
When it fails, it's either still On, or completely shutdown.
In both cases I have to reboot it manuallyIf it is still on, the WAN port flashes every one to two seconds.
That's the only additional observation I made.The filesystem test was OK.
And there was no problem pumping Gigabytes through the ports.
I did it with iperf3 for over an hour and 100 Connections simultaneous. Worked perfectly fine.Do you know of any other log, Hardware or Software related, where I could get more granular details?
Thanks Stephen.
-
If it shutsdown completely without logging anything that's probably a hardware issue.
If it's still powered on is anything shown on the console?
Do you ever see a crash report on the dashboard after you reboot it?
-
I have attached a monitor to it, so I can see what's going on.
When it's still powered on but frozen, the console is black and the pfsense shell menu disappeared. But that's because the Monitor doesn't get any signal for display, and therefore the monitor goes black...but the pfesense "hardware" it's still powered on, because I can see the light of the Power-Button and the Ports too. And in this state, I can't ssh into it anymore.
I never saw a crash report after/during the reboot process.
What I did in the meantime was a backup of the configuration, reset it to "manufacturing" status and reloaded the configuration.
Since then (yesterday evening) it did not crash or shutdown anymore. I will do a iperf3 test today to see if it's stable.
I'l let you know how it plays out Stephen.
Thanks for your time, much appreciated.
-
That sounds like a hardware error. You should still output on the local VGA console if the device is running in any way. You would normally still see output even if the OS crashed hard.
-
Jup, I changed the VIDEO Cable now to VGA instead of HDMI.
Who knows, maybe I see something on VGA? §8-) -
Maybe. The VGA port is often the default/primary output.
-
And here's the end of the odyssey §8-)
I found out, that it's definitely a hardware problem.
I tried to install Pop!OS, which I run as my daily driver.
During the setup process it crashed 5 times in a row.
And just to be sure, I tried Ubuntu & FBSD too.And that's the end of the crappy buy.
Now, I grab a coffee, jump on my couch and see if I'l buy a Protectli or a Netgate appliance.
What are you using yourself, or would you recommend to use?
My internet speed is fiber (1GB). -
@chicknbread said in pfsense is shutting down or hang randomly:
Netgate appliance.
Right now we have the 4100, 6100 and 8200 shipping with a free 6-month subscription of TAC Professional (email based support for software configurations)
With 1Gbps service a 4100 should be a perfect fit but your other needs might push you to the 6100. What else are you hoping/planning to do with your firewall?
-
Hi Ryan
Thanks for your suggestions.
I only need it for my home office where max. 5 clients are connected. The Fiber connection I have here is a little overkill for what I need. So, maybe even the 2100 would be a good fit for this little infra here.
Unfortunately I live in Switzerland and the shipping costs versus the product costs are quite a bit...
However, I want to give a huge thanks to @stephenw10, who was helping me here. Thanks very much, and I for sure take this into my thoughts when I buy a new product.
Cheerio you both & have a wonderful happy new year.
-
@chicknbread said in pfsense is shutting down or hang randomly:
Unfortunately I live in Switzerland and the shipping costs versus the product costs are quite a bit...
We might have a partner by you with the hardware, but the TAC offer is only for purchases from our store.
-
Ryan, I ordered directly from netgate.
Thanks all.
