PFsense hanging since version 2.4.4
-
Not sure that dmesg is going to show us much, might be better to get us a copy of the system logs if at all possible.
Looking over what you have posted us a few things I would do.
1.) The message
WARNING: / was not properly dismountedmakes me think you should look into booting into single user mode and running FSCK a few times.2.) There was an issue in unbound, and memory leaks, you should upgrade to 2.4.4-p1
-
Thanks for your reply :)
Yes, i have been running a FSCK a few times and repaired the system. But it did crash again after a few days anyway. That message is because of the crashes yesterday i suppose :)
Is unbound the DNS resolver? I'm running that.. But I'm also running version 2.4.4-p1 though.. Maybe i should try upgrading to something else?
What kind of system logs would be useful?
-
It's tricky to time it right, if you have an external logging server this is where it get's helpful.
Basically you want to get a copy of the system log right after a reboot, sometimes there is info logged in /var/crash that can be helpful.
If there is nothing logged, chances are high it's hardware related, and it's starting to fail out (RAM, HDD, etc).
-
I wish I could point you to something more specific, but I can tell you that I've been running PFsense for 10 years since v1.2.3 and every problem I've had in that time frame has been linked to hardware issues.
I had similar symptoms a few weeks back where it would appear to crash every 3-7 days. In my situation, the symptoms started with no internet, next I found I couldn't ping PFsense, next while investigating the box I would find the PS fan whining loudly on full blast. My box is headless so I couldn't see the console, but a reboot seemed to set everything right for another 3-7 days.
After each crash, there was a crash log waiting for me in the GUI, which I'm not familiar with parsing, so I started investigating my hardware before I posted my crash logs in the forums. Upon opening the case, I found my CPU and front intake fans caked with dust and a loose SATA cable on my HDD. So, after blowing out all the dust in the case, re-seating the SATA cable on my HDD, re-seating the RAM, and replacing the PS... the frequent crashing went away and PFsense has now been stable for the last 3+ weeks.
Before my recent hardware issues, my box would stay up for 6-12 months at a time.... only going down due to forced reboots on firmware updates. Otherwise, there's no telling how long my uptimes would be.
No software is perfect, but my money is on something in your hardware. I would do a deep dive into your hardware (e.g. blow out dust, re-seat connections, re-seat the RAM, try new RAM, etc, etc). After that, post the crash log here.
-
@chrismacmahon said in PFsense hanging since version 2.4.4:
It's tricky to time it right, if you have an external logging server this is where it get's helpful.
Basically you want to get a copy of the system log right after a reboot, sometimes there is info logged in /var/crash that can be helpful.
When you say system log is that the system.log in /var/log? In that case it does not contain anything during the crash or before when it happens... /var/crash is empty so i guess it points to a hardware issue..
If there is nothing logged, chances are high it's hardware related, and it's starting to fail out (RAM, HDD, etc).
I have taken the mini pc apart. No dust in it since its fan-less system. I removed the M2 drive and RAM and put them back again just in case.
I also removed the Wifi chip in it since i do not use it anymore and one less problem cause to worry about :)
-
our book has all the information needed on logs: https://www.netgate.com/docs/pfsense/monitoring/system-logs.html
If you want to view from the CLI: https://www.netgate.com/docs/pfsense/monitoring/working-with-binary-circular-logs-clog.html
-
@marvosa said in PFsense hanging since version 2.4.4:
No software is perfect, but my money is on something in your hardware. I would do a deep dive into your hardware (e.g. blow out dust, re-seat connections, re-seat the RAM, try new RAM, etc, etc). After that, post the crash log here.
Yeah.. I had a nightmare with another system here a few months ago.. Windows system. It did bluescreen now and then.. After about 1-2 months searching for the fault, checking drivers, viruses, replaced hd, reinstalled windows on another hd, replaced the RAM sticks, changed GFX card and finally i replaced the PSU and that solved the problem! .. I have never in my 25-30 year PC career have had a faulty PSU that created blue screens... i even measured the old PSU with a Fluke measuring tool and the voltages seemed fine and no fluctuation what i could find... But bluescreens happened sporadically .. 1-3 days apart...
Anyway... thats off topic.. haha..
I opened the mini-pc.. no dust (fan less) removed RAM and Disk and put back again... Removed WIFI-mini chip. I will let it run for a few days now and see... If it crashes again i will move the machine to a stable 12V power supply to rule out the PSU (thats the easiest thing for me to test right now) .. After that i have to decide if i should buy new RAM or HDD first... What do you think? :)
-
How old is the system?
-
@chrismacmahon said in PFsense hanging since version 2.4.4:
How old is the system?
It's from 1st of April 2017 so not very old..
It is ordered from Aliexpress.com though...........
-
Computer hanged again.. After 1day and 8 hours. I will update to 2.4.4 p2 and if it hangs again i will connect it to another PSU to test one thing each time.. Will let you know in here to help out others maybe :)
-
It sounds like it's hardware.
I'm not in your shoes, but chasing hardware faults is difficult at best sometimes.
When I had this happen 10 years ago, in my home, my wife insisted i fix it.... I ended up buying new hardware as that was the fastest path to resolution. Good luck!
-
It hung again. So now im running on another PSU. Also running a memtest now with Memtest86+... Will keep you updated... :) When it hung the console screen were just frozen and no special messages.
If it freezes again next step is to run it from USB stick i guess...
If it frezes then i will test older releases of pfsense. -
This is a hardware fault.
If you have another device to swap out I would do so, or if you can run a VM to test I would.
-
@chrismacmahon said in PFsense hanging since version 2.4.4:
This is a hardware fault.
If you have another device to swap out I would do so, or if you can run a VM to test I would.
A VM to test what exactly? A VM of my current installation?
The router hanged again an hour ago. Im now making a USB stick to try to run my config from it if possible... Live.
-
If you have the hardware to spin up a virtual machine, you can import your working config into the VM and run off of that.
-
@chrismacmahon said in PFsense hanging since version 2.4.4:
If you have the hardware to spin up a virtual machine, you can import your working config into the VM and run off of that.
ok.. i dont have a ESXi machine at home.. but at work.. i will try to reinstall 2.4.4.. on the computer first.. or run 2.3 if that failes.
-
i tried to create a USB stick with 2.4.4 p1 but after installing i get into that crappy serial console bug.. i pressed ESC to type "set kern.vty=sc" but then booting into multi user mode i seem to get a crash.. text scrolls by too fast for me to read.. so after ESC i want to boot into single user mode to be able to see the boot errors.. but how do i boot from CLI (ESC) into single user mode.. i have tried to google for it but cannot find anything... i tried like boot single and such without success.. please help!
I really wanted to try 2.4.4 before i go back and try 2.3 or something that might work better...
-
Not sure, I would re-burn the image, try again...if it happens again the hardware issue is the problem.
We are a fan of Etcher.io
-
Found it... "boot -s" it is :) .. Will try it and review the logs.
-
Had to run fsck -y a few times due to unclean system because i did a power off because of the display issue in 2.4.4.. i did not realize the system would start up anyway..
So now the system is up .. 2.4.4 p1 with basic settings.. Nothing changed except password.. will let it run like that to see what happens.. if it hangs again i will try to go back to an old version where i know it was working... to rule out software too...
Is there anyone that has an older image available? 2.4 or 2.4.1 or 2.4.2? that i could have to test?
