How to debug a nano pfsense that crashes every 65+ hours?
-
Hardware:
- D2500CC
- 4 gb of ram
- nano install
- 1 apc ups connected via ups
Configuration:
- 1 Lan (gigabit)
- 1 Wan (pppoe, gigabit, 800mbps+ down, 600mbps+ up)
- OpenVPN server
- apcupsd installed but disabled
- nut installed and working
- ssh enabled
General usage
- 2-10% state table
- 10-25% MBUF Usage
- Load averages around 0.15-0.25
- cpu usage 5-60%
- memory usage 25%
Simptoms: every few days, around 65-72 hours after last restart:
- can ping lan ip of pfsense
- can resolve outside dns
- can't access internet resources
- can't connect to ssh
- can't connect to web interface
- attached usb keyboard does not do anything (all 3 status leds are full on and hitting caps lock / num lock / etc does not do anything)
- no output to screen
How can i enable full loging to debug this issue? Can i set something to write full logs to the usb drive? Or maybe to a second usb drive (i can format and mount a second drive easily).
-
Bad fan? heat? Failing board or memory? Failing psu?
-
Tested with an IR thermometer - 35 degrees C. I will try to swap out ram this evening (ordered two new 2gb pieces).
I was wondering if there is any way to see from logs what is happening? -
Are you showing core dumps?
-
As far as i can tell all logs are stored in /var (wich is a ramdisk), so when it freezes i loose them all. Also ordered a serial cable to try to see if i can get into pfsense when it locks up.
-
Are you running out of ram (ramdisk)? Resources exhausted?
This seems abit high, but not exhausted:
10-25% MBUF Usage
-
Those are values during peak hours during the day. 5-10 desktop / mobile / tablet user plus a few servers doing low traffic.
Everything is ipv4 only (disable ipv6).
I switched the ram and we'll see in 3 days if it crashes again.I was wondering if there is a way to send logs to another system or if there is a script/option to copy logs automatically with scp/sftp/ftp..
-
There is actually a way to offload all the logging. If seen people describe it but I've never done it.
Wait a bit and I'm sure someone will tell you how.
-
You can enable an external syslog server for long term logging:
https://doc.pfsense.org/index.php/Copying_Logs_to_a_Remote_Host_with_Syslog
Of course if the box looses it's network it can't send logs….Steve
-
Thank you everyone for your help. As usual, I learned many things from the forum.
It seems the motherboard was broken. It crashed with 4 different ram sticks. I got a replacement today and hopefully I will have no more issues.
Now I just have build a low-powered system for syslog and general loging…
-
I should have just stuck with my first guess… haha
