How to debug a nano pfsense that crashes every 65+ hours?



  • Hardware:

    • D2500CC
    • 4 gb of ram
    • nano install
    • 1 apc ups connected via ups

    Configuration:

    • 1 Lan (gigabit)
    • 1 Wan (pppoe, gigabit, 800mbps+ down, 600mbps+ up)
    • OpenVPN server
    • apcupsd installed but disabled
    • nut installed and working
    • ssh enabled

    General usage

    • 2-10% state table
    • 10-25% MBUF Usage
    • Load averages around 0.15-0.25
    • cpu usage 5-60%
    • memory usage 25%

    Simptoms: every few days, around 65-72 hours after last restart:

    • can ping lan ip of pfsense
    • can resolve outside dns
    • can't access internet resources
    • can't connect to ssh
    • can't connect to web interface
    • attached usb keyboard does not do anything (all 3 status leds are full on and hitting caps lock / num lock / etc does not do anything)
    • no output to screen

    How can i enable full loging to debug this issue? Can i set something to write full logs to the usb drive? Or maybe to a second usb drive (i can format and mount a second drive easily).



  • Bad fan? heat?  Failing board or memory? Failing psu?



  • Tested with an IR thermometer - 35 degrees C.  I will try to swap out ram this evening (ordered two new 2gb pieces). 
    I was wondering if there is any way to see from logs what is happening?



  • Are you showing core dumps?



  • As far as i can tell all logs are stored in /var (wich is a ramdisk), so when it freezes i loose them all. Also ordered a serial cable to try to see if i can get into pfsense when it locks up.



  • Are you running out of ram (ramdisk)?  Resources exhausted?

    This seems abit high, but not exhausted:

    10-25% MBUF Usage



  • Those are values during peak hours during the day. 5-10 desktop / mobile / tablet user plus a few servers doing low traffic.
    Everything is ipv4 only (disable ipv6).
    I switched the ram and we'll see in 3 days if it crashes again.

    I was wondering if there is a way to send logs to another system or if there is a script/option to copy logs automatically with scp/sftp/ftp..



  • There is actually a way to offload all the logging.  If seen people describe it but I've never done it.

    Wait a bit and I'm sure someone will tell you how.


  • Netgate Administrator

    You can enable an external syslog server for long term logging:
    https://doc.pfsense.org/index.php/Copying_Logs_to_a_Remote_Host_with_Syslog
    Of course if the box looses it's network it can't send logs….

    Steve



  • Thank you everyone for your help. As usual, I learned many things from the forum.

    It seems the motherboard was broken. It crashed with 4 different ram sticks. I got a replacement today and hopefully I will have no more issues.

    Now I just have build a low-powered system for syslog and general loging…



  • I should have just stuck with my first guess…  haha