PfSense Crashing after no changes and years of stability



  • Hey Folks,
    I have a dedicate little computer thats been running PfSense for a few years problem free. I login into it periodically and update and make a few tweaks and it has been stable. I have not touched it in about a month at this point. In the last couple days I have notice my internet connection dropping. Troubleshooting it appears to be PfSense is hanging. I am not able to ping it and rebooting it solves the issue. When I login to it I get the following in a yellow bar "pfSense has detected a crash report or programming bug. Click here for more information."

    Here are the first few lines of the report. At a glance it appears that this is old though. I have made a copy of it and I will clear it now. The crashing seems to be happening ever day or two... So I will monitor it. In the mean time. Any suggestions on troubleshooting this issues? I am a newbie to PfSense and only learned enough to get it running and doing what I need. I should also mention that this is a cheap little Celeron NUC type of box. So it could be a hardware issue maybe (just a guess though).

    Crash report begins.  Anonymous machine information:
    
    amd64
    11.2-RELEASE-p10
    FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019     root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    No PHP errors found.
    
    Filename: /var/crash/info.0
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 129024
      Blocksize: 512
      Dumptime: Wed Mar  4 15:26:41 2020
      Hostname: pfSense.localdomain
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019
        root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfS
      Panic String: 
      Dump Parity: 763370498
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000013630007401  7064 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command; use "help" to list available commands
    db:1:lockinfo>  show alllocks
    No such command; use "help" to list available commands
    db:1:lockinfo>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe00f0243380
    curthread    = 0xfffff800049b7620: pid 12 "irq263: re1"
    curpcb       = 0xfffffe008d7e4cc0
    fpcurthread  = none
    idlethread   = 0xfffff800044de620: tid 100004 "idle: cpu1"
    curpmap      = 0xffffffff82b85998
    tssp         = 0xffffffff82bb6878
    commontssp   = 0xffffffff82bb6878
    rsp0         = 0xfffffe008d7e4cc0
    gs32p        = 0xffffffff82bbd0d0
    ldt          = 0xffffffff82bbd110
    tss          = 0xffffffff82bbd100
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100045 td 0xfffff800049b7620
    m_copym() at m_copym+0x30/frame 0xfffffe008d7e44b0
    ip_fragment() at ip_fragment+0x2d5/frame 0xfffffe008d7e4550
    pf_test() at pf_test+0x23d6/frame 0xfffffe008d7e4790
    pf_check_out() at pf_check_out+0x1d/frame 0xfffffe008d7e47b0
    pfil_run_hooks() at pfil_run_hooks+0x90/frame 0xfffffe008d7e4840
    ip_tryforward() at ip_tryforward+0x230/frame 0xfffffe008d7e48b0
    ip_input() at ip_input+0x3bf/frame 0xfffffe008d7e4910
    netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4960
    ether_demux() at ether_demux+0x173/frame 0xfffffe008d7e4990
    ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe008d7e49f0
    netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4a40
    ether_input() at ether_input+0x26/frame 0xfffffe008d7e4a60
    re_rxeof() at re_rxeof+0x601/frame 0xfffffe008d7e4ad0
    re_intr_msi() at re_intr_msi+0xfc/frame 0xfffffe008d7e4b20
    intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe008d7e4b60
    ithread_loop() at ithread_loop+0xe7/frame 0xfffffe008d7e4bb0
    fork_exit() at fork_exit+0x83/frame 0xfffffe008d7e4bf0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008d7e4bf0
    


  • @bedpan Please post Status > System logs >General.



  • I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt

    I greatly appreciate you having a look!

    Thanks!



  • @bedpan said in PfSense Crashing after no changes and years of stability:

    I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt

    I greatly appreciate you having a look!

    Thanks!

    From the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.

    You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.



  • @bmeeks said in PfSense Crashing after no changes and years of stability:

    rom the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.
    You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.

    Thanks I will have a look at whats involved in making that happen. Odd that I have not updated in awhile and never had issue until the last few weeks.

    Lets see what I can figure out!



  • @bedpan: I agree it's weird, but I think the same has happened to a few other folks in the last year according to that thread. Could be a traffic related thing, too, I suppose.



  • @bmeeks
    Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-)



  • @bedpan said in PfSense Crashing after no changes and years of stability:

    @bmeeks
    Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-)

    I'm not a FreeBSD kernel expert either. User @stephenw10 is, though. Perhaps he will drop by this thread since I mentioned his name and chime in with some assistance or advice.



  • Well I found this thread: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/15
    seems to match what I have going on. Thanks for pointing me in the right direction. I have applied the driver and confirmed it is loaded.. Lets see what happens!

    Thanks

    ...

    On review I notice its the same thread you linked to... I like to do things the hard way and found it on my own to ;-)


Log in to reply