PfSense Crashing after no changes and years of stability

bedpan

Hey Folks,
I have a dedicate little computer thats been running PfSense for a few years problem free. I login into it periodically and update and make a few tweaks and it has been stable. I have not touched it in about a month at this point. In the last couple days I have notice my internet connection dropping. Troubleshooting it appears to be PfSense is hanging. I am not able to ping it and rebooting it solves the issue. When I login to it I get the following in a yellow bar "pfSense has detected a crash report or programming bug. Click here for more information."

Here are the first few lines of the report. At a glance it appears that this is old though. I have made a copy of it and I will clear it now. The crashing seems to be happening ever day or two... So I will monitor it. In the mean time. Any suggestions on troubleshooting this issues? I am a newbie to PfSense and only learned enough to get it running and doing what I need. I should also mention that this is a cheap little Celeron NUC type of box. So it could be a hardware issue maybe (just a guess though).

Crash report begins.  Anonymous machine information:

amd64
11.2-RELEASE-p10
FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019     root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/label/swap0
  Architecture: amd64
  Architecture Version: 1
  Dump Length: 129024
  Blocksize: 512
  Dumptime: Wed Mar  4 15:26:41 2020
  Hostname: pfSense.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019
    root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfS
  Panic String: 
  Dump Parity: 763370498
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000013630007401  7064 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe00f0243380
curthread    = 0xfffff800049b7620: pid 12 "irq263: re1"
curpcb       = 0xfffffe008d7e4cc0
fpcurthread  = none
idlethread   = 0xfffff800044de620: tid 100004 "idle: cpu1"
curpmap      = 0xffffffff82b85998
tssp         = 0xffffffff82bb6878
commontssp   = 0xffffffff82bb6878
rsp0         = 0xfffffe008d7e4cc0
gs32p        = 0xffffffff82bbd0d0
ldt          = 0xffffffff82bbd110
tss          = 0xffffffff82bbd100
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100045 td 0xfffff800049b7620
m_copym() at m_copym+0x30/frame 0xfffffe008d7e44b0
ip_fragment() at ip_fragment+0x2d5/frame 0xfffffe008d7e4550
pf_test() at pf_test+0x23d6/frame 0xfffffe008d7e4790
pf_check_out() at pf_check_out+0x1d/frame 0xfffffe008d7e47b0
pfil_run_hooks() at pfil_run_hooks+0x90/frame 0xfffffe008d7e4840
ip_tryforward() at ip_tryforward+0x230/frame 0xfffffe008d7e48b0
ip_input() at ip_input+0x3bf/frame 0xfffffe008d7e4910
netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4960
ether_demux() at ether_demux+0x173/frame 0xfffffe008d7e4990
ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe008d7e49f0
netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4a40
ether_input() at ether_input+0x26/frame 0xfffffe008d7e4a60
re_rxeof() at re_rxeof+0x601/frame 0xfffffe008d7e4ad0
re_intr_msi() at re_intr_msi+0xfc/frame 0xfffffe008d7e4b20
intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe008d7e4b60
ithread_loop() at ithread_loop+0xe7/frame 0xfffffe008d7e4bb0
fork_exit() at fork_exit+0x83/frame 0xfffffe008d7e4bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008d7e4bf0

NollipfSense

@bedpan Please post Status > System logs >General.

bedpan

I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt

I greatly appreciate you having a look!

Thanks!

bmeeks

@bedpan said in PfSense Crashing after no changes and years of stability:

I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt

I greatly appreciate you having a look!

Thanks!

From the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.

You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.

bedpan

@bmeeks said in PfSense Crashing after no changes and years of stability:

rom the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.
You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.

Thanks I will have a look at whats involved in making that happen. Odd that I have not updated in awhile and never had issue until the last few weeks.

Lets see what I can figure out!

bmeeks

@bedpan: I agree it's weird, but I think the same has happened to a few other folks in the last year according to that thread. Could be a traffic related thing, too, I suppose.

bedpan

@bmeeks
Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-)

bmeeks

@bedpan said in PfSense Crashing after no changes and years of stability:

@bmeeks
Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-)

I'm not a FreeBSD kernel expert either. User @stephenw10 is, though. Perhaps he will drop by this thread since I mentioned his name and chime in with some assistance or advice.

bedpan

Well I found this thread: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/15
seems to match what I have going on. Thanks for pointing me in the right direction. I have applied the driver and confirmed it is loaded.. Lets see what happens!

Thanks

...

On review I notice its the same thread you linked to... I like to do things the hard way and found it on my own to ;-)