PfSense Crashing after no changes and years of stability
-
Hey Folks,
I have a dedicate little computer thats been running PfSense for a few years problem free. I login into it periodically and update and make a few tweaks and it has been stable. I have not touched it in about a month at this point. In the last couple days I have notice my internet connection dropping. Troubleshooting it appears to be PfSense is hanging. I am not able to ping it and rebooting it solves the issue. When I login to it I get the following in a yellow bar "pfSense has detected a crash report or programming bug. Click here for more information."Here are the first few lines of the report. At a glance it appears that this is old though. I have made a copy of it and I will clear it now. The crashing seems to be happening ever day or two... So I will monitor it. In the mean time. Any suggestions on troubleshooting this issues? I am a newbie to PfSense and only learned enough to get it running and doing what I need. I should also mention that this is a cheap little Celeron NUC type of box. So it could be a hardware issue maybe (just a guess though).
Crash report begins. Anonymous machine information: amd64 11.2-RELEASE-p10 FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/info.0 Dump header from device: /dev/label/swap0 Architecture: amd64 Architecture Version: 1 Dump Length: 129024 Blocksize: 512 Dumptime: Wed Mar 4 15:26:41 2020 Hostname: pfSense.localdomain Magic: FreeBSD Text Dump Version String: FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfS Panic String: Dump Parity: 763370498 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013630007401 7064 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command; use "help" to list available commands db:1:lockinfo> show alllocks No such command; use "help" to list available commands db:1:lockinfo> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe00f0243380 curthread = 0xfffff800049b7620: pid 12 "irq263: re1" curpcb = 0xfffffe008d7e4cc0 fpcurthread = none idlethread = 0xfffff800044de620: tid 100004 "idle: cpu1" curpmap = 0xffffffff82b85998 tssp = 0xffffffff82bb6878 commontssp = 0xffffffff82bb6878 rsp0 = 0xfffffe008d7e4cc0 gs32p = 0xffffffff82bbd0d0 ldt = 0xffffffff82bbd110 tss = 0xffffffff82bbd100 db:0:kdb.enter.default> bt Tracing pid 12 tid 100045 td 0xfffff800049b7620 m_copym() at m_copym+0x30/frame 0xfffffe008d7e44b0 ip_fragment() at ip_fragment+0x2d5/frame 0xfffffe008d7e4550 pf_test() at pf_test+0x23d6/frame 0xfffffe008d7e4790 pf_check_out() at pf_check_out+0x1d/frame 0xfffffe008d7e47b0 pfil_run_hooks() at pfil_run_hooks+0x90/frame 0xfffffe008d7e4840 ip_tryforward() at ip_tryforward+0x230/frame 0xfffffe008d7e48b0 ip_input() at ip_input+0x3bf/frame 0xfffffe008d7e4910 netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4960 ether_demux() at ether_demux+0x173/frame 0xfffffe008d7e4990 ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe008d7e49f0 netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe008d7e4a40 ether_input() at ether_input+0x26/frame 0xfffffe008d7e4a60 re_rxeof() at re_rxeof+0x601/frame 0xfffffe008d7e4ad0 re_intr_msi() at re_intr_msi+0xfc/frame 0xfffffe008d7e4b20 intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe008d7e4b60 ithread_loop() at ithread_loop+0xe7/frame 0xfffffe008d7e4bb0 fork_exit() at fork_exit+0x83/frame 0xfffffe008d7e4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008d7e4bf0
-
@bedpan Please post Status > System logs >General.
-
I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt
I greatly appreciate you having a look!
Thanks!
-
@bedpan said in PfSense Crashing after no changes and years of stability:
I don't see an easy way to export. I will keep poking at it but in the mean time here is the last 2000 lines from Status > System logs >General - pfsense general log.txt
I greatly appreciate you having a look!
Thanks!
From the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.
You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.
-
@bmeeks said in PfSense Crashing after no changes and years of stability:
rom the entries in that log file, you apparently have one or more Realtek NICs. That driver has problems on FreeBSD. You probably need to try the newest Realtek binary driver discussed here: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/.
You can try installing that driver to see if it helps. In the log snippet you posted I see a large number of watchdog timer timeouts on your Realtek NIC. Those will cause loss of connectivity and could lead to the crashes as well.Thanks I will have a look at whats involved in making that happen. Odd that I have not updated in awhile and never had issue until the last few weeks.
Lets see what I can figure out!
-
@bedpan: I agree it's weird, but I think the same has happened to a few other folks in the last year according to that thread. Could be a traffic related thing, too, I suppose.
-
@bmeeks
Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-) -
@bedpan said in PfSense Crashing after no changes and years of stability:
@bmeeks
Not arguing, just strange. But your right it does happen.. Again I am not a PfSense or Linux guy so its some hurdles to figure out. Just started googling ;-)I'm not a FreeBSD kernel expert either. User @stephenw10 is, though. Perhaps he will drop by this thread since I mentioned his name and chime in with some assistance or advice.
-
Well I found this thread: https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release/15
seems to match what I have going on. Thanks for pointing me in the right direction. I have applied the driver and confirmed it is loaded.. Lets see what happens!Thanks
...
On review I notice its the same thread you linked to... I like to do things the hard way and found it on my own to ;-)