Regular pfSense crash

rudolfb

pfSense 2.7.2-RELEASE (amd64)
Zotac ZBOX CI337 nano
Crucial CT16G48C40S5 RAM
Transcend 400S 256 GB M.2 2242 SSD

Intermitantly pfSense crashes. I turn of the machine, turn it back on again, and after about 30 seconds everything is fine again. Sometimes the crashes occur often, i.e. a few times a day, sometimes things go well for a week or more without any issue.

The problem happens on two identical sets of hardware. I have two Zotac's running with the identical hardware, identical versions of pfSense, with the identical settings, and the issue happens intermitantly with both Zotac firewalls.

Grateful for anyone that can give me an indication what the root cause might be, and where I can start to look.

info.0
info.0

textdump.tar.0
textdump.tar.0

stephenw10

Panic:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address	= 0x18
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d7d70d
stack pointer	        = 0x0:0xfffffe00c574cc70
frame pointer	        = 0x0:0xfffffe00c574cce0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 2)
rdi: fffff8001097eb80 rsi: fffff8045478682e rdx: 0000acac1c110260
rcx: 0000000000000008  r8: 0000acac1c110260  r9: 0000000000000008
rax: fffff8001097eb60 rbx: fffff8001097eb88 rbp: fffffe00c574cce0

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address	= 0x18
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d7ef85
stack pointer	        = 0x28:0xfffffe00c5751c50
frame pointer	        = 0x28:0xfffffe00c5751c90
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 1)
r10: fffffe00c574cc9c r11: 0000000000000004 r12: 0000000000000028
rdi: fffff804549aa034 rsi: 0000000000000000 rdx: 0000000000000001
rcx: 000000000000667e  r8: 000000011a094b73  r9: fffffe009d600ce0
rax: 0000000000000000 rbx: fffff803454cda00 rbp: fffffe00c5751c90
r10: fffff80002195000 r11: 000000002a7b70f9 r12: 0000000000000000
r13: 0000000000000028 r14: 0000000000000028 r15: 0000000000000014
trap number		= 12
panic: page fault
cpuid = 2
time = 1721928035
KDB: enter: panic

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100042 td 0xfffffe002053ee40
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c5751930
vpanic() at vpanic+0x163/frame 0xfffffe00c5751a60
panic() at panic+0x43/frame 0xfffffe00c5751ac0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00c5751b20
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c5751b80
calltrap() at calltrap+0x8/frame 0xfffffe00c5751b80
--- trap 0xc, rip = 0xffffffff80d7ef85, rsp = 0xfffffe00c5751c50, rbp = 0xfffffe00c5751c90 ---
m_apply() at m_apply+0x95/frame 0xfffffe00c5751c90
in_cksum_skip() at in_cksum_skip+0x2f/frame 0xfffffe00c5751cb0
udp_input() at udp_input+0x2f0/frame 0xfffffe00c5751d90
ip_input() at ip_input+0x26b/frame 0xfffffe00c5751df0
swi_net() at swi_net+0x128/frame 0xfffffe00c5751e60
ithread_loop() at ithread_loop+0x257/frame 0xfffffe00c5751ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c5751f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5751f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

That doesn't look familiar but what I also see is:

<6>re1: watchdog timeout
<6>re1: link state changed to DOWN
<6>re1: link state changed to UP
<6>re1: watchdog timeout
<6>re1: link state changed to DOWN
<6>re1: link state changed to UP
<6>re1: watchdog timeout
<6>re1: link state changed to DOWN
<6>re1: link state changed to UP
<6>re1: watchdog timeout
<6>re1: link state changed to DOWN
<6>re1: link state changed to UP

Any time you see that you should try the alternative Realtek driver pkg.

Steve

rudolfb

@stephenw10
Hi Steve,

Thanks. Not sure how to update the driver at the moment, but at least I now have a starting point, and can do some research in order to learn how load a new network driver. Very much appreciated!

Rudolf

stephenw10

At the command line (the real command line!) run the following commands:

pkg-static install realtek-re-kmod

Then:

echo 'if_re_load="YES"' >> /boot/loader.conf.local
echo 'if_re_name="/boot/modules/if_re.ko"' >> /boot/loader.conf.local

Then reboot and check the boot logs show the 1.98 driver:

re0: <Realtek PCIe GbE Family Controller> port 0x1000-0x10ff mem 0xf7a00000-0xf7a00fff,0xf7900000-0xf7903fff irq 16 at device 0.0 on pci1
re0: Using Memory Mapping!
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: version:1.98.00_3
re0: Ethernet address: 00:0d:b9:38:30:10

rudolfb

@stephenw10
Hi Steve,

Ok, I got around to this this morning. Your steps were all that was needed to install the alternative Realtek driver.

Was able to

install the new driver,
activate it,
and was able to see from the log that the new had been loaded successfully.

Both firewalls are now updated. I will need to wait a while to see if this resolves my crash issue. Hopefully this will.

Learnt a bit about freebsd in the process.

Again, thank you for your generous help.

Regards

Rudolf