pfSense Crashing

obsidiancomm

Greetings.

I am running pfSense on a Supermicro CSE-512L-200B that has an Xeon W3520 CPU, 8GB ram, with 22.01-RELEASE (I realize 22.05 is available but cannot upgrade until after the working day is over) at home. I currently just have 4x Intel 1GbE nics (Gitabit CT 82574L) in it as I'm awaiting delivery of a supermicro riser to be install to install a 10GbE NIC.

Current Setup:
em0 -> WAN
em1 -> LAN (multiple vlans)
em2 -> LAN (multiple vlans)
em3 -> LAN (multiple vlans)

None of the vlans or interfaces are currently using more than 80% capacity, if that. All of a sudden I'm starting to see crashes. I'm attaching the crash information.
textdump.tar (1).0 info (1).0

Just wondering if anyone sees anything that I should be looking at? If more information is needed, please let me know.

For reference, I use pfSense in my home network / home lab, but as I WFH, crashes cause me to be offline for work. I've thought about setting up pfSense in a HA way since I have an identical system that I'm using a a Vyos router in order to do some "trickery" with Spectrum, but this has nothing to do with the pfSense crashing. As far as pfSense is concerned, it just knows that traffic for certain IPs exit out interface em3.900 and when the primary WAN goes down, it uses that interface for backup.

Thanks in advance!

obsidiancomm

And just to expand.

Spectrum likes to use 192.168.1/24 for their IP range and depending on the device they provide, this cannot be changed. I already have devices numbered in this range and renumbering wasn't something I wanted to do.

We only use Spectrum for backup purposes, and to be able to watch Spectrum's cable service via their app (since we have no cable boxes from them). Spectrum requires traffic to come from your "home network" in order to view whatever TV package you have. So I use a Vyos box that is connected to Spectrum and the Vyos box also connects to my switch.

In pfSense, I have defined an alias that lists the IPs that Spectrum uses for streaming services. I then set firewall rules that matches any Destination for the SPECTRUM_STREAMING_IP alias and overrides the Gateway to SpectrumPrimary. This has worked fine without any issue since I set it up.

I'm at a lost as to what is happening. Funny enough, when the system crashed, the only traffic was inter-vlan streaming traffic where the traffic was not leaving my network.

stephenw10

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 4329 tid 101784 td 0xfffff8011f8f2000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0067edffe0
vpanic() at vpanic+0x197/frame 0xfffffe0067ee0030
panic() at panic+0x43/frame 0xfffffe0067ee0090
trap_fatal() at trap_fatal+0x391/frame 0xfffffe0067ee00f0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0067ee0140
trap() at trap+0x286/frame 0xfffffe0067ee0250
calltrap() at calltrap+0x8/frame 0xfffffe0067ee0250
--- trap 0xc, rip = 0xffffffff810a5a22, rsp = 0xfffffe0067ee0320, rbp = 0xfffffe0067ee0920 ---
pfioctl() at pfioctl+0x2982/frame 0xfffffe0067ee0920
devfs_ioctl() at devfs_ioctl+0xb0/frame 0xfffffe0067ee0970
VOP_IOCTL_APV() at VOP_IOCTL_APV+0x7b/frame 0xfffffe0067ee09a0
vn_ioctl() at vn_ioctl+0x16c/frame 0xfffffe0067ee0ab0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0067ee0ad0
kern_ioctl() at kern_ioctl+0x2b7/frame 0xfffffe0067ee0b30
sys_ioctl() at sys_ioctl+0x101/frame 0xfffffe0067ee0c00
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe0067ee0d30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0067ee0d30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800492d4a, rsp = 0x7fffffffe608, rbp = 0x7fffffffe680 ---

Nothing really definitive there or in the message buffer. This isn't good though:

<6>pid 22872 (ntopng), jid 0, uid 0: exited on signal 11 (core dumped)

Do you have any further crash reports to compare it with?

If this seemed to just start happening I would suspect a hardware issue in a system of that age. If it is the crash reports will likely be all different.

Steve

obsidiancomm

@stephenw10 Attached is the second crash:
textdump.tar (2).0 info.0

During this crash I was actually in front of the rack and watched the screen just started scrolling -- no way to pause or anything, then after a few minutes the system rebooted.

obsidiancomm

@stephenw10
Checking these new logs, I see again:

<6>pid 22872 (ntopng), jid 0, uid 0: exited on signal 11 (core dumped)

And a little further, the actual panic:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address	= 0x8000011
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff810a5a22
stack pointer	        = 0x28:0xfffffe0067ee0320
frame pointer	        = 0x28:0xfffffe0067ee0920
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 4329 (pfctl)
trap number		= 12
panic: page fault
cpuid = 1
time = 1674233447
KDB: enter: panic

What would be causing ntopng to coredump?

obsidiancomm

I disabled ntopng as I'm not currently using it and will only enable when I specifically need it. Not ideal, but if this stops the crashing I will deal with it.

stephenw10

That appears to be the same crash report file. Do you have a different one?