Crash Report

jimp

<6>pid 80017 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 86369 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 14742 (netstat), jid 0, uid 0: exited on signal 11
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 43961 (php-cgi), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 84237 (grep), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 43089 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 38635 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 55275 (awk), jid 0, uid 0: exited on signal 10 (core dumped)
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 85766 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 8376 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped)
<6>pid 53863 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 346 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped)
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 2169 (awk), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 71483 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 67881 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 17006 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 674 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 53229 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 22051 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 65419 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 70832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 11432 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 37415 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 25600 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 27832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 46837 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 79236 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 67326 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 25607 (php), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 33071 (grep), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 48416 (php_pfb), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 347 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 23422 (grep), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 34032 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 55850 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 57865 (grep), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 6265 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 39844 (netstat), jid 0, uid 0: exited on signal 10
<5>arp: packet with unknown hardware format 0x06 received on cxl1
kernel trap 9 with interrupts disabled


Fatal trap 9: general protection fault while in kernel mode
cpuid = 9; apic id = 09
instruction pointer	= 0x20:0xffffffff80daa116
stack pointer	        = 0x28:0xfffffe008fcc6128
frame pointer	        = 0x28:0xff7ffe008fcc6170
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 19853 (telegraf)
trap number		= 9
panic: general protection fault
cpuid = 9
time = 1638104130
KDB: enter: panic

db:0:kdb.enter.default>  show pcpu
cpuid        = 9
dynamic pcpu = 0xfffffe0080d67380
curthread    = 0xfffff802d4585740: pid 19853 tid 100759 "telegraf"
curpcb       = 0xfffff802d4585ce0
fpcurthread  = 0xfffff802d4585740: pid 19853 "telegraf"
idlethread   = 0xfffff80004597740: tid 100012 "idle: cpu9"
curpmap      = 0xfffff802d4524138
tssp         = 0xffffffff837179c8
commontssp   = 0xffffffff837179c8
rsp0         = 0xfffffe008fcc6540
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff8371e1e0
ldt          = 0xffffffff8371e220
tss          = 0xffffffff8371e210
tlb gen      = 365236
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 19853 tid 100759 td 0xfffff802d4585740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe008fcc5e30
vpanic() at vpanic+0x197/frame 0xfffffe008fcc5e80
panic() at panic+0x43/frame 0xfffffe008fcc5ee0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe008fcc5f40
trap() at trap+0x67/frame 0xfffffe008fcc6050
calltrap() at calltrap+0x8/frame 0xfffffe008fcc6050
--- trap 0x9, rip = 0xffffffff80daa116, rsp = 0xfffffe008fcc6128, rbp = 0xff7ffe008fcc6170 ---
mi_switch() at mi_switch+0x106/frame 0xff7ffe008fcc6170

That is a rather odd backtrace. Very short. Between that and the number of core dumps in the message buffer it suggests it's more of a hardware issue, such as heat, power, or faulty hardware.

madbeefer

@jimp well the entire thing was too long to post, honestly, I am not sure I picked out important part. Its all brand new hardware.. I tested the RAM before hand, I would hope its not heat, since its all new and its got the biggest CPU cooler I could find that would fit the case, and the CPU doesn't get put under much load.

madbeefer

Now I am getting this message.. PHP ERROR: Type: 1, File: /etc/inc/config.lib.inc, Line: 172, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2251799813795840 bytes) @ 2021-11-28 22:38:20

stephenw10

@madbeefer said in Crash Report:

tried to allocate 2251799813795840 bytes

Mmm, something very wrong there. It's hitting the 512MB limit in PHP which can happen if the config file becomes corrupt or you attempt to run a command with a huge output. However here it's trying to allocate 2048TiB! That cannot be right even if your config file ludicrously large.

How big is your config file? What were you attempting to do there?

Steve

madbeefer

@stephenw10
It looks like my config file is about 200k. My setup is nothing too fancy. I have a couple VLANs, snort, and a handful of other things going on. It actually worked just fine on my old hardware. The biggest difference between my old hardware and new was I went from an 8th gen intel CPU to a 12th gen, and changed from the onboard intel NIC to a Chelsio T520-BT T5 NIC. I have some 10g network functionality so I wanted to include my pfsense into that.

stephenw10

Mmm, 200k is fine.

Any idea what exactly you were attempting when it threw that error?

jimp

I'd still lean toward a hardware issue for that -- could easily be filesystem corruption or a bad read from the disk that led to that.

I'd be very suspicious of the hardware, even if it's new, that's when things are most likely to be bad (DOA) or fail.

madbeefer

@stephenw10
Nothing in particular, I've been asleep a lot of the times I've gotten the errors. It will cause it to randomly reboot as well.

stephenw10

Hmm, I would certainly try disabling some of those packages and seeing if that prevents it. I'd start with ntopng.

madbeefer

@stephenw10

I'll give that a shot and see if it helps. Any suggestions to keep my setup in case I have a corrupted file system.

stephenw10

If it fails to mount root because of filesystem damage you can run a manual check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Steve