Crash Report
-
I'm not really sure, but I figured this was as good as place as any to post my crash report. info.0 textdump.tar.0
-
@madbeefer Post your crash report...no one would want to download it.
-
The thing is too big to post all of it.. So I am guessing this is the important part. It looks like telegraf might be the issue.
<118> __
<118> _ __ / |__ ___ _ __ ___ ___
<118>| '_ | |/ _|/ _ \ ' / __|/ _
<118>| |) | _ \ / | | _ \ /
<118>| ./|| |/_|| ||/_|
<118>|_|
<118>
<118>
<118>Welcome to pfSense 2.5.2-RELEASE...
<118>
<118>No core dumps found.
<118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.32/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>>>> Removing vital flag from php74... done.
uhub0: 24 ports with 24 removable, self powered
<118>External config loader 1.0 is now starting... nvd0p1 nvd0p2 nvd0p3
<118>Launching the init system...Updating CPU Microcode...
ugen0.2: <CPS OR1500LCDRM1U> at usbus0
CPU: 11th Gen Intel(R) Core(TM) i5-11600K @ 3.90GHz (3912.22-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xa0671 Family=0x6 Model=0xa7 Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0xf2bf67eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL>
Structured Extended Features2=0x405f5e<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,RDPID>
Structured Extended Features3=0xbc000410<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME>
VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
<118>Done.
<118>...
ugen0.3: <vendor 0x05e3 USB2.0 Hub> at usbus0
<118>.
uhub1 on uhub0
uhub1: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/60.90, addr 2> on usbus0
<118> done.
<118>Initializing.................. done.
<118>Starting device manager (devd)...
uhub1: 4 ports with 4 removable, self powered
ig4iic0: <Intel Tiger Lake-H I2C Controller-1> at device 21.0 on pci0
ig4iic0: Using MSI
iicbus0: <Philips I2C bus (ACPI-hinted)> on ig4iic0
ugen0.4: <AsusTek Computer Inc. AURA LED Controller> at usbus0
uhid0 on uhub0
uhid0: <AsusTek Computer Inc. AURA LED Controller, class 0/0, rev 2.00/1.00, addr 3> on usbus0
<118>done.
<118>Loading configuration......done.
<118>Updating configuration...done.
<118>Checking config backups consistency.................................done.
<118>Setting up extended sysctls...done.
padlock0: No ACE support.
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256> on motherboard
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
<118>Setting timezone...done.
<118>Configuring loopback interface...
<6>lo0: link state changed to UP
<118>done.
<118>Starting syslog...done.
<118>Starting Secure Shell Services...done.
<118>Setting up interfaces microcode...done.
<118>Starting PC/SC Smart Card Services...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring LAGG interfaces...done.
<118>Configuring VLAN interfaces...
ugen0.2: <CPS OR1500LCDRM1U> at usbus0 (disconnected)
<6>vlan0: changing name to 'cxl1.60'
<6>vlan1: changing name to 'cxl1.100'
<6>vlan2: changing name to 'cxl1.99'
<6>vlan3: changing name to 'cxl1.98'
<6>vlan4: changing name to 'cxl1.123'
<118>done.
<118>Configuring QinQ interfaces...done.
<118>Configuring WAN interface...done.
<118>Configuring LAN interface...done.
<118>Configuring VLANCAMERAS60 interface...done.
<118>Configuring IOTJUNK interface...done.
<118>Configuring ROOMMATEINET interface...done.
<118>Configuring TEMP interface...done.
<118>Configuring IPsec VTI interfaces...done.
<118>Configuring CARP settings...done.
<118>Syncing OpenVPN settings...
<6>tun2: changing name to 'ovpns2'
<118>done.
<6>ovpns2: link state changed to UP
<6>pflog0: promiscuous mode enabled
<118>Configuring firewall.....Starting PFLOG...done.
<118>Setting up gateway monitors...done.
<118>Setting up static routes...
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
<6>cxl1: link state changed to UP
<6>cxl1.100: link state changed to UP
<6>cxl1.98: link state changed to UP
<6>cxl1.99: link state changed to UP
<6>cxl1.123: link state changed to UP
<6>cxl1.60: link state changed to UP
<6>cxl0: link state changed to UP
ugen0.2: <CPS OR1500LCDRM1U> at usbus0
<118>done.
<118>Setting up DNSs...
<118>Starting DNS Resolver...done.
<118>Synchronizing user settings...done.
<118>Starting webConfigurator...done.
<118>Configuring CRON...done.
<118>Starting NTP Server...done.
<118>Starting DHCP service...done.
<118>Configuring firewall.....
ugen0.2: <CPS OR1500LCDRM1U> at usbus0 (disconnected)
ugen0.2: <CPS OR1500LCDRM1U> at usbus0
<118>Generating RRD graphs...done.
<118>Starting syslog...done.
<118>Starting CRON... done.
<6>pid 66643 (xmllint), jid 0, uid 0: exited on signal 10 (core dumped)
<118>Bus error (core dumped)
<118> Starting package snort...
<6>pid 57250 (php-cgi), jid 0, uid 0: exited on signal 10 (core dumped)
<118>done.
<118> Starting package darkstat...done.
<6>cxl1: promiscuous mode enabled
<6>cxl1.60: promiscuous mode enabled
<6>cxl0: promiscuous mode enabled
<118> Starting package nut...done.
<118> Starting package Traffic Totals...done.
<118> Starting package OpenVPN Client Export Utility...done.
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 80693 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<118> Starting package ntopng...done.
<118> Starting package suricata...done.
<118> Starting package iperf...done.
<118> Starting package pfBlockerNG-devel...done.
<118> Starting package Backup...done.
<118> Starting package bandwidthd...done.
<118> Starting package Telegraf...done.
<118> Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done.
<118> Starting /usr/local/etc/rc.d/pfb_filter.sh...done.
<118> Starting /usr/local/etc/rc.d/shutdown.nut.sh...done.
<118>pfSense 2.5.2-RELEASE amd64 Fri Jul 02 15:33:00 EDT 2021
<118>Bootup complete
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 80017 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 86369 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 14742 (netstat), jid 0, uid 0: exited on signal 11
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 43961 (php-cgi), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 84237 (grep), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 43089 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 38635 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 55275 (awk), jid 0, uid 0: exited on signal 10 (core dumped)
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 85766 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 8376 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped)
<6>pid 53863 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 346 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped)
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<5>arp: packet with unknown hardware format 0x06 received on cxl1
<6>pid 2169 (awk), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 71483 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 67881 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 17006 (php), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 674 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 53229 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 22051 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 65419 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 70832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 11432 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 37415 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 25600 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 27832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 46837 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 79236 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 67326 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 25607 (php), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 33071 (grep), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 48416 (php_pfb), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 347 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 23422 (grep), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 34032 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 55850 (netstat), jid 0, uid 0: exited on signal 10
<6>pid 57865 (grep), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 6265 (sh), jid 0, uid 0: exited on signal 10 (core dumped)
<6>pid 39844 (netstat), jid 0, uid 0: exited on signal 10
<5>arp: packet with unknown hardware format 0x06 received on cxl1
kernel trap 9 with interrupts disabledFatal trap 9: general protection fault while in kernel mode
cpuid = 9; apic id = 09
instruction pointer = 0x20:0xffffffff80daa116
stack pointer = 0x28:0xfffffe008fcc6128
frame pointer = 0x28:0xff7ffe008fcc6170
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 19853 (telegraf)
trap number = 9
panic: general protection fault
cpuid = 9
time = 1638104130
KDB: enter: panic -
<6>pid 80017 (php), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 86369 (php), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 14742 (netstat), jid 0, uid 0: exited on signal 11 <5>arp: packet with unknown hardware format 0x06 received on cxl1 <5>arp: packet with unknown hardware format 0x06 received on cxl1 <6>pid 43961 (php-cgi), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 84237 (grep), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 43089 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 38635 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 55275 (awk), jid 0, uid 0: exited on signal 10 (core dumped) <5>arp: packet with unknown hardware format 0x06 received on cxl1 <6>pid 85766 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 8376 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped) <6>pid 53863 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 346 (php-fpm), jid 0, uid 0: exited on signal 6 (core dumped) <5>arp: packet with unknown hardware format 0x06 received on cxl1 <5>arp: packet with unknown hardware format 0x06 received on cxl1 <6>pid 2169 (awk), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 71483 (php), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 67881 (php), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 17006 (php), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 674 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 53229 (sh), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 22051 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 65419 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 70832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 11432 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 37415 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 25600 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 27832 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 46837 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 79236 (sh), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 67326 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 25607 (php), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 33071 (grep), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 48416 (php_pfb), jid 0, uid 0: exited on signal 11 (core dumped) <6>pid 347 (php-fpm), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 23422 (grep), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 34032 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 55850 (netstat), jid 0, uid 0: exited on signal 10 <6>pid 57865 (grep), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 6265 (sh), jid 0, uid 0: exited on signal 10 (core dumped) <6>pid 39844 (netstat), jid 0, uid 0: exited on signal 10 <5>arp: packet with unknown hardware format 0x06 received on cxl1 kernel trap 9 with interrupts disabled Fatal trap 9: general protection fault while in kernel mode cpuid = 9; apic id = 09 instruction pointer = 0x20:0xffffffff80daa116 stack pointer = 0x28:0xfffffe008fcc6128 frame pointer = 0x28:0xff7ffe008fcc6170 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 19853 (telegraf) trap number = 9 panic: general protection fault cpuid = 9 time = 1638104130 KDB: enter: panic
db:0:kdb.enter.default> show pcpu cpuid = 9 dynamic pcpu = 0xfffffe0080d67380 curthread = 0xfffff802d4585740: pid 19853 tid 100759 "telegraf" curpcb = 0xfffff802d4585ce0 fpcurthread = 0xfffff802d4585740: pid 19853 "telegraf" idlethread = 0xfffff80004597740: tid 100012 "idle: cpu9" curpmap = 0xfffff802d4524138 tssp = 0xffffffff837179c8 commontssp = 0xffffffff837179c8 rsp0 = 0xfffffe008fcc6540 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff8371e1e0 ldt = 0xffffffff8371e220 tss = 0xffffffff8371e210 tlb gen = 365236 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 19853 tid 100759 td 0xfffff802d4585740 kdb_enter() at kdb_enter+0x37/frame 0xfffffe008fcc5e30 vpanic() at vpanic+0x197/frame 0xfffffe008fcc5e80 panic() at panic+0x43/frame 0xfffffe008fcc5ee0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe008fcc5f40 trap() at trap+0x67/frame 0xfffffe008fcc6050 calltrap() at calltrap+0x8/frame 0xfffffe008fcc6050 --- trap 0x9, rip = 0xffffffff80daa116, rsp = 0xfffffe008fcc6128, rbp = 0xff7ffe008fcc6170 --- mi_switch() at mi_switch+0x106/frame 0xff7ffe008fcc6170
That is a rather odd backtrace. Very short. Between that and the number of core dumps in the message buffer it suggests it's more of a hardware issue, such as heat, power, or faulty hardware.
-
@jimp well the entire thing was too long to post, honestly, I am not sure I picked out important part. Its all brand new hardware.. I tested the RAM before hand, I would hope its not heat, since its all new and its got the biggest CPU cooler I could find that would fit the case, and the CPU doesn't get put under much load.
-
Now I am getting this message.. PHP ERROR: Type: 1, File: /etc/inc/config.lib.inc, Line: 172, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2251799813795840 bytes) @ 2021-11-28 22:38:20
-
@madbeefer said in Crash Report:
tried to allocate 2251799813795840 bytes
Mmm, something very wrong there. It's hitting the 512MB limit in PHP which can happen if the config file becomes corrupt or you attempt to run a command with a huge output. However here it's trying to allocate 2048TiB! That cannot be right even if your config file ludicrously large.
How big is your config file? What were you attempting to do there?
Steve
-
@stephenw10
It looks like my config file is about 200k. My setup is nothing too fancy. I have a couple VLANs, snort, and a handful of other things going on. It actually worked just fine on my old hardware. The biggest difference between my old hardware and new was I went from an 8th gen intel CPU to a 12th gen, and changed from the onboard intel NIC to a Chelsio T520-BT T5 NIC. I have some 10g network functionality so I wanted to include my pfsense into that. -
Mmm, 200k is fine.
Any idea what exactly you were attempting when it threw that error?
-
I'd still lean toward a hardware issue for that -- could easily be filesystem corruption or a bad read from the disk that led to that.
I'd be very suspicious of the hardware, even if it's new, that's when things are most likely to be bad (DOA) or fail.
-
@stephenw10
Nothing in particular, I've been asleep a lot of the times I've gotten the errors. It will cause it to randomly reboot as well. -
Hmm, I would certainly try disabling some of those packages and seeing if that prevents it. I'd start with ntopng.
-
I'll give that a shot and see if it helps. Any suggestions to keep my setup in case I have a corrupted file system.
-
If it fails to mount root because of filesystem damage you can run a manual check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-checkSteve