pfSense crashes randomly - new setup
-
Hello. I am new to pfSense and I am considering making the switch from my current setup which is a Linksys WRT3200ACM with DD-WRT firmware. I have been using that for about 10 years but feel that I want to upgrade to pfSense for added control and network security. I have been watching videos by lawrence systems on youtube as well as many others for about a year now, and I think I have a pretty good grasp of setting up pfSense and decided to pull the trigger.
I have an HP T730 thin client with 8GB ram and a 32 GB SSD that i purchased from Ebay. I also installed a 4-port NIC, which is an IBM 39Y6138. I purchased the NIC on r/homelabsales from a reliable seller and from what I can tell it is a genuine NIC and not a knock off. I updated the T730 BIOS to the latest version but that didn't help either. I ran a system memory test and a hard drive test, and it passed both tests without any errors.
The issue that I have is that pfSense will randomly freeze. It can take 2 minutes or it can take 45 minutes, but sooner or later it freezes. It can happen when its completely idle or when I am setting things up in the GUI, it doesn't matter. It will not respond to pings, The monitor connected to the device shows the default screen, but the keyboard becomes unresponsive, and the web interface stops responding and times out. If left in this state for a few minutes (5-10 minutes maybe?) the system will all of a sudden start outputting lines to the monitor so fast that they cant be read. Then it restarts and there is a crash log when you log back into the web GUI.
I am an absolute beginner with this type of hardware so I don't understand the logs at all. From what I can tell, there is an issue called "spin lock held too long" which causes a system "panic". The log outputs are below, if anyone can help be decipher what is going on. The only thing that I can think of is maybe the NIC isnt compatible?
from the msgbuf.txt in the crash report:
---<<BOOT>>--- Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/amd64.amd64/sys/pfSense amd64 FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git llvmorg-15.0.7-0-g8dfdcc7b7bf6) VT(efifb): resolution 800x600 CPU: AMD RX-427BB with AMD Radeon(tm) R7 Graphics (2695.09-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x630f01 Family=0x15 Model=0x30 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0xfebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC,<b25>,DBE,PTSC> Structured Extended Features=0x9<FSGSBASE,BMI1> XSAVE Features=0x1<XSAVEOPT> AMD Extended Feature Extensions ID EBX=0x1000<IBPB> SVM: (disabled in BIOS) NP,NRIP,VClean,AFlush,DAssist,NAsids=65536 TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 7125319680 (6795 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <HPQOEM SLIC-WKS> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20221020/tbfadt-796) ioapic0 <Version 2.1> irqs 0-23 ioapic1 <Version 2.1> irqs 24-55 Launching APs: 2 1 3 TCP_ratelimit: Is now initialized wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80748430, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff807484e0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80748590, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff807659c0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765a70, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80765b20, 0) error 1 random: entropy device external interface kbd1 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0. netgate0: <unknown hardware> efirtc0: <EFI Realtime Clock> efirtc0: registered as a time-of-day clock, resolution 1.000000s smbios0: <System Management BIOS> at iomem 0xf04c0-0xf04de smbios0: Version: 2.8, BCD Revision: 2.7 acpi0: <HPQOEM SLIC-WKS> acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 450 Event timer "HPET2" frequency 14318180 Hz quality 450 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xd0000000-0xdfffffff,0xe0000000-0xe07fffff,0xfeb00000-0xfeb3ffff irq 26 at device 1.0 on pci0 vgapci0: Boot video device hdac0: <ATI (0x1308) HDA Controller> mem 0xfeb64000-0xfeb67fff irq 27 at device 1.1 on pci0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.1 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 pcib3: <PCI-PCI bridge> at device 2.0 on pci2 pci3: <PCI bus> on pcib3 em0: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0xd020-0xd03f mem 0xfe9a0000-0xfe9bffff,0xfe980000-0xfe99ffff irq 27 at device 0.0 on pci3 em0: EEPROM V5.10-2 em0: Using 1024 TX descriptors and 1024 RX descriptors em0: Using an MSI interrupt <6>em0: Ethernet address: 00:15:17:d2:22:65 <6>em0: netmap queues/slots: TX 1/1024, RX 1/1024 em1: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0xd000-0xd01f mem 0xfe940000-0xfe95ffff,0xfe920000-0xfe93ffff irq 26 at device 0.1 on pci3 em1: EEPROM V5.10-2 em1: Using 1024 TX descriptors and 1024 RX descriptors em1: Using an MSI interrupt <6>em1: Ethernet address: 00:15:17:d2:22:64 <6>em1: netmap queues/slots: TX 1/1024, RX 1/1024 pcib4: <PCI-PCI bridge> at device 4.0 on pci2 pci4: <PCI bus> on pcib4 em2: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0xc020-0xc03f mem 0xfe8a0000-0xfe8bffff,0xfe880000-0xfe89ffff irq 25 at device 0.0 on pci4 em2: EEPROM V5.10-2 em2: Using 1024 TX descriptors and 1024 RX descriptors em2: Using an MSI interrupt <6>em2: Ethernet address: 00:15:17:d2:22:67 <6>em2: netmap queues/slots: TX 1/1024, RX 1/1024 em3: <Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper)> port 0xc000-0xc01f mem 0xfe840000-0xfe85ffff,0xfe820000-0xfe83ffff irq 24 at device 0.1 on pci4 em3: EEPROM V5.10-2 em3: Using 1024 TX descriptors and 1024 RX descriptors em3: Using an MSI interrupt <6>em3: Ethernet address: 00:15:17:d2:22:66 <6>em3: netmap queues/slots: TX 1/1024, RX 1/1024 pcib5: <ACPI PCI-PCI bridge> irq 18 at device 3.2 on pci0 pci5: <ACPI PCI bus> on pcib5 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xfea04000-0xfea04fff,0xfea00000-0xfea03fff irq 36 at device 0.0 on pci5 re0: Using 1 MSI-X message re0: Chip rev. 0x54000000 re0: MAC rev. 0x00100000 miibus0: <MII bus> on re0 rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow <6>re0: Using defaults for TSO: 65518/35/2048 <6>re0: Ethernet address: 7c:d3:0a:3c:9b:74 <6>re0: netmap queues/slots: TX 1/256, RX 1/256 xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb6a000-0xfeb6bfff irq 18 at device 16.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 xhci1: <AMD FCH USB 3.0 controller> mem 0xfeb68000-0xfeb69fff irq 17 at device 16.1 on pci0 xhci1: 32 bytes context size, 64-bit DMA usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xf190-0xf197,0xf180-0xf183,0xf170-0xf177,0xf160-0xf163,0xf150-0xf15f mem 0xfeb70000-0xfeb707ff irq 19 at device 17.0 on pci0 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ohci0: <AMD FCH USB Controller> mem 0xfeb6f000-0xfeb6ffff irq 18 at device 18.0 on pci0 usbus2 on ohci0 usbus2: 12Mbps Full Speed USB v1.0 ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb6e000-0xfeb6e0ff irq 17 at device 18.2 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 usbus3: 480Mbps High Speed USB v2.0 ohci1: <AMD FCH USB Controller> mem 0xfeb6d000-0xfeb6dfff irq 18 at device 19.0 on pci0 usbus4 on ohci1 usbus4: 12Mbps Full Speed USB v1.0 ehci1: <AMD FCH USB 2.0 controller> mem 0xfeb6c000-0xfeb6c0ff irq 17 at device 19.2 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci1 usbus5: 480Mbps High Speed USB v2.0 atapci0: <AMD Hudson-2 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf100-0xf10f irq 17 at device 20.1 on pci0 ata0: <ATA channel> at channel 0 on atapci0 ata1: <ATA channel> at channel 1 on atapci0 hdac1: <AMD Hudson-2 HDA Controller> mem 0xfeb60000-0xfeb63fff irq 16 at device 20.2 on pci0 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 pcib6: <ACPI PCI-PCI bridge> at device 20.4 on pci0 pci6: <ACPI PCI bus> on pcib6 acpi_button0: <Power Button> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] driver bug: Unable to set devclass (class: atkbdc devname: (unknown)) ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: <Parallel port bus> on ppc0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 orm0: <ISA Option ROM> at iomem 0xcf800-0xd07ff pnpid ORM0000 on isa0 hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Timecounter "TSC-low" frequency 1347482132 Hz quality 1000 Timecounters tick every 1.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) ugen2.1: <AMD OHCI root HUB> at usbus2 ugen1.1: <AMD XHCI root HUB> at usbus1 ugen3.1: <AMD EHCI root HUB> at usbus3 ugen5.1: <AMD EHCI root HUB> at usbus5 uhub0 on usbus2 uhub1 on usbus1 uhub2 on usbus5 uhub0: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 uhub3 on usbus3 uhub3: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5 uhub1: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ugen4.1: <AMD OHCI root HUB> at usbus4 hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0 hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0 pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0 pcm1: <ATI R6xx (HDMI)> at nid 5 on hdaa0 pcm2: <ATI R6xx (HDMI)> at nid 7 on hdaa0 pcm3: <ATI R6xx (HDMI)> at nid 9 on hdaa0 hdacc1: <Realtek ALC221 HDA CODEC> at cad 0 on hdac1 hdaa1: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc1 pcm4: <Realtek ALC221 (Analog 2.0+HP/2.0)> at nid 20,33 and 24,27 on hdaa1 pcm5: <Realtek ALC221 (Front Analog Headphones)> at nid 26 on hdaa1 Trying to mount root from zfs:pfSense/ROOT/default []... uhub4 on usbus4 uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen0.1: <AMD XHCI root HUB> at usbus0 uhub5 on usbus0 uhub5: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub0: 5 ports with 5 removable, self powered uhub4: 5 ports with 5 removable, self powered uhub1: 4 ports with 4 removable, self powered uhub5: 4 ports with 4 removable, self powered Root mount waiting for: CAM usbus3 usbus5 uhub2: 5 ports with 5 removable, self powered uhub3: 5 ports with 5 removable, self powered Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <PSSBN032GA27MC1 SBFH01H0> ACS-4 ATA SATA 3.x device ada0: Serial Number 5A54078A0BC400125322 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 30533MB (62533296 512 byte sectors) <118>Configuring crash dumps... <118>Using /dev/ada0p3 for dump device. <118>/dev/gpt/efiboot0: 6 files, 259 MiB free (16549 clusters) <118>FIXED <118>/dev/gpt/efiboot0: MARKING FILE SYSTEM CLEAN <118>Filesystems are clean, continuing... <118>Mounting filesystems... <118>Mounting ZFS boot environment... done. <118> <118> __ <118> _ __ / _|___ ___ _ __ ___ ___ <118>| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \ <118>| |_) | _\__ \ __/ | | \__ \ __/ <118>| .__/|_| |___/\___|_| |_|___/\___| <118>|_| <118> <118> <118>Welcome to pfSense 2.7.0-RELEASE... <118> <118>No core dumps found. <118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE <118>32-bit compatibility ldconfig path: <118>done. <118>>>> Removing vital flag from php82... done. <118>External config loader 1.0 is now starting... ada0p1 ada0p2 ada0p4 <118>Launching the init system...Updating CPU Microcode... CPU: AMD RX-427BB with AMD Radeon(tm) R7 Graphics (2694.96-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x630f01 Family=0x15 Model=0x30 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0xfebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC,<b25>,DBE,PTSC> Structured Extended Features=0x9<FSGSBASE,BMI1> XSAVE Features=0x1<XSAVEOPT> AMD Extended Feature Extensions ID EBX=0x1000<IBPB> SVM: (disabled in BIOS) NP,NRIP,VClean,AFlush,DAssist,NAsids=65536 TSC: P-state invariant, performance statistics <118>Done. <118> done. <118>Initializing.................. done. <118>Starting device manager (devd)... intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0 smbus0: <System Management Bus> on intsmb0 <118>done. <118>Loading configuration....done. <118>Updating configuration...done. <118>Checking config backups consistency.................................done. <118>Setting up extended sysctls...done. <118>Setting timezone...done. <118>Configuring loopback interface... <6>lo0: link state changed to UP <118>done. <118>Starting syslog...done. <118>Setting up interfaces microcode...done. <118>Removed leftover dhcp6c lock file: /tmp/dhcp6c_lock <118>Configuring loopback interface...done. <118>Configuring VLAN interfaces... <6>vlan0: changing name to 'em2.107' <6>vlan1: changing name to 'em2.17' <118>done. <118>Configuring WAN interface...done. <118>Configuring LAN interface...done. <118>Configuring WIFILAN interface...done. <118>Configuring VLAN107 interface...done. <118>Configuring VLAN17_GUEST interface...done. <118>Configuring LAN2 interface... [fib_algo] inet.0 (bsearch4#34) rebuild_fd_flm: switching algo to radix4_lockless <118>done. <118>Configuring CARP settings...done. <118>Starting Secure Shell Services...done. <118>Syncing OpenVPN settings...done. <6>pflog0: promiscuous mode enabled <118>Configuring firewall......done. <118>Starting PFLOG...done. <118>Setting up gateway monitors...done. <118>Setting up static routes...route: writing to routing socket: Network is unreachable <118>done. <118>Setting up DNSs... <118>Starting DNS Resolver... <6>em1: link state changed to UP <118>done. <118>Synchronizing user settings...done. <118>Configuring CRON...done. <118>Bootstrapping clock...done. <118>Starting NTP Server...done. <118>Starting webConfigurator...done. <118>Starting DHCP service...done. <118>Starting DHCPv6 service...done. <118>Configuring firewall......done. <118>Generating RRD graphs...done. <118>Starting syslog...done. <118>Starting CRON... done. <118>pfSense 2.7.0-RELEASE amd64 Wed Jun 28 03:53:34 UTC 2023 <118>Bootup complete <6>em0: link state changed to UP <6>em0: link state changed to DOWN <6>em0: link state changed to UP spin lock 0xffffffff8303fa60 (et_hw_mtx) held by 0xfffffe000fd84ac0 (tid 100073) too long spin lock 0xffffffff8303fa60 (et_hw_mtx) held by 0xfffffe000fd84ac0 (tid 100073) too long panic: spin lock held too long cpuid = 3 time = 1688679387 KDB: enter: panic -
I have a bit of an update and maybe some more questions:
Running pciconf, I have determined that that I am using the em drivers:
em0@pci0:3:0:0: class=0x020000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x10bc subvendor=0x8086 subdevice=0x11bc
vendor = 'Intel Corporation'
device = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
class = network
subclass = ethernetI also read that that in the past ema nd igb drivers were merged. Could I be using the wrong drivers? Do I need to use igb drivers? I dont know how to tell pfsense to use a different driver. Or does this all just mean that my NIC is not compatible?
Thanks for the help.
-
That's the correct driver.
We need to see the backtrace from the ddb.txt file in the crash report to know more.
So for example:
db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x532100 curthread = 0xfffff800033a0000: pid 11 "idle: cpu0" curpcb = 0xfffffe0059bc3cc0 fpcurthread = none idlethread = 0xfffff800033a0000: tid 100003 "idle: cpu0" curpmap = 0xffffffff820f89a0 tssp = 0xffffffff82113890 commontssp = 0xffffffff82113890 rsp0 = 0xfffffe0059bc3cc0 gs32p = 0xffffffff821152e8 ldt = 0xffffffff82115328 tss = 0xffffffff82115318 db:0:kdb.enter.default> bt Tracing pid 11 tid 100003 td 0xfffff800033a0000 callout_process() at callout_process+0x1a0/frame 0xfffffe0059bc38b0 handleevents() at handleevents+0x18e/frame 0xfffffe0059bc3910 timercb() at timercb+0x318/frame 0xfffffe0059bc3970 lapic_handle_timer() at lapic_handle_timer+0x9c/frame 0xfffffe0059bc39a0 Xtimerint() at Xtimerint+0x8c/frame 0xfffffe0059bc39a0 --- interrupt, rip = 0xffffffff80f84316, rsp = 0xfffffe0059bc3a70, rbp = 0xfffffe0059bc3a70 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0059bc3a70 acpi_cpu_idle() at acpi_cpu_idle+0x15a/frame 0xfffffe0059bc3ac0 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0059bc3ae0 cpu_idle() at cpu_idle+0x90/frame 0xfffffe0059bc3b00 sched_idletd() at sched_idletd+0x1d5/frame 0xfffffe0059bc3bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe0059bc3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0059bc3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> psSteve
-
here is what is in my ddb.txt file. I had to use pastebin because it was too long: https://pastebin.com/vWni3cmP
Thanks.
-
OK so:
db:0:kdb.enter.default> show pcpu cpuid = 3 dynamic pcpu = 0xfffffe008cd085c0 curthread = 0xfffffe000fdc93a0: pid 16 tid 100079 critnest 1 "usbus1" curpcb = 0xfffffe000fdc98c0 fpcurthread = none idlethread = 0xfffffe000fcbce40: tid 100006 "idle: cpu3" self = 0xffffffff84013000 curpmap = 0xffffffff8303ef30 tssp = 0xffffffff84013384 rsp0 = 0xfffffe007ca93000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff84013404 ldt = 0xffffffff84013444 tss = 0xffffffff84013434 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 16 tid 100079 td 0xfffffe000fdc93a0 kdb_enter() at kdb_enter+0x32/frame 0xfffffe007ca92bb0 vpanic() at vpanic+0x183/frame 0xfffffe007ca92c00 panic() at panic+0x43/frame 0xfffffe007ca92c60 _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x67/frame 0xfffffe007ca92c70 _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0xd5/frame 0xfffffe007ca92ce0 cpu_new_callout() at cpu_new_callout+0x2a2/frame 0xfffffe007ca92d30 callout_reset_sbt_on() at callout_reset_sbt_on+0x1a8/frame 0xfffffe007ca92d90 sleepq_set_timeout_sbt() at sleepq_set_timeout_sbt+0xbd/frame 0xfffffe007ca92dd0 _sleep() at _sleep+0x178/frame 0xfffffe007ca92e50 pause_sbt() at pause_sbt+0xff/frame 0xfffffe007ca92e80 usb_pause_mtx() at usb_pause_mtx+0x55/frame 0xfffffe007ca92eb0 usb_process() at usb_process+0xd7/frame 0xfffffe007ca92ef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe007ca92f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007ca92f30 --- trap 0xd9738ee, rip = 0x85d0315689903142, rsp = 0x22e7aa0baea7aa0e, rbp = 0x9ed8862492988620 ---So it looks to be something USB related but I don't see any USB devices in your log other than controllers and hubs. Do you have any USB devices connected?
-
@stephenw10
No, the only USB device plugged in is the keyboard. However, i tried running it completely headless without the keyboard or monitor connected. With just the power and one cable on the LAN side connected to a laptop for access the GUI. No WAN connection, as i am trying to work out the bugs befpre i disconmect my current internet setup. Even with only the LAN and power cable, it still gets a panic error.Also, i just got 2 brand new RAM sticks today 2x8GB and still have the problem so im guessing its not related to the RAM.
-
If it's a RAM issue the panics will be more random. Do all your crash reports show that same backtrace?
It could be a driver issue with one of the USB controllers. You might try disabling the USB3 (xhci) controller in the BIOS if you can.
-
It looks like this latest crash is different.
db:0:kdb.enter.default> show registers cs 0x20 ds 0x3b es 0x3b fs 0x13 gs 0x1b ss 0x28 rax 0x12 rcx 0x1 rdx 0xfffffe001b7e4690 rbx 0x100 rsp 0xfffffe001b7e4a70 rbp 0xfffffe001b7e4a70 rsi 0x32 rdi 0xffffffff82d82918 vt_conswindow+0x10 r8 0 r9 0x1e6b00 r10 0xffffffff82d82908 vt_conswindow r11 0x15f r12 0 r13 0xfffffe001e255c80 r14 0xfffffe001b7e4b00 r15 0xfffffe001e2563a0 rip 0xffffffff80d43122 kdb_enter+0x32 rflags 0x86 kdb_enter+0x32: movq $0,0x2347ce3(%rip) db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command; use "help" to list available commands db:1:lockinfo> show alllocks No such command; use "help" to list available commands db:1:lockinfo> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x10865c0 curthread = 0xfffffe001e2563a0: pid 11 tid 100003 critnest 3 "idle: cpu0" curpcb = 0xfffffe001e2568c0 fpcurthread = none idlethread = 0xfffffe001e2563a0: tid 100003 "idle: cpu0" self = 0xffffffff84010000 curpmap = 0xffffffff8303ef30 tssp = 0xffffffff84010384 rsp0 = 0xfffffe001b7e5000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff84010404 ldt = 0xffffffff84010444 tss = 0xffffffff84010434 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 11 tid 100003 td 0xfffffe001e2563a0 kdb_enter() at kdb_enter+0x32/frame 0xfffffe001b7e4a70 vpanic() at vpanic+0x183/frame 0xfffffe001b7e4ac0 panic() at panic+0x43/frame 0xfffffe001b7e4b20 _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x67/frame 0xfffffe001b7e4b30 _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0xd5/frame 0xfffffe001b7e4ba0 handleevents() at handleevents+0x2cb/frame 0xfffffe001b7e4be0 timercb() at timercb+0x25b/frame 0xfffffe001b7e4c30 hpet_intr_single() at hpet_intr_single+0x1b0/frame 0xfffffe001b7e4c60 intr_event_handle() at intr_event_handle+0x123/frame 0xfffffe001b7e4cd0 intr_execute_handlers() at intr_execute_handlers+0x4a/frame 0xfffffe001b7e4d00 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe001b7e4d00 --- interrupt, rip = 0xffffffff8125b026, rsp = 0xfffffe001b7e4dd0, rbp = 0xfffffe001b7e4dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe001b7e4dd0 acpi_cpu_idle() at acpi_cpu_idle+0x2fe/frame 0xfffffe001b7e4e10 cpu_idle_acpi() at cpu_idle_acpi+0x48/frame 0xfffffe001b7e4e30 cpu_idle() at cpu_idle+0x9e/frame 0xfffffe001b7e4e50 sched_idletd() at sched_idletd+0x4d1/frame 0xfffffe001b7e4ef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe001b7e4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001b7e4f30 --- trap 0x552ee2ab, rip = 0xdd69eb03d129eb07, rsp = 0x7a5e704f761e704b, rbp = 0xc6615c61ca215c65 --- db:0:kdb.enter.default> ps -
Hmm still the same spin lock issue though but from something else.
Given that platform is known I would try removing the Intel NIC and running with only the Realtek NIC for a few days, see if it still crashes.
-
@stephenw10
Ok thanks for the advice. I will try that now and report back after a few days. If there are no crashes, then i guess ill blame the NIC. -
I took the NIC out and it has been running for 6+ hours with no issues. Ill let it run longer to make sure, but before it never made it past 2-1/2 hours without a kernel panic. Im guessing that the NIC was the issue. I guess i need to find a new NIC. I will report back tomorrow after i run it all night.
-
So ive just passed 24 hrs runnung without the NIC installed and no kernel panics. Im assuming the NIC was causing the problems. I guess the IBM Pro/1000 PT Quad NIC is not compatible, even though it is listed as a compatible device? I guess ill be looking for an i350-T4 as they seem to be the best.
Thanks for your help!
-
@farizno said in pfSense crashes randomly - new setup:
So ive just passed 24 hrs runnung without the NIC installed and no kernel panics. Im assuming the NIC was causing the problems. I guess the IBM Pro/1000 PT Quad NIC is not compatible, even though it is listed as a compatible device? I guess ill be looking for an i350-T4 as they seem to be the best.
Thanks for your help!
It could just be a faulty card and not compatibility issue.
-
Or some low level compatibility with that particular device.
Or a power or heat issue there with the expansion card.
-
@stephenw10 I do appreciate all the assistance. I will order an i350-T4 this week and report back after trying that card. Thanks again.
-
After installing an i350-T4 card, I can confirm that there have been no more kernel panics. I think this definitely points to an issue with the IBM/Intel PRO/1000 PT 82571EB/82571GB card that I have. I am not sure if the card is faulty and I don't really know how to test it. I guess I can try installing it in a Windows desktop PC that I have and see if it causes my desktop to crash, but the desktop that I have is connected on wireless (I don't have an ethernet drop near where it is located) so I am not sure if just having it installed will tell me if it functions properly.
Anyways, thanks for all the assistance stephenw10.
-
Yup, testing the card in a different host is really the only way to know for sure.
-
X xMrMurderx referenced this topic
