pfSense randomly freezing
-
My pfSense box at home started randomly freezing in February. At first I thought the motherboard was dying (it was running on the original Supermicro A1SRi-2558F, with the problematic Atom), so I installed pfSense to PCEngines APU 2d3 and restored the configuration, but the random freezes are still happening.
When this started happening, pfSense would freeze every few hours. At the time I didn't have it fully updated, so I did that first, but it hadn't helped much (the freezes became less frequent, but still happened). Switching to APU seemed to help at first, but then it also started freezing, although it only happens every few weeks (I also tested the Supermicro board with Linux, where it runs stable).
There's nothing useful in logs when pfSense freezes - the entries just end, then resume after reset. I also left serial console open, and there was nothing printed to it when pfSense froze (there also was no response to any keypresses; it similarly didn't respond to any keypresses on the SuperMicro board when it froze).
Any ideas how to further troubleshoot this?
Here's system.log from the time of the freeze:
Apr 13 23:32:12 pfsense apcupsd[77792]: Communications with UPS lost. Apr 13 23:42:16 pfsense apcupsd[77792]: Communications with UPS lost. Apr 13 23:52:19 pfsense apcupsd[77792]: Communications with UPS lost. Apr 14 11:22:09 pfsense syslogd: kernel boot file is /boot/kernel/kernel Apr 14 11:22:09 pfsense kernel: Copyright (c) 1992-2018 The FreeBSD Project. Apr 14 11:22:09 pfsense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Apr 14 11:22:09 pfsense kernel: The Regents of the University of California. All rights reserved. Apr 14 11:22:09 pfsense kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Apr 14 11:22:09 pfsense kernel: FreeBSD 11.2-RELEASE-p6 #3 518496b29ae(RELENG_2_4_4): Wed Dec 12 07:41:44 EST 2018 Apr 14 11:22:09 pfsense kernel: root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64 Apr 14 11:22:09 pfsense kernel: FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) Apr 14 11:22:09 pfsense kernel: VT(vga): resolution 640x480 Apr 14 11:22:09 pfsense kernel: CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU) Apr 14 11:22:09 pfsense kernel: Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 Apr 14 11:22:09 pfsense kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Apr 14 11:22:09 pfsense kernel: Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> Apr 14 11:22:09 pfsense kernel: AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> Apr 14 11:22:09 pfsense kernel: AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I> Apr 14 11:22:09 pfsense kernel: Structured Extended Features=0x8<BMI1> Apr 14 11:22:09 pfsense kernel: XSAVE Features=0x1<XSAVEOPT> Apr 14 11:22:09 pfsense kernel: SVM: NP,NRIP,AFlush,DAssist,NAsids=8 Apr 14 11:22:09 pfsense kernel: TSC: P-state invariant, performance statistics Apr 14 11:22:09 pfsense kernel: real memory = 4815060992 (4592 MB) Apr 14 11:22:09 pfsense kernel: avail memory = 4071936000 (3883 MB) Apr 14 11:22:09 pfsense kernel: Event timer "LAPIC" quality 600 Apr 14 11:22:09 pfsense kernel: ACPI APIC Table: <CORE COREBOOT> Apr 14 11:22:09 pfsense kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Apr 14 11:22:09 pfsense kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) Apr 14 11:22:09 pfsense kernel: ioapic1: Changing APIC ID to 5 Apr 14 11:22:09 pfsense kernel: ioapic0 <Version 2.1> irqs 0-23 on motherboard Apr 14 11:22:09 pfsense kernel: ioapic1 <Version 2.1> irqs 24-55 on motherboard Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #3 Launched! Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #2 Launched! Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #1 Launched! Apr 14 11:22:09 pfsense kernel: Timecounter "TSC" frequency 998149794 Hz quality 1000 Apr 14 11:22:09 pfsense kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Apr 14 11:22:09 pfsense kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80680430, 0) error 1 Apr 14 11:22:09 pfsense kernel: random: entropy device external interface Apr 14 11:22:09 pfsense kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Apr 14 11:22:09 pfsense kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806804e0, 0) error 1 Apr 14 11:22:09 pfsense kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Apr 14 11:22:09 pfsense kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80680590, 0) error 1 Apr 14 11:22:09 pfsense kernel: iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Apr 14 11:22:09 pfsense kernel: iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806a7460, 0) error 1 Apr 14 11:22:09 pfsense kernel: iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Apr 14 11:22:09 pfsense kernel: iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806a7510, 0) error 1 Apr 14 11:22:09 pfsense kernel: iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Apr 14 11:22:09 pfsense kernel: iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff806a75c0, 0) error 1 Apr 14 11:22:09 pfsense kernel: wlan: mac acl policy registered Apr 14 11:22:09 pfsense kernel: kbd0 at kbdmux0 Apr 14 11:22:09 pfsense kernel: netmap: loaded module Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff81209800, 0) error 19 Apr 14 11:22:09 pfsense kernel: nexus0 Apr 14 11:22:09 pfsense kernel: vtvga0: <VT VGA driver> on motherboard Apr 14 11:22:09 pfsense kernel: cryptosoft0: <software crypto> on motherboard Apr 14 11:22:09 pfsense kernel: padlock0: No ACE support. Apr 14 11:22:09 pfsense kernel: acpi0: <CORE COREBOOT> on motherboard Apr 14 11:22:09 pfsense kernel: acpi0: Power Button (fixed) Apr 14 11:22:09 pfsense kernel: cpu0: <ACPI CPU> on acpi0 Apr 14 11:22:09 pfsense kernel: cpu1: <ACPI CPU> on acpi0 Apr 14 11:22:09 pfsense kernel: cpu2: <ACPI CPU> on acpi0 Apr 14 11:22:09 pfsense kernel: cpu3: <ACPI CPU> on acpi0 Apr 14 11:22:09 pfsense kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Apr 14 11:22:09 pfsense kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s Apr 14 11:22:09 pfsense kernel: Event timer "RTC" frequency 32768 Hz quality 0 Apr 14 11:22:09 pfsense kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Apr 14 11:22:09 pfsense kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Apr 14 11:22:09 pfsense kernel: Event timer "i8254" frequency 1193182 Hz quality 100 Apr 14 11:22:09 pfsense kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 Apr 14 11:22:09 pfsense kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0 Apr 14 11:22:09 pfsense kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Apr 14 11:22:09 pfsense kernel: Timecounter "HPET" frequency 14318180 Hz quality 950 Apr 14 11:22:09 pfsense kernel: acpi_button0: <Power Button> on acpi0 Apr 14 11:22:09 pfsense kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 Apr 14 11:22:09 pfsense kernel: pci0: <ACPI PCI bus> on pcib0 Apr 14 11:22:09 pfsense kernel: pcib1: <ACPI PCI-PCI bridge> at device 2.2 on pci0 Apr 14 11:22:09 pfsense kernel: pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff Apr 14 11:22:09 pfsense kernel: pci1: <ACPI PCI bus> on pcib1 Apr 14 11:22:09 pfsense kernel: igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci1 Apr 14 11:22:09 pfsense kernel: igb0: Using MSIX interrupts with 5 vectors Apr 14 11:22:09 pfsense kernel: igb0: Ethernet address: 00:0d:b9:41:03:98 Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 0 to cpu 0 Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 1 to cpu 1 Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 2 to cpu 2 Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 3 to cpu 3 Apr 14 11:22:09 pfsense kernel: igb0: netmap queues/slots: TX 4/1024, RX 4/1024 Apr 14 11:22:09 pfsense kernel: pcib2: <ACPI PCI-PCI bridge> at device 2.3 on pci0 Apr 14 11:22:09 pfsense kernel: pci2: <ACPI PCI bus> on pcib2 Apr 14 11:22:09 pfsense kernel: igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2000-0x201f mem 0xfe700000-0xfe71ffff,0xfe720000-0xfe723fff at device 0.0 on pci2 Apr 14 11:22:09 pfsense kernel: igb1: Using MSIX interrupts with 5 vectors Apr 14 11:22:09 pfsense kernel: igb1: Ethernet address: 00:0d:b9:41:03:99 Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 0 to cpu 0 Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 1 to cpu 1 Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 2 to cpu 2 Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 3 to cpu 3 Apr 14 11:22:09 pfsense kernel: igb1: netmap queues/slots: TX 4/1024, RX 4/1024 Apr 14 11:22:09 pfsense kernel: pcib3: <ACPI PCI-PCI bridge> at device 2.4 on pci0 Apr 14 11:22:09 pfsense kernel: pci3: <ACPI PCI bus> on pcib3 Apr 14 11:22:09 pfsense kernel: igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x3000-0x301f mem 0xfe800000-0xfe81ffff,0xfe820000-0xfe823fff at device 0.0 on pci3 Apr 14 11:22:09 pfsense kernel: igb2: Using MSIX interrupts with 5 vectors Apr 14 11:22:09 pfsense kernel: igb2: Ethernet address: 00:0d:b9:41:03:9a Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 0 to cpu 0 Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 1 to cpu 1 Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 2 to cpu 2 Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 3 to cpu 3 Apr 14 11:22:09 pfsense kernel: igb2: netmap queues/slots: TX 4/1024, RX 4/1024 Apr 14 11:22:09 pfsense kernel: pci0: <encrypt/decrypt> at device 8.0 (no driver attached) Apr 14 11:22:09 pfsense kernel: xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb22000-0xfeb23fff at device 16.0 on pci0 Apr 14 11:22:09 pfsense kernel: xhci0: 32 bytes context size, 64-bit DMA Apr 14 11:22:09 pfsense kernel: xhci0: Unable to map MSI-X table Apr 14 11:22:09 pfsense kernel: usbus0 on xhci0 Apr 14 11:22:09 pfsense kernel: usbus0: 5.0Gbps Super Speed USB v3.0 Apr 14 11:22:09 pfsense kernel: ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xfeb25000-0xfeb253ff at device 17.0 on pci0 Apr 14 11:22:09 pfsense kernel: ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS Apr 14 11:22:09 pfsense kernel: ahcich0: <AHCI channel> at channel 0 on ahci0 Apr 14 11:22:09 pfsense kernel: ahcich1: <AHCI channel> at channel 1 on ahci0 Apr 14 11:22:09 pfsense kernel: ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb25400-0xfeb254ff at device 19.0 on pci0 Apr 14 11:22:09 pfsense kernel: usbus1: EHCI version 1.0 Apr 14 11:22:09 pfsense kernel: usbus1 on ehci0 Apr 14 11:22:09 pfsense kernel: usbus1: 480Mbps High Speed USB v2.0 Apr 14 11:22:09 pfsense kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0 Apr 14 11:22:09 pfsense kernel: isa0: <ISA bus> on isab0 Apr 14 11:22:09 pfsense kernel: sdhci_pci0: <Generic SD HCI> mem 0xfeb25500-0xfeb255ff at device 20.7 on pci0 Apr 14 11:22:09 pfsense kernel: sdhci_pci0: 1 slot(s) allocated Apr 14 11:22:09 pfsense kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 Apr 14 11:22:09 pfsense kernel: uart0: console (115200,n,8,1) Apr 14 11:22:09 pfsense kernel: orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0 Apr 14 11:22:09 pfsense kernel: ppc0: cannot reserve I/O port range Apr 14 11:22:09 pfsense kernel: uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0 Apr 14 11:22:09 pfsense kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Apr 14 11:22:09 pfsense kernel: Timecounters tick every 1.000 msec Apr 14 11:22:09 pfsense kernel: ugen0.1: <0x1022 XHCI root HUB> at usbus0 Apr 14 11:22:09 pfsense kernel: ugen1.1: <AMD EHCI root HUB> at usbus1 Apr 14 11:22:09 pfsense kernel: uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 Apr 14 11:22:09 pfsense kernel: uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 Apr 14 11:22:09 pfsense kernel: uhub0: 4 ports with 4 removable, self powered Apr 14 11:22:09 pfsense kernel: uhub1: 2 ports with 2 removable, self powered Apr 14 11:22:09 pfsense kernel: ugen1.2: <vendor 0x0438 product 0x7900> at usbus1 Apr 14 11:22:09 pfsense kernel: uhub2 on uhub1 Apr 14 11:22:09 pfsense kernel: uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1 Apr 14 11:22:09 pfsense kernel: uhub2: 4 ports with 4 removable, self powered Apr 14 11:22:09 pfsense kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 Apr 14 11:22:09 pfsense kernel: ada0: <Crucial CT250MX200SSD3 MU05> ACS-3 ATA SATA 3.x device Apr 14 11:22:09 pfsense kernel: ada0: Serial Number 154310E1FB49 Apr 14 11:22:09 pfsense kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) Apr 14 11:22:09 pfsense kernel: ada0: Command Queueing enabled Apr 14 11:22:09 pfsense kernel: ada0: 238475MB (488397168 512 byte sectors) Apr 14 11:22:09 pfsense kernel: Trying to mount root from ufs:/dev/ufsid/5c66f7dbb14e16a8 [rw]... Apr 14 11:22:09 pfsense kernel: WARNING: / was not properly dismounted Apr 14 11:22:09 pfsense kernel: random: unblocking device. Apr 14 11:22:09 pfsense kernel: CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU) Apr 14 11:22:09 pfsense kernel: Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 Apr 14 11:22:09 pfsense kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Apr 14 11:22:09 pfsense kernel: Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> Apr 14 11:22:09 pfsense kernel: AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> Apr 14 11:22:09 pfsense kernel: AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I> Apr 14 11:22:09 pfsense kernel: Structured Extended Features=0x8<BMI1> Apr 14 11:22:09 pfsense kernel: XSAVE Features=0x1<XSAVEOPT> Apr 14 11:22:09 pfsense kernel: SVM: NP,NRIP,AFlush,DAssist,NAsids=8 Apr 14 11:22:09 pfsense kernel: TSC: P-state invariant, performance statistics Apr 14 11:22:09 pfsense kernel: padlock0: No ACE support. Apr 14 11:22:09 pfsense kernel: aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard Apr 14 11:22:09 pfsense kernel: amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb5 Apr 14 11:22:09 pfsense kernel: vlan0: changing name to 'igb3.5' Apr 14 11:22:09 pfsense kernel: vlan1: changing name to 'igb3.1' Apr 14 11:22:09 pfsense kernel: vlan2: changing name to 'igb3.10' Apr 14 11:22:09 pfsense root: Restoring contents of RAM disk store... Apr 14 11:22:09 pfsense check_reload_status: starting sshd Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/dhcpleases.tgz Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/logs.tgz Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/rrd.tgz Apr 14 11:22:09 pfsense root: RAM disk restore succeeded. Apr 14 11:22:09 pfsense kernel: done. Apr 14 11:22:09 pfsense php-cgi: rc.bootup: Resyncing OpenVPN instances. Apr 14 11:22:09 pfsense kernel: Apr 14 11:22:09 pfsense kernel: tun1: changing name to 'ovpns1' Apr 14 11:22:10 pfsense kernel: ovpns1: link state changed to UP Apr 14 11:22:10 pfsense kernel: pflog0: promiscuous mode enabled Apr 14 11:22:10 pfsense check_reload_status: rc.newwanip starting ovpns1 Apr 14 11:22:10 pfsense xinetd[36596]: xinetd Version 2.3.15 started with libwrap loadavg options compiled in. Apr 14 11:22:10 pfsense kernel: .. Apr 14 11:22:10 pfsense xinetd[36596]: Started working: 48 available services Apr 14 11:22:10 pfsense kernel: .. Apr 14 11:22:10 pfsense sshd[48755]: Server listening on :: port 665. Apr 14 11:22:10 pfsense sshd[48755]: Server listening on 0.0.0.0 port 665. Apr 14 11:22:10 pfsense syslogd: Logging subprocess 49043 (exec /usr/local/sbin/sshguard) exited due to signal 15. Apr 14 11:22:10 pfsense kernel: .done. Apr 14 11:22:10 pfsense kernel: done. Apr 14 11:22:11 pfsense php-fpm[368]: /rc.newwanip: rc.newwanip: Info: starting on ovpns1. Apr 14 11:22:11 pfsense php-fpm[368]: /rc.newwanip: rc.newwanip: on (IP address: 10.255.254.1) (interface: []) (real interface: ovpns1). Apr 14 11:22:12 pfsense php-cgi: rc.bootup: Default gateway setting as default. Apr 14 11:22:12 pfsense rc.gateway_alarm[58622]: >>> Gateway alarm: Default (Addr:84.255.192.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Apr 14 11:22:12 pfsense check_reload_status: updating dyndns Default Apr 14 11:22:12 pfsense rc.gateway_alarm[59987]: >>> Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Apr 14 11:22:12 pfsense check_reload_status: Restarting ipsec tunnels Apr 14 11:22:12 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces Apr 14 11:22:12 pfsense check_reload_status: Reloading filter Apr 14 11:22:12 pfsense check_reload_status: updating dyndns WANGWv6 Apr 14 11:22:12 pfsense check_reload_status: Restarting ipsec tunnels Apr 14 11:22:12 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces Apr 14 11:22:12 pfsense check_reload_status: Reloading filter Apr 14 11:22:12 pfsense php-cgi: rc.bootup: Default gateway setting T-2 IPv6 as default. Apr 14 11:22:13 pfsense kernel: done. Apr 14 11:22:13 pfsense kernel: done. Apr 14 11:22:14 pfsense kernel: Apr 14 11:22:14 pfsense kernel: igb0: link state changed to UP Apr 14 11:22:14 pfsense check_reload_status: Linkup starting igb0 Apr 14 11:22:14 pfsense check_reload_status: Linkup starting igb1 Apr 14 11:22:14 pfsense kernel: igb1: link state changed to UP Apr 14 11:22:14 pfsense php-cgi: rc.bootup: NTPD is starting up. Apr 14 11:22:14 pfsense kernel: done. Apr 14 11:22:14 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:22:14 pfsense xinetd[36596]: Swapping defaults Apr 14 11:22:14 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:22:14 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:22:14 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:22:14 pfsense kernel: done. Apr 14 11:22:14 pfsense kernel: done. Apr 14 11:22:14 pfsense check_reload_status: Updating all dyndns Apr 14 11:22:15 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:22:15 pfsense xinetd[36596]: Swapping defaults Apr 14 11:22:15 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:22:15 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:22:15 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:22:16 pfsense kernel: . Apr 14 11:22:16 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:22:16 pfsense xinetd[36596]: Swapping defaults Apr 14 11:22:16 pfsense kernel: . Apr 14 11:22:16 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:22:16 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:22:16 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:22:16 pfsense kernel: .. Apr 14 11:22:17 pfsense kernel: .done. Apr 14 11:22:17 pfsense kernel: done Apr 14 11:22:19 pfsense kernel: done Apr 14 11:22:21 pfsense php-cgi: rc.bootup: Creating rrd update script Apr 14 11:22:21 pfsense kernel: done. Apr 14 11:22:22 pfsense root: /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one Apr 14 11:22:24 pfsense syslogd: exiting on signal 15 Apr 14 11:22:24 pfsense syslogd: kernel boot file is /boot/kernel/kernel Apr 14 11:22:24 pfsense kernel: done. Apr 14 11:22:25 pfsense php-fpm[367]: /rc.start_packages: Restarting/Starting all packages. Apr 14 11:22:26 pfsense apcupsd[77091]: apcupsd 3.14.14 (31 May 2016) freebsd startup succeeded Apr 14 11:22:26 pfsense apcupsd[77091]: NIS server startup succeeded Apr 14 11:22:40 pfsense apcupsd[77091]: Communications with UPS lost. Apr 14 11:22:58 pfsense rc.gateway_alarm[86673]: >>> Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:0 RTT:9.651ms RTTsd:56.200ms Loss:17%) Apr 14 11:22:58 pfsense check_reload_status: updating dyndns WANGWv6 Apr 14 11:22:58 pfsense check_reload_status: Restarting ipsec tunnels Apr 14 11:22:58 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces Apr 14 11:22:58 pfsense check_reload_status: Reloading filter Apr 14 11:23:00 pfsense php-fpm[73516]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGWv6. Apr 14 11:23:00 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:23:00 pfsense xinetd[36596]: Swapping defaults Apr 14 11:23:00 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:23:00 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:23:00 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:23:01 pfsense rc.gateway_alarm[30338]: >>> Gateway alarm: Default (Addr:84.255.192.1 Alarm:0 RTT:2.919ms RTTsd:3.327ms Loss:17%) Apr 14 11:23:01 pfsense check_reload_status: updating dyndns Default Apr 14 11:23:01 pfsense check_reload_status: Restarting ipsec tunnels Apr 14 11:23:01 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces Apr 14 11:23:01 pfsense check_reload_status: Reloading filter Apr 14 11:23:02 pfsense php-fpm[73516]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use Default. Apr 14 11:23:03 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:23:03 pfsense xinetd[36596]: Swapping defaults Apr 14 11:23:03 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:23:03 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:23:03 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:23:11 pfsense sshd[74432]: user root login class [preauth] Apr 14 11:23:11 pfsense sshd[74432]: user root login class [preauth] Apr 14 11:23:11 pfsense sshd[74432]: user root login class [preauth] Apr 14 11:23:11 pfsense sshd[74432]: user root login class [preauth] Apr 14 11:23:11 pfsense sshd[74432]: user root login class [preauth] Apr 14 11:23:11 pfsense sshd[74432]: Accepted publickey for root from 172.16.255.5 port 1075 ssh2: RSA SHA256:Yjy3YU+A17QAOM0xbL7kxDs6bLmSllQREeK49/Rz2t0 Apr 14 11:23:15 pfsense php-fpm[368]: /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing. Apr 14 11:23:15 pfsense check_reload_status: Reloading filter Apr 14 11:23:16 pfsense xinetd[36596]: Starting reconfiguration Apr 14 11:23:16 pfsense xinetd[36596]: Swapping defaults Apr 14 11:23:16 pfsense xinetd[36596]: readjusting service 19000-tcp […] Apr 14 11:23:16 pfsense xinetd[36596]: readjusting service 19047-tcp Apr 14 11:23:16 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services) Apr 14 11:23:40 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli SET ntopng.user.admin.password 'e6e19dec0e290613c39cc3c4de6542ff'' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Operation timed out' Apr 14 11:23:41 pfsense php-fpm[29042]: /index.php: Successful login for user 'admin' from: 172.16.255.5 (Local Database) Apr 14 11:24:42 pfsense check_reload_status: Syncing firewall Apr 14 11:24:42 pfsense apcupsd[77091]: apcupsd exiting, signal 15 Apr 14 11:24:42 pfsense apcupsd[77091]: apcupsd shutdown succeeded Apr 14 11:24:47 pfsense apcupsd[70362]: apcupsd 3.14.14 (31 May 2016) freebsd startup succeeded Apr 14 11:24:47 pfsense apcupsd[70362]: NIS server startup succeeded Apr 14 11:24:55 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli SET ntopng.user.admin.group administrator' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Operation timed out' Apr 14 11:25:12 pfsense check_reload_status: Syncing firewall Apr 14 11:25:13 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli save' returned exit code '15', the output was '' Apr 14 11:25:13 pfsense check_reload_status: Syncing firewall Apr 14 11:25:14 pfsense check_reload_status: Syncing firewall -
Usually the culprit for something like this is ReakTEK NICs. I took a look at the specs of your new board and couldn't find the exact model. The 2dX models all seem to use Intel NICs. Double check to see if that is the case. That's where I would start.
Got the specs at this link: https://www.pcengines.ch/apu2d2.htm
-
@ender_ said in pfSense randomly freezing:
Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:0 RTT:9.651ms RTTsd:56.200ms Loss:17%)
This catches my eye right away.. Can you try turning off IPv6 for a bit and test?
-
@tim-mcmanus The specs are here: https://www.pcengines.ch/apu2d4.htm - board has Intel NICs (igb driver). My board isn't really new - I bought it some time ago, but mostly used it for playing around (only problem I had was one WiFi NIC not being detected, but I removed those before installing pfSense).
@chpalmer I've been using IPv6 with pfSense since 2011 without any problems (also have it set up at work, though on different hardware). There's a similar entry for the IPv4 gateway:
Gateway alarm: Default (Addr:84.255.192.1 Alarm:0 RTT:2.919ms RTTsd:3.327ms Loss:17%)I can try disabling IPv6, but I'd rather not leave it like that (the previous time pfSense froze was on March 15th - as I said, these freezes did become far less frequent) -
@ender_ said in pfSense randomly freezing:
My pfSense box at home started randomly freezing in February
Well, geez, what do you expect at that time of the year?
When I have suspicious hardware, I fire up a utility called memtest and let it run for several hours. If it stays up, it's likely not a mom board issue.
That said, I've never seen pfSense lock up
-
@JKnott I left memtest running a few days after I moved pfSense to APU, it found no problems (and didn't lock up). The old board had ECC RAM, too, and there were no errors logged in IPMI.
