pfSense randomly freezing



  • My pfSense box at home started randomly freezing in February. At first I thought the motherboard was dying (it was running on the original Supermicro A1SRi-2558F, with the problematic Atom), so I installed pfSense to PCEngines APU 2d3 and restored the configuration, but the random freezes are still happening.

    When this started happening, pfSense would freeze every few hours. At the time I didn't have it fully updated, so I did that first, but it hadn't helped much (the freezes became less frequent, but still happened). Switching to APU seemed to help at first, but then it also started freezing, although it only happens every few weeks (I also tested the Supermicro board with Linux, where it runs stable).

    There's nothing useful in logs when pfSense freezes - the entries just end, then resume after reset. I also left serial console open, and there was nothing printed to it when pfSense froze (there also was no response to any keypresses; it similarly didn't respond to any keypresses on the SuperMicro board when it froze).

    Any ideas how to further troubleshoot this?

    Here's system.log from the time of the freeze:

    Apr 13 23:32:12 pfsense apcupsd[77792]: Communications with UPS lost.
    Apr 13 23:42:16 pfsense apcupsd[77792]: Communications with UPS lost.
    Apr 13 23:52:19 pfsense apcupsd[77792]: Communications with UPS lost.
    Apr 14 11:22:09 pfsense syslogd: kernel boot file is /boot/kernel/kernel
    Apr 14 11:22:09 pfsense kernel: Copyright (c) 1992-2018 The FreeBSD Project.
    Apr 14 11:22:09 pfsense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    Apr 14 11:22:09 pfsense kernel: The Regents of the University of California. All rights reserved.
    Apr 14 11:22:09 pfsense kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
    Apr 14 11:22:09 pfsense kernel: FreeBSD 11.2-RELEASE-p6 #3 518496b29ae(RELENG_2_4_4): Wed Dec 12 07:41:44 EST 2018
    Apr 14 11:22:09 pfsense kernel: root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64
    Apr 14 11:22:09 pfsense kernel: FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
    Apr 14 11:22:09 pfsense kernel: VT(vga): resolution 640x480
    Apr 14 11:22:09 pfsense kernel: CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU)
    Apr 14 11:22:09 pfsense kernel: Origin="AuthenticAMD"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1
    Apr 14 11:22:09 pfsense kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
    Apr 14 11:22:09 pfsense kernel: Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
    Apr 14 11:22:09 pfsense kernel: AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
    Apr 14 11:22:09 pfsense kernel: AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I>
    Apr 14 11:22:09 pfsense kernel: Structured Extended Features=0x8<BMI1>
    Apr 14 11:22:09 pfsense kernel: XSAVE Features=0x1<XSAVEOPT>
    Apr 14 11:22:09 pfsense kernel: SVM: NP,NRIP,AFlush,DAssist,NAsids=8
    Apr 14 11:22:09 pfsense kernel: TSC: P-state invariant, performance statistics
    Apr 14 11:22:09 pfsense kernel: real memory  = 4815060992 (4592 MB)
    Apr 14 11:22:09 pfsense kernel: avail memory = 4071936000 (3883 MB)
    Apr 14 11:22:09 pfsense kernel: Event timer "LAPIC" quality 600
    Apr 14 11:22:09 pfsense kernel: ACPI APIC Table: <CORE   COREBOOT>
    Apr 14 11:22:09 pfsense kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    Apr 14 11:22:09 pfsense kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
    Apr 14 11:22:09 pfsense kernel: ioapic1: Changing APIC ID to 5
    Apr 14 11:22:09 pfsense kernel: ioapic0 <Version 2.1> irqs 0-23 on motherboard
    Apr 14 11:22:09 pfsense kernel: ioapic1 <Version 2.1> irqs 24-55 on motherboard
    Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #3 Launched!
    Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #2 Launched!
    Apr 14 11:22:09 pfsense kernel: SMP: AP CPU #1 Launched!
    Apr 14 11:22:09 pfsense kernel: Timecounter "TSC" frequency 998149794 Hz quality 1000
    Apr 14 11:22:09 pfsense kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Apr 14 11:22:09 pfsense kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80680430, 0) error 1
    Apr 14 11:22:09 pfsense kernel: random: entropy device external interface
    Apr 14 11:22:09 pfsense kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Apr 14 11:22:09 pfsense kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806804e0, 0) error 1
    Apr 14 11:22:09 pfsense kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Apr 14 11:22:09 pfsense kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80680590, 0) error 1
    Apr 14 11:22:09 pfsense kernel: iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Apr 14 11:22:09 pfsense kernel: iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806a7460, 0) error 1
    Apr 14 11:22:09 pfsense kernel: iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Apr 14 11:22:09 pfsense kernel: iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806a7510, 0) error 1
    Apr 14 11:22:09 pfsense kernel: iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Apr 14 11:22:09 pfsense kernel: iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff806a75c0, 0) error 1
    Apr 14 11:22:09 pfsense kernel: wlan: mac acl policy registered
    Apr 14 11:22:09 pfsense kernel: kbd0 at kbdmux0
    Apr 14 11:22:09 pfsense kernel: netmap: loaded module
    Apr 14 11:22:09 pfsense kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff81209800, 0) error 19
    Apr 14 11:22:09 pfsense kernel: nexus0
    Apr 14 11:22:09 pfsense kernel: vtvga0: <VT VGA driver> on motherboard
    Apr 14 11:22:09 pfsense kernel: cryptosoft0: <software crypto> on motherboard
    Apr 14 11:22:09 pfsense kernel: padlock0: No ACE support.
    Apr 14 11:22:09 pfsense kernel: acpi0: <CORE COREBOOT> on motherboard
    Apr 14 11:22:09 pfsense kernel: acpi0: Power Button (fixed)
    Apr 14 11:22:09 pfsense kernel: cpu0: <ACPI CPU> on acpi0
    Apr 14 11:22:09 pfsense kernel: cpu1: <ACPI CPU> on acpi0
    Apr 14 11:22:09 pfsense kernel: cpu2: <ACPI CPU> on acpi0
    Apr 14 11:22:09 pfsense kernel: cpu3: <ACPI CPU> on acpi0
    Apr 14 11:22:09 pfsense kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
    Apr 14 11:22:09 pfsense kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s
    Apr 14 11:22:09 pfsense kernel: Event timer "RTC" frequency 32768 Hz quality 0
    Apr 14 11:22:09 pfsense kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
    Apr 14 11:22:09 pfsense kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
    Apr 14 11:22:09 pfsense kernel: Event timer "i8254" frequency 1193182 Hz quality 100
    Apr 14 11:22:09 pfsense kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    Apr 14 11:22:09 pfsense kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0
    Apr 14 11:22:09 pfsense kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
    Apr 14 11:22:09 pfsense kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
    Apr 14 11:22:09 pfsense kernel: acpi_button0: <Power Button> on acpi0
    Apr 14 11:22:09 pfsense kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
    Apr 14 11:22:09 pfsense kernel: pci0: <ACPI PCI bus> on pcib0
    Apr 14 11:22:09 pfsense kernel: pcib1: <ACPI PCI-PCI bridge> at device 2.2 on pci0
    Apr 14 11:22:09 pfsense kernel: pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff
    Apr 14 11:22:09 pfsense kernel: pci1: <ACPI PCI bus> on pcib1
    Apr 14 11:22:09 pfsense kernel: igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci1
    Apr 14 11:22:09 pfsense kernel: igb0: Using MSIX interrupts with 5 vectors
    Apr 14 11:22:09 pfsense kernel: igb0: Ethernet address: 00:0d:b9:41:03:98
    Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 0 to cpu 0
    Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 1 to cpu 1
    Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 2 to cpu 2
    Apr 14 11:22:09 pfsense kernel: igb0: Bound queue 3 to cpu 3
    Apr 14 11:22:09 pfsense kernel: igb0: netmap queues/slots: TX 4/1024, RX 4/1024
    Apr 14 11:22:09 pfsense kernel: pcib2: <ACPI PCI-PCI bridge> at device 2.3 on pci0
    Apr 14 11:22:09 pfsense kernel: pci2: <ACPI PCI bus> on pcib2
    Apr 14 11:22:09 pfsense kernel: igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2000-0x201f mem 0xfe700000-0xfe71ffff,0xfe720000-0xfe723fff at device 0.0 on pci2
    Apr 14 11:22:09 pfsense kernel: igb1: Using MSIX interrupts with 5 vectors
    Apr 14 11:22:09 pfsense kernel: igb1: Ethernet address: 00:0d:b9:41:03:99
    Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 0 to cpu 0
    Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 1 to cpu 1
    Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 2 to cpu 2
    Apr 14 11:22:09 pfsense kernel: igb1: Bound queue 3 to cpu 3
    Apr 14 11:22:09 pfsense kernel: igb1: netmap queues/slots: TX 4/1024, RX 4/1024
    Apr 14 11:22:09 pfsense kernel: pcib3: <ACPI PCI-PCI bridge> at device 2.4 on pci0
    Apr 14 11:22:09 pfsense kernel: pci3: <ACPI PCI bus> on pcib3
    Apr 14 11:22:09 pfsense kernel: igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x3000-0x301f mem 0xfe800000-0xfe81ffff,0xfe820000-0xfe823fff at device 0.0 on pci3
    Apr 14 11:22:09 pfsense kernel: igb2: Using MSIX interrupts with 5 vectors
    Apr 14 11:22:09 pfsense kernel: igb2: Ethernet address: 00:0d:b9:41:03:9a
    Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 0 to cpu 0
    Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 1 to cpu 1
    Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 2 to cpu 2
    Apr 14 11:22:09 pfsense kernel: igb2: Bound queue 3 to cpu 3
    Apr 14 11:22:09 pfsense kernel: igb2: netmap queues/slots: TX 4/1024, RX 4/1024
    Apr 14 11:22:09 pfsense kernel: pci0: <encrypt/decrypt> at device 8.0 (no driver attached)
    Apr 14 11:22:09 pfsense kernel: xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb22000-0xfeb23fff at device 16.0 on pci0
    Apr 14 11:22:09 pfsense kernel: xhci0: 32 bytes context size, 64-bit DMA
    Apr 14 11:22:09 pfsense kernel: xhci0: Unable to map MSI-X table
    Apr 14 11:22:09 pfsense kernel: usbus0 on xhci0
    Apr 14 11:22:09 pfsense kernel: usbus0: 5.0Gbps Super Speed USB v3.0
    Apr 14 11:22:09 pfsense kernel: ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xfeb25000-0xfeb253ff at device 17.0 on pci0
    Apr 14 11:22:09 pfsense kernel: ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS
    Apr 14 11:22:09 pfsense kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
    Apr 14 11:22:09 pfsense kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
    Apr 14 11:22:09 pfsense kernel: ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb25400-0xfeb254ff at device 19.0 on pci0
    Apr 14 11:22:09 pfsense kernel: usbus1: EHCI version 1.0
    Apr 14 11:22:09 pfsense kernel: usbus1 on ehci0
    Apr 14 11:22:09 pfsense kernel: usbus1: 480Mbps High Speed USB v2.0
    Apr 14 11:22:09 pfsense kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0
    Apr 14 11:22:09 pfsense kernel: isa0: <ISA bus> on isab0
    Apr 14 11:22:09 pfsense kernel: sdhci_pci0: <Generic SD HCI> mem 0xfeb25500-0xfeb255ff at device 20.7 on pci0
    Apr 14 11:22:09 pfsense kernel: sdhci_pci0: 1 slot(s) allocated
    Apr 14 11:22:09 pfsense kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    Apr 14 11:22:09 pfsense kernel: uart0: console (115200,n,8,1)
    Apr 14 11:22:09 pfsense kernel: orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
    Apr 14 11:22:09 pfsense kernel: ppc0: cannot reserve I/O port range
    Apr 14 11:22:09 pfsense kernel: uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
    Apr 14 11:22:09 pfsense kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0
    Apr 14 11:22:09 pfsense kernel: Timecounters tick every 1.000 msec
    Apr 14 11:22:09 pfsense kernel: ugen0.1: <0x1022 XHCI root HUB> at usbus0
    Apr 14 11:22:09 pfsense kernel: ugen1.1: <AMD EHCI root HUB> at usbus1
    Apr 14 11:22:09 pfsense kernel: uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    Apr 14 11:22:09 pfsense kernel: uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
    Apr 14 11:22:09 pfsense kernel: uhub0: 4 ports with 4 removable, self powered
    Apr 14 11:22:09 pfsense kernel: uhub1: 2 ports with 2 removable, self powered
    Apr 14 11:22:09 pfsense kernel: ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
    Apr 14 11:22:09 pfsense kernel: uhub2 on uhub1
    Apr 14 11:22:09 pfsense kernel: uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1
    Apr 14 11:22:09 pfsense kernel: uhub2: 4 ports with 4 removable, self powered
    Apr 14 11:22:09 pfsense kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    Apr 14 11:22:09 pfsense kernel: ada0: <Crucial CT250MX200SSD3 MU05> ACS-3 ATA SATA 3.x device
    Apr 14 11:22:09 pfsense kernel: ada0: Serial Number 154310E1FB49
    Apr 14 11:22:09 pfsense kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    Apr 14 11:22:09 pfsense kernel: ada0: Command Queueing enabled
    Apr 14 11:22:09 pfsense kernel: ada0: 238475MB (488397168 512 byte sectors)
    Apr 14 11:22:09 pfsense kernel: Trying to mount root from ufs:/dev/ufsid/5c66f7dbb14e16a8 [rw]...
    Apr 14 11:22:09 pfsense kernel: WARNING: / was not properly dismounted
    Apr 14 11:22:09 pfsense kernel: random: unblocking device.
    Apr 14 11:22:09 pfsense kernel: CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU)
    Apr 14 11:22:09 pfsense kernel: Origin="AuthenticAMD"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1
    Apr 14 11:22:09 pfsense kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
    Apr 14 11:22:09 pfsense kernel: Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
    Apr 14 11:22:09 pfsense kernel: AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
    Apr 14 11:22:09 pfsense kernel: AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I>
    Apr 14 11:22:09 pfsense kernel: Structured Extended Features=0x8<BMI1>
    Apr 14 11:22:09 pfsense kernel: XSAVE Features=0x1<XSAVEOPT>
    Apr 14 11:22:09 pfsense kernel: SVM: NP,NRIP,AFlush,DAssist,NAsids=8
    Apr 14 11:22:09 pfsense kernel: TSC: P-state invariant, performance statistics
    Apr 14 11:22:09 pfsense kernel: padlock0: No ACE support.
    Apr 14 11:22:09 pfsense kernel: aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
    Apr 14 11:22:09 pfsense kernel: amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb5
    Apr 14 11:22:09 pfsense kernel: vlan0: changing name to 'igb3.5'
    Apr 14 11:22:09 pfsense kernel: vlan1: changing name to 'igb3.1'
    Apr 14 11:22:09 pfsense kernel: vlan2: changing name to 'igb3.10'
    Apr 14 11:22:09 pfsense root: Restoring contents of RAM disk store...
    Apr 14 11:22:09 pfsense check_reload_status: starting sshd
    Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/dhcpleases.tgz
    Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/logs.tgz
    Apr 14 11:22:09 pfsense root: RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/rrd.tgz
    Apr 14 11:22:09 pfsense root: RAM disk restore succeeded.
    Apr 14 11:22:09 pfsense kernel: done.
    Apr 14 11:22:09 pfsense php-cgi: rc.bootup: Resyncing OpenVPN instances.
    Apr 14 11:22:09 pfsense kernel:
    Apr 14 11:22:09 pfsense kernel: tun1: changing name to 'ovpns1'
    Apr 14 11:22:10 pfsense kernel: ovpns1: link state changed to UP
    Apr 14 11:22:10 pfsense kernel: pflog0: promiscuous mode enabled
    Apr 14 11:22:10 pfsense check_reload_status: rc.newwanip starting ovpns1
    Apr 14 11:22:10 pfsense xinetd[36596]: xinetd Version 2.3.15 started with libwrap loadavg options compiled in.
    Apr 14 11:22:10 pfsense kernel: ..
    Apr 14 11:22:10 pfsense xinetd[36596]: Started working: 48 available services
    Apr 14 11:22:10 pfsense kernel: ..
    Apr 14 11:22:10 pfsense sshd[48755]: Server listening on :: port 665.
    Apr 14 11:22:10 pfsense sshd[48755]: Server listening on 0.0.0.0 port 665.
    Apr 14 11:22:10 pfsense syslogd: Logging subprocess 49043 (exec /usr/local/sbin/sshguard) exited due to signal 15.
    Apr 14 11:22:10 pfsense kernel: .done.
    Apr 14 11:22:10 pfsense kernel: done.
    Apr 14 11:22:11 pfsense php-fpm[368]: /rc.newwanip: rc.newwanip: Info: starting on ovpns1.
    Apr 14 11:22:11 pfsense php-fpm[368]: /rc.newwanip: rc.newwanip: on (IP address: 10.255.254.1) (interface: []) (real interface: ovpns1).
    Apr 14 11:22:12 pfsense php-cgi: rc.bootup: Default gateway setting  as default.
    Apr 14 11:22:12 pfsense rc.gateway_alarm[58622]: >>> Gateway alarm: Default (Addr:84.255.192.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
    Apr 14 11:22:12 pfsense check_reload_status: updating dyndns Default
    Apr 14 11:22:12 pfsense rc.gateway_alarm[59987]: >>> Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
    Apr 14 11:22:12 pfsense check_reload_status: Restarting ipsec tunnels
    Apr 14 11:22:12 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces
    Apr 14 11:22:12 pfsense check_reload_status: Reloading filter
    Apr 14 11:22:12 pfsense check_reload_status: updating dyndns WANGWv6
    Apr 14 11:22:12 pfsense check_reload_status: Restarting ipsec tunnels
    Apr 14 11:22:12 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces
    Apr 14 11:22:12 pfsense check_reload_status: Reloading filter
    Apr 14 11:22:12 pfsense php-cgi: rc.bootup: Default gateway setting T-2 IPv6 as default.
    Apr 14 11:22:13 pfsense kernel: done.
    Apr 14 11:22:13 pfsense kernel: done.
    Apr 14 11:22:14 pfsense kernel:
    Apr 14 11:22:14 pfsense kernel: igb0: link state changed to UP
    Apr 14 11:22:14 pfsense check_reload_status: Linkup starting igb0
    Apr 14 11:22:14 pfsense check_reload_status: Linkup starting igb1
    Apr 14 11:22:14 pfsense kernel: igb1: link state changed to UP
    Apr 14 11:22:14 pfsense php-cgi: rc.bootup: NTPD is starting up.
    Apr 14 11:22:14 pfsense kernel: done.
    Apr 14 11:22:14 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:22:14 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:22:14 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:22:14 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:22:14 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:22:14 pfsense kernel: done.
    Apr 14 11:22:14 pfsense kernel: done.
    Apr 14 11:22:14 pfsense check_reload_status: Updating all dyndns
    Apr 14 11:22:15 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:22:15 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:22:15 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:22:15 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:22:15 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:22:16 pfsense kernel: .
    Apr 14 11:22:16 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:22:16 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:22:16 pfsense kernel: .
    Apr 14 11:22:16 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:22:16 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:22:16 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:22:16 pfsense kernel: ..
    Apr 14 11:22:17 pfsense kernel: .done.
    Apr 14 11:22:17 pfsense kernel: done
    Apr 14 11:22:19 pfsense kernel: done
    Apr 14 11:22:21 pfsense php-cgi: rc.bootup: Creating rrd update script
    Apr 14 11:22:21 pfsense kernel: done.
    Apr 14 11:22:22 pfsense root: /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
    Apr 14 11:22:24 pfsense syslogd: exiting on signal 15
    Apr 14 11:22:24 pfsense syslogd: kernel boot file is /boot/kernel/kernel
    Apr 14 11:22:24 pfsense kernel: done.
    Apr 14 11:22:25 pfsense php-fpm[367]: /rc.start_packages: Restarting/Starting all packages.
    Apr 14 11:22:26 pfsense apcupsd[77091]: apcupsd 3.14.14 (31 May 2016) freebsd startup succeeded
    Apr 14 11:22:26 pfsense apcupsd[77091]: NIS server startup succeeded
    Apr 14 11:22:40 pfsense apcupsd[77091]: Communications with UPS lost.
    Apr 14 11:22:58 pfsense rc.gateway_alarm[86673]: >>> Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:0 RTT:9.651ms RTTsd:56.200ms Loss:17%)
    Apr 14 11:22:58 pfsense check_reload_status: updating dyndns WANGWv6
    Apr 14 11:22:58 pfsense check_reload_status: Restarting ipsec tunnels
    Apr 14 11:22:58 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces
    Apr 14 11:22:58 pfsense check_reload_status: Reloading filter
    Apr 14 11:23:00 pfsense php-fpm[73516]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGWv6.
    Apr 14 11:23:00 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:23:00 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:23:00 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:23:00 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:23:00 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:23:01 pfsense rc.gateway_alarm[30338]: >>> Gateway alarm: Default (Addr:84.255.192.1 Alarm:0 RTT:2.919ms RTTsd:3.327ms Loss:17%)
    Apr 14 11:23:01 pfsense check_reload_status: updating dyndns Default
    Apr 14 11:23:01 pfsense check_reload_status: Restarting ipsec tunnels
    Apr 14 11:23:01 pfsense check_reload_status: Restarting OpenVPN tunnels/interfaces
    Apr 14 11:23:01 pfsense check_reload_status: Reloading filter
    Apr 14 11:23:02 pfsense php-fpm[73516]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use Default.
    Apr 14 11:23:03 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:23:03 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:23:03 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:23:03 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:23:03 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:23:11 pfsense sshd[74432]: user root login class  [preauth]
    Apr 14 11:23:11 pfsense sshd[74432]: user root login class  [preauth]
    Apr 14 11:23:11 pfsense sshd[74432]: user root login class  [preauth]
    Apr 14 11:23:11 pfsense sshd[74432]: user root login class  [preauth]
    Apr 14 11:23:11 pfsense sshd[74432]: user root login class  [preauth]
    Apr 14 11:23:11 pfsense sshd[74432]: Accepted publickey for root from 172.16.255.5 port 1075 ssh2: RSA SHA256:Yjy3YU+A17QAOM0xbL7kxDs6bLmSllQREeK49/Rz2t0
    Apr 14 11:23:15 pfsense php-fpm[368]: /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Apr 14 11:23:15 pfsense check_reload_status: Reloading filter
    Apr 14 11:23:16 pfsense xinetd[36596]: Starting reconfiguration
    Apr 14 11:23:16 pfsense xinetd[36596]: Swapping defaults
    Apr 14 11:23:16 pfsense xinetd[36596]: readjusting service 19000-tcp
    […]
    Apr 14 11:23:16 pfsense xinetd[36596]: readjusting service 19047-tcp
    Apr 14 11:23:16 pfsense xinetd[36596]: Reconfigured: new=0 old=48 dropped=0 (services)
    Apr 14 11:23:40 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli SET ntopng.user.admin.password 'e6e19dec0e290613c39cc3c4de6542ff'' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Operation timed out'
    Apr 14 11:23:41 pfsense php-fpm[29042]: /index.php: Successful login for user 'admin' from: 172.16.255.5 (Local Database)
    Apr 14 11:24:42 pfsense check_reload_status: Syncing firewall
    Apr 14 11:24:42 pfsense apcupsd[77091]: apcupsd exiting, signal 15
    Apr 14 11:24:42 pfsense apcupsd[77091]: apcupsd shutdown succeeded
    Apr 14 11:24:47 pfsense apcupsd[70362]: apcupsd 3.14.14 (31 May 2016) freebsd startup succeeded
    Apr 14 11:24:47 pfsense apcupsd[70362]: NIS server startup succeeded
    Apr 14 11:24:55 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli SET ntopng.user.admin.group administrator' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Operation timed out'
    Apr 14 11:25:12 pfsense check_reload_status: Syncing firewall
    Apr 14 11:25:13 pfsense php-fpm[367]: /rc.start_packages: The command '/usr/local/bin/redis-cli save' returned exit code '15', the output was ''
    Apr 14 11:25:13 pfsense check_reload_status: Syncing firewall
    Apr 14 11:25:14 pfsense check_reload_status: Syncing firewall
    


  • Usually the culprit for something like this is ReakTEK NICs. I took a look at the specs of your new board and couldn't find the exact model. The 2dX models all seem to use Intel NICs. Double check to see if that is the case. That's where I would start.

    Got the specs at this link: https://www.pcengines.ch/apu2d2.htm



  • @ender_ said in pfSense randomly freezing:

    Gateway alarm: WANGWv6 (Addr:2a01:260:4094::5 Alarm:0 RTT:9.651ms RTTsd:56.200ms Loss:17%)

    This catches my eye right away.. Can you try turning off IPv6 for a bit and test?



  • @tim-mcmanus The specs are here: https://www.pcengines.ch/apu2d4.htm - board has Intel NICs (igb driver). My board isn't really new - I bought it some time ago, but mostly used it for playing around (only problem I had was one WiFi NIC not being detected, but I removed those before installing pfSense).

    @chpalmer I've been using IPv6 with pfSense since 2011 without any problems (also have it set up at work, though on different hardware). There's a similar entry for the IPv4 gateway: Gateway alarm: Default (Addr:84.255.192.1 Alarm:0 RTT:2.919ms RTTsd:3.327ms Loss:17%) I can try disabling IPv6, but I'd rather not leave it like that (the previous time pfSense froze was on March 15th - as I said, these freezes did become far less frequent)



  • @ender_ said in pfSense randomly freezing:

    My pfSense box at home started randomly freezing in February

    Well, geez, what do you expect at that time of the year? 😝

    When I have suspicious hardware, I fire up a utility called memtest and let it run for several hours. If it stays up, it's likely not a mom board issue.

    That said, I've never seen pfSense lock up



  • @JKnott I left memtest running a few days after I moved pfSense to APU, it found no problems (and didn't lock up). The old board had ECC RAM, too, and there were no errors logged in IPMI.


Log in to reply