21.02 Sudden lockup
-
@mcury Can you provide the full output instead of just the provided snippet? I need everything, not just a chunk, to provide to our devs.
-
@kphillips Sure, but now you are asking for the system log, right?
Because the console output is complete there, from the boot start to finish, ok? -
@kphillips The last few posts in this thread point to pfBlockerNG as a possible culprit. I have been running since noon yesterday with pfBlockerNG. I was also running Snort, but as reported in a previous message, it aborted just after midnight. Other than the fact that Snort isn't running, pfsense seems to be functioning correctly. I'd reboot and connect to the console, but I have a whole lot of people in Zoom calls who will kill me if I disrupt what they're doing.
-
@rloeb I have neither PFBlocker or Snort packages installed and I am also impacted by this freezing WAN/LAN issue on my SG3100.
-
Confirms what I was trying to say. This is broader than an app.
-
@rloeb Our engineers have a half dozen different SG-3100s in various configurations running non-stop on 21.02. None of them have experienced lockups yet, so we're still trying to track down what is unique here. Full console output from a frozen device would be helpful.
-
@kphillips Kindly check and advise if this log can help you, or if is there something else that I can do
Feb 18 15:42:16 kernel ---<<BOOT>>--- Feb 18 15:42:16 kernel Copyright (c) 1992-2020 The FreeBSD Project. Feb 18 15:42:16 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Feb 18 15:42:16 kernel The Regents of the University of California. All rights reserved. Feb 18 15:42:16 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Feb 18 15:42:16 kernel FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense-SG-3100 arm Feb 18 15:42:16 kernel FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) Feb 18 15:42:16 kernel CPU: ARM Cortex-A9 r4p1 (ECO: 0x00000000) Feb 18 15:42:16 kernel CPU Features: Feb 18 15:42:16 kernel Multiprocessing, Thumb2, Security, VMSAv7, Coherent Walk Feb 18 15:42:16 kernel Optional instructions: Feb 18 15:42:16 kernel UMULL, SMULL, SIMD(ext) Feb 18 15:42:16 kernel LoUU:2 LoC:2 LoUIS:2 Feb 18 15:42:16 kernel Cache level 1: Feb 18 15:42:16 kernel 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc Feb 18 15:42:16 kernel 32KB/32B 4-way instruction cache Read-Alloc Feb 18 15:42:16 kernel real memory = 2147479552 (2047 MB) Feb 18 15:42:16 kernel avail memory = 2071031808 (1975 MB) Feb 18 15:42:16 kernel FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs Feb 18 15:42:16 kernel arc4random: no preloaded entropy cache Feb 18 15:42:16 kernel ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Feb 18 15:42:16 kernel ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc014deb0, 0) error 1 Feb 18 15:42:16 kernel ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Feb 18 15:42:16 kernel ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc014df60, 0) error 1 Feb 18 15:42:16 kernel iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Feb 18 15:42:16 kernel iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (iwi_bss_fw, 0xc0156f50, 0) error 1 Feb 18 15:42:16 kernel iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Feb 18 15:42:16 kernel iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (iwi_ibss_fw, 0xc0157000, 0) error 1 Feb 18 15:42:16 kernel iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Feb 18 15:42:16 kernel iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (iwi_monitor_fw, 0xc01570b0, 0) error 1 Feb 18 15:42:16 kernel ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Feb 18 15:42:16 kernel ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Feb 18 15:42:16 kernel module_register_init: MOD_LOAD (ipw_bss_fw, 0xc014de00, 0) error 1 Feb 18 15:42:16 kernel wlan: mac acl policy registered Feb 18 15:42:16 kernel random: entropy device external interface Feb 18 15:42:16 kernel [ath_hal] loaded Feb 18 15:42:16 kernel ofwbus0: <Open Firmware Device Tree> Feb 18 15:42:16 kernel simplebus0: <Flattened device tree simple bus> on ofwbus0 Feb 18 15:42:16 kernel simplebus1: <Flattened device tree simple bus> on simplebus0 Feb 18 15:42:16 kernel l2cache0: <PL310 L2 cache controller> mem 0x8000-0x8fff on simplebus1 Feb 18 15:42:16 kernel l2cache0: cannot allocate IRQ, not using interrupt Feb 18 15:42:16 kernel l2cache0: Part number: 0x3, release: 0x9 Feb 18 15:42:16 kernel l2cache0: L2 Cache enabled: 1024KB/32B 16 ways Feb 18 15:42:16 kernel gic0: <ARM Generic Interrupt Controller> mem 0xd000-0xdfff,0xc100-0xc1ff on simplebus1 Feb 18 15:42:16 kernel gic0: pn 0x39, arch 0x1, rev 0x2, implementer 0x43b irqs 192 Feb 18 15:42:16 kernel mpic0: <Marvell Integrated Interrupt Controller> mem 0x20a00-0x20ccf,0x21070-0x210c7 irq 17 on simplebus1 Feb 18 15:42:16 kernel gpio0: <Marvell Integrated GPIO Controller> mem 0x18100-0x1813f,0x181c0-0x181c7 irq 9,10,11,12 on simplebus1 Feb 18 15:42:16 kernel gpio0: 4 IRQs available Feb 18 15:42:16 kernel gpio0: Disable interrupts (offset = 0 + EDGE(0x18) Feb 18 15:42:16 kernel gpio0: Disable interrupts (offset = 0 + LEV(0x1C)) Feb 18 15:42:16 kernel gpio0: Setup intr 0 Feb 18 15:42:16 kernel gpio0: Setup intr 1 Feb 18 15:42:16 kernel gpio0: Setup intr 2 Feb 18 15:42:16 kernel gpio0: Setup intr 3 Feb 18 15:42:16 kernel gpio0: Clear int status (offset = 0) Feb 18 15:42:16 kernel gpiobus0: <GPIO bus> on gpio0 Feb 18 15:42:16 kernel gpio1: <Marvell Integrated GPIO Controller> mem 0x18140-0x1817f,0x181c8-0x181cf irq 13,14,15,16 on simplebus1 Feb 18 15:42:16 kernel gpio1: 4 IRQs available Feb 18 15:42:16 kernel gpio1: Disable interrupts (offset = 0 + EDGE(0x18) Feb 18 15:42:16 kernel gpio1: Disable interrupts (offset = 0 + LEV(0x1C)) Feb 18 15:42:16 kernel gpio1: Setup intr 0 Feb 18 15:42:16 kernel gpio1: Setup intr 1 Feb 18 15:42:16 kernel gpio1: Setup intr 2 Feb 18 15:42:16 kernel gpio1: Setup intr 3 Feb 18 15:42:16 kernel gpio1: Clear int status (offset = 0) Feb 18 15:42:16 kernel gpiobus1: <GPIO bus> on gpio1 Feb 18 15:42:16 kernel mp_tmr0: <ARM MPCore Timers> mem 0xc200-0xc21f irq 3 on simplebus1 Feb 18 15:42:16 kernel Timecounter "MPCore" frequency 800000000 Hz quality 800 Feb 18 15:42:16 kernel mp_tmr1: <ARM MPCore Timers> mem 0xc600-0xc61f irq 4 on simplebus1 Feb 18 15:42:16 kernel Event timer "MPCore" frequency 800000000 Hz quality 1000 Feb 18 15:42:16 kernel twsi0: <Marvell Integrated I2C Bus Controller> mem 0x11000-0x1101f irq 5 on simplebus1 Feb 18 15:42:16 kernel iicbus0: <OFW I2C bus> on twsi0 Feb 18 15:42:16 kernel iic0: <I2C generic I/O> on iicbus0 Feb 18 15:42:16 kernel gpio2: <NXP PCA9552 LED driver> at addr 0xc0 on iicbus0 Feb 18 15:42:16 kernel device_attach: gpio2 attach returned 6 Feb 18 15:42:16 kernel gpio2: <ISSI IS31FL3199 9 channel light effect LED driver> at addr 0xce on iicbus0 Feb 18 15:42:16 kernel gpiobus2: <OFW GPIO bus> on gpio2 Feb 18 15:42:16 kernel gpioc2: <GPIO controller> on gpio2 Feb 18 15:42:16 kernel uart0: <16550 or compatible> mem 0x12000-0x120ff irq 7 on simplebus1 Feb 18 15:42:16 kernel uart0: console (-1,n,8,1) Feb 18 15:42:16 kernel uart1: <16550 or compatible> mem 0x12100-0x121ff irq 8 on simplebus1 Feb 18 15:42:16 kernel gpioc0: <GPIO controller> on gpio0 Feb 18 15:42:16 kernel gpioc1: <GPIO controller> on gpio1 Feb 18 15:42:16 kernel wdt0: <Marvell Watchdog Timer> mem 0x20300-0x20333,0x20704-0x20707,0x18260-0x18263 irq 24,25 on simplebus1 Feb 18 15:42:16 kernel pmsu0: <Power Management Service Unit> mem 0x22000-0x22fff on simplebus1 Feb 18 15:42:16 kernel mvneta0: <NETA controller> mem 0x30000-0x33fff irq 26 on simplebus1 Feb 18 15:42:16 kernel mvneta0: version is 10 Feb 18 15:42:16 kernel mvneta0: Ethernet address: 00:08:a2:0c:c4:1b Feb 18 15:42:16 kernel miibus0: <MII bus> on mvneta0 Feb 18 15:42:16 kernel mv88e151x0: <Marvell 88E1512 Gigabit PHY> PHY 1 on miibus0 Feb 18 15:42:16 kernel mv88e151x0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto Feb 18 15:42:16 kernel mvneta1: <NETA controller> mem 0x34000-0x37fff irq 27 on simplebus1 Feb 18 15:42:16 kernel mvneta1: version is 10 Feb 18 15:42:16 kernel mvneta1: Ethernet address: 00:08:a2:0c:c4:1c Feb 18 15:42:16 kernel mdio0: <MDIO> on mvneta1 Feb 18 15:42:16 kernel e6000sw0: <Marvell 88E6141> on mdio0 Feb 18 15:42:16 kernel e6000sw0: single-chip addressing mode Feb 18 15:42:16 kernel e6000sw0: PHY at port 1 Feb 18 15:42:16 kernel miibus1: <MII bus> on e6000sw0 Feb 18 15:42:16 kernel e1000phy0: <Marvell 88E1000 Gigabit PHY> PHY 17 on miibus1 Feb 18 15:42:16 kernel e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto Feb 18 15:42:16 kernel e6000sw0: PHY at port 2 Feb 18 15:42:16 kernel miibus2: <MII bus> on e6000sw0 Feb 18 15:42:16 kernel e1000phy1: <Marvell 88E1000 Gigabit PHY> PHY 18 on miibus2 Feb 18 15:42:16 kernel e1000phy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto Feb 18 15:42:16 kernel e6000sw0: PHY at port 3 Feb 18 15:42:16 kernel miibus3: <MII bus> on e6000sw0 Feb 18 15:42:16 kernel e1000phy2: <Marvell 88E1000 Gigabit PHY> PHY 19 on miibus3 Feb 18 15:42:16 kernel e1000phy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto Feb 18 15:42:16 kernel e6000sw0: PHY at port 4 Feb 18 15:42:16 kernel miibus4: <MII bus> on e6000sw0 Feb 18 15:42:16 kernel e1000phy3: <Marvell 88E1000 Gigabit PHY> PHY 20 on miibus4 Feb 18 15:42:16 kernel e1000phy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto Feb 18 15:42:16 kernel e6000sw0: CPU port at 5 Feb 18 15:42:16 kernel e6000sw0: fixed port at 5 Feb 18 15:42:16 kernel e6000sw0: switch is ready. Feb 18 15:42:16 kernel etherswitch0: <Switch controller> on e6000sw0 Feb 18 15:42:16 kernel mvneta2: <NETA controller> mem 0x70000-0x73fff irq 28 on simplebus1 Feb 18 15:42:16 kernel mvneta2: version is 10 Feb 18 15:42:16 kernel mvneta2: Ethernet address: 00:08:a2:0c:c4:1d Feb 18 15:42:16 kernel miibus5: <MII bus> on mvneta2 Feb 18 15:42:16 kernel mv88e151x1: <Marvell 88E1512 Gigabit PHY> PHY 0 on miibus5 Feb 18 15:42:16 kernel mv88e151x1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto Feb 18 15:42:16 kernel ehci0: <Marvell Integrated USB 2.0 controller> mem 0x58000-0x584ff irq 29 on simplebus1 Feb 18 15:42:16 kernel usbus0: EHCI version 1.0 Feb 18 15:42:16 kernel usbus0 on ehci0 Feb 18 15:42:16 kernel cesa0: <Marvell Cryptographic Engine and Security Accelerator> mem 0x90000-0x9ffff irq 30,31 on simplebus1 Feb 18 15:42:16 kernel rtc0: <Marvell Integrated RTC> mem 0xa3800-0xa381f,0x184a0-0x184ab irq 32 on simplebus1 Feb 18 15:42:16 kernel rtc0: registered as a time-of-day clock, resolution 1.000000s Feb 18 15:42:16 kernel ahci0: <Marvell AHCI Controller> mem 0xa8000-0xa9fff irq 33 on simplebus1 Feb 18 15:42:16 kernel ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS Feb 18 15:42:16 kernel ahci0: quirks=0x200010<2CH,MRVL_SR_DEL> Feb 18 15:42:16 kernel ahcich0: <AHCI channel> at channel 0 on ahci0 Feb 18 15:42:16 kernel ahcich1: <AHCI channel> at channel 1 on ahci0 Feb 18 15:42:16 kernel armada_thermal0: <Armada380 Thermal Control> mem 0xe4078-0xe407b,0xe4070-0xe4077 on simplebus1 Feb 18 15:42:16 kernel sdhci_fdt0: <ARMADA38X SDHCI controller> mem 0xd8000-0xd8fff,0xdc000-0xdc0ff,0x18454-0x18457 irq 36 on simplebus1 Feb 18 15:42:16 kernel sdhci_fdt0: 1 slot(s) allocated Feb 18 15:42:16 kernel xhci0: <Marvell Integrated USB 3.0 controller> mem 0xf8000-0xfbfff,0xfc000-0xfffff irq 38 on simplebus1 Feb 18 15:42:16 kernel xhci0: 32 bytes context size, 32-bit DMA Feb 18 15:42:16 kernel usbus1 on xhci0 Feb 18 15:42:16 kernel cesa1: <Marvell Cryptographic Engine and Security Accelerator> mem 0x90000-0x9ffff irq 39,40 on simplebus1 Feb 18 15:42:16 kernel spi0: <Marvell SPI controller> mem 0xf001000000010600-0xf00100000001064f irq 1 on simplebus0 Feb 18 15:42:16 kernel cpulist0: <Open Firmware CPU Group> on ofwbus0 Feb 18 15:42:16 kernel cpu0: <Open Firmware CPU> on cpulist0 Feb 18 15:42:16 kernel cpu1: <Open Firmware CPU> on cpulist0 Feb 18 15:42:16 kernel cryptosoft0: <software crypto> Feb 18 15:42:16 kernel Timecounters tick every 1.000 msec Feb 18 15:42:16 kernel mvneta1: link state changed to UP Feb 18 15:42:16 kernel spibus0: <OFW SPI bus> on spi0 Feb 18 15:42:16 kernel mx25l0: <M25Pxx Flash Family> at cs 0 mode 0 on spibus0 Feb 18 15:42:16 kernel mx25l0: device type n25q128, size 16384K in 256 sectors of 64K, erase size 4K Feb 18 15:42:16 kernel Release APs Feb 18 15:42:16 kernel e6000sw0port1: link state changed to DOWN Feb 18 15:42:16 kernel usbus0: Feb 18 15:42:16 kernel e6000sw0port2: link state changed to DOWN Feb 18 15:42:16 kernel 480Mbps High Speed USB v2.0 Feb 18 15:42:16 kernel e6000sw0port3: link state changed to DOWN Feb 18 15:42:16 kernel usbus1: Feb 18 15:42:16 kernel e6000sw0port4: link state changed to DOWN Feb 18 15:42:16 kernel 5.0Gbps Super Speed USB v3.0 Feb 18 15:42:16 kernel ugen0.1: <Marvell EHCI root HUB> at usbus0 Feb 18 15:42:16 kernel uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 Feb 18 15:42:16 kernel ugen1.1: <Marvell XHCI root HUB> at usbus1 Feb 18 15:42:16 kernel uhub1: <Marvell XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 Feb 18 15:42:16 kernel arc4random: no preloaded entropy cache Feb 18 15:42:16 kernel Trying to mount root from ufs:/dev/diskid/DISK-D4EC8498s2a [rw,noatime]... Feb 18 15:42:16 kernel Root mount waiting for: usbus0 CAM usbus1 Feb 18 15:42:16 kernel uhub1: 2 ports with 2 removable, self powered Feb 18 15:42:16 kernel mmc0: <MMC/SD bus> on sdhci_fdt0 Feb 18 15:42:16 kernel mmcsd0: 8GB <MMCHC 008GE0 0.0 SN D4EC8498 MFG 06/2015 by 17 0x0000> at mmc0 50.0MHz/8bit/65535-block Feb 18 15:42:16 kernel mmcsd0boot0: 4MB partition 1 at mmcsd0 Feb 18 15:42:16 kernel mmcsd0boot1: 4MB partition 2 at mmcsd0 Feb 18 15:42:16 kernel mmcsd0rpmb: 4MB partition 3 at mmcsd0 Feb 18 15:42:16 kernel uhub0: 1 port with 1 removable, self powered Feb 18 15:42:16 kernel ugen1.2: <INNO TECH USB to Serial> at usbus1 Feb 18 15:42:16 kernel e6000sw0port2: link state changed to UP Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: Feb 18 15:42:16 kernel e6000sw0port1: link state changed to UP Feb 18 15:42:16 kernel CAM Feb 18 15:42:16 kernel e6000sw0port3: link state changed to UP Feb 18 15:42:16 kernel e6000sw0port4: link state changed to UP Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel Root mount waiting for: CAM Feb 18 15:42:16 kernel mountroot: waiting for device /dev/diskid/DISK-D4EC8498s2a... Feb 18 15:42:16 kernel random: unblocking device. Feb 18 15:42:16 kernel lo0: link state changed to UP Feb 18 15:42:16 check_reload_status 362 Linkup starting $e6000sw0port1 Feb 18 15:42:16 kernel e6000sw0port1: link state changed to DOWN Feb 18 15:42:16 check_reload_status 362 Linkup starting $e6000sw0port2 Feb 18 15:42:16 kernel e6000sw0port2: link state changed to DOWN Feb 18 15:42:16 check_reload_status 362 Linkup starting $e6000sw0port3 Feb 18 15:42:16 kernel e6000sw0port3: link state changed to DOWN Feb 18 15:42:16 check_reload_status 362 Linkup starting $e6000sw0port4 Feb 18 15:42:16 kernel e6000sw0port4: link state changed to DOWN Feb 18 15:42:16 kernel vlan0: changing name to 'mvneta1.100' Feb 18 15:42:16 kernel vlan1: changing name to 'mvneta1.10' Feb 18 15:42:16 check_reload_status 362 Linkup starting mvneta2 Feb 18 15:42:16 kernel vlan2: changing name to 'mvneta1.20' Feb 18 15:42:16 kernel mvneta2: link state changed to UP Feb 18 15:42:17 sshd 22090 Server listening on :: port 22. Feb 18 15:42:17 sshd 22090 Server listening on 0.0.0.0 port 22. Feb 18 15:42:17 check_reload_status 362 rc.newwanip starting mvneta2 Feb 18 15:42:17 php 366 rc.bootup: calling interface_dhcpv6_configure. Feb 18 15:42:17 php 366 rc.bootup: Accept router advertisements on interface mvneta2 Feb 18 15:42:17 php 366 rc.bootup: Starting rtsold process Feb 18 15:42:18 php-fpm 348 /rc.newwanip: rc.newwanip: Info: starting on mvneta2. Feb 18 15:42:18 php-fpm 348 /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: WAN[wan]) (real interface: mvneta2). Feb 18 15:42:18 php-fpm 348 /rc.newwanip: Accept router advertisements on interface mvneta2 Feb 18 15:42:18 php-fpm 348 /rc.newwanip: Starting rtsold process Feb 18 15:42:19 check_reload_status 362 Linkup starting $e6000sw0port2 Feb 18 15:42:19 kernel e6000sw0port2: link state changed to UP Feb 18 15:42:20 kernel done. Feb 18 15:42:20 check_reload_status 362 Linkup starting $e6000sw0port1 Feb 18 15:42:20 kernel e6000sw0port1: link state changed to UP Feb 18 15:42:20 kernel e6000sw0port3: link state changed to UP Feb 18 15:42:20 kernel e6000sw0port4: link state changed to UP Feb 18 15:42:20 check_reload_status 362 Linkup starting $e6000sw0port3 Feb 18 15:42:20 check_reload_status 362 Linkup starting $e6000sw0port4 Feb 18 15:42:20 php-fpm 348 /rc.newwanip: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_mvneta2.pid -M /var/etc/rtsold_mvneta2_script.sh -O /var/etc/rtsold_mvneta2_script.sh mvneta2' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Feb 18 15:42:21 check_reload_status 362 rc.newwanip starting mvneta2 Feb 18 15:42:21 php 366 rc.bootup: calling interface_dhcpv6_configure. Feb 18 15:42:21 php 366 rc.bootup: Accept router advertisements on interface mvneta2 Feb 18 15:42:21 php 366 rc.bootup: Killing running rtsold process Feb 18 15:42:22 php-fpm 349 /rc.newwanip: rc.newwanip: Info: starting on mvneta2. Feb 18 15:42:22 php-fpm 349 /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: WAN[wan]) (real interface: mvneta2). Feb 18 15:42:22 php-fpm 349 /rc.newwanip: Accept router advertisements on interface mvneta2 Feb 18 15:42:22 php-fpm 349 /rc.newwanip: Starting rtsold process Feb 18 15:42:23 php 366 rc.bootup: Starting rtsold process Feb 18 15:42:25 php 366 rc.bootup: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_mvneta2.pid -M /var/etc/rtsold_mvneta2_script.sh -O /var/etc/rtsold_mvneta2_script.sh mvneta2' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Feb 18 15:42:25 php 366 rc.bootup: Resyncing OpenVPN instances. Feb 18 15:42:25 kernel pflog0: promiscuous mode enabled Feb 18 15:42:25 rtsold 41791 Received RA specifying route for interface wan(mvneta2) Feb 18 15:42:25 rtsold 42392 Starting dhcp6 client for interface wan(mvneta2) Feb 18 15:42:27 php-fpm 348 /rc.newwanipv6: rc.newwanipv6: Info: starting on mvneta2. Feb 18 15:42:27 php-fpm 348 /rc.newwanipv6: rc.newwanipv6: on (IP address: X.X.X.X.X.X.X) (interface: wan) (real interface: mvneta2). Feb 18 15:42:28 php-fpm 348 /rc.newwanipv6: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Feb 18 15:42:28 php-fpm 348 /rc.newwanipv6: Default gateway setting Interface WAN_DHCP Gateway as default. Feb 18 15:42:28 php-fpm 348 /rc.newwanipv6: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 18 15:42:39 kernel pid 366 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 15:42:39 php-fpm 349 /rc.start_packages: Restarting/Starting all packages. Feb 18 15:42:39 php-fpm 349 /rc.start_packages: Starting service nut Feb 18 15:42:39 upsmon 64296 Startup successful Feb 18 15:42:40 blazer_usb 65265 Startup successful Feb 18 15:42:40 upsd 65608 listening on 192.168.255.249 port 3493 Feb 18 15:42:40 upsd 65608 listening on ::1 port 3493 Feb 18 15:42:40 upsd 65608 listening on 127.0.0.1 port 3493 Feb 18 15:42:40 upsd 65608 Connected to UPS [ups]: blazer_usb-ups Feb 18 15:42:40 upsd 65816 Startup successful Feb 18 15:42:42 upsmon 66974 Startup successful Feb 18 15:42:43 lighttpd_pfb 74974 [pfBlockerNG] DNSBL Webserver started Feb 18 15:42:44 upsd 81219 listening on 192.168.255.249 port 3493 Feb 18 15:42:44 upsd 81219 listening on ::1 port 3493 Feb 18 15:42:44 upsd 81219 listening on 127.0.0.1 port 3493 Feb 18 15:42:44 upsd 81219 Can't connect to UPS [ups] (blazer_usb-ups): Connection refused Feb 18 15:42:44 upsd 81333 Startup successful Feb 18 15:42:44 php-fpm 349 [pfBlockerNG] Starting firewall filter daemon Feb 18 15:42:44 php 79331 [pfBlockerNG] DNSBL parser daemon started Feb 18 15:42:44 root 89009 Bootup complete Feb 18 15:42:45 lighttpd_pfb 90034 [pfBlockerNG] DNSBL Webserver started Feb 18 15:42:45 php 88877 [pfBlockerNG] filterlog daemon started Feb 18 15:42:45 php 88309 [pfBlockerNG] filterlog daemon started Feb 18 15:42:46 php 91697 [pfBlockerNG] DNSBL parser daemon started Feb 18 15:42:46 upsd 81333 User local-monitor@::1 logged into UPS [ups] Feb 18 15:42:46 upsmon 67756 Poll UPS [ups] failed - Driver not connected Feb 18 15:42:46 upsmon 67756 Communications with UPS ups lost Feb 18 15:42:46 login 98062 login on ttyu0 as root Feb 18 15:42:48 upsd 81333 User monuser@192.168.255.252 logged into UPS [ups] Feb 18 15:42:48 blazer_usb 13450 Startup successful Feb 18 15:42:50 upsd 81333 Connected to UPS [ups]: blazer_usb-ups Feb 18 15:42:51 upsmon 67756 Communications with UPS ups established Feb 18 15:43:44 upsd 81333 User monuser2@192.168.255.251 logged into UPS [ups] Feb 18 15:46:10 login 29288 login on ttyu0 as root Feb 18 15:46:21 php 35698 rc.restart_webgui: Creating rrd update script Feb 18 15:46:33 php-fpm 348 /index.php: Successful login for user 'admin' from: 192.168.255.250 (Local Database Fallback) Feb 18 15:46:39 check_reload_status 362 Syncing firewall Feb 18 15:46:41 check_reload_status 362 Syncing firewall Feb 18 15:46:41 check_reload_status 362 Syncing firewall Feb 18 15:46:42 check_reload_status 362 Reloading filter Feb 18 15:46:42 php-fpm 3213 [pfBlockerNG] Stopping firewall filter daemon Feb 18 15:46:42 check_reload_status 362 Syncing firewall Feb 18 15:46:42 check_reload_status 362 Syncing firewall Feb 18 15:46:44 upsmon 67756 Poll UPS [ups] failed - Write error: Permission denied Feb 18 15:46:44 upsmon 67756 Communications with UPS ups lost Feb 18 15:46:49 upsd 81333 User local-monitor@::1 logged into UPS [ups] Feb 18 15:46:49 upsmon 67756 Communications with UPS ups established Feb 18 15:46:52 php 64330 rc.initial.reboot: Stopping all packages. Feb 18 15:46:52 upsmon 67756 Signal 15: exiting Feb 18 15:46:52 upsd 81333 User local-monitor@::1 logged out from UPS [ups] Feb 18 15:46:52 upsd 81333 mainloop: Interrupted system call Feb 18 15:46:52 upsd 81333 Signal 15: exiting Feb 18 15:46:52 blazer_usb 13450 Signal 15: exiting Feb 18 15:46:55 reboot 73891 rebooted by root Feb 18 15:46:55 syslogd exiting on signal 15 Feb 18 15:47:48 syslogd kernel boot file is /boot/kernel/kernel Feb 18 15:47:48 kernel ---<<BOOT>>--- Feb 18 15:47:48 kernel Copyright (c) 1992-2020 The FreeBSD Project. Feb 18 15:47:48 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Feb 18 15:47:48 kernel The Regents of the University of California. All rights reserved. Feb 18 15:47:48 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Feb 18 15:47:48 kernel FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense-SG-3100 arm -
@mcury Thank you! Was the last reboot you initiated after it had froze up?
-
Understood. Tough to find a phantom bug! Some of the reports above say that there wasn't any output when it locked up. What is a signal 11 and what creates it? Is the core dump from a signal 11 of any use? (I have no idea how to retrieve it, but I thought I'd ask.)
-
@kphillips Yes, this is the first reboot after installing pfblockerng-devel.
Installed the program, ran the wizard, and reboot, and the problem happens.
To fix, got into the console, pressed 11 to restart the GUI, connected to it and disabled pfblockerng.
Then reboot and it's all fine. -
@mcury OK thank you. So to confirm, the webConfigurator is inaccessible and you cannot pass traffic at all when this happens? What if you ping by IP address rather than by hostname from a device attached to the LAN? If you ping 8.8.8.8, for example, does it succeed but DNS queries fail? Since pfBlockerNG ties into unbound (DNS Resolver) I'm wondering if its crashing the DNS Resolver causing your issue.
-
If you're still able to connect to the command line at the console but nothing else try to connect out. Can you ping anything local on any interface?
What does
ifconfig -vashow?What about
netstat -rndo you still have valid routes?Or
etherswitchcfgis the switch still responding?Steve
-
@stephenw10 Important to note: Do this from the USB serial from the option 8 shell and after the device has locked up BEFORE rebooting it, please.
-
@kphillips said in 21.02 Sudden lockup:
@mcury OK thank you. So to confirm, the webConfigurator is inaccessible and you cannot pass traffic at all when this happens? What if you ping by IP address rather than by hostname from a device attached to the LAN? If you ping 8.8.8.8, for example, does it succeed but DNS queries fail? Since pfBlockerNG ties into unbound (DNS Resolver) I'm wondering if its crashing the DNS Resolver causing your issue.
Yes, webConfigurator is inaccessible and I cannot pass traffic.
Ping to 8.8.8.8 fails, or any other IP on the internet (it's not DNS).I'll perform the tests through the console.
I'll reproduce the problem, give me a few minutes.
1 - enable pfblockerng-devel again
2 - Reboot
3 - Test again the ping to 8.8.8.8, ping the pfsense LAN interface.Then, I'll perform the tests @stephenw10 asked
1 - perform ifconfig -va
2 - netstat -rn
3 - etherswitchcfg to check the switch -
@rloeb no output for me when it locked up.
Once it reboots I can see the normal boot process.@kphillips Anything I can capture to help out?
-
After enabling pfBlockerNG, rebooted the system and the problem didn't happen.
So I accessed the GUI, clicked in Firewall>pfBlockerNG>Update>Reload all>RunThen I rebooted again, and the problem happened.
It seems that it happens hourly after the cron update from pfblockerng, or if I manually force a reload or an update, and it's triggered during the first boot after that.- GUI is inaccessible.
- SSH from a device in LAN to pfsense LAN interface - Works
- SSH from VLAN WIFI to devices in another VLAN - Don't work.
- Ping to pfsense LAN interface from a PC in the LAN - Works.
- Ping from a PC in the LAN to 8.8.8.8 - Don't work.
ifconfig -va
21.02-RELEASE][root@pfsense.local.lan]/root: ifconfig -va mvneta0: flags=8a02<BROADCAST,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE> ether 00:08:a2:0c:c4:1b media: Ethernet autoselect (none) status: no carrier nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> mvneta1: flags=88a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500 description: mgmt options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> ether 00:08:a2:0c:c4:1c inet6 fe80::208:a2ff:fe0c:c41c%mvneta1 prefixlen 64 scopeid 0x2 inet 172.16.200.1 netmask 0xfffffff8 broadcast 172.16.200.7 media: Ethernet 2500Base-KX <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> mvneta2: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE> ether 00:08:a2:0c:c4:1d inet6 fe80::208:a2ff:fe0c:c41d%mvneta2 prefixlen 64 scopeid 0x8 inet6 X:X:X:X:X:X:X:X prefixlen 64 autoconf inet6 X:X:X::X prefixlen 128 inet X.X.X.X netmask 0xfffffc00 broadcast X.X.X.X inet 192.168.100.2 netmask 0xffffffff broadcast 192.168.100.2 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> enc0: flags=0<> metric 0 mtu 1536 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa inet 127.0.0.1 netmask 0xff000000 inet 10.10.10.1 netmask 0xffffffff groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=100<PROMISC> metric 0 mtu 33184 groups: pflog pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync mvneta1.100: flags=88843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500 description: lan options=3<RXCSUM,TXCSUM> ether 00:08:a2:0c:c4:1c inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.100 prefixlen 64 scopeid 0xd inet 192.168.255.249 netmask 0xfffffff8 broadcast 192.168.255.255 groups: vlan vlan: 100 vlanpcp: 0 parent interface: mvneta1 media: Ethernet Other <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> mvneta1.10: flags=88843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500 description: wifi options=3<RXCSUM,TXCSUM> ether 00:08:a2:0c:c4:1c inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.10 prefixlen 64 scopeid 0xe inet6 fe80::1:1%mvneta1.10 prefixlen 64 scopeid 0xe inet6 X:X:X:X:X:X:X:X prefixlen 64 inet 192.168.10.1 netmask 0xfffffff0 broadcast 192.168.10.15 groups: vlan vlan: 10 vlanpcp: 0 parent interface: mvneta1 media: Ethernet Other <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> mvneta1.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: guest options=3<RXCSUM,TXCSUM> ether 00:08:a2:0c:c4:1c inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.20 prefixlen 64 scopeid 0xf inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 groups: vlan vlan: 20 vlanpcp: 0 parent interface: mvneta1 media: Ethernet Other <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>netstat -rn
[21.02-RELEASE][root@pfsense.local.lan]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default X.X.X.X UGS mvneta2 10.10.10.1 link#10 UH lo0 127.0.0.1 link#10 UH lo0 172.16.200.0/29 link#2 U mvneta1 172.16.200.1 link#2 UHS lo0 X.X.X.X/22 link#8 U mvneta2 X.X.X.X link#8 UHS lo0 192.168.10.0/28 link#14 U mvneta1. 192.168.10.1 link#14 UHS lo0 192.168.20.0/24 link#15 U mvneta1. 192.168.20.1 link#15 UHS lo0 192.168.100.2 link#8 UHS lo0 192.168.100.2/32 link#8 U mvneta2 192.168.255.248/29 link#13 U mvneta1. 192.168.255.249 link#13 UHS lo0 Internet6: Destination Gateway Flags Netif Expire default fe80::2c8:8bff:fed8:3419%mvneta2 UG mvneta2 ::1 link#10 UH lo0 X:X:X::/64 link#8 U mvneta2 X:X:X::X link#8 UHS lo0 X:X:X:X:X:X:X:X link#8 UHS lo0 X:X:X:X::/64 link#14 U mvneta1. X:X:X:X:X:X:X:X link#14 UHS lo0 fe80::%mvneta1/64 link#2 U mvneta1 fe80::208:a2ff:fe0c:c41c%mvneta1 link#2 UHS lo0 fe80::%mvneta2/64 link#8 U mvneta2 fe80::208:a2ff:fe0c:c41d%mvneta2 link#8 UHS lo0 fe80::%lo0/64 link#10 U lo0 fe80::1%lo0 link#10 UHS lo0 fe80::%mvneta1.100/64 link#13 U mvneta1. fe80::208:a2ff:fe0c:c41c%mvneta1.100 link#13 UHS lo0 fe80::%mvneta1.10/64 link#14 U mvneta1. fe80::1:1%mvneta1.10 link#14 UHS lo0 fe80::208:a2ff:fe0c:c41c%mvneta1.10 link#14 UHS lo0 fe80::%mvneta1.20/64 link#15 U mvneta1. fe80::208:a2ff:fe0c:c41c%mvneta1.20 link#15 UHS lo0etherswitchcfg
21.02-RELEASE][root@pfsense.local.lan]/root: etherswitchcfg etherswitch0: VLAN mode: DOT1Q port1: pvid: 100 state=8<FORWARDING> flags=0<> media: Ethernet autoselect (1000baseT <full-duplex>) status: active port2: pvid: 100 state=8<FORWARDING> flags=0<> media: Ethernet autoselect (100baseTX <full-duplex>) status: active port3: pvid: 100 state=8<FORWARDING> flags=0<> media: Ethernet autoselect (1000baseT <full-duplex,master>) status: active port4: pvid: 1 state=8<FORWARDING> flags=0<> media: Ethernet autoselect (1000baseT <full-duplex>) status: active port5: pvid: 1 state=8<FORWARDING> flags=1<CPUPORT> media: Ethernet 2500Base-KX <full-duplex> status: active vlangroup0: vlan: 1 members 4,5 vlangroup1: vlan: 100 members 1,2,3,4t,5t vlangroup2: vlan: 10 members 4t,5t vlangroup3: vlan: 20 members 4t,5t -
@mcury Awesome find! It kinda makes sense as it was happening to me hourly. I have pfblockerng disable now. I do wonder if the corn schedule still kicks in?
EDIT: It does. I just had to reboot the box.
Looks like I have to find a way to disable the cron. -
+1 also experiencing the hang described here. Disabling pfBlockerNg helped.
-
Are you running the dev version of pfBlocker?
Does it 'lock up' as it's running the reload or once it's completed?
How large are you aliases/blocklists?
I have the non-dev package running and am not seeing that but only have limited dnsbl lists loaded.
Steve
-
@stephenw10 I am running pfBlockerNG-devel 3.0.0_10. The alias/blocklists are whatever is OOTB, I didn't customize anything.
By "lockup" I mean the LAN/WAN interfaces are completely unresponsive. Console access is fine.
P.S. I cross posted to this thread to make sure that @BBcan177 (the developer of pfBlockerNG) is aware.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.