21.02 Sudden lockup
-
I am waiting for it to happen again - I've had a console open and logging since last night. Once it does I will post the output.
-
@rloeb said in 21.02 Sudden lockup:
Been running for 18+ hours. However, just noticed that Snort is NOT running and aborted just after midnight:
Feb 18 00:30:17 kernel pid 76998 (php), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 18 00:30:14 php 76998 [Snort] Building new sid-msg.map file for WAN...
Something is very wrong with this release!
Getting similar errors but with pfblockerng, during boot.
https://forum.netgate.com/post/964587
Feb 18 02:05:29 kernel pid 49475 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:09:02 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:16:21 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:39:03 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:44:59 kernel pid 377 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:52:02 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 03:07:38 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) -
@mcury Can someone provide serial console output? We've asked for this a few times and until someone gives us diagnostics information we can't move forward.
-
@kphillips said in 21.02 Sudden lockup:
@mcury Can someone provide serial console output? We've asked for this a few times and until someone gives us diagnostics information we can't move forward.
Sure, the only problem during boot is the Configuring Firewall.Segmentation fault (core dumped). This only happens after the pfblocker installation, and after a reboot.
Let me install the pfblockerng-devel again, and reboot to provide you the logs.
One moment please. -
@mcury Thank you. So, to confirm, this issue is only present when you are running pfBlockerNG and you don't experience the issue when you are not running pfBlockerNG?
-
I’m getting the same problem on my SG-3100, and I’m not using pfBlocker or Snort. I only have HAProxy (nearly idle) and OpenVPN packages installed.
-
@yammering Can you please provide serial console output for your appliance when one of these lockups occurs?
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/connect-to-console.html
-
@kphillips Exactly, the problem is when pfblockerng is enabled.
If it's installed, but disabled, the problem doesn't happen. -
Installed pfblockerng-devel, ran the wizard, didn't change anything else, I'll reboot now to get you the logs
UPDATE PROCESS START [ v3.0.0_10 ] [ 02/18/21 15:29:26 ] ===[ DNSBL Process ]================================================ Missing DNSBL stats and/or Unbound DNSBL files - Rebuilding Loading DNSBL SafeSearch... disabled Loading DNSBL Whitelist... completed [ EasyList ] Downloading update .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 10871 10871 0 0 0 10871 ---------------------------------------------------------------------- [ EasyPrivacy ] Downloading update [ 02/18/21 15:29:31 ] .. 200 OK. Whitelist: collector-cdn.github.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3037 3033 1 1 0 3031 ---------------------------------------------------------------------- [ Adaway ] Downloading update [ 02/18/21 15:29:34 ] .. 200 OK. Whitelist: aan.amazon-adsystem.com|aax-eu-retail-direct.amazon-adsystem.com|aax-fe-sin.amazon-adsystem.com|aax-fe.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax-us.amazon-adsystem.com|aax.amazon-adsystem.com|amidt.adsafeprotected.com|appvast.adsafeprotected.com|banners.itunes.apple.com|bs.eyeblaster.akadns.net|bs.serving-sys.com|bsla.eyeblaster.akadns.net|c.amazon-adsystem.com|ca.iadsdk.apple.com|cdn-a.amazon-adsystem.com|cdn.adsafeprotected.com|cf.iadsdk.apple.com|control.kochava.com|daldt.adsafeprotected.com|device-metrics-us-2.amazon.com|dt.adsafeprotected.com|dtvc.adsafeprotected.com|fls-eu.amazon-adsystem.com|fls-fe.amazon-adsystem.com|fls-na.amazon-adsystem.com|fls-na.amazon.com|fw.adsafeprotected.com|fwvc.adsafeprotected.com|iadsdk.apple.com|imp.control.kochava.com|mads.amazon-adsystem.com|mobile-static.adsafeprotected.com|mobile.adsafeprotected.com|nyidt.adsafeprotected.com|orfw.adsafeprotected.com|orpixel.adsafeprotected.com|pixel.adsafeprotected.com|px.moatads.com|s.amazon-adsystem.com|secure-gl.imrworldwide.com|sgfw.adsafeprotected.com|sgpixel.adsafeprotected.com|sjedt.adsafeprotected.com|spixel.adsafeprotected.com|static.adsafeprotected.com|tr.iadsdk.apple.com|unified-cdn.adsafeprotected.com|unified.adsafeprotected.com|ut.iadsdk.apple.com|vaes.amazon-adsystem.com|vafw.adsafeprotected.com|vapixel.adsafeprotected.com|vast.adsafeprotected.com|video.adsafeprotected.com|web-sdk.control.kochava.com|wildcard.moatads.com.edgekey.net|wrapper-cdn.adsafeprotected.com|wrapper-vast.adsafeprotected.com|ws-eu.amazon-adsystem.com|z-eu.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 9167 9167 193 61 0 8913 ---------------------------------------------------------------------- [ D_Me_ADs ] Downloading update [ 02/18/21 15:29:36 ] .. 200 OK. Whitelist: advertising.apple.com|amazon-adsystem.com|iadsdk.apple.com|pixel.adsafeprotected.com|qwapi.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2701 2701 196 5 0 2500 ---------------------------------------------------------------------- [ D_Me_Tracking ] Downloading update [ 02/18/21 15:29:38 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 34 34 11 0 0 23 ---------------------------------------------------------------------- [ Yoyo ] Downloading update [ 02/18/21 15:29:39 ] .. 200 OK. Whitelist: adsafeprotected.com|amazon-adsystem.com|pixel.adsafeprotected.com|securemetrics.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3571 3571 2089 4 0 1478 ---------------------------------------------------------------------- [ C19_CTC ] Downloading update [ 02/18/21 15:29:42 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 122596 122596 1 0 0 122595 ---------------------------------------------------------------------- [ Krisk_C19 ] Downloading update [ 02/18/21 15:29:53 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1999 1999 11 0 0 1988 ---------------------------------------------------------------------- [ SWC ] Downloading update [ 02/18/21 15:29:57 ] .. 200 OK. Whitelist: aax-cpm.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax-us-pdx.amazon-adsystem.com|aax.amazon-adsystem.com|amazon-adsystem.com|anycast.dt.adsafeprotected.com|bs.serving-sys.com|c.amazon-adsystem.com|dra.amazon-adsystem.com|dt.adsafeprotected.com|fls-na.amazon-adsystem.com|fw.adsafeprotected.com|images-aud.sourceforge.net|ir-de.amazon-adsystem.com|ir-na.amazon-adsystem.com|localhost.localdomain|mads.amazon-adsystem.com|pixel.adsafeprotected.com|ps-us.amazon-adsystem.com|px.moatads.com|rcm-na.amazon-adsystem.com|static.adsafeprotected.com|wms-eu.amazon-adsystem.com|wms-na.amazon-adsystem.com|ws-ea.amazon-adsystem.com|ws-na.amazon-adsystem.com|z-na.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 14667 14638 1125 27 0 13486 ---------------------------------------------------------------------- [ D_Me_Malv ] Downloading update [ 02/18/21 15:30:06 ] .. 200 OK. Whitelist: advertising.apple.com|amazon-adsystem.com|iadsdk.apple.com|pixel.adsafeprotected.com|qwapi.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2735 2735 2729 5 0 1 ---------------------------------------------------------------------- [ D_Me_Malw ] Downloading update [ 02/18/21 15:30:09 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1 1 1 0 0 0 ---------------------------------------------------------------------- [ Juniper ] Downloading update [ 02/18/21 15:30:12 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 9 6 0 0 0 6 ---------------------------------------------------------------------- [ Maltrail_BD ] Downloading update [ 02/18/21 15:30:16 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 145596 145596 2286 0 0 143310 ---------------------------------------------------------------------- [ MDS ] Downloading update [ 02/18/21 15:30:29 ] .. 404 Not Found [ DNSBL_Malicious - MDS ] Download FAIL [ 02/18/21 15:30:31 ] Firewall and/or IDS (Legacy mode only) are not blocking download. [ MDS_Immortal ] Downloading update .. 404 Not Found [ DNSBL_Malicious - MDS_Immortal ] Download FAIL Firewall and/or IDS (Legacy mode only) are not blocking download. [ MDL ] Downloading update .. 200 OK No Domains Found! Ensure only domain based Feeds are used for DNSBL! [ MVPS ] Downloading update [ 02/18/21 15:30:33 ] .. 200 OK. Whitelist: 5726.bapi.adsafeprotected.com|6063.bapi.adsafeprotected.com|aax-eu.amazon-adsystem.com|aax-us-east-rtb.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax.amazon-adsystem.com|bs.serving-sys.com|c.amazon-adsystem.com|cdn.adsafeprotected.com|dt.adsafeprotected.com|fls-eu.amazon-adsystem.com|fls-na.amazon-adsystem.com|fw.adsafeprotected.com|images-aud.sourceforge.net|ir-de.amazon-adsystem.com|ir-na.amazon-adsystem.com|ir-uk.amazon-adsystem.com|pixel.adsafeprotected.com|pm.adsafeprotected.com|ps-eu.amazon-adsystem.com|ps-us.amazon-adsystem.com|px.moatads.com|rcm-eu.amazon-adsystem.com|rcm-na.amazon-adsystem.com|s.amazon-adsystem.com|secure-gl.imrworldwide.com|spixel.adsafeprotected.com|static.adsafeprotected.com|wms-eu.amazon-adsystem.com|wms-na.amazon-adsystem.com|ws-na.amazon-adsystem.com|z-na.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 8815 8815 2112 32 0 6671 ---------------------------------------------------------------------- [ Spam404 ] Downloading update [ 02/18/21 15:30:38 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 7066 7064 65 0 0 6999 ---------------------------------------------------------------------- [ SFS_Toxic_BD ] Downloading update [ 02/18/21 15:30:41 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 27957 27956 12 0 0 27944 ---------------------------------------------------------------------- Saving DNSBL statistics... completed [ 02/18/21 15:30:50 ] ------------------------------------------------------------------------ Assembling DNSBL database...... completed [ 02/18/21 15:30:53 ] Adding DNSBL Unbound mode (Resolver adv. setting) DNS Resolver ( enabled ) unbound.conf modifications: Added DNSBL Unbound mode Saving new DNSBL web server configuration to port [ 8081 and 8443 ] VIP address(es) configured New DNSBL certificate created Restarting DNSBL Service Stopping Unbound Resolver Unbound stopped in 1 sec. Additional mounts: No changes required. Starting Unbound Resolver... completed [ 02/18/21 15:31:09 ] DNSBL update [ 349816 | PASSED ]... completed [ 02/18/21 15:31:10 ] ------------------------------------------------------------------------ ===[ GeoIP Process ]============================================ ===[ IPv4 Process ]================================================= [ Abuse_Feodo_C2_v4 ] Downloading update .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 118 118 118 [ Pass ] ----------------------------------------------------------------- [ Abuse_IPBL_v4 ] Downloading update [ 02/18/21 15:31:12 ] .. 200 OK. completed .. Empty file, Adding '127.1.7.7' to avoid download failure. ------------------------------ Original Master Final ------------------------------ 0 1 1 [ Pass ] ----------------------------------------------------------------- [ Abuse_SSLBL_v4 ] Downloading update [ 02/18/21 15:31:13 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 99 94 94 [ Pass ] ----------------------------------------------------------------- [ CINS_army_v4 ] Downloading update .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 15000 15000 15000 [ Pass ] ----------------------------------------------------------------- [ ET_Block_v4 ] Downloading update [ 02/18/21 15:31:15 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 1338 1223 1223 [ Pass ] ----------------------------------------------------------------- [ ET_Comp_v4 ] Downloading update [ 02/18/21 15:31:16 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 4961 4884 4884 [ Pass ] ----------------------------------------------------------------- [ ISC_Block_v4 ] Downloading update [ 02/18/21 15:31:18 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 21 22 22 [ Pass ] ----------------------------------------------------------------- [ Spamhaus_Drop_v4 ] Downloading update [ 02/18/21 15:31:20 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 972 2 2 [ Pass ] ----------------------------------------------------------------- [ Spamhaus_eDrop_v4 ] Downloading update [ 02/18/21 15:31:21 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 84 76 76 [ Pass ] ----------------------------------------------------------------- [ Talos_BL_v4 ] Downloading update [ 02/18/21 15:31:22 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 796 730 730 [ Pass ] ----------------------------------------------------------------- ===[ Aliastables / Rules ]================================ Firewall rule changes found, applying Filter Reload ** Restarting firewall filter daemon ** ===[ FINAL Processing ]===================================== [ Original IP count ] [ 23387 ] [ Final IP Count ] [ 22150 ] ===[ Deny List IP Counts ]=========================== 22150 total 15000 /var/db/pfblockerng/deny/CINS_army_v4.txt 4884 /var/db/pfblockerng/deny/ET_Comp_v4.txt 1223 /var/db/pfblockerng/deny/ET_Block_v4.txt 730 /var/db/pfblockerng/deny/Talos_BL_v4.txt 118 /var/db/pfblockerng/deny/Abuse_Feodo_C2_v4.txt 94 /var/db/pfblockerng/deny/Abuse_SSLBL_v4.txt 76 /var/db/pfblockerng/deny/Spamhaus_eDrop_v4.txt 22 /var/db/pfblockerng/deny/ISC_Block_v4.txt 2 /var/db/pfblockerng/deny/Spamhaus_Drop_v4.txt 1 /var/db/pfblockerng/deny/Abuse_IPBL_v4.txt ====================[ Empty Lists w/127.1.7.7 ]================== Abuse_IPBL_v4.txt ===[ DNSBL Domain/IP Counts ] =================================== 349816 total 143310 /var/db/pfblockerng/dnsbl/Maltrail_BD.txt 122595 /var/db/pfblockerng/dnsbl/C19_CTC.txt 27944 /var/db/pfblockerng/dnsbl/SFS_Toxic_BD.txt 13486 /var/db/pfblockerng/dnsbl/SWC.txt 10871 /var/db/pfblockerng/dnsbl/EasyList.txt 8913 /var/db/pfblockerng/dnsbl/Adaway.txt 6999 /var/db/pfblockerng/dnsbl/Spam404.txt 6671 /var/db/pfblockerng/dnsbl/MVPS.txt 3031 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt 2500 /var/db/pfblockerng/dnsbl/D_Me_ADs.txt 1988 /var/db/pfblockerng/dnsbl/Krisk_C19.txt 1478 /var/db/pfblockerng/dnsbl/Yoyo.txt 23 /var/db/pfblockerng/dnsbl/D_Me_Tracking.txt 6 /var/db/pfblockerng/dnsbl/Juniper.txt 1 /var/db/pfblockerng/dnsbl/D_Me_Malv.txt 0 /var/db/pfblockerng/dnsbl/MDS_Immortal.fail 0 /var/db/pfblockerng/dnsbl/MDS.fail 0 /var/db/pfblockerng/dnsbl/MDL.txt 0 /var/db/pfblockerng/dnsbl/D_Me_Malw.txt ====================[ IPv4/6 Last Updated List Summary ]============== Feb 11 04:49 Spamhaus_eDrop_v4 Feb 17 02:30 ET_Block_v4 Feb 17 02:30 ET_Comp_v4 Feb 18 03:33 Spamhaus_Drop_v4 Feb 18 14:18 CINS_army_v4 Feb 18 14:19 ISC_Block_v4 Feb 18 15:05 Talos_BL_v4 Feb 18 15:25 Abuse_SSLBL_v4 Feb 18 15:30 Abuse_Feodo_C2_v4 Feb 18 15:31 Abuse_IPBL_v4 ====================[ DNSBL Last Updated List Summary ]============== Jul 31 2015 D_Me_Tracking Sep 5 2018 Juniper Jan 31 2020 D_Me_ADs Jul 10 2020 D_Me_Malw Jul 10 2020 D_Me_Malv Nov 12 19:17 MDL Dec 15 05:07 MVPS Feb 15 02:18 Adaway Feb 15 05:50 Yoyo Feb 16 05:48 SWC Feb 18 09:30 Krisk_C19 Feb 18 13:50 C19_CTC Feb 18 15:00 SFS_Toxic_BD Feb 18 15:02 EasyPrivacy Feb 18 15:21 EasyList Feb 18 15:30 Maltrail_BD Feb 18 15:30 Spam404 =============================================================== Database Sanity check [ PASSED ] ------------------------ Masterfile/Deny folder uniq check Deny folder/Masterfile uniq check Sync check (Pass=No IPs reported) ---------- Alias table IP Counts ----------------------------- 22150 /var/db/aliastables/pfB_PRI1_v4.txt pfSense Table Stats ------------------- table-entries hard limit 400000 Table Usage Count 22176 UPDATE PROCESS ENDED [ 02/18/21 15:31:28 ] -
@mcury Thank you very much. I'll keep an eye on this thread for your response.
-
Removed the public IP addresses, serial number and device ID from the output.
FreeBSD/arm (pfsense.local.lan) (ttyu0) Netgate SG-3100 - Serial: X - Netgate Device ID: X *** Welcome to Netgate pfSense Plus 21.02-RELEASE (arm) on pfsense *** WAN (wan) -> mvneta2 -> v4/DHCP4: X.X.X.X/22 v6/DHCP6: X:X:X:X:X:X:X:X/64/64 MGMT (lan) -> mvneta1 -> v4: 172.16.200.1/29 OPT1 (opt1) -> mvneta0 -> LAN (opt2) -> mvneta1.100 -> v4: 192.168.255.249/29 WIFI (opt3) -> mvneta1.10 -> v4: 192.168.10.1/28 v6/t6: X:X:X:X:X:X:X:X/64 GUEST (opt4) -> mvneta1.20 -> v4: 192.168.20.1/24 0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) PHP shell + Netgate pfSense Plus tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option: 5 Netgate pfSense Plus will reboot. This may take a few minutes, depending on your hardware. Do you want to proceed? Y/y: Reboot normally R/r: Reroot (Stop processes, remount disks, re-run startup sequence) Enter: Abort Enter an option: y Netgate pfSense Plus is rebooting now. Stopping package AWS VPC Wizard...done. Stopping package IPsec Profile Wizard...done. Stopping package acme...done. Stopping package nut...done. Stopping package pfBlockerNG-devel...done. Stopping /usr/local/etc/rc.d/pfb_dnsbl.sh...done. Stopping /usr/local/etc/rc.d/pfb_filter.sh...done. Stopping /usr/local/etc/rc.d/shutdown.nut.sh...done. Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining... 2 1 0 done Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done All buffers synced. Uptime: 1m48s Rebooting... BootROM - 1.73 Booting from SPI flash General initialization - Version: 1.0.0 AVS selection from EFUSE disabled (Skip reading EFUSE values) Overriding default AVS value to: 0x23 Detected Device ID 6820 High speed PHY - Version: 2.0 Init Customer board board SerDes lanes topology details: | Lane # | Speed| Type | ------------------------------| | 0 | 3 | SATA0 | | 1 | 5 | PCIe0 | | 2 | 3 | SATA1 | | 3 | 4 | SGMII2 | | 4 | 5 | PCIe1 | | 5 | 5 | USB3 HOST1 | ------------------------------- PCIe, Idx 0: detected no link PCIe, Idx 1: detected no link High speed PHY - Ended Successfully mv_ddr: mv_ddr-armada-17.06.1-g07f8294 (Aug 11 2017 - 13:12:46) DDR4 Training Sequence - Switching XBAR Window to FastPath Window mv_ddr: completed successfully BootROM: Image checksum verification PASSED __ __ _ _ | \/ | __ _ _ ____ _____| | | | |\/| |/ _` | '__\ \ / / _ \ | | | | | | (_| | | \ V / __/ | | |_| |_|\__,_|_| \_/ \___|_|_| _ _ ____ _ | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |___| |_) | (_) | (_) | |_ \___/ |____/ \___/ \___/ \__| ** LOADER ** U-Boot 2013.01-02873-g5e4ac3c (Aug 11 2017 - 13:12:40) Marvell version: devel-17.06.0 Board: Rogue-1 SoC: MV88F6820 Rev A0 running 2 CPUs CPU: ARM Cortex A9 MPCore (Rev 1) LE CPU 0 CPU @ 1600 [MHz] L2 @ 800 [MHz] TClock @ 250 [MHz] DDR4 @ 800 [MHz] DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled DRAM: 2 GiB MMC: mv_sdh: 0 DBG: Calling spi_flash_probe from env_relocate_spec() SF: Probing bus 0 cs 0 @ 20000000Hz mode 3 SF: Detected N25Q128 with page size 64 KiB, total 16 MiB PCI-e 0: Detected No Link. PCI-e 1: Detected No Link. USB2.0 0: Host Mode USB3.0 1: Host Mode Map: Code: 0x7fedc000:0x7ff975a8 BSS: 0x7ffef600 Stack: 0x7f4cbf20 Heap: 0x7f4cc000:0x7fedc000 U-Boot Environment: 0x00100000:0x00110000 (SPI) Board configuration detected: Net: | port | Interface | PHY address | |--------|-----------|--------------| | egiga0 | RGMII | 0x00 | | egiga1 | RGMII | 0x01 | | egiga2 | SGMII | In-Band | egiga0 [PRIME], egiga1, egiga2 Hit any key to stop autoboot: 0 reading ubldr.bin 228648 bytes read in 8 ms (27.3 MiB/s) ## Starting application at 0x00200000 ... Consoles: U-Boot console Compatible U-Boot API signature found @0x7f4dc280 FreeBSD/armv6 U-Boot loader, Revision 1.2 (Tue Jul 10 10:26:23 EDT 2018 root@buildbot3) DRAM: 2048MB Number of U-Boot devices: 4 U-Boot env: loaderdev not set, will probe all devices. Found U-Boot device: disk Probing all disk devices... Checking unit=0 slice=<auto> partition=<auto>...disk0: read failed, error=1 Checking unit=1 slice=<auto> partition=<auto>...disk1: read failed, error=1 Checking unit=2 slice=<auto> partition=<auto>... good. Booting from disk2s2a: Loading /boot/defaults/loader.conf console comconsole is invalid! no valid consoles! Available consoles: uboot /boot/kernel/kernel text=0xa8d650 data=0x11d47b4 data=0x0+0x54000 syms=[0x4+0xa8f50+0x4+0xf377f] / Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... /boot/dtb/armada-385-80300-0148-G00-X100.dtb size=0x60ed Loaded DTB from file 'armada-385-80300-0148-G00-X100.dtb'. Kernel entry at 0x400180... Kernel args: (null) SOC: Marvell 88F6820, TClock 250MHz, Frequency 1600MHz Instruction cache prefetch enabled, data cache prefetch disabled ---<<BOOT>>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense-SG-3100 arm FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) CPU: ARM Cortex-A9 r4p1 (ECO: 0x00000000) CPU Features: Multiprocessing, Thumb2, Security, VMSAv7, Coherent Walk Optional instructions: UMULL, SMULL, SIMD(ext) LoUU:2 LoC:2 LoUIS:2 Cache level 1: 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 4-way instruction cache Read-Alloc real memory = 2147479552 (2047 MB) avail memory = 2071031808 (1975 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs arc4random: no preloaded entropy cache ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc014deb0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc014df60, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xc0156f50, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xc0157000, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xc01570b0, 0) error 1 ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xc014de00, 0) error 1 wlan: mac acl policy registered random: entropy device external interface [ath_hal] loaded ofwbus0: <Open Firmware Device Tree> simplebus0: <Flattened device tree simple bus> on ofwbus0 simplebus1: <Flattened device tree simple bus> on simplebus0 l2cache0: <PL310 L2 cache controller> mem 0x8000-0x8fff on simplebus1 l2cache0: cannot allocate IRQ, not using interrupt l2cache0: Part number: 0x3, release: 0x9 l2cache0: L2 Cache enabled: 1024KB/32B 16 ways gic0: <ARM Generic Interrupt Controller> mem 0xd000-0xdfff,0xc100-0xc1ff on simplebus1 gic0: pn 0x39, arch 0x1, rev 0x2, implementer 0x43b irqs 192 mpic0: <Marvell Integrated Interrupt Controller> mem 0x20a00-0x20ccf,0x21070-0x210c7 irq 17 on simplebus1 gpio0: <Marvell Integrated GPIO Controller> mem 0x18100-0x1813f,0x181c0-0x181c7 irq 9,10,11,12 on simplebus1 gpio0: 4 IRQs available gpio0: Disable interrupts (offset = 0 + EDGE(0x18) gpio0: Disable interrupts (offset = 0 + LEV(0x1C)) gpio0: Setup intr 0 gpio0: Setup intr 1 gpio0: Setup intr 2 gpio0: Setup intr 3 gpio0: Clear int status (offset = 0) gpiobus0: <GPIO bus> on gpio0 gpio1: <Marvell Integrated GPIO Controller> mem 0x18140-0x1817f,0x181c8-0x181cf irq 13,14,15,16 on simplebus1 gpio1: 4 IRQs available gpio1: Disable interrupts (offset = 0 + EDGE(0x18) gpio1: Disable interrupts (offset = 0 + LEV(0x1C)) gpio1: Setup intr 0 gpio1: Setup intr 1 gpio1: Setup intr 2 gpio1: Setup intr 3 gpio1: Clear int status (offset = 0) gpiobus1: <GPIO bus> on gpio1 mp_tmr0: <ARM MPCore Timers> mem 0xc200-0xc21f irq 3 on simplebus1 Timecounter "MPCore" frequency 800000000 Hz quality 800 mp_tmr1: <ARM MPCore Timers> mem 0xc600-0xc61f irq 4 on simplebus1 Event timer "MPCore" frequency 800000000 Hz quality 1000 twsi0: <Marvell Integrated I2C Bus Controller> mem 0x11000-0x1101f irq 5 on simplebus1 iicbus0: <OFW I2C bus> on twsi0 iic0: <I2C generic I/O> on iicbus0 gpio2: <NXP PCA9552 LED driver> at addr 0xc0 on iicbus0 device_attach: gpio2 attach returned 6 gpio2: <ISSI IS31FL3199 9 channel light effect LED driver> at addr 0xce on iicbus0 gpiobus2: <OFW GPIO bus> on gpio2 gpioc2: <GPIO controller> on gpio2 uart0: <16550 or compatible> mem 0x12000-0x120ff irq 7 on simplebus1 uart0: console (-1,n,8,1) uart1: <16550 or compatible> mem 0x12100-0x121ff irq 8 on simplebus1 gpioc0: <GPIO controller> on gpio0 gpioc1: <GPIO controller> on gpio1 wdt0: <Marvell Watchdog Timer> mem 0x20300-0x20333,0x20704-0x20707,0x18260-0x18263 irq 24,25 on simplebus1 pmsu0: <Power Management Service Unit> mem 0x22000-0x22fff on simplebus1 mvneta0: <NETA controller> mem 0x30000-0x33fff irq 26 on simplebus1 mvneta0: version is 10 mvneta0: Ethernet address: 00:08:a2:0c:c4:1b miibus0: <MII bus> on mvneta0 mv88e151x0: <Marvell 88E1512 Gigabit PHY> PHY 1 on miibus0 mv88e151x0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto mvneta1: <NETA controller> mem 0x34000-0x37fff irq 27 on simplebus1 mvneta1: version is 10 mvneta1: Ethernet address: 00:08:a2:0c:c4:1c mdio0: <MDIO> on mvneta1 e6000sw0: <Marvell 88E6141> on mdio0 e6000sw0: single-chip addressing mode e6000sw0: PHY at port 1 miibus1: <MII bus> on e6000sw0 e1000phy0: <Marvell 88E1000 Gigabit PHY> PHY 17 on miibus1 e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto e6000sw0: PHY at port 2 miibus2: <MII bus> on e6000sw0 e1000phy1: <Marvell 88E1000 Gigabit PHY> PHY 18 on miibus2 e1000phy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto e6000sw0: PHY at port 3 miibus3: <MII bus> on e6000sw0 e1000phy2: <Marvell 88E1000 Gigabit PHY> PHY 19 on miibus3 e1000phy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto e6000sw0: PHY at port 4 miibus4: <MII bus> on e6000sw0 e1000phy3: <Marvell 88E1000 Gigabit PHY> PHY 20 on miibus4 e1000phy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto e6000sw0: CPU port at 5 e6000sw0: fixed port at 5 e6000sw0: switch is ready. etherswitch0: <Switch controller> on e6000sw0 mvneta2: <NETA controller> mem 0x70000-0x73fff irq 28 on simplebus1 mvneta2: version is 10 mvneta2: Ethernet address: 00:08:a2:0c:c4:1d miibus5: <MII bus> on mvneta2 mv88e151x1: <Marvell 88E1512 Gigabit PHY> PHY 0 on miibus5 mv88e151x1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto ehci0: <Marvell Integrated USB 2.0 controller> mem 0x58000-0x584ff irq 29 on simplebus1 usbus0: EHCI version 1.0 usbus0 on ehci0 cesa0: <Marvell Cryptographic Engine and Security Accelerator> mem 0x90000-0x9ffff irq 30,31 on simplebus1 rtc0: <Marvell Integrated RTC> mem 0xa3800-0xa381f,0x184a0-0x184ab irq 32 on simplebus1 rtc0: registered as a time-of-day clock, resolution 1.000000s ahci0: <Marvell AHCI Controller> mem 0xa8000-0xa9fff irq 33 on simplebus1 ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS ahci0: quirks=0x200010<2CH,MRVL_SR_DEL> ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 armada_thermal0: <Armada380 Thermal Control> mem 0xe4078-0xe407b,0xe4070-0xe4077 on simplebus1 sdhci_fdt0: <ARMADA38X SDHCI controller> mem 0xd8000-0xd8fff,0xdc000-0xdc0ff,0x18454-0x18457 irq 36 on simplebus1 sdhci_fdt0: 1 slot(s) allocated xhci0: <Marvell Integrated USB 3.0 controller> mem 0xf8000-0xfbfff,0xfc000-0xfffff irq 38 on simplebus1 xhci0: 32 bytes context size, 32-bit DMA usbus1 on xhci0 cesa1: <Marvell Cryptographic Engine and Security Accelerator> mem 0x90000-0x9ffff irq 39,40 on simplebus1 spi0: <Marvell SPI controller> mem 0xf001000000010600-0xf00100000001064f irq 1 on simplebus0 cpulist0: <Open Firmware CPU Group> on ofwbus0 cpu0: <Open Firmware CPU> on cpulist0 cpu1: <Open Firmware CPU> on cpulist0 cryptosoft0: <software crypto> Timecounters tick every 1.000 msec mvneta1: link state changed to UP spibus0: <OFW SPI bus> on spi0 mx25l0: <M25Pxx Flash Family> at cs 0 mode 0 on spibus0 mx25l0: device type n25q128, size 16384K in 256 sectors of 64K, erase size 4K Release APs e6000sw0port1: link state changed to DOWN usbus0: e6000sw0port2: link state changed to DOWN 480Mbps High Speed USB v2.0 e6000sw0port3: link state changed to DOWN usbus1: e6000sw0port4: link state changed to DOWN 5.0Gbps Super Speed USB v3.0 ugen0.1: <Marvell EHCI root HUB> at usbus0 uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ugen1.1: <Marvell XHCI root HUB> at usbus1 uhub1: <Marvell XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 arc4random: no preloaded entropy cache Trying to mount root from ufs:/dev/diskid/DISK-D4EC8498s2a [rw,noatime]... Root mount waiting for: usbus0 CAM usbus1 uhub1: 2 ports with 2 removable, self powered mmc0: <MMC/SD bus> on sdhci_fdt0 mmcsd0: 8GB <MMCHC 008GE0 0.0 SN D4EC8498 MFG 06/2015 by 17 0x0000> at mmc0 50.0MHz/8bit/65535-block mmcsd0boot0: 4MB partition 1 at mmcsd0 mmcsd0boot1: 4MB partition 2 at mmcsd0 mmcsd0rpmb: 4MB partition 3 at mmcsd0 uhub0: 1 port with 1 removable, self powered ugen1.2: <INNO TECH USB to Serial> at usbus1 e6000sw0port2: link state changed to UP Root mount waiting for: CAM Root mount waiting for:e6000sw0port1: link state changed to UP CAMe6000sw0port3: link state changed to UP e6000sw0port4: link state changed to UP Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM mountroot: waiting for device /dev/diskid/DISK-D4EC8498s2a... Configuring crash dumps... No suitable dump device was found. /dev/diskid/DISK-D4EC8498s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/diskid/DISK-D4EC8498s2a: clean, 1516070 free (1078 frags, 189374 blocks, 0.1% fragmentation) Filesystems are clean, continuing... Mounting filesystems... random: unblocking device. __ _ __ / _|___ ___ _ __ ___ ___ _ | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \ _| |_ | |_) | _\__ \ __/ | | \__ \ __/ |_ _| | .__/|_| |___/\___|_| |_|___/\___| |_| |_| Welcome to Netgate pfSense Plus 21.02-RELEASE... ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/lib/engines /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE Soft Float compatibility ldconfig path: done. External config loader 1.0 is now starting... Launching the init system....... done. Initializing.................... done. Starting device manager (devd)...done. Loading configuration......done. Updating configuration...done. Checking config backups consistency.................................done. Setting up extended sysctls...done. Setting timezone...done. Configuring looplo0: link state changed to UP back interface...done. Starting syslog...done. Starting Secure Shell Services...done. Configuring switch...done. Setting up interfaces microcode...done. Starting PC/SC Smart Card Services...done. Configuring loopback interface...done. Creating wireless clone interfaces...done. Configuring LAGG interfaces...done. Configuring VLAN interfaces...done. Configuring QinQ interfaces...done. Configuring IPsec VTI interfaces...done. Configuring WAN interface...done. Configuring MGMT interface...done. Configuring LAN interface...done. Configuring GUEST interface...done. Configuring WIFI interface...done. Configuring WAN interface...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. Configuring firewall.Segmentation fault (core dumped) Starting CRON... done. Starting package AWS VPC Wizard...done. Starting package IPsec Profile Wizard...done. Starting package acme...done. Starting package nut...done. Starting package pfBlockerNG-devel...done. Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done. Starting /usr/local/etc/rc.d/pfb_filter.sh...done. Starting /usr/local/etc/rc.d/shutdown.nut.sh...done. Netgate pfSense Plus 21.02-RELEASE arm Tue Feb 16 08:56:55 EST 2021 Bootup complete FreeBSD/arm (pfsense.local.lan) (ttyu0) Netgate SG-3100 - Serial: X - Netgate Device ID: X *** Welcome to Netgate pfSense Plus 21.02-RELEASE (arm) on pfsense *** WAN (wan) -> mvneta2 -> v4/DHCP4: X.X.X.X/X v6/DHCP6: X:X:X:X:X:X:X:X/64 MGMT (lan) -> mvneta1 -> v4: 172.16.200.1/29 OPT1 (opt1) -> mvneta0 -> LAN (opt2) -> mvneta1.100 -> v4: 192.168.255.249/29 WIFI (opt3) -> mvneta1.10 -> v4: 192.168.10.1/28 v6/t6: X:X:X:X:X:X:X:X/64 GUEST (opt4) -> mvneta1.20 -> v4: 192.168.20.1/24 0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) PHP shell + Netgate pfSense Plus tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell -
System logs:
Feb 18 15:42:17 php 366 rc.bootup: calling interface_dhcpv6_configure. Feb 18 15:42:17 php 366 rc.bootup: Accept router advertisements on interface mvneta2 Feb 18 15:42:17 php 366 rc.bootup: Starting rtsold process Feb 18 15:42:21 php 366 rc.bootup: calling interface_dhcpv6_configure. Feb 18 15:42:21 php 366 rc.bootup: Accept router advertisements on interface mvneta2 Feb 18 15:42:21 php 366 rc.bootup: Killing running rtsold process Feb 18 15:42:23 php 366 rc.bootup: Starting rtsold process Feb 18 15:42:25 php 366 rc.bootup: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_mvneta2.pid -M /var/etc/rtsold_mvneta2_script.sh -O /var/etc/rtsold_mvneta2_script.sh mvneta2' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Feb 18 15:42:25 php 366 rc.bootup: Resyncing OpenVPN instances.Feb 18 15:42:39 kernel pid 366 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
-
@mcury I think we have a misunderstanding. I need the USB Serial console output for the device when a lock up occurs. Not the system logs.
-
@kphillips I came here looking for solutions to the same crashing behavior after upgrading today. I uninstalled pfblockerNG, and while it ran longer (~20 min vs 3-5min) it still "crashed". With the serial cable connected and viewing the console in putty, there is no output when it becomes unreachable. It IS still reachable via the console but nothing else is happening.
-
@kphillips Hm, this is what I have, no more logs to provide you unfortunately, pfsense no longer responds, and to access the GUI, I have to Restart the webConfigurator, disable pfblocker, and restart..
Doing so, Internet works, GUI responds, normal usage. -
@mcury Can you attach a USB cable to the firewall and see what the USB console is doing?
Instructions for doing so can be found here:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/connect-to-console.htmlEven if the webConfigurator is locked up, this should always respond.
-
I am in the same boat as many here. SG-3100 sudden lockup after upgrade with no console output.
I am guessing a kernel panic as I am unable to see anything via serial console or logs. -
@kphillips That log is from the console, connected through USB cable to pfsense, using putty serial. The problem start exactly here when configuring Firewall as per below, boot proceeds to the end, but GUI and Internet don't work.
As per mentioned above, I have to restart the GUI through the console with the option 11, disable pfblockerng and reboot, then everything works.Syncing OpenVPN settings...done.
Configuring firewall.Segmentation fault (core dumped) <<<
Starting CRON... done. -
@kphillips It shows nothing for me.
I cant even get to the console. This is why I suspect a kernel panic. -
@mcury Can you provide the full output instead of just the provided snippet? I need everything, not just a chunk, to provide to our devs.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.