21.02 Sudden lockup
-
Yeah, I just had the same thing happen. I reported this back in 2.5 beta, it seems to only occur on the 3100 series. I still have IPv4/IPv6 addresses on all of my interfaces, but I get total connectivity failure. I had about 6 hours of uptime before this happened. It's completely random.
I tried to get logs of it, but since its a total loss of network communication, my log servers never get anything, and the local logs never showed anything.
I have no packages running btw, just openvpn export.
A power cycle will fix it, but i used a console connection to manually reboot. Everything came right back up.
I will leave a console session open as jimp has suggested.
-
its indeed really weird. Because it was late yesterday (EU) I wanted to do the rollback / reinstall this morning but it didn't crashed during the night. Maybe because of not much load maybe random?
Only thing I changed yesterday was to disable all packages, which actually werent so much;
pfblocker, avahi, service watchdock, lldp
But after the crash the same picture for me, besides the logs posted above no other clue.
Let's see how it run during the day without packages enabled.
-
Same issue here, install went without issues. Device was working for about 30-45 minutes before it froze/locked up the first time. Now I need to power cycle it every 10-60 minutes. Tried removing all unnecessary packages, but without success.
When it freezes I can't even ping it via LAN.
This has now happened 5 times.I'm opening a support ticket in order to get access to the image, so I can test if reinstall solves the issues...
Netgate: SG-3100.
-
@kuser
Added a ticket and got hold of 21.02 image, reflashed the device and reimported backup.
Same issue after about 65 minutes.
The device doesn't actually freeze, but something happens with internal switch/interface.
It stops responding to WAN/LAN, however usb-console is available.I've requested 2.4.5p1 image from NetGate.
-
Has anyone monitored the console yet when this happens? The system log wouldn't have the same information printed to the console necessarily.
And that also would let you check easily if it's actually locked up vs still being responsive at the console but losing connectivity.
-
I can confirm that the console was available the last time I lost LAN/WAN. I didn't find anything interesting in the logs(dmesg), but I do suspect it might be related to the internal switch. But I'm not really sure I know what I was looking for. I am currently connected to the console and can provide some debug information if it locks up again. Anything particular I should check?
I tried service netif restart but that seemed to hang.
-
Every time this has happened to me the console is accessible. Both interfaces also keep their ipv6/ipv4 addresses. It "feels" like routes are randomly disappearing, but I should still be able to ping stuff on the local connected network if that was the issue, and I can't even do that. Traffic pretty much stops.
-
Try to disable pfblockerng, I'm getting similar behavior, and it's working with it disabled.
-
@behemyth If the console is accessible like you said, can you please provide the output?
-
I am also experiencing these same issues with loss of LAN/WAN on my 3100 after upgrade last night to 21.02. I am not running any special packages aside from DHCP, DNS, NTP and UPnP.
-
Been running for 18+ hours. However, just noticed that Snort is NOT running and aborted just after midnight:
Feb 18 00:30:17 kernel pid 76998 (php), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 18 00:30:14 php 76998 [Snort] Building new sid-msg.map file for WAN...
Something is very wrong with this release!
-
I am waiting for it to happen again - I've had a console open and logging since last night. Once it does I will post the output.
-
@rloeb said in 21.02 Sudden lockup:
Been running for 18+ hours. However, just noticed that Snort is NOT running and aborted just after midnight:
Feb 18 00:30:17 kernel pid 76998 (php), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 18 00:30:14 php 76998 [Snort] Building new sid-msg.map file for WAN...
Something is very wrong with this release!
Getting similar errors but with pfblockerng, during boot.
https://forum.netgate.com/post/964587
Feb 18 02:05:29 kernel pid 49475 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:09:02 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:16:21 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:39:03 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:44:59 kernel pid 377 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 02:52:02 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 18 03:07:38 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) -
@mcury Can someone provide serial console output? We've asked for this a few times and until someone gives us diagnostics information we can't move forward.
-
@kphillips said in 21.02 Sudden lockup:
@mcury Can someone provide serial console output? We've asked for this a few times and until someone gives us diagnostics information we can't move forward.
Sure, the only problem during boot is the Configuring Firewall.Segmentation fault (core dumped). This only happens after the pfblocker installation, and after a reboot.
Let me install the pfblockerng-devel again, and reboot to provide you the logs.
One moment please. -
@mcury Thank you. So, to confirm, this issue is only present when you are running pfBlockerNG and you don't experience the issue when you are not running pfBlockerNG?
-
I’m getting the same problem on my SG-3100, and I’m not using pfBlocker or Snort. I only have HAProxy (nearly idle) and OpenVPN packages installed.
-
@yammering Can you please provide serial console output for your appliance when one of these lockups occurs?
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/connect-to-console.html
-
@kphillips Exactly, the problem is when pfblockerng is enabled.
If it's installed, but disabled, the problem doesn't happen. -
Installed pfblockerng-devel, ran the wizard, didn't change anything else, I'll reboot now to get you the logs
UPDATE PROCESS START [ v3.0.0_10 ] [ 02/18/21 15:29:26 ] ===[ DNSBL Process ]================================================ Missing DNSBL stats and/or Unbound DNSBL files - Rebuilding Loading DNSBL SafeSearch... disabled Loading DNSBL Whitelist... completed [ EasyList ] Downloading update .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 10871 10871 0 0 0 10871 ---------------------------------------------------------------------- [ EasyPrivacy ] Downloading update [ 02/18/21 15:29:31 ] .. 200 OK. Whitelist: collector-cdn.github.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3037 3033 1 1 0 3031 ---------------------------------------------------------------------- [ Adaway ] Downloading update [ 02/18/21 15:29:34 ] .. 200 OK. Whitelist: aan.amazon-adsystem.com|aax-eu-retail-direct.amazon-adsystem.com|aax-fe-sin.amazon-adsystem.com|aax-fe.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax-us.amazon-adsystem.com|aax.amazon-adsystem.com|amidt.adsafeprotected.com|appvast.adsafeprotected.com|banners.itunes.apple.com|bs.eyeblaster.akadns.net|bs.serving-sys.com|bsla.eyeblaster.akadns.net|c.amazon-adsystem.com|ca.iadsdk.apple.com|cdn-a.amazon-adsystem.com|cdn.adsafeprotected.com|cf.iadsdk.apple.com|control.kochava.com|daldt.adsafeprotected.com|device-metrics-us-2.amazon.com|dt.adsafeprotected.com|dtvc.adsafeprotected.com|fls-eu.amazon-adsystem.com|fls-fe.amazon-adsystem.com|fls-na.amazon-adsystem.com|fls-na.amazon.com|fw.adsafeprotected.com|fwvc.adsafeprotected.com|iadsdk.apple.com|imp.control.kochava.com|mads.amazon-adsystem.com|mobile-static.adsafeprotected.com|mobile.adsafeprotected.com|nyidt.adsafeprotected.com|orfw.adsafeprotected.com|orpixel.adsafeprotected.com|pixel.adsafeprotected.com|px.moatads.com|s.amazon-adsystem.com|secure-gl.imrworldwide.com|sgfw.adsafeprotected.com|sgpixel.adsafeprotected.com|sjedt.adsafeprotected.com|spixel.adsafeprotected.com|static.adsafeprotected.com|tr.iadsdk.apple.com|unified-cdn.adsafeprotected.com|unified.adsafeprotected.com|ut.iadsdk.apple.com|vaes.amazon-adsystem.com|vafw.adsafeprotected.com|vapixel.adsafeprotected.com|vast.adsafeprotected.com|video.adsafeprotected.com|web-sdk.control.kochava.com|wildcard.moatads.com.edgekey.net|wrapper-cdn.adsafeprotected.com|wrapper-vast.adsafeprotected.com|ws-eu.amazon-adsystem.com|z-eu.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 9167 9167 193 61 0 8913 ---------------------------------------------------------------------- [ D_Me_ADs ] Downloading update [ 02/18/21 15:29:36 ] .. 200 OK. Whitelist: advertising.apple.com|amazon-adsystem.com|iadsdk.apple.com|pixel.adsafeprotected.com|qwapi.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2701 2701 196 5 0 2500 ---------------------------------------------------------------------- [ D_Me_Tracking ] Downloading update [ 02/18/21 15:29:38 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 34 34 11 0 0 23 ---------------------------------------------------------------------- [ Yoyo ] Downloading update [ 02/18/21 15:29:39 ] .. 200 OK. Whitelist: adsafeprotected.com|amazon-adsystem.com|pixel.adsafeprotected.com|securemetrics.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3571 3571 2089 4 0 1478 ---------------------------------------------------------------------- [ C19_CTC ] Downloading update [ 02/18/21 15:29:42 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 122596 122596 1 0 0 122595 ---------------------------------------------------------------------- [ Krisk_C19 ] Downloading update [ 02/18/21 15:29:53 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1999 1999 11 0 0 1988 ---------------------------------------------------------------------- [ SWC ] Downloading update [ 02/18/21 15:29:57 ] .. 200 OK. Whitelist: aax-cpm.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax-us-pdx.amazon-adsystem.com|aax.amazon-adsystem.com|amazon-adsystem.com|anycast.dt.adsafeprotected.com|bs.serving-sys.com|c.amazon-adsystem.com|dra.amazon-adsystem.com|dt.adsafeprotected.com|fls-na.amazon-adsystem.com|fw.adsafeprotected.com|images-aud.sourceforge.net|ir-de.amazon-adsystem.com|ir-na.amazon-adsystem.com|localhost.localdomain|mads.amazon-adsystem.com|pixel.adsafeprotected.com|ps-us.amazon-adsystem.com|px.moatads.com|rcm-na.amazon-adsystem.com|static.adsafeprotected.com|wms-eu.amazon-adsystem.com|wms-na.amazon-adsystem.com|ws-ea.amazon-adsystem.com|ws-na.amazon-adsystem.com|z-na.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 14667 14638 1125 27 0 13486 ---------------------------------------------------------------------- [ D_Me_Malv ] Downloading update [ 02/18/21 15:30:06 ] .. 200 OK. Whitelist: advertising.apple.com|amazon-adsystem.com|iadsdk.apple.com|pixel.adsafeprotected.com|qwapi.apple.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2735 2735 2729 5 0 1 ---------------------------------------------------------------------- [ D_Me_Malw ] Downloading update [ 02/18/21 15:30:09 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1 1 1 0 0 0 ---------------------------------------------------------------------- [ Juniper ] Downloading update [ 02/18/21 15:30:12 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 9 6 0 0 0 6 ---------------------------------------------------------------------- [ Maltrail_BD ] Downloading update [ 02/18/21 15:30:16 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 145596 145596 2286 0 0 143310 ---------------------------------------------------------------------- [ MDS ] Downloading update [ 02/18/21 15:30:29 ] .. 404 Not Found [ DNSBL_Malicious - MDS ] Download FAIL [ 02/18/21 15:30:31 ] Firewall and/or IDS (Legacy mode only) are not blocking download. [ MDS_Immortal ] Downloading update .. 404 Not Found [ DNSBL_Malicious - MDS_Immortal ] Download FAIL Firewall and/or IDS (Legacy mode only) are not blocking download. [ MDL ] Downloading update .. 200 OK No Domains Found! Ensure only domain based Feeds are used for DNSBL! [ MVPS ] Downloading update [ 02/18/21 15:30:33 ] .. 200 OK. Whitelist: 5726.bapi.adsafeprotected.com|6063.bapi.adsafeprotected.com|aax-eu.amazon-adsystem.com|aax-us-east-rtb.amazon-adsystem.com|aax-us-east.amazon-adsystem.com|aax.amazon-adsystem.com|bs.serving-sys.com|c.amazon-adsystem.com|cdn.adsafeprotected.com|dt.adsafeprotected.com|fls-eu.amazon-adsystem.com|fls-na.amazon-adsystem.com|fw.adsafeprotected.com|images-aud.sourceforge.net|ir-de.amazon-adsystem.com|ir-na.amazon-adsystem.com|ir-uk.amazon-adsystem.com|pixel.adsafeprotected.com|pm.adsafeprotected.com|ps-eu.amazon-adsystem.com|ps-us.amazon-adsystem.com|px.moatads.com|rcm-eu.amazon-adsystem.com|rcm-na.amazon-adsystem.com|s.amazon-adsystem.com|secure-gl.imrworldwide.com|spixel.adsafeprotected.com|static.adsafeprotected.com|wms-eu.amazon-adsystem.com|wms-na.amazon-adsystem.com|ws-na.amazon-adsystem.com|z-na.amazon-adsystem.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 8815 8815 2112 32 0 6671 ---------------------------------------------------------------------- [ Spam404 ] Downloading update [ 02/18/21 15:30:38 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 7066 7064 65 0 0 6999 ---------------------------------------------------------------------- [ SFS_Toxic_BD ] Downloading update [ 02/18/21 15:30:41 ] .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 27957 27956 12 0 0 27944 ---------------------------------------------------------------------- Saving DNSBL statistics... completed [ 02/18/21 15:30:50 ] ------------------------------------------------------------------------ Assembling DNSBL database...... completed [ 02/18/21 15:30:53 ] Adding DNSBL Unbound mode (Resolver adv. setting) DNS Resolver ( enabled ) unbound.conf modifications: Added DNSBL Unbound mode Saving new DNSBL web server configuration to port [ 8081 and 8443 ] VIP address(es) configured New DNSBL certificate created Restarting DNSBL Service Stopping Unbound Resolver Unbound stopped in 1 sec. Additional mounts: No changes required. Starting Unbound Resolver... completed [ 02/18/21 15:31:09 ] DNSBL update [ 349816 | PASSED ]... completed [ 02/18/21 15:31:10 ] ------------------------------------------------------------------------ ===[ GeoIP Process ]============================================ ===[ IPv4 Process ]================================================= [ Abuse_Feodo_C2_v4 ] Downloading update .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 118 118 118 [ Pass ] ----------------------------------------------------------------- [ Abuse_IPBL_v4 ] Downloading update [ 02/18/21 15:31:12 ] .. 200 OK. completed .. Empty file, Adding '127.1.7.7' to avoid download failure. ------------------------------ Original Master Final ------------------------------ 0 1 1 [ Pass ] ----------------------------------------------------------------- [ Abuse_SSLBL_v4 ] Downloading update [ 02/18/21 15:31:13 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 99 94 94 [ Pass ] ----------------------------------------------------------------- [ CINS_army_v4 ] Downloading update .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 15000 15000 15000 [ Pass ] ----------------------------------------------------------------- [ ET_Block_v4 ] Downloading update [ 02/18/21 15:31:15 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 1338 1223 1223 [ Pass ] ----------------------------------------------------------------- [ ET_Comp_v4 ] Downloading update [ 02/18/21 15:31:16 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 4961 4884 4884 [ Pass ] ----------------------------------------------------------------- [ ISC_Block_v4 ] Downloading update [ 02/18/21 15:31:18 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 21 22 22 [ Pass ] ----------------------------------------------------------------- [ Spamhaus_Drop_v4 ] Downloading update [ 02/18/21 15:31:20 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 972 2 2 [ Pass ] ----------------------------------------------------------------- [ Spamhaus_eDrop_v4 ] Downloading update [ 02/18/21 15:31:21 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 84 76 76 [ Pass ] ----------------------------------------------------------------- [ Talos_BL_v4 ] Downloading update [ 02/18/21 15:31:22 ] .. 200 OK. completed .. ------------------------------ Original Master Final ------------------------------ 796 730 730 [ Pass ] ----------------------------------------------------------------- ===[ Aliastables / Rules ]================================ Firewall rule changes found, applying Filter Reload ** Restarting firewall filter daemon ** ===[ FINAL Processing ]===================================== [ Original IP count ] [ 23387 ] [ Final IP Count ] [ 22150 ] ===[ Deny List IP Counts ]=========================== 22150 total 15000 /var/db/pfblockerng/deny/CINS_army_v4.txt 4884 /var/db/pfblockerng/deny/ET_Comp_v4.txt 1223 /var/db/pfblockerng/deny/ET_Block_v4.txt 730 /var/db/pfblockerng/deny/Talos_BL_v4.txt 118 /var/db/pfblockerng/deny/Abuse_Feodo_C2_v4.txt 94 /var/db/pfblockerng/deny/Abuse_SSLBL_v4.txt 76 /var/db/pfblockerng/deny/Spamhaus_eDrop_v4.txt 22 /var/db/pfblockerng/deny/ISC_Block_v4.txt 2 /var/db/pfblockerng/deny/Spamhaus_Drop_v4.txt 1 /var/db/pfblockerng/deny/Abuse_IPBL_v4.txt ====================[ Empty Lists w/127.1.7.7 ]================== Abuse_IPBL_v4.txt ===[ DNSBL Domain/IP Counts ] =================================== 349816 total 143310 /var/db/pfblockerng/dnsbl/Maltrail_BD.txt 122595 /var/db/pfblockerng/dnsbl/C19_CTC.txt 27944 /var/db/pfblockerng/dnsbl/SFS_Toxic_BD.txt 13486 /var/db/pfblockerng/dnsbl/SWC.txt 10871 /var/db/pfblockerng/dnsbl/EasyList.txt 8913 /var/db/pfblockerng/dnsbl/Adaway.txt 6999 /var/db/pfblockerng/dnsbl/Spam404.txt 6671 /var/db/pfblockerng/dnsbl/MVPS.txt 3031 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt 2500 /var/db/pfblockerng/dnsbl/D_Me_ADs.txt 1988 /var/db/pfblockerng/dnsbl/Krisk_C19.txt 1478 /var/db/pfblockerng/dnsbl/Yoyo.txt 23 /var/db/pfblockerng/dnsbl/D_Me_Tracking.txt 6 /var/db/pfblockerng/dnsbl/Juniper.txt 1 /var/db/pfblockerng/dnsbl/D_Me_Malv.txt 0 /var/db/pfblockerng/dnsbl/MDS_Immortal.fail 0 /var/db/pfblockerng/dnsbl/MDS.fail 0 /var/db/pfblockerng/dnsbl/MDL.txt 0 /var/db/pfblockerng/dnsbl/D_Me_Malw.txt ====================[ IPv4/6 Last Updated List Summary ]============== Feb 11 04:49 Spamhaus_eDrop_v4 Feb 17 02:30 ET_Block_v4 Feb 17 02:30 ET_Comp_v4 Feb 18 03:33 Spamhaus_Drop_v4 Feb 18 14:18 CINS_army_v4 Feb 18 14:19 ISC_Block_v4 Feb 18 15:05 Talos_BL_v4 Feb 18 15:25 Abuse_SSLBL_v4 Feb 18 15:30 Abuse_Feodo_C2_v4 Feb 18 15:31 Abuse_IPBL_v4 ====================[ DNSBL Last Updated List Summary ]============== Jul 31 2015 D_Me_Tracking Sep 5 2018 Juniper Jan 31 2020 D_Me_ADs Jul 10 2020 D_Me_Malw Jul 10 2020 D_Me_Malv Nov 12 19:17 MDL Dec 15 05:07 MVPS Feb 15 02:18 Adaway Feb 15 05:50 Yoyo Feb 16 05:48 SWC Feb 18 09:30 Krisk_C19 Feb 18 13:50 C19_CTC Feb 18 15:00 SFS_Toxic_BD Feb 18 15:02 EasyPrivacy Feb 18 15:21 EasyList Feb 18 15:30 Maltrail_BD Feb 18 15:30 Spam404 =============================================================== Database Sanity check [ PASSED ] ------------------------ Masterfile/Deny folder uniq check Deny folder/Masterfile uniq check Sync check (Pass=No IPs reported) ---------- Alias table IP Counts ----------------------------- 22150 /var/db/aliastables/pfB_PRI1_v4.txt pfSense Table Stats ------------------- table-entries hard limit 400000 Table Usage Count 22176 UPDATE PROCESS ENDED [ 02/18/21 15:31:28 ]
