21.02 Sudden lockup
-
@stephenw10 I am running pfBlockerNG-devel 3.0.0_10. The alias/blocklists are whatever is OOTB, I didn't customize anything.
By "lockup" I mean the LAN/WAN interfaces are completely unresponsive. Console access is fine.
P.S. I cross posted to this thread to make sure that @BBcan177 (the developer of pfBlockerNG) is aware.
-
@stephenw10 pfBlockerNG net 2.1.4_24
I do have several lists loading.
My Max table Entry is: 4000000Another thing I notice is that vnstad keeps crashing:
vnstatd Status Traffic Totals data collection daemon -
@stephenw10 said in 21.02 Sudden lockup:
Are you running the dev version of pfBlocker?
Does it 'lock up' as it's running the reload or once it's completed?
How large are you aliases/blocklists?
I have the non-dev package running and am not seeing that but only have limited dnsbl lists loaded.
Steve
Running the pfblockerng-devel 3.0.0_10, didn't customize anything, just ran the wizard, so it's a normal amount of aliases/blocklists.
No, it doesn't 'lock up' as it's running the reload or once it's completed, it finishes the process, and it's triggered if I reboot in this phase:Syncing OpenVPN settings...done.
Configuring firewall.Segmentation fault (core dumped) <<<
Starting CRON... done. -
Ah, OK so part way through the boot after loading the alises/lists?
Can you give us an idea of the numbers? I have:
===[ Native List IP Counts ] =================================== 8513 total 7311 /var/db/pfblockerng/native/Google.txt 968 /var/db/pfblockerng/native/Spamhaus_drop.txt 181 /var/db/pfblockerng/native/Facebook.txt 53 /var/db/pfblockerng/native/Netflix.txt ===[ DNSBL Domain/IP Counts ] =================================== 21346 total 16998 /var/db/pfblockerng/dnsbl/Easylist_Default.txt 4342 /var/db/pfblockerng/dnsbl/Easylist_Privacy.txt 6 /var/db/pfblockerng/dnsbl/Custom_List_custom.txtSteve
-
@stephenw10 said in 21.02 Sudden lockup:
Ah, OK so part way through the boot after loading the alises/lists?
Exactly.
Packages installed: Acme, NUT, pfBlockerng-devel, aws-wizard, ipsec-profile-wizard.
It doesn't seem to be a memory issue:
===[ Deny List IP Counts ]=========================== 22138 total 15000 /var/db/pfblockerng/deny/CINS_army_v4.txt 4884 /var/db/pfblockerng/deny/ET_Comp_v4.txt 1223 /var/db/pfblockerng/deny/ET_Block_v4.txt 732 /var/db/pfblockerng/deny/Talos_BL_v4.txt 118 /var/db/pfblockerng/deny/Abuse_Feodo_C2_v4.txt 94 /var/db/pfblockerng/deny/Abuse_SSLBL_v4.txt 76 /var/db/pfblockerng/deny/Spamhaus_eDrop_v4.txt 8 /var/db/pfblockerng/deny/ISC_Block_v4.txt 2 /var/db/pfblockerng/deny/Spamhaus_Drop_v4.txt 1 /var/db/pfblockerng/deny/Abuse_IPBL_v4.txt ====================[ Empty Lists w/127.1.7.7 ]================== Abuse_IPBL_v4.txt ===[ DNSBL Domain/IP Counts ] =================================== 349821 total 143310 /var/db/pfblockerng/dnsbl/Maltrail_BD.txt 122595 /var/db/pfblockerng/dnsbl/C19_CTC.txt 27951 /var/db/pfblockerng/dnsbl/SFS_Toxic_BD.txt 13486 /var/db/pfblockerng/dnsbl/SWC.txt 10871 /var/db/pfblockerng/dnsbl/EasyList.txt 8913 /var/db/pfblockerng/dnsbl/Adaway.txt 6999 /var/db/pfblockerng/dnsbl/Spam404.txt 6671 /var/db/pfblockerng/dnsbl/MVPS.txt 3031 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt 2500 /var/db/pfblockerng/dnsbl/D_Me_ADs.txt 1986 /var/db/pfblockerng/dnsbl/Krisk_C19.txt 1478 /var/db/pfblockerng/dnsbl/Yoyo.txt 23 /var/db/pfblockerng/dnsbl/D_Me_Tracking.txt 6 /var/db/pfblockerng/dnsbl/Juniper.txt 1 /var/db/pfblockerng/dnsbl/D_Me_Malv.txt 0 /var/db/pfblockerng/dnsbl/MDS_Immortal.fail 0 /var/db/pfblockerng/dnsbl/MDS.fail 0 /var/db/pfblockerng/dnsbl/MDL.txt 0 /var/db/pfblockerng/dnsbl/D_Me_Malw.txt ====================[ IPv4/6 Last Updated List Summary ]============== Feb 11 04:49 Spamhaus_eDrop_v4 Feb 17 02:30 ET_Block_v4 Feb 17 02:30 ET_Comp_v4 Feb 18 03:33 Spamhaus_Drop_v4 Feb 18 18:05 Talos_BL_v4 Feb 18 18:18 CINS_army_v4 Feb 18 18:39 ISC_Block_v4 Feb 18 18:55 Abuse_Feodo_C2_v4 Feb 18 18:55 Abuse_SSLBL_v4 Feb 18 19:01 Abuse_IPBL_v4 ====================[ DNSBL Last Updated List Summary ]============== Jul 31 2015 D_Me_Tracking Sep 5 2018 Juniper Jan 31 2020 D_Me_ADs Jul 10 2020 D_Me_Malw Jul 10 2020 D_Me_Malv Nov 12 19:17 MDL Dec 15 05:07 MVPS Feb 15 02:18 Adaway Feb 15 05:50 Yoyo Feb 16 05:48 SWC Feb 18 15:02 EasyPrivacy Feb 18 15:30 Krisk_C19 Feb 18 17:50 C19_CTC Feb 18 18:50 EasyList Feb 18 19:00 SFS_Toxic_BD Feb 18 19:00 Maltrail_BD Feb 18 19:00 Spam404 =============================================================== Database Sanity check [ PASSED ] ------------------------ Masterfile/Deny folder uniq check Deny folder/Masterfile uniq check Sync check (Pass=No IPs reported) ---------- Alias table IP Counts ----------------------------- 22138 /var/db/aliastables/pfB_PRI1_v4.txt pfSense Table Stats ------------------- table-entries hard limit 400000 Table Usage Count 22164 UPDATE PROCESS ENDED [ 02/18/21 19:01:58 ]
-
@stephenw10 My SG-3100 locked up again with pfblocker disabled. no cron running.
Unless the cron is running even though is been told to disable.
You guys have a bigger issue here. -
@mcury I don't think is a pfblocker issue TBH.
Mine keeps locking up even though is fully disabled, I am now going to uninstall it. -
@ffuentes Try to set the pfblocker cron settings to run once a day, instead of one hour to confirm if the problem will happen to you again.
Or completely remove the pfblockerng package. If it happens again even without the package installed (not only disabled). We will have a better understanding about it.. -
@mcury I just uninstalled it. Let see how far the rabbit holes go. :/
-
@ffuentes said in 21.02 Sudden lockup:
@mcury I just uninstalled it. Let see how far the rabbit holes go. :/
Let's hope that we all here have the same problem, one fix is faster than two or more :)
-
@mcury I also disabled and got a lockup again just now. This is bad news, folks. I don’t think it’s pfBlockerNG either.
How do I downgrade back to 2.4.5p1?
-
@bldnightowl said in 21.02 Sudden lockup:
@mcury I also disabled and got a lockup again just now. This is bad news, folks. I don’t think it’s pfBlockerNG either.
How do I downgrade back to 2.4.5p1?
Try to remove pfblockerng package, just to make sure tick that setting to remove the configuration too, this option is inside pfblockerng.
-
How do I downgrade back to 2.4.5p1?
Reinstall & restore Config Backup
-
@mcury said in 21.02 Sudden lockup:
@ffuentes said in 21.02 Sudden lockup:
@mcury I just uninstalled it. Let see how far the rabbit holes go. :/
Let's hope that we all here have the same problem, one fix is faster than two or more :)
Completely agree. I am trying to avoid going back to the previous version to try and help narrow this down.
But is really causing a lot of pain as it seems to be very consistent. (That's good news in a technical way)I have a console up and running via USB to see if I can catch anything, but is a HARD lockup. You can't get any output at all.
-
@ffuentes said in 21.02 Sudden lockup:
@mcury said in 21.02 Sudden lockup:
@ffuentes said in 21.02 Sudden lockup:
@mcury I just uninstalled it. Let see how far the rabbit holes go. :/
Let's hope that we all here have the same problem, one fix is faster than two or more :)
Completely agree. I am trying to avoid going back to the previous version to try and help narrow this down.
But is really causing a lot of pain as it seems to be very consistent. (That's good news in a technical way)I have a console up and running via USB to see if I can catch anything, but is a HARD lockup. You can't get any output at all.
Hm, the console also stops? You can't reboot it through the console?
Did you upgrade from 2.4.5p1 or performed a clean 21.02 install? -
@mcury I did an upgrade. 2.4.5p1 -> 21.02.
One hour later, instant lockup, and happily ever since (not... but you get the idea
)Worth mentioning that after hours of troubleshooting I did do a factory default, and reconfigure everything. Still the same behavir.
-
@ffuentes I've been watching this thread all day. It seems the focus is turning to pfblockerNG, yet I have uninstalled all my packages. Removing PfblockerNG-devel did lengthen the time between lockups, it did not eliminate the problem.
edit: I am rebooting via USB cable->putty->console
-
@bulfinch Thanks for this update.
I suspect not a pfblocker issue, but something a bit deeper.
When we get lockups like this, sometimes it hints a kernel issue.
Maybe a wild process, but even then you get a ping response when the OS is running low on resources. -
@ffuentes @Bulfinch I tried to upgrade from 2.4.5p1, got a lot of errors, so decided to perform a clean install and configure everything from scratch..
I suggest you guys do the same, I lost a few hours yesterday reinstalling everything.. but I'm not getting this 'lockups'.
The only thing I restored from my 2.4.5p1 backup was the aliases..I saved a screenshot from my update from 2.4.5p1 to 21.02 (this screen doesn't show all the errors, there were plenty more..
-
@mcury For me, the console has never frozen. And I can still access the WAN from the console (pings, etc.) -- it's the LAN and the DNS Resolver that go down.
I have fully removed pfBlockerNG and will give this one more shot before throwing in the towel. But if the root cause really is pfBlockerNG somehow, I hope that gets resolved very quickly -- it's very important to have that.
How can I downgrade though to 2.4.5p1 though if I can't reach the internet because my pfSense is down? I did backup the configuration before I upgraded to 21.02, but that backup doesn't include the full pfSense software does it?
