*SOLVED* pfSense freezing for a second or two every 15 minutes
-
I have a pfSense 2.4.5p1 VM running on ESXi, has been working fine for years. It is running with 2 cores of L5520 @ 2.27GHz and 2GB of RAM.
Recently though, I have been experiencing freezing during video meetings, which I've traced back to pfSense which is having some abnormal utilization spikes every 15 minutes, at exactly :00, :15, :30 and :45 minutes.
I see that there is a cronjob called /etc/rc.filter_configure_sync every 15 minutes. What does this do?
I also noticed that while running top from cli that pfctl would spike the CPU at 99.67% for a few seconds at the exact time the "freezing" is occurring.What I did find while looking through the Status / Monitoring was that the Queues display was showing huge spikes exactly every 15 minutes (see qVoIP in graph below), despite there not being the same level of traffic to back these up.
I believe the issue might be related to the pfctl invocation, where it is because it is flushing the queues or something because while the freezing occurs there isn't any actual traffic lost, just delayed a lot; I see 4000-5000ms ping times of the LAN interface at the time the problem occurs.Any suggestions on how to troubleshoot this would be appreciated.
VMWare performance chat of the pfSense instance, showing CPU spikes
-
@awebster said in pfSense freezing for a second or two every 15 minutes:
filter_configure_sync
That runs every 15mins if you have any scheduled firewall rules defined. It updates the ruleset and removes open states.
It wouldn't normally be noticeable though.
If you you just reload the ruleset from Status > Filter Reload do you see the same spike/lagging?The description looks exactly like this:
https://redmine.pfsense.org/issues/10414
But that is fixed in 2.4.5p1. Are you sure you're still on 2.4.5?Steve
-
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
If you you just reload the ruleset from Status > Filter Reload do you see the same spike/lagging?
Good suggestion. Filter Reload does indeed cause the problem on demand.
The system is on 2.4.5p1
Admittedly this issue does tick many of the same boxes seen in https://redmine.pfsense.org/issues/10414, however, the IPv6 bogons table flush/load timing test that was used to see if the problem was triggered does not appear to be the case here.
-
Can you test the same mitigation shown there, setting a single CPU core only?
That would confirm it's likely the same cause at least.
What does
uname -ashow? I wonder if you have somehow ended up still running a 2.4.5 kernel.Steve
-
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
What does uname -a show? I wonder if you have somehow ended up still running a 2.4.5 kernel.
FreeBSD portal-vm 11.3-STABLE FreeBSD 11.3-STABLE #243 abf8cba50ce(RELENG_2_4_5): Tue Jun 2 17:53:37 EDT 2020 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense amd64
I'll try out the 1 core suggestion when I can reboot the box without affecting the other users here.
-
Hmm, definitely 2.4.5p1 then.
-
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
Can you test the same mitigation shown there, setting a single CPU core only?
Tested the setup with one CPU core, same problem, it doesn't seem to be related to the number of cores.
-
I mean it could be just a very large ruleset? If your config especially complex?
Also I assume you do have some scheduled rules that are triggering this every 15m?
This just started happening in 2.4.5p1? There was no significant change that created it?
Steve
-
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
I mean it could be just a very large ruleset? If your config especially complex?
The config is complex-ish. I am using pfBlocker so it does add some load with large tables. I'll try turning it off and see if it makes a difference and report back.
Also I assume you do have some scheduled rules that are triggering this every 15m?
No scheduled rules, but /etc/rc.filter_configure_sync runs every 15 minutes out of /etc/crontab
This just started happening in 2.4.5p1? There was no significant change that created it?
Yes, I can't pinpoint exactly when, but it appears to have started in the last 3-4 weeks. No changes other than what pfBlocker might be doing.
When doing the filter reload as far as I can tell, it seems to get bogged down after the Setting up SCRUB information. Is there a way to get the reload status output on the CLI since the GUI refresh is affected by the packet delay problem.
Generating ALTQ queues Loading filter rules Setting up logging information Setting up SCRUB information Processing down interface states Running plugins Running plugins (pf) Plugins completed. Done -
This crontab entry:
0,15,30,45 * * * * root /etc/rc.filter_configure_syncIs only generated by having an active scheduled firewall rule AFAIK.
pfBlocker adding something maybe?
Steve
-
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
Is only generated by having an active scheduled firewall rule AFAIK.
Yup, found a scheduled rule and removed it, so the cronjob is gone.
While it doesn't resolve the underlying problem of packet delay during afilter reload, at least it isn't happening every 15 minutes, so the Zoom/Teams/Meet users are happy now
Thanks @stephenw10 for helping narrow down the source of the problem.
-
@awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
@stephenw10 said in pfSense freezing for a second or two every 15 minutes:
Is only generated by having an active scheduled firewall rule AFAIK.
Yup, found a scheduled rule and removed it, so the cronjob is gone.
While it doesn't resolve the underlying problem of packet delay during afilter reload, at least it isn't happening every 15 minutes, so the Zoom/Teams/Meet users are happy now
Thanks @stephenw10 for helping narrow down the source of the problem.My guess is you have pfBlockerNG setup with GEO blocking, and you have some enourmous blocking lists that causes a CPU spike whenever the list needs to be reloaded.
One typical mistake with GEO blocking is that many people create a block rule for almost the entire world. That will cause a HUGE block list of addresses to load every time.
The real way to solve that problem is to reverse the problem - create a allow rule from the few countries you wish to allow access from, and let the rest of the world hit the default deny rule in your firewall.
This is how it’s done:
1: Disable pfBlockerNG automatic rule generation
2: Have pfBlocker generate a ALLOW ALIAS list with the few regions/countries needed.
3: Use that list as source on your public facing services. -
@keyser said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
One typical mistake with GEO blocking is that many people create a block rule for almost the entire world. That will cause a HUGE block list of addresses to load every time.
The real way to solve that problem is to reverse the problem - create a allow rule from the few countries you wish to allow access from, and let the rest of the world hit the default deny rule in your firewall.Agreed, and that is how I've configured it; my rule is to allow only North American IP blocks (v4 and v6) in toward the published services behind the firewall. Perhaps those lists have just gotten too big?!
-
The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.
Steve
-
@stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.
Yup, I think the smoking gun might be pfB_NAmerica_v6.txt (only USA and Canada selected). If IPv6 list grows much more I can see that it'll be problematic for many people running pfBlocker.
Oddly GeoIP hasn't updated since Aug 5th, but it appears to only update once a month.
I ran/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcto force it to update, and the next pfblocker cron run reduced the size of the resulting tables (see 17:00:00 pfblocker update log output below), namely this:Updating: pfB_NAmerica_v6 538 addresses added.97343 addresses deleted.Sadly, despite the updated DB, the filter reload still causes packet delay/loss.
I disabled any rules referencing pfB_*_v6, but the problem persists.
Question: Does the fact that a rule is disabled still load the tables with pfctl ?Here is the output of the pfblocker log before and after GeoIP update:
[ firehol3 ] ( No remote timestamp/md5 unchanged ) Update not required [ adware ] Previous download failed. Re-attempt download UPDATE PROCESS START ===[ DNSBL Process ]================================================ [ EasyList ] exists. [ adware ] Downloading update .. 404 Not Found [ DNSBL_Adware - adware ] Download FAIL [ 08/20/21 16:00:01 ] Firewall and/or IDS are not blocking download. Restoring previously downloaded file . ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 175 163 3 0 0 160 ---------------------------------------------------------------------- [ DNSBL_IP ] Updating aliastable... no changes. Total IP count = 10 ------------------------------------------ Assembling database... completed Validating database... completed Reloading Unbound.... completed DNSBL update [ 2206 | PASSED ]... completed [ 08/20/21 16:00:05 ] ------------------------------------------ ===[ Continent Process ]============================================ [ pfB_Asia_v4 ] exists. [ pfB_Asia_v6 ] exists. [ pfB_Europe_v4 ] exists. [ pfB_Europe_v6 ] exists. [ 08/20/21 16:00:06 ] [ pfB_NAmerica_v4 ] exists. [ pfB_NAmerica_v6 ] exists. [ 08/20/21 16:00:09 ] [ pfB_Top_v4 ] exists. [ 08/20/21 16:00:10 ] [ pfB_Top_v6 ] exists. ===[ IPv4 Process ]================================================= [ firehol3 ] exists. [ shdrop ] exists. ===[ IPv6 Process ]================================================= [ shdrop6_v6 ] exists. ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload No Changes to Aliases, Skipping pfctl Update ===[ Kill States ]================================================== No matching states found ====================================================================== ===[ FINAL Processing ]===================================== [ Original IP count ] [ 986408 ] ===[ Permit List IP Counts ]========================= 716191 total 613306 /var/db/pfblockerng/permit/pfB_NAmerica_v6.txt 33299 /var/db/pfblockerng/permit/pfB_Europe_v4.txt 32977 /var/db/pfblockerng/permit/pfB_Europe_v6.txt 23984 /var/db/pfblockerng/permit/pfB_NAmerica_v4.txt 7102 /var/db/pfblockerng/permit/pfB_Asia_v4.txt 5523 /var/db/pfblockerng/permit/pfB_Asia_v6.txt ===[ Deny List IP Counts ]=========================== 94563 total 62130 /var/db/pfblockerng/deny/pfB_Top_v4.txt 17782 /var/db/pfblockerng/deny/firehol3.txt 13609 /var/db/pfblockerng/deny/pfB_Top_v6.txt 1004 /var/db/pfblockerng/deny/shdrop.txt 38 /var/db/pfblockerng/deny/shdrop6_v6.txt ===[ DNSBL Domain/IP Counts ] =================================== 2216 total 2046 /var/db/pfblockerng/dnsbl/EasyList.txt 160 /var/db/pfblockerng/dnsbl/adware.txt 10 /var/db/pfblockerng/dnsbl/EasyList.ip 0 /var/db/pfblockerng/dnsbl/adware.fail ====================[ Last Updated List Summary ]============== Aug 5 03:00 pfB_Asia_v4 Aug 5 03:00 pfB_Asia_v6 Aug 5 03:00 pfB_Europe_v4 Aug 5 03:00 pfB_Europe_v6 Aug 5 03:00 pfB_NAmerica_v4 Aug 5 03:02 pfB_NAmerica_v6 Aug 5 03:02 pfB_Top_v4 Aug 5 03:02 pfB_Top_v6 Aug 17 00:05 shdrop Aug 20 00:00 shdrop6_v6 Aug 20 07:00 firehol3 IPv4 alias tables IP count ----------------------------- 145349 IPv6 alias tables IP count ----------------------------- 665415 Alias table IP Counts ----------------------------- 810764 total 613306 /var/db/aliastables/pfB_NAmerica_v6.txt 62130 /var/db/aliastables/pfB_Top_v4.txt 33299 /var/db/aliastables/pfB_Europe_v4.txt 32977 /var/db/aliastables/pfB_Europe_v6.txt 23984 /var/db/aliastables/pfB_NAmerica_v4.txt 17782 /var/db/aliastables/pfB_firehol3.txt 13609 /var/db/aliastables/pfB_Top_v6.txt 7102 /var/db/aliastables/pfB_Asia_v4.txt 5523 /var/db/aliastables/pfB_Asia_v6.txt 1004 /var/db/aliastables/pfB_SpamhausDROP.txt 38 /var/db/aliastables/pfB_SpamhausIPv6DROP.txt 10 /var/db/aliastables/pfB_DNSBLIP.txt pfSense Table Stats ------------------- table-entries hard limit 2000000 Table Usage Count 924925 UPDATE PROCESS ENDED [ 08/20/21 16:00:12 ] CRON PROCESS START [ 08/20/21 17:00:00 ] [ firehol3 ] ( No remote timestamp/md5 unchanged ) Update not required [ adware ] Previous download failed. Re-attempt download UPDATE PROCESS START ===[ DNSBL Process ]================================================ [ EasyList ] exists. [ adware ] Downloading update .. 404 Not Found [ DNSBL_Adware - adware ] Download FAIL [ 08/20/21 17:00:01 ] Firewall and/or IDS are not blocking download. Restoring previously downloaded file . ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 175 163 3 0 0 160 ---------------------------------------------------------------------- [ DNSBL_IP ] Updating aliastable... no changes. Total IP count = 10 ------------------------------------------ Assembling database... completed Validating database... completed Reloading Unbound.... completed DNSBL update [ 2206 | PASSED ]... completed [ 08/20/21 17:00:05 ] ------------------------------------------ ===[ Continent Process ]============================================ [ pfB_Asia_v4 ] Changes found... Updating Aggregation Stats: ------------------ Original Final ------------------ 7100 7099 ------------------ [ pfB_Asia_v6 ] Changes found... Updating [ pfB_Europe_v4 ] Changes found... Updating Aggregation Stats: ------------------ Original Final ------------------ 92785 33496 ------------------ [ pfB_Europe_v6 ] Changes found... Updating [ pfB_NAmerica_v4 ] Changes found... Updating Aggregation Stats: ------------------ Original Final ------------------ 134087 23023 ------------------ [ pfB_NAmerica_v6 ] Changes found... Updating [ pfB_Top_v4 ] Changes found... Updating Aggregation Stats: ------------------ Original Final ------------------ 74426 62411 ------------------ [ pfB_Top_v6 ] Changes found... Updating ===[ IPv4 Process ]================================================= [ firehol3 ] exists. [ 08/20/21 17:02:36 ] [ shdrop ] exists. ===[ IPv6 Process ]================================================= [ shdrop6_v6 ] exists. ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload Updating: pfB_Asia_v4 33 addresses added.36 addresses deleted. Updating: pfB_Asia_v6 107 addresses added.23 addresses deleted. Updating: pfB_Europe_v4 621 addresses added.424 addresses deleted. Updating: pfB_Europe_v6 3551 addresses added.2828 addresses deleted. Updating: pfB_NAmerica_v4 467 addresses added.1428 addresses deleted. Updating: pfB_NAmerica_v6 538 addresses added.97343 addresses deleted. Updating: pfB_Top_v4 1088 addresses added.807 addresses deleted. Updating: pfB_Top_v6 10751 addresses added.8019 addresses deleted. ===[ Kill States ]================================================== No matching states found ====================================================================== ===[ FINAL Processing ]===================================== [ Original IP count ] [ 896803 ] ===[ Permit List IP Counts ]========================= 616774 total 513839 /var/db/pfblockerng/permit/pfB_NAmerica_v6.txt 33710 /var/db/pfblockerng/permit/pfB_Europe_v6.txt 33496 /var/db/pfblockerng/permit/pfB_Europe_v4.txt 23023 /var/db/pfblockerng/permit/pfB_NAmerica_v4.txt 7099 /var/db/pfblockerng/permit/pfB_Asia_v4.txt 5607 /var/db/pfblockerng/permit/pfB_Asia_v6.txt ===[ Deny List IP Counts ]=========================== 97576 total 62411 /var/db/pfblockerng/deny/pfB_Top_v4.txt 17782 /var/db/pfblockerng/deny/firehol3.txt 16341 /var/db/pfblockerng/deny/pfB_Top_v6.txt 1004 /var/db/pfblockerng/deny/shdrop.txt 38 /var/db/pfblockerng/deny/shdrop6_v6.txt ===[ DNSBL Domain/IP Counts ] =================================== 2216 total 2046 /var/db/pfblockerng/dnsbl/EasyList.txt 160 /var/db/pfblockerng/dnsbl/adware.txt 10 /var/db/pfblockerng/dnsbl/EasyList.ip 0 /var/db/pfblockerng/dnsbl/adware.fail ====================[ Last Updated List Summary ]============== Aug 17 00:05 shdrop Aug 20 00:00 shdrop6_v6 Aug 20 07:00 firehol3 Aug 20 17:00 pfB_Asia_v4 Aug 20 17:00 pfB_Asia_v6 Aug 20 17:00 pfB_Europe_v4 Aug 20 17:00 pfB_Europe_v6 Aug 20 17:00 pfB_NAmerica_v4 Aug 20 17:02 pfB_NAmerica_v6 Aug 20 17:02 pfB_Top_v4 Aug 20 17:02 pfB_Top_v6 IPv4 alias tables IP count ----------------------------- 144863 IPv6 alias tables IP count ----------------------------- 569497 Alias table IP Counts ----------------------------- 714360 total 513839 /var/db/aliastables/pfB_NAmerica_v6.txt 62411 /var/db/aliastables/pfB_Top_v4.txt 33710 /var/db/aliastables/pfB_Europe_v6.txt 33496 /var/db/aliastables/pfB_Europe_v4.txt 23023 /var/db/aliastables/pfB_NAmerica_v4.txt 17782 /var/db/aliastables/pfB_firehol3.txt 16341 /var/db/aliastables/pfB_Top_v6.txt 7099 /var/db/aliastables/pfB_Asia_v4.txt 5607 /var/db/aliastables/pfB_Asia_v6.txt 1004 /var/db/aliastables/pfB_SpamhausDROP.txt 38 /var/db/aliastables/pfB_SpamhausIPv6DROP.txt 10 /var/db/aliastables/pfB_DNSBLIP.txt pfSense Table Stats ------------------- table-entries hard limit 2000000 Table Usage Count 831173 UPDATE PROCESS ENDED [ 08/20/21 17:02:43 ]–A.
-
@awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
@stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.
Yup, I think the smoking gun might be pfB_NAmerica_v6.txt (only USA and Canada selected). If IPv6 list grows much more I can see that it'll be problematic for many people running pfBlocker.
Oddly GeoIP hasn't updated since Aug 5th, but it appears to only update once a month.
I ran/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcto force it to update, and the next pfblocker cron run reduced the size of the resulting tables (see 17:00:00 pfblocker update log output below), namely this:Updating: pfB_NAmerica_v6 538 addresses added.97343 addresses deleted.Sadly, despite the updated DB, the filter reload still causes packet delay/loss.
I disabled any rules referencing pfB_*_v6, but the problem persists.
Question: Does the fact that a rule is disabled still load the tables with pfctl ?Seems likely that list is causing your problem. Some 600.000 entries is no small amount (and more than one third of the hard limit in pfBlockerNG).
There is the question of hardware as well. If you are trying to do this on fx. A SG-1100/SG-2100 or smaller, you have to remember that the single thread CPU core performance and memory bandwidth of that ARM processor, is less than half of fx. an Intel Atom based box. And the Atom is once again less than half as fast as a common 7th or 8th gen Intel Core CPU. Boxes with that caliber of CPU might not even register loading the same list.
EDIT: Just noticed you are running in a VM on a rather old Intel XEON. The virtualised single thread performance of that CPU is likely struggeling to reach a modern level Atom CPU core. So Hardware may well be part of the problem.
-
@keyser Yeah, just looked it up - Its way slower than a modern Atom, even before virtualisation.
So hardware is very likely pressing the issue. -
@awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
I disabled any rules referencing pfB_*_v6, but the problem persists.
Question: Does the fact that a rule is disabled still load the tables with pfctl ?No, I would not expect it to populate any tables that are not actually in use.
The other table that can be massive is bogons v6. But we have many users loading those on relatively low powered systems.
Steve
-
@stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:
The other table that can be massive is bogons v6.
Not too horrible at 124297 entries.
Some tables as pfctl seems them
-pa-r-- bogonsv6 Addresses: 124297 Cleared: Fri Aug 20 01:07:38 2021 References: [ Anchors: 0 Rules: 1 ] Evaluations: [ NoMatch: 537822 Match: 0 ] In/Block: [ Packets: 0 Bytes: 0 ] In/Pass: [ Packets: 0 Bytes: 0 ] In/XPass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ] Out/XPass: [ Packets: 0 Bytes: 0 ] -pa---- pfB_NAmerica_v6 Addresses: 505938 Cleared: Fri Aug 20 01:07:38 2021 References: [ Anchors: 0 Rules: 0 ] Evaluations: [ NoMatch: 2 Match: 12 ] In/Block: [ Packets: 0 Bytes: 0 ] In/Pass: [ Packets: 0 Bytes: 0 ] In/XPass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ] Out/XPass: [ Packets: 0 Bytes: 0 ] -pa-r-- pfB_Top_v6 Addresses: 16341 Cleared: Fri Aug 20 01:07:38 2021 References: [ Anchors: 0 Rules: 3 ] Evaluations: [ NoMatch: 51426 Match: 150 ] In/Block: [ Packets: 150 Bytes: 10800 ] In/Pass: [ Packets: 0 Bytes: 0 ] In/XPass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ] Out/XPass: [ Packets: 0 Bytes: 0 ]
