Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5

tman222

@BBcan177 said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@jwj and others
Try to reduce the setting in: pfSense > Advanced > Firewall & NAT > Firewall Maximum Table Entries as per this thread:
https://www.reddit.com/r/PFSENSE/comments/fsx4mx/fix_for_tmprulesdebug28_cannot_define_table_pfb/

Thanks @BBcan177 - mine is currently set at the default (2 Million). Should it be reduced below that to help mitigate the load and latency spikes?

Thanks again.

BBcan177

@tman222
I have no idea. This is why I am asking users to try lower values and see how it responds.

bmeeks

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

So, we already have the "fix".

It appears to me, from reading the FreeBSD 11/STABLE Release Notes, that the patch submitted for pfctl was accepted and incorporated. So I would assume that patch also made its way into pfSense-2.4.5.

It's possible that patch has some unintended side effects. And since the problems seem to be triggered by users doing things that cause pfctl to come into play (reloading filters or updating alias tables), that is more circumstantial evidence of a possible link to recent pfctl changes and the utilization/latency issue.

@tman222 I just played about with this. Removed block bogons on my WAN. Tried various table size settings from 100000 to, what for me was the default. 4000000. Didn't notice any difference. Checked the system log to look for memory allocation issues, there were none at each setting.

BBcan177

@jwj
You need to run a "Filter Reload" and/or a Reboot to fully ensure it took effect.

@BBcan177 I did, filter reload. It warns you if you increase it such that a reboot is required.

@BBcan177 @bmeeks Thanks for all your help and involvement since 2.4.5 was released. Your packages are fine but you have done your best to help as many as possible.

nzkiwi68

I'm experiencing the same issues as reported here under my post "Upgrade HA cluster 2.4.4-p3 to 2.4.5 - persistent CARP maintenance mode causes gateway instability" https://forum.netgate.com/topic/151698/upgrade-ha-cluster-2-4-4-p3-to-2-4-5-persistent-carp-maintenance-mode-causes-gateway-instability

2 sites now running really badly. Both sites running 10Gbase-SR with multi VLANs on the 10 Gb interfaces.

SiteA - main problem
Cannot have both firewalls up, primary and backup. If you do, zero VPN traffic passes over direct traditional site to site IPSEC or over the VTI routed FRR interfaces.
Left with the backup firewall powered off and the site is working.

SiteB - main problem
Massive instability following a reboot, and it just carries on and on, with all three gateways on both the primary and secondary firewall going nuts. The firewalls stagger and drop packets. In the end left the backup firewall powered off and after about 10-15 minutes following a reboot, the gateways stop going offline and the firewall settles down and becomes stable.

@bmeeks @BBcan177

I turned off bogons. I deactivated pfblocker. I set the max table size to 60000. Rebooted. The reboot was quick again, like 2.4.4-p3. I ping6 google.com from a lan side device while doing a filter reload. The ping times don't change by any meaningful amount.

Makes me wonder if the pfctl fix was overlooked when the release was built?

Set max table size to 100000. Reboot. Reboot is longer again. Turn on pfblocker and some ip lists that add up to more than 60000 but less than 100000. Problem returns. Latency when reloading the filters. Bigger tables bigger problem.

Edited to add: the max table size doesn't appear to make any difference, other than setting a hard limit on table size. The actual size of the aliases/tables is what triggers the problem.

It sure looks to me that anything over that sixty some thousand mark causes the issue. I could be wrong, wouldn't be the first time, but this sure looks like the issue.

bmeeks

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks @BBcan177

I turned off bogons. I deactivated pfblocker. I set the max table size to 60000. Rebooted. The reboot was quick again, like 2.4.4-p3. I ping6 google.com from a lan side device while doing a filter reload. The ping times don't change by any meaningful amount.

Makes me wonder if the pfctl fix was overlooked when the release was built?

Set max table size to 100000. Reboot. Reboot is longer again. Turn on pfblocker and some ip lists that add up to more than 60000 but less than 100000. Problem returns. Latency when reloading the filters. Bigger tables bigger problem.

Edited to add: the max table size doesn't appear to make any difference, other than setting a hard limit on table size. The actual size of the aliases/tables is what triggers the problem.

It sure looks to me that anything over that sixty some thousand mark causes the issue. I could be wrong, wouldn't be the first time, but this sure looks like the issue.

The actual patch file can be accessed here: https://security.FreeBSD.org/patches/EN-20:04/pfctl.patch. What the patch does is remove the former arbitrary hardcoded limit of 65,535 (defined as PF_TABLES_MAX_REQUEST) and allows the use of a sysctl parameter instead. Deeper research into the other pf related source code would be required to determine if allowing that larger PF_TABLES_MAX_REQUEST value has an adverse impact.

Looking a bit farther into what the patch actually does gives me a theory. The 65,535 number does not appear to be a limit on the number of IP addresses in a given table. It appears, instead, to be a limit on the number of tables or addresses you can add to the firewall during a single call to the corresponding ioctl() function. That limit was formerly hardcoded to 65,535. Now, with the addition of a sysctl variable for customizing this limit, I can envision a scenario where with a very high value for this new sysctl value that you are overloading the other pf areas. In particular, you would be requesting "too many tables and/or addresses in a single ioctl() call". So this may well be why lowering the value improves performance! You are no longer "overloading" the other ioctl routines that are actually creating the tables or addresses in RAM.

getcom

@getcom said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

The system is not accessible for minutes if anything changed.
As I added some new VLANs it never came back, I had to go onsite for a reboot which is not so easy at the moment because everybody is working from home.

It happened again: I need a VLAN tag on one interface which was previously working as normal device.

created a new VLAN
changed name of parent device because name should be reused in the VLAN device
changed static IPv4 to none
press Save
apply change
bamm...connection interrupted, VPN stays connected, but without any traffic...it does not come back, VPN reconnection not working
no chance to connect over the second gateway
again I have to go onsite

This is a dual WAN/tier 1/tier 2 setup.
With 2.4.5 in a multiwan setup you cannot change anything on the interfaces without loosing the firewall.
This is a show blocker and loosing larger lists too.
Tomorrow I will drive to the customer and rollback to the 2.4.4-p3 release and the previous pfB until there is a fix available..

Seeing as this appears to affect so many people, is it worth writing up a decent Redmine ticket and submitting it?

bmeeks

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks @BBcan177

I turned off bogons. I deactivated pfblocker. I set the max table size to 60000. Rebooted. The reboot was quick again, like 2.4.4-p3. I ping6 google.com from a lan side device while doing a filter reload. The ping times don't change by any meaningful amount.

Makes me wonder if the pfctl fix was overlooked when the release was built?

Set max table size to 100000. Reboot. Reboot is longer again. Turn on pfblocker and some ip lists that add up to more than 60000 but less than 100000. Problem returns. Latency when reloading the filters. Bigger tables bigger problem.

Edited to add: the max table size doesn't appear to make any difference, other than setting a hard limit on table size. The actual size of the aliases/tables is what triggers the problem.

It sure looks to me that anything over that sixty some thousand mark causes the issue. I could be wrong, wouldn't be the first time, but this sure looks like the issue.

See my later edit to my post here: https://forum.netgate.com/topic/151690/increased-memory-and-cpu-spikes-causing-latency-outage-with-2-4-5/91. I have a theory why a value less that 65,536 seems to work better.

getcom

This post is deleted!

@bmeeks Makes sense, since they're all loaded with one call. Total number of all tables would have to be under that number.

Also makes me take note of the fact that the "fork" of pfsense is still on 11.2 even as they have been more aggressive with pursuing freebsd upgrades historically.

nzkiwi68

@getcom Yep, that looks like my SiteA issue, if both firewalls are on, zero VPN traffic passes yet CARP is fine.

Power down the backup firewall and VPN traffic instantly starts passing, or
Power down the primary firewall and after failover stuff occurs, VPN traffic starts passing

bmeeks

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks Makes sense, since they're all loaded with one call. Total number of all tables would have to be under that number.

Also makes me take note of the fact that the "fork" of pfsense is still on 11.2 even as they have been more aggressive with pursuing freebsd upgrades historically.

A possible solution for loading very large numbers of tables or IP addresses would be split them up into chunks of 65,000 or less and iterate through a loop making a series of ioctl() function calls to create the tables or addresses.

getcom

@bmeeks said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks @BBcan177

I turned off bogons. I deactivated pfblocker. I set the max table size to 60000. Rebooted. The reboot was quick again, like 2.4.4-p3. I ping6 google.com from a lan side device while doing a filter reload. The ping times don't change by any meaningful amount.

Makes me wonder if the pfctl fix was overlooked when the release was built?

Set max table size to 100000. Reboot. Reboot is longer again. Turn on pfblocker and some ip lists that add up to more than 60000 but less than 100000. Problem returns. Latency when reloading the filters. Bigger tables bigger problem.

Edited to add: the max table size doesn't appear to make any difference, other than setting a hard limit on table size. The actual size of the aliases/tables is what triggers the problem.

It sure looks to me that anything over that sixty some thousand mark causes the issue. I could be wrong, wouldn't be the first time, but this sure looks like the issue.

The actual patch file can be accessed here: https://security.FreeBSD.org/patches/EN-20:04/pfctl.patch. What the patch does is remove the former arbitrary hardcoded limit of 65,535 (defined as PF_TABLES_MAX_REQUEST) and allows the use of a sysctl parameter instead. Deeper research into the other pf related source code would be required to determine if allowing that larger PF_TABLES_MAX_REQUEST value has an adverse impact.

Looking a bit farther into what the patch actually does gives me a theory. The 65,535 number does not appear to be a limit on the number of IP addresses in a given table. It appears, instead, to be a limit on the number of tables or addresses you can add to the firewall during a single call to the corresponding ioctl() function. That limit was formerly hardcoded to 65,535. Now, with the addition of a sysctl variable for customizing this limit, I can envision a scenario where with a very high value for this new sysctl value that you are overloading the other pf areas. In particular, you would be requesting "too many tables and/or addresses in a single ioctl() call". So this may well be why lowering the value improves performance! You are no longer "overloading" the other ioctl routines that are actually creating the tables or addresses in RAM.

This makes sense, but as a consequence, this then makes pfBlockerNG unusable or with larger values, pfSense itself makes it unusable. The choice is yours...if you currently need both, you will need a fix or rollback in the near future.

bmeeks

@getcom said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@jwj said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

@bmeeks @BBcan177

I turned off bogons. I deactivated pfblocker. I set the max table size to 60000. Rebooted. The reboot was quick again, like 2.4.4-p3. I ping6 google.com from a lan side device while doing a filter reload. The ping times don't change by any meaningful amount.

Makes me wonder if the pfctl fix was overlooked when the release was built?

Set max table size to 100000. Reboot. Reboot is longer again. Turn on pfblocker and some ip lists that add up to more than 60000 but less than 100000. Problem returns. Latency when reloading the filters. Bigger tables bigger problem.

Edited to add: the max table size doesn't appear to make any difference, other than setting a hard limit on table size. The actual size of the aliases/tables is what triggers the problem.

It sure looks to me that anything over that sixty some thousand mark causes the issue. I could be wrong, wouldn't be the first time, but this sure looks like the issue.

The actual patch file can be accessed here: https://security.FreeBSD.org/patches/EN-20:04/pfctl.patch. What the patch does is remove the former arbitrary hardcoded limit of 65,535 (defined as PF_TABLES_MAX_REQUEST) and allows the use of a sysctl parameter instead. Deeper research into the other pf related source code would be required to determine if allowing that larger PF_TABLES_MAX_REQUEST value has an adverse impact.

Looking a bit farther into what the patch actually does gives me a theory. The 65,535 number does not appear to be a limit on the number of IP addresses in a given table. It appears, instead, to be a limit on the number of tables or addresses you can add to the firewall during a single call to the corresponding ioctl() function. That limit was formerly hardcoded to 65,535. Now, with the addition of a sysctl variable for customizing this limit, I can envision a scenario where with a very high value for this new sysctl value that you are overloading the other pf areas. In particular, you would be requesting "too many tables and/or addresses in a single ioctl() call". So this may well be why lowering the value improves performance! You are no longer "overloading" the other ioctl routines that are actually creating the tables or addresses in RAM.

This makes sense, but as a consequence, this then makes pfBlockerNG unusable or with larger values, pfSense itself makes it unusable. The choice is yours...if you currently need both, you will need a fix or rollback in the near future.

I'm going to go back and research how this was coded in FreeBSD 11.2/RELEASE (which is what 2.4.4_p3 was based on). Will be interesting to see what may have changed from 11.2 to 11.3 of FreeBSD.

LATER EDIT: The PF_TABLES_MAX_REQUEST limit was a hardcoded value (a defined constant, actually) in the 11.2-RELEASE of FreeBSD. The recent FreeBSD 11.3/STABLE patch I referenced in an earlier post above is titled "Missing Tuneable", but I think it might really should have been titled "New Tuneable". Using "Missing" in the title could be taken by some to mean it was there previously in earlier versions and was erroneously removed and is being restored by the patch. Instead, it appears to me the patch added this system tuneable as a new feature and removed the former hardcoded limit. Anecdotal evidence from posters in this thread indicates allowing that former hard limit to now be essentially unlimited can produce undesirable side effects.

@bmeeks

If I'm not mistaken that "tunable" wasn't in 2.4.4-p3 in System->Advanced->Firewall tab. Could someone double check that please.

So, in 2.4.4-p3 how did the big, ~110k, bogonsv6 table get loaded in 2.4.4-p3? If I set it to 60000 and then turn on bogons I get the can not allocate memory error and Alert.