updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr'

pirsig · Feb 21, 2022, 2:12 AM

Hi, I recently upgraded/re-installed pfsense 22.01 on my SG-1100 a couple of days ago. Since then I have noticed consistently high CPU activity (50-100%).

htop shows the culprit appears to be '/sbin/pfctl -vvsr'

Based on previous threads I could find on similar issues I tried limiting state table to 65,000 to no effect.
Attached is my /tmp/rules/debug file

rules.debug.txt

I also have the following packages installed:

acme
Avahi
haproxy
pfblockerNG-devel
snort

I ran the offending command

time pfctl -vvsr

which gave a massive output:
pfctl_output.txt
ending:

5.677u 4.668s 0:13.07 79.0%	194+179k 1+0io 0pf+0w

I'm not sure what the issue is, can anyone help me?
Thanks so much for your help!

rcoleman-netgate · Feb 21, 2022, 2:58 AM

@pirsig You have those packages installed, but what are running? pfblocker? snort? The two of those running together will likely make it choke, and snort is not a good idea for the eMMC that is in the 1100, either. In fact those and haproxy are fairly CPU intense.

SteveITS · Feb 21, 2022, 4:22 AM

@rcoleman-netgate said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

snort is not a good idea for the eMMC

Is your comment based on the assumption of high logging volume? Just thinking about this thread on eMMC.

@pirsig, the vv would be verbose output. Is this running all the time to generate the consistently high CPU? I haven't noticed consistent CPU usage on the 2100 I've upgraded, nor do I see that process (though I didn't watch for a long time). I did notice this regression though, in this thread.

Another thing I've seen is if something is generating a lot of logs, multiple bzip instances run to compress them, and therefore Netgate recommends turning off log compression on slower CPUs. But, you'd see bzip in the process list if that was the issue.

pirsig · Feb 21, 2022, 4:32 AM

@rcoleman-netgate

thanks for the information. I was running all the packages listed simultaneously. I uninstalled snort since I don't think I need to be running it.

However it didn't seem to affect the CPU usage and the pfctl process is still running.
@SteveITS - Yes the process is running continuously. And yes, I also have the same latency issue with the filter reload process.

pirsig · Feb 21, 2022, 5:35 AM

This post is deleted!

pirsig · Feb 21, 2022, 6:11 AM

@pirsig
I have narrowed down the issue to pfblockerNG-devel. Uninstalling it removes the CPU load and pfctl process.

SteveITS · Feb 21, 2022, 2:55 PM

@pirsig Interesting, were you using DNSBL? We have not used that at our clients, but from what I read here it can take a lot of memory and CPU to process large block lists. We use pfBlocker-devel for geoIP aliases and block feeds.

pirsig · Feb 21, 2022, 3:09 PM

@steveits

I am using DNSBL as well as several aliases for selective routing purposes. However it is not the DNSBL functionality since when I go to Status -> Services and stop the 'pfb_filter' service, leaving the 'pfb_dnsbl' service running the CPU usage drops immediately.

I should mention that I was previously running the pfsense 22.01 beta build with the same configuration (pfBlockerNG-devel + DNSBL, OpenVPN, torrents, snort, etc) with no issue, so I think it has to do with the new stable version.

SteveITS · Feb 21, 2022, 3:19 PM

@pirsig Hm, I'm not seeing that on three routers with 22.01 or 2.6. I'd guess it's related to the regression discussed above.

serbus · Feb 21, 2022, 4:07 PM

@steveits said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

from what I read here it can take a lot of memory and CPU to process large block lists

There is Wildcard Blocking (TLD) option on the Firewall > pfBlockerNG > DNSBL settings page that can really increase the load. Check the infoblock text for that option.

@pirsig said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

/sbin/pfctl -vvsr

I did a really quick check and dont see this command being called from anywhere except the hidden status page (after logging in, browse to <pfsense url>/status.php). Not sure what would be calling it or why it would be pinning the CPU...

Edit: it looks like the pfblocker widget uses a variation of that call - "pfctl -vv -sr"...?

John

rcoleman-netgate · Feb 21, 2022, 4:11 PM

@steveits said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

Is your comment based on the assumption of high logging volume? Just thinking about this thread on eMMC.

Yes. Most users that are doing that on an eMMC have stated they were unaware of the recommendation to use an SSD/HDD for that task (and since the 1100 lacks that ability) they had a configuration that destroyed their eMMC in weeks, or months.

SteveITS · Feb 21, 2022, 4:38 PM

@rcoleman-netgate Interesting. I surely was unaware of that recommendation. Weeks to months seems absurdly low and makes we wonder about defaulting to ZFS per that thread and one or two other threads I've seen.

I'll pursue in that thread.

pirsig · Feb 21, 2022, 4:44 PM

@serbus said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

There is Wildcard Blocking (TLD) option on the Firewall > pfBlockerNG > DNSBL settings page that can really increase the load. Check the infoblock text for that option.

I checked the DNSBL settings and wildcard blocking is unchecked.

@serbus said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

Edit: it looks like the pfblocker widget uses a variation of that call - "pfctl -vv -sr"...?

Thanks for the suggestion, I deleted the pfblocker widget from my home page, but the CPU is still pinned :/

However I did check the hidden status page as you suggested, and found a little more detail on the process, it seems to be called from the filterlog script?

 0.0  4.4  62604 43828  -  S    10:10     0:19.21 |-- /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
32.2  0.3  11904  3192  -  R    11:39     0:03.80 | `-- /sbin/pfctl -vvsr

rcoleman-netgate · Feb 21, 2022, 4:46 PM

@steveits It comes from this published document:
https://www.netgate.com/supported-pfsense-plus-packages

The applications with the high logging or writes to the eMMC are noted as recommending an SSD or HDD.
There are, obviously, ways around that but they have their own caveats.

BBcan177 · Feb 22, 2022, 4:29 PM

@pirsig

See here for a patch:

https://www.reddit.com/r/pfBlockerNG/comments/sk9txi/ip_block_logging_not_working_pfsense_260rc/hvv99s1/?utm_source=reddit&utm_medium=web2x&context=3

pirsig · Feb 22, 2022, 4:56 PM

@bbcan177 said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

https://www.reddit.com/r/pfBlockerNG/comments/sk9txi/ip_block_logging_not_working_pfsense_260rc/hvv99s1/?utm_source=reddit&utm_medium=web2x&context=3

Installed the patch and it solved it! Thanks!