2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl

luckman212

@jimp This is wonderful news! Good luck on the endeavor. Can I ask 2 questions?

is there any way to remotely downgrade to 2.4.4 from 2.4.5? I think I have a remote SG3100 hitting this issue and it was on 2.4.2 earlier today, I upgraded it... it's 20 miles away :(
in case (1) is not possible, is this bug also present in current 2.5.0 builds?

sorry if these q's are already answered but I'm on mobile and so haven't read the whole thread (came here via a reddit link)

Rico

AFAIK there is no way to downgrade online. The only thing you can do is reflash 2.4.4 with usb thumb.

-Rico

teiva

@andrew_241 Not sure if you the post about a FreeBSD bug affecting this version, but since I've dropped my vCPU count from 4 to 1 and although the firewall is busier than usual when loading or doing a filter reload the server is not locking up anymore like it was before. Anyway just thought i'd let you know.

teiva

@luckman212 said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:

if these q's are already answered but I'm on mobile and so haven't read the whole thread (came here via a reddit link)

This is great news. Dropping to 1vCPU has temporarily mitigated my issue.

jimp

@luckman212 said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:

is there any way to remotely downgrade to 2.4.4 from 2.4.5? I think I have a remote SG3100 hitting this issue and it was on 2.4.2 earlier today, I upgraded it... it's 20 miles away :(

No

in case (1) is not possible, is this bug also present in current 2.5.0 builds?

We haven't tested 2.5.0, but I don't think it does. That could change, though, as we're getting the 2.5.0 builds up onto stable/12 and it may be there.

luckman212

@jimp Thank you again. Reading through redmine #10414 it seems like the temporary workaround is:

set System > Advanced > Firewall & NAT > Firewall Maximum Table Entries to <65535 — e.g. 65000
disable Block bogon networks on all interfaces

The thing is, I've done both of those things on my only 2.4.5 system (a remote SG-3100) and I believe I am still hitting this problem.

Take a look at this gateway monitoring graph — never seen spikes like this! They're almost all exactly 20 minutes apart. I checked /etc/crontab for any possible jobs that might be running on 20 minute intervals (found nothing). I also searched the filesystem for any references to 1200 seconds and found just one, in /usr/local/www/interfaces_bridge_edit.php stating "...the timeout of address cache entries [..] default is 1200 seconds". Don't know if that's anything.

Multiple conversations with the ISP and they are assuring me the problem is "on my end" — of course. I'd normally set up some Wireshark captures between the ISP equipment and pfSense in this type of situation, but since I'm remote that isn't possible.

It seems like people are also reporting success on virtual machines by setting CPU cores to 1. Is there any boot flag that we can set here to disable SMP e.g. kern.smp.disabled=1 or hint.lapic.1.disable=1 or ~~is that not necessary~~?

update: see below -- disabling SMP seems to have helpred.

jimp

Not just bogons but anything that loads large tables. It could be a URL table alias, pfBlockerNG, or something else.

luckman212

@jimp said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:

anything that loads large tables. It could be a URL table alias, pfBlockerNG

This unit doesn't have any aliases defined, and pfBNG is not installed (no packages installed actually).

akm22562

@luckman212 In my case, I had CARP configured with bogons blocked on the WAN interface.

I can't afford to disable CARP. Disabling bogons was a big help.

Anyway, just my $0.02.

carl2187

@jimp amazing detective work to have already isolated it down to a specific upstream change in freebsd!

Please let the community know if theres anything we can do to help, test, build kernels etc.

Thanks for all you do!

luckman212

Seems like the fix for this will land in 2.4.5-p1 which is coming soon. But, this person was desperate. So as a test, I put the following in their /boot/loader.conf.local:

kern.smp.disabled=1

After rebooting, the problem is gone. It's only been an hour, but not a single hiccup so far (fingers crossed).This is on an SG-3100.

Cool_Corona

I can confirm this fixes the issue completely!

Its not enough to edit -> system -> tunables.

You have to edit /boot/loader.conf.local manually.

Its the same as limiting a VM to only 1 core.

nzkiwi68

kern.smp.disabled=1
After rebooting, the problem is gone. It's only been an hour, but not a single hiccup so far (fingers crossed).This is on an SG-3100.

But, a quick look online suggests to me that this is disabling all multi CPU support. On busy systems this could be a problem switching your multi core (example XG-1537 with 8 cores plus hyper threading) into a single CPU system!

I recommend if you can wait on 2.4.4-p3 or limp along on 2.4.5 if you can.

For me, I shall wait for the official patch / release.

luckman212

@nzkiwi68 said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:

this is disabling all multi CPU support [..] I recommend if you can wait on 2.4.4-p3 or limp along on 2.4.5 if you can.

100% good advice. In this case I had to upgrade due to another problem, so I was "stuck" on 2.4.5 with a remote system and had no other option. When 2.4.5-p1 / 2.5.0 come out this should not be needed. Losing 1 core is a fair trade for regaining the stability.

Cool_Corona

@luckman212

Agreed. Limiting to 1 core is a viable option on a home network or on a small B2B setup. A busy connection running IDS/IPS would be running full load and not have spare ressources left.

DD

@luckman212 In version 2.5 everything is working ok. I switched from 2.4.4-p3 to 2.4.5 and I have problem with this bug (my FW is on Hyper-V), then I switched to 2.5 and everything is ok. This week was pfSense 2.5 changed to FreeBSD 12.1-STABLE.

Mr_JinX

@DD I've just done a clean install from the latest public download, and it shows FreeBSD version 11.3 stable, on their site it shows 12.0 stable https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html

Update:
I've just updated to the development release and it's now running 12.1 stable, do we believe the issues are not present in FreeBSD 12.1 stable?

ghosterius

Is there any development on this situation? Are we having some 2.4.5_p1 coming up soon to solve this?
I have the pfSense running on an Hyper-V and there's absolutely nothing I can do on the pfSense without having an huge impact (outage, traffic gone, website and console unresponsive).

I've attempted reverting back to 2.4.4_p3 but... unfortunately you guys removed the image available for me to reinstall it so... oops!

jimp

Read the thread. All the info is here. At least read from https://forum.netgate.com/post/908806 down.

ghosterius

@jimp said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:

Read the thread. All the info is here. At least read from https://forum.netgate.com/post/908806 down.

Instead of replying arrogantly, why not assuming I've just done that 5 times and try to give a more "to the point" answer?
Why do you think I am asking if there's any development over this situation? Maybe because of your post stating that you're "assessing the next steps" (the same you just gave me a link to go and read again...).

Reducing my system to a single core installation is not doable (done that... performance went to the ground) and unfortunately reducing the tables to 65000 entries makes pfBlockerNG go wild and start throwing errors.

So... Expecting any development over this situation? is there any kind of patch we can do? can we "override" the setting that exists that causes this problem?