2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl
-
@jimp amazing detective work to have already isolated it down to a specific upstream change in freebsd!
Please let the community know if theres anything we can do to help, test, build kernels etc.
Thanks for all you do!
-
Seems like the fix for this will land in 2.4.5-p1 which is coming soon. But, this person was desperate. So as a test, I put the following in their /boot/loader.conf.local:
kern.smp.disabled=1After rebooting, the problem is gone. It's only been an hour, but not a single hiccup so far (
fingers crossed).This is on an SG-3100. -
I can confirm this fixes the issue completely!
Its not enough to edit -> system -> tunables.
You have to edit /boot/loader.conf.local manually.
Its the same as limiting a VM to only 1 core.
-
kern.smp.disabled=1After rebooting, the problem is gone. It's only been an hour, but not a single hiccup so far (
fingers crossed).This is on an SG-3100.But, a quick look online suggests to me that this is disabling all multi CPU support. On busy systems this could be a problem switching your multi core (example XG-1537 with 8 cores plus hyper threading) into a single CPU system!
I recommend if you can wait on 2.4.4-p3 or limp along on 2.4.5 if you can.
For me, I shall wait for the official patch / release.
-
@nzkiwi68 said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:
this is disabling all multi CPU support [..] I recommend if you can wait on 2.4.4-p3 or limp along on 2.4.5 if you can.
100% good advice. In this case I had to upgrade due to another problem, so I was "stuck" on 2.4.5 with a remote system and had no other option. When 2.4.5-p1 / 2.5.0 come out this should not be needed. Losing 1 core is a fair trade for regaining the stability.
-
Agreed. Limiting to 1 core is a viable option on a home network or on a small B2B setup. A busy connection running IDS/IPS would be running full load and not have spare ressources left.
-
@luckman212 In version 2.5 everything is working ok. I switched from 2.4.4-p3 to 2.4.5 and I have problem with this bug (my FW is on Hyper-V), then I switched to 2.5 and everything is ok. This week was pfSense 2.5 changed to FreeBSD 12.1-STABLE.
-
@DD I've just done a clean install from the latest public download, and it shows FreeBSD version 11.3 stable, on their site it shows 12.0 stable https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html
Update:
I've just updated to the development release and it's now running 12.1 stable, do we believe the issues are not present in FreeBSD 12.1 stable? -
Is there any development on this situation? Are we having some 2.4.5_p1 coming up soon to solve this?
I have the pfSense running on an Hyper-V and there's absolutely nothing I can do on the pfSense without having an huge impact (outage, traffic gone, website and console unresponsive).I've attempted reverting back to 2.4.4_p3 but... unfortunately you guys removed the image available for me to reinstall it so... oops!
-
Read the thread. All the info is here. At least read from https://forum.netgate.com/post/908806 down.
-
@jimp said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:
Read the thread. All the info is here. At least read from https://forum.netgate.com/post/908806 down.
Instead of replying arrogantly, why not assuming I've just done that 5 times and try to give a more "to the point" answer?
Why do you think I am asking if there's any development over this situation? Maybe because of your post stating that you're "assessing the next steps" (the same you just gave me a link to go and read again...).Reducing my system to a single core installation is not doable (done that... performance went to the ground) and unfortunately reducing the tables to 65000 entries makes pfBlockerNG go wild and start throwing errors.
So... Expecting any development over this situation? is there any kind of patch we can do? can we "override" the setting that exists that causes this problem?
-
While terse, my response was not "arrogant". It's not arrogant to expect people to read a thread with all of the information (and links to more information). Far too often people pop in and expect others to do the leg work for them when all of the information is here. If you'd read from my linked post down and followed the links, you'd have all your answers. I just skimmed them and checked. The info is here, and in the links (Like https://redmine.pfsense.org/issues/10414 linked in https://forum.netgate.com/post/909130).
There are multiple comments with suggested workarounds and how to enact them.
-
Working in IT as I do, I understand your point and where you're coming from... However would have been much nicer to start with a: "Have you read the thread already? Did you notice that we've already pointed to possible workarounds?"
To which I'd reply that I've seen the workarounds and that none of them work for me (unfortunately I must add) at least without having another impact....Thanks for the link to the redmine part, I did not notice that one previously, apologies!
I've noticed that there's a patch for the kernel and good results are visible... Do we know or have an idea when that's coming out?
Thanks for your time.
-
We don't all have time to be nice, especially when there is no indication of what the person posting has done. If you had included the additional info about exactly what you had tried in your first comment, that would have been even more helpful. We're not mind readers.
The workarounds do work, at a possible performance penalty (hurts different deployments worse than others). The main workaround is reducing the CPU cores to 1, which is mentioned several times, and that will work 100% of the time for everyone. If that was too much of a performance hit, then you will need to disable all the large tables, move it to hardware with faster single cores, or go back to 2.4.4-p3.
No ETA on 2.4.5-p1 other than "Soon" (as in Weeks, not months). Still some testing left to do on other issues being rolled into 2.4.5-p1 to address other issues discovered in the release.
-
@ghosterius said in 2.4.5.a.20200110.1421 and earlier: High CPU usage from pfctl:
Is there any development on this situation? Are we having some 2.4.5_p1 coming up soon to solve this?
I have the pfSense running on an Hyper-V and there's absolutely nothing I can do on the pfSense without having an huge impact (outage, traffic gone, website and console unresponsive).I've attempted reverting back to 2.4.4_p3 but... unfortunately you guys removed the image available for me to reinstall it so... oops!
Sure. Just reduce to one virtual CPU. EOF
-
So glad I found this thread.
I have had this with my 2.4.5 install using proxmox.
Now reverted to 1 core for the time being which seems to just about cope.
Thread followed to watch out for 2.4.5-p1.
Thanks for debugging this guys! -
https://redmine.pfsense.org/projects/pfsense/roadmap#2.4.5-p1 looks very soon.
-Rico
-
No ETA on 2.4.5-p1 other than "Soon" (as in Weeks, not months)
looks very soon.
now you have done it..
it seems that @techpro2004 is still away ... I'm sure he will come back to ask.. when ... "Soon" (as in Weeks, not months)! -
I bet he can't be far away since at least you now pinged him.
-Rico
-
I love that I am very new to this forum but still know exaclty what you are talking about.
