Netgate 2100 Max CPu pings 100% when download large files
-
Hello - if this is the wrong board let me know where I should be posting. I have a Netgate 2100 max that's been issue free for until now. Lately when I download a large file (multiple gigs) my cpu jumps to 100% and my Pfsense dashboard crashes with a 50x error and it wont come back until I kill the download. Even then the cpu is running 70% or more. I don't have tons of traffic constantly hitting my nics so I am not sure how to troubleshoot. Attached are some screen shots. Let me know what I can do to get anyone logs or whatever to help.
Thanks - Scott
-
Forgot system activity.
Normal ops:
During download and when the pfsense+ dashboard stops working before the 50x error.
-
@northernsky what pfSense version? 25.03 will have a fix for dashboard CPU usage.
And what Internet speed?
-
@SteveITS said in Netgate 2100 Max CPu pings 100% when download large files:
@northernsky what pfSense version? 25.03 will have a fix for dashboard CPU usage.
And what Internet speed?
Hello,
Pfsense+ 24.11 and my internet speed is around half a gig but the cpu ping to 100% and the dashboard error usually during any download.
-
@northernsky try without the dashboard visible.
-
Heya - Yeah I did try that while downloading while in the system activity page to watch the CPU activity. The page just froze and stopped updating.
-
Are you using limiters?
-
Nope no limiters.
-
Suggestions :
The dashboard isn't a static page, content get refreshed every x seconds, and the data collection process costs CPU cycles.
Remove the totally useless process, like "servicewathdog", it's a real PITA.
I'm pretty sure most of your traffic is "TLS" (https, mail over TLS, etc) so you can stop using "ClamAV" as it can't see and check the payload of the traffic : it's encrypted.
I presume you don't visit any http sites anymore.Imho : remove "squid" also. It can be useful, bit normally you would have opted for a big iron, not a little arm processor.
About the 'dpinfger' pings that get lost : ICMP (ping) is a low priority protocol.
When you download at the max avaible ISP speed, the pipe "from the Internet to your pfSense is full". In that case, upstream, the decision is made for you : higher priotty traffic comes first, lower priority passes when there is room avaible. You wind up having packet loss as ping was the looser.
This can have a nasty side effect : loss is 100 %, and if the default action is : reset the WAN interface to re establish a good connection, the things get even worse : as now all interface (WAN) related processes get restarted, eating away even more CPU cycles. This included the resolver, that get restarted .... (and now the servciewathdog- mess kicks in and makes thing even more worse)
Solution : create limiters to leave some spare room for ICMP ?! Or just live with it an disable the action :
-
@northernsky The 2100 can do around/roughly 600 Mbps without additional packages. You could try disabling Clam and/squid (which is deprecated anyway) and testing. The web GUI not responding seems like it’s really overloaded? Try “top” at a command line.
-
@Gertjan Thank you! I took your advice and removed the packages you suggested. I did not disable the gateway action and will do some research in limiters to see if I even need them and if not I will disable the gateway action.
Thanks - Scott
-
@SteveITS I will putty in and run top when I test it again. I appreciate the guidance on this forum.
-
Yup, that. Try without the webgui connected at all.
The usage page you showed though has all the CPU usage in passing traffic as you'd expect for a large file maxing out the WAN bandwidth.
This was something that just started happening lately? Anything changed? You updated pfSense maybe?
-
@stephenw10
Nope no changes not since I updated to the latest patch a few days after it came out. -
@northernsky said in Netgate 2100 Max CPu pings 100% when download large files:
I took your advice and removed the packages you suggested. I did not disable the gateway action and will do some research in limiters to see if I even need them and if not I will disable the gateway action.
@Gertjan gave some excellent advice. Removing clamav and squid (and anything associated with squid) was an excellent decision.
FWIW, I would like to second the recommendation to disable the Gateway Monitoring Action. You have a single WAN, so there is usually no downside to doing this. All the monitoring action ends up doing is restarting a bunch of processes that usually don't need to be restarted in a single WAN configuration, which can result in a cascade failure as @Gertjan described.
-
@SteveITS so I deleted the Clam and squid packages and I ran the download closed out of the webgui with just putty running and the cpu looks fine. Unless someone sees something i don't.
I also ran it again with the download going with the webgui up and putty overlayed with steam capped at 60 megs. I was able to reload the webgui without issues or it giving me the 50x error message, but the cpu on that stills pings at 100% but in reality top is saying 17% for system. I guess don't believe the dashboard widgets? Also capping steam helps from saturating my pipe.
-
@northernsky said in Netgate 2100 Max CPu pings 100% when download large files:
@SteveITS so I deleted the Clam and squid packages and I ran the download closed out of the webgui with just putty running and the cpu looks fine. Unless someone sees something i don't.
Your CPU is still 100% pegged. 76% in interrupt, which seems really high to me... @stephenw10, does this seem high to you?
-
Yeah it's showing 0% idle, so 100% used. You need to use
top -HaSPto see everything using CPU cycles there.That seems very high usage if it's 60Mbps. It's in the ball park if that's 60MBps.
-
@stephenw10 Ok So here is the top again with the switch and the MB/s.
Steams cap setting:
Stream download:
-
@northernsky try 50000.
