SG2100 100% CPU usage post upgrade to 24.11

mtarbox

Cross posted on the pfsense reddit.
Upgraded from 24.3 to 24.11 on a SG2100 this morning.
Noted high, like pegged,100% usage post update. Rebooted, same. Let it sit for 5+ hours waiting for it to settle out, same.
Ideas? Logfiles you might need?
Minimal packages, ipsec and pfBlockerNG are the primary and ipsec currently has nothing connected.
firewall 24_11.JPG

SteveITS

@mtarbox Possibly helpful, https://forum.netgate.com/topic/194267/high-cpu-on-2100-after-upgrade-to-rc-from-beta/. Have not upgraded any yet myself.

mtarbox

@SteveITS Kea was enabled from before with no DNS registration/Early DNS registration.
I enabled them, and will sit on it for a bit.
I have a whopping 21 leases.

marcosm

The tldr is:
There was an issue with dashboard widgets not refreshing at the intended intervals. That's been fixed, but it also means more requests / higher resource usage while the dashboard is opened. This is likely what's happening in your case. You may ignore it (monitor usage over SSH instead) or bump up the widget intervals.

For reference and troubleshooting tips, see:
https://forum.netgate.com/topic/190824/cpu-load-on-1100/

mtarbox

@marcosm and @SteveITS thank you for the responses and links.
@jimp thank you for moving my post to the appropriate spot.
I nuked the dashboard. I disabled update check in the system widget, and CPU usage dropped significantly. Definitely feel more comfortable.

Of note, this unit is almost 4 years old, and was due to be replaced in next years budget. I kept my boss in the loop, and when Sharon@netgate sent out the email yesterday with the sale price, I was told to buy a new 4200. Just waiting on a response from sales.

I'll decom this one, and keep it as a spare. Maybe fire it up to update as needed.

bumperjeep

I'm having the same issue, any tips?

I'm running this at home, so I have pretty minimal packages.
wireguard
vnstatd
syslogd
nut
kea-dhcp4
avahi

Any tips?

stephenw10

See this post: https://forum.netgate.com/post/1191398

mtarbox

@bumperjeep Like I wrote, I nuked my dashboard, and removed every widget, disabled update check in the system widget, and then added widgets as see what happened with each one. So far, so good on my end.
I went from 99% to 30-50%, which was where I was normally. YMMV.

smirkis

this was an odd issue that seemed to clear up after removing all widgets and re-adding them back.

mtarbox

@smirkis I noted the same thing. Luckily our new 4200 arrives on Monday. Already have the new config from TAC. I’ll decom this one and install a new storage drive and install the new image from TAC as well, and keep it as a spare.

tve

I have 100% CPU on SG-2440 dashboard post 24.11 upgrade when GUI is open. From ssh shell top command CPU has about 0% idle when GUI active. GUI is very unresponsive and has crashed twice. As soon as I log out from GUI, CPU idle starts to change with tops over 90% idle.
No Wireguard in use. KEA was enabled on 24.03.
This is old device and several packages active, Most active are ntopng and unbound. Snort and pfBlocker also active. Memory usage is reasonable 50% of 4GB.
Not SG2100 or SG1100 https://forum.netgate.com/topic/190824/cpu-load-on-1100/24 but similar behavior.

stephenw10

What widgets do you have on the dash?

Did you try the suggested patch to revert the widget refresh method?

mhijazi

Hello. I was searching the forums for the same issue before posting a new thread and I found this one. I am not entirely sure if this issue is after a specific version update, but the CPU utilization is always very high, and when I download large files, it will be always at 100% and the UI will stop responding until I stop the download. I am not really running any CPU intensive services or packages. it's pretty much at default settings. Even on idle there will always be those 2 processes that are always high CPU usage. Below is an output:

last pid: 34956; load averages: 0.47, 0.45, 0.35 up 24+03:17:11 22:13:02
158 threads: 3 running, 136 sleeping, 19 waiting
CPU: 0.3% user, 0.5% nice, 1.7% system, 0.9% interrupt, 96.6% idle
Mem: 12M Active, 1571M Inact, 551M Wired, 328M Buf, 1205M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 187 ki31 0B 32K CPU0 0 556.7H 94.68% [idle{idle: cpu0}]
11 root 187 ki31 0B 32K RUN 1 562.5H 93.16% [idle{idle: cpu1}]

these 2 PID will always be this high.

Any ideas ?

Thanks in advance.

Edit: I just noticed that those 2 PID are the % of being idle so this is normal. However when I download large files, the interrupt percentage goes above 95% . is this normal ?

last pid: 20075; load averages: 1.20, 1.13, 0.98 up 24+03:41:21 22:37:12
53 processes: 1 running, 52 sleeping
CPU: 0.3% user, 0.0% nice, 0.7% system, 97.4% interrupt, 1.6% idle
Mem: 41M Active, 1550M Inact, 552M Wired, 328M Buf, 1195M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
84564 root 1 20 0 14M 4192K CPU1 1 0:00 0.58% top
96531 root 1 20 0 13M 2868K kqread 1 10:11 0.15% tail
99019 root 1 20 0 13M 2796K select 1 7:56 0.12% tail
73582 root 5 68 0 13M 3116K uwait 0 10:06 0.05% dpinger
43948 root 1 20 0 22M 13M select 1 0:00 0.03% sshd
9858 root 2 20 0 23M 9376K select 0 6:46 0.02% ntpd
23923 root 7 20 0 46M 25M select 1 7:05 0.02% kea-dhcp4
5822 root 1 20 0 107M 38M kqread 0 1:48 0.01% php-fpm
53672 root 1 20 0 13M 3992K bpf 0 2:57 0.00% filterlog
63103 root 1 68 20 13M 3360K wait 1 16:12 0.00% sh

SteveITS

@mhijazi What's your Internet speed? The 2100 maxes out around 600 Mbps give or take due to its CPU.

mhijazi

@SteveITS it's 1Gbps . I guess that is the reason. I was checking the product specs, you are referring to this ?

IMIX Traffic: 594 Mbps

SteveITS

@mhijazi So when they write:

L3 Forwarding

    IPERF3 Traffic: 2.20 Gbps
    IMIX Traffic: 594 Mbps

Firewall
(10k ACLs)

    IPERF3 Traffic: 964 Mbps
    IMIX Traffic: 249 Mbps

IPsec VPN
(AES-GCM-128 / AES-NI w/ SafeXcel)

    IPERF3 Traffic: 254 Mbps
    IMIX Traffic: 90 Mbps

...the L3 forwarding is without firewall or NAT, IIRC. My general rule of thumb for "firewall" is to expect a number for "download speed" that is about halfway between the IPERF3 and IMIX "firewall" benchmark numbers. The VPN of course uses CPU for encryption so is more limited.

stephenw10

Yup the interrupt loading you see there is almost certainly just the firewall pushing packets.

If you run top -HaSP at the CLI you may see more.

mhijazi

@SteveITS Thanks for the information Steve. I guess then it's normal to have 100% CPU utilization when downloading files at 600-700 Mbps. When I bought this device my internet speed was 500 Mbps. so all were good. but since I upgraded to 1 Gbps , I started seeing this behavior. I guess it's time to shop for a better product

mhijazi

@stephenw10 Thanks for the information. I overlooked the device capabilities while troubleshooting and thought something was wrong.