PfSense High CPU Load Out Of Nowhere
-
Hello,
I have been running my pfSense router just fine now for about a year. But in the last couple days, around when I updated to the most recent version of 2.1.3-RELEASE (i386), my CPU load jumped from its normal 23% up to 50-100%. There is no extra traffic, nothing has changed. I tried rebooting, no change.
Here is my debugging:# systat -vmstat 1 1 users Load 1.00 1.01 1.00 Jun 8 08:11 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 70792 14960 194996 20048 372676 count All 99776 18416 2328312 28432 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 3996 total 40 1M 8 841 156 zfod ata0 irq14 ozfod 1998 cpu0: time 49.8%Sys 0.0%Intr 0.4%User 0.0%Nice 49.8%Idle %ozfod 1998 cpu1: time | | | | | | | | | | | daefr ========================= prcfr 16 dtbuf 1 totfr Namei Name-cache Dir-cache 34708 desvn react Calls hits % hits % 1050 numvn pdwak 3 3 100 134 frevn pdpgs intrn Disks ad0 md0 55240 wire KB/t 0.00 0.00 39064 act tps 0 1 24836 inact MB/s 0.00 0.00 240 cache %busy 0 0 372436 free 28944 buf# top -aSH last pid: 26062; load averages: 1.00, 1.00, 1.00 up 2+12:47:33 08:12:26 118 processes: 4 running, 93 sleeping, 21 waiting CPU: 0.0% user, 0.0% nice, 50.0% system, 0.0% interrupt, 50.0% idle Mem: 37M Active, 24M Inact, 54M Wired, 240K Cache, 28M Buf, 365M Free Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 19 root 171 ki-6 0K 8K CPU0 0 60.6H 100.00% [idlepoll] 11 root 171 ki31 0K 16K RUN 1 56.2H 85.35% [idle{idle: cpu1}] 11 root 171 ki31 0K 16K RUN 0 204:41 13.38% [idle{idle: cpu0}] 252 root 76 20 3352K 1196K kqread 0 12:04 0.00% /usr/local/sbin/check_reload_status 12 root -32 - 0K 168K WAIT 0 2:40 0.00% [intr{swi4: clock}] 22951 root 64 20 3264K 1252K select 0 1:44 0.00% /usr/local/sbin/apinger -c /var/etc/apinger.conf 91743 root 45 0 31060K 23480K accept 0 1:19 0.00% /usr/local/bin/php{php} 0 root -16 0 0K 72K sched 1 0:46 0.00% [kernel{swapper}] 79987 root 44 0 3412K 1436K select 0 0:23 0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/ 14 root -16 - 0K 8K - 0 0:23 0.00% [yarrow] 55697 root 47 0 34260K 24560K accept 0 0:22 0.00% /usr/local/bin/php{php} 12 root -32 - 0K 168K WAIT 0 0:21 0.00% [intr{swi4: clock}] 51103 root 76 20 3644K 1512K wait 0 0:18 0.00% /bin/sh /var/db/rrd/updaterrd.sh 57889 root 76 20 3644K 1512K wait 0 0:17 0.00% /bin/sh /var/db/rrd/updaterrd.sh 28829 root 44 0 8004K 5540K kqread 1 0:15 0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.c 3 root -8 - 0K 8K - 0 0:09 0.00% [g_up] 21 root 20 - 0K 8K syncer 1 0:07 0.00% [syncer] 18223 root 44 0 3264K 872K piperd 0 0:06 0.00% logger -t pf -p local0.info 18189 root 44 0 5868K 3344K bpf 1 0:06 0.00% /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0 4 root -8 - 0K 8K - 0 0:05 0.00% [g_down] 74794 root 64 20 6048K 6068K select 1 0:05 0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 8 root -16 - 0K 8K pftm 1 0:03 0.00% [pfpurge] 12 root -64 - 0K 168K WAIT 0 0:03 0.00% [intr{irq14: ata0}] 66 root -8 - 0K 8K mdwait 1 0:02 0.00% [md0] 15 root -40 - 0K 160K - 1 0:01 0.00% [usb{usbus4}] 36 root -8 - 0K 16K l2arc_ 1 0:01 0.00% [zfskern{l2arc_feed_threa}] 36 root -8 - 0K 16K arc_re 1 0:01 0.00% [zfskern{arc_reclaim_thre}] 10033 root 56 0 3324K 1300K select 0 0:01 0.00% dhclient: bge0 [priv] (dhclient) 23 root -16 - 0K 8K sdflus 1 0:01 0.00% [softdepflush] 22 root -16 - 0K 8K vlruwt 1 0:01 0.00% [vnlru] 20 root -16 - 0K 8K psleep 1 0:01 0.00% [bufdaemon] 15 root -40 - 0K 160K - 1 0:01 0.00% [usb{usbus1}] 15 root -40 - 0K 160K - 1 0:01 0.00% [usb{usbus0}] 15 root -40 - 0K 160K - 1 0:01 0.00% [usb{usbus3}] 15 root -40 - 0K 160K - 1 0:01 0.00% [usb{usbus2}] 16084 _dhcp 44 0 3324K 1400K select 1 0:01 0.00% dhclient: bge0 (dhclient) 80937 root 44 0 3352K 1332K nanslp 1 0:00 0.00% /usr/sbin/cron -s 89780 root 61 0 3264K 1028K nanslp 1 0:00 0.00% minicron: helper /usr/local/bin/ping_hosts.sh (minicron) 49939 nobody 44 0 5512K 2876K select 0 0:00 0.00% /usr/local/sbin/dnsmasq --all-servers --dns-forward-max=5000 16 root -16 - 0K 8K psleep 1 0:00 0.00% [pagedaemon] 95253 root 44 0 3324K 1204K nanslp 1 0:00 0.00% /usr/libexec/getty Pc ttyv0I am not sure what I should do. Should I reinstall or is it fine?
Thanks,
Trevor -
Do you have device polling enabled in System: Advanced: Networking:? Disable it if you do.
https://forum.pfsense.org/index.php?topic=30817.0
Steve
-
Hello,
We had a problem with high cpu as well. We were running snort and openvpn. The cpu went up every time the network traffic went up. for example if we tried to download a large file, the cpu would jump to 80%. We fixed it by enabling device polling. from System: Advanced: Networking. Enabling device polling is actually a good idea if your hardware is being used close to its capacity. see http://www.cyberciti.biz/faq/freebsd-device-polling-network-polling-tutorial/ and https://blog.pfsense.org/?p=115. Enabling polling disables cpu interrupts, which take a lot of cpu resources. After we enabled device polling the cpu went down to 40%. We also disabled ipv6 under System: Advanced: Networking. see https://forum.pfsense.org/index.php?topic=77493.msg422407#msg422407. If you are running pfsense in a virtualized environment you can get a performance boot by using virtio drivers. https://doc.pfsense.org/index.php/VirtIO_Driver_Support.
Nadir Latif
-
You may like to read this thread:
https://forum.pfsense.org/index.php?topic=83861.0;all
The symptoms you dedcribe sound very similar.Steve
-
On high speed networks we can get a performance boost by moving the tcp/ip processing from the operating system to the network card hardware. This is called TCP Offloading. see http://en.wikipedia.org/wiki/TCP_offload_engine. FreeBSD has good support for TCP offloading. In Pfsense it can be enabled by unchecking the "Disable hardware checksum offload" option. In pfsense this option is unchecked by default.
-
The hardware offloading features available in the System: Advanced: Networking: section of the webgui do not include a complete TOE as referenced in that Wiki page. They only offload smaller functions: TSO, LRO and checksum.
It looks like there is at least some support for TOE in FreeBSD but you would need to enable in manually in pfSense. Importantly I have no idea how it would interact with pf. As referenced in the wiki article once you've handed off the entire TCP stack to hardware much of the OS internal networking features are by-passed. It could be potentially completely redundant in pfSense.
Steve
