High CPU load, after reset everthing ok for another 6hours
-
Hello there,
Iam using Pfsense for 8 Years now. For 2 or 4 weeks my Pfsense has high cpu load (60-80%), but i cant find whats using that much cpu with top.
CPU: system is high (see pictures).
If i run : killall -9 php; killall -9 lighttpd; /etc/rc.restart_webgui , everything is ok again (4-7% CPU Comsuption).ps uxawww (but nothing to find there)
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 277.0 0.0 0 64 - RL 11Mar18 40509:50.79 [idle] root 12 8.9 0.0 0 912 - WL 11Mar18 2338:51.72 [intr] root 82217 1.0 9.6 1088684 695580 - SNs 15:06 16:18.71 /usr/local/bin/snort -R 4983 -D -l /var/log/snort/snort_vmx04983 --pid-path /var/run --nolock-pidfile -G 4983 -c /usr/local/etc/snort/snort_4983_vmx0/snort.conf -i vmx0 root 0 0.0 0.0 0 224 - DLs 11Mar18 0:13.11 [kernel] root 1 0.0 0.0 5028 140 - ILs 11Mar18 0:00.89 /sbin/init -- root 2 0.0 0.0 0 16 - DL 11Mar18 0:00.00 [crypto] root 3 0.0 0.0 0 16 - DL 11Mar18 0:00.00 [crypto returns] root 4 0.0 0.0 0 32 - DL 11Mar18 104:53.96 [cam] root 5 0.0 0.0 0 16 - DL 11Mar18 0:00.00 [mpt_recovery0] root 6 0.0 0.0 0 16 - DL 11Mar18 0:00.81 [soaiod1] root 7 0.0 0.0 0 16 - DL 11Mar18 0:01.16 [soaiod2] root 8 0.0 0.0 0 16 - DL 11Mar18 0:00.63 [soaiod3] root 9 0.0 0.0 0 16 - DL 11Mar18 0:00.87 [soaiod4] root 10 0.0 0.0 0 16 - DL 11Mar18 0:00.00 [audit] root 13 0.0 0.0 0 64 - DL 11Mar18 0:17.96 [ng_queue] root 14 0.0 0.0 0 48 - DL 11Mar18 0:00.03 [geom] root 15 0.0 0.0 0 160 - DL 11Mar18 109:39.83 [usb] root 16 0.0 0.0 0 16 - DL 11Mar18 0:00.00 [sctp_iterator] root 17 0.0 0.0 0 16 - DL 11Mar18 8:58.42 [pf purge] root 18 0.0 0.0 0 16 - DL 11Mar18 11:52.77 [rand_harvestq] root 19 0.0 0.0 0 48 - DL 11Mar18 53:24.37 [pagedaemon] root 20 0.0 0.0 0 16 - DL 11Mar18 0:02.31 [vmdaemon] root 21 0.0 0.0 0 16 - DL 11Mar18 0:00.04 [pagezero] root 22 0.0 0.0 0 16 - DL 11Mar18 2:27.42 [bufspacedaemon] root 23 0.0 0.0 0 48 - DL 11Mar18 8:50.17 [bufdaemon] root 24 0.0 0.0 0 16 - DL 11Mar18 0:34.64 [vnlru] root 25 0.0 0.0 0 16 - DL 11Mar18 13:09.64 [syncer] root 59 0.0 0.0 0 16 - DL 11Mar18 0:03.73 [md0] squid 163 0.0 0.0 33780 3564 - S Tue11 0:21.36 (pinger) (pinger) root 309 0.0 0.2 286836 17964 - Ss 11Mar18 4:16.76 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) root 347 0.0 0.0 19444 1180 - INs 11Mar18 0:00.23 /usr/local/sbin/check_reload_status root 349 0.0 0.0 19444 0 - IWN - 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 361 0.0 0.0 9560 536 - Ss 11Mar18 1:06.39 /sbin/devd -q -f /etc/pfSense-devd.conf root 416 0.0 0.0 8224 336 - I 11Mar18 0:00.01 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron) root 1488 0.0 0.0 8224 336 - I 11Mar18 0:00.11 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) squid 5319 0.0 0.0 33780 2560 - S 12Mar18 2:33.86 (pinger) (pinger) root 5921 0.0 0.0 10532 1928 - Is 11Mar18 0:05.96 dhclient: vmx2 [priv] (dhclient) squid 7485 0.0 0.0 33780 3564 - S Mon04 0:39.86 (pinger) (pinger) root 7634 0.0 0.0 12580 2476 - Is 04:00 0:00.00 /usr/local/sbin/upsmon uucp 7978 0.0 0.0 12580 2480 - S 04:00 0:03.02 /usr/local/sbin/upsmon root 10079 0.0 0.4 290936 31216 - S 21:46 0:00.31 php-fpm: pool nginx (php-fpm) _dhcp 10621 0.0 0.0 10532 1944 - Ss 11Mar18 0:16.81 dhclient: vmx2 (dhclient) root 11502 0.0 0.2 74576 13308 - S 11Mar18 31:07.48 /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-vm-tools/plugins/vmsvc root 13219 0.0 0.1 53496 4620 - Is 11Mar18 0:00.31 /usr/sbin/sshd squid 13900 0.0 0.0 33780 3564 - S Sun00 0:51.21 (pinger) (pinger) nobody 14268 0.0 0.0 19064 2988 - SNs 04:00 11:22.04 /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobody squid 14680 0.0 0.0 33780 3564 - S Sun13 0:44.60 (pinger) (pinger) squid 16606 0.0 0.0 33780 3564 - S Wed00 0:16.66 (pinger) (pinger) root 16793 0.0 0.1 44844 8276 - Ss 04:00 0:01.74 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/pppoe_opt1.pid -s ppp pppoeclient squid 17467 0.0 0.0 33780 2560 - S 12Mar18 2:42.55 (pinger) (pinger) squid 18099 0.0 0.0 33780 3556 - S Fri00 1:36.17 (pinger) (pinger) root 19428 0.0 0.1 35664 6604 - Is Sat15 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 19536 0.0 0.4 247864 30696 - SN Tue13 13:48.86 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 19594 0.0 0.1 37712 8296 - S Sat15 1:09.01 nginx: worker process (nginx) root 19762 0.0 0.1 37712 8264 - S Sat15 1:59.67 nginx: worker process (nginx) root 22098 0.0 0.0 8232 2220 - Is 04:00 0:02.29 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d zuhause -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /etc/hosts root 23291 0.0 0.4 247864 30740 - SN Tue13 13:56.19 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 25644 0.0 0.0 33780 3556 - S Sat10 1:00.57 (pinger) (pinger) root 28606 0.0 0.0 10584 2312 - Is Sat15 0:00.36 /usr/local/sbin/sshlockout_pf 15 squid 29439 0.0 0.0 33780 3564 - S Tue13 0:22.17 (pinger) (pinger) unbound 29604 0.0 1.3 120424 92292 - Ss 04:00 6:34.52 /usr/local/sbin/unbound -c /var/unbound/unbound.conf root 30058 0.0 0.4 249912 31428 - RN Mon04 24:32.48 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 30188 0.0 0.4 249912 31236 - SN Wed08 8:55.71 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 30448 0.0 0.0 13084 2368 - I 04:00 0:02.33 /bin/sh /usr/local/pkg/sqpmon.sh clamav 31875 0.0 12.0 963296 870840 - Ss 11Mar18 173:38.14 /usr/local/sbin/clamd --config-file=/usr/local/etc/clamd.conf root 32862 0.0 0.1 20356 5832 - Ss 11Mar18 0:24.74 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 33195 0.0 0.4 249912 30940 - SN Wed04 10:18.78 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 33560 0.0 0.1 33780 3656 - S 04:00 0:08.56 (pinger) (pinger) clamav 36016 0.0 0.0 21288 2596 - Ss 11Mar18 1:46.02 /usr/local/bin/c-icap -f /usr/local/etc/c-icap/c-icap.conf squid 36422 0.0 0.0 33780 2580 - S 13Mar18 2:19.80 (pinger) (pinger) root 37246 0.0 0.1 20356 4712 - Ss 11Mar18 0:17.80 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf squid 39137 0.0 0.0 33780 2588 - S 13Mar18 1:57.87 (pinger) (pinger) root 40519 0.0 0.0 12700 1952 - Ss 11Mar18 5:03.88 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid squid 40766 0.0 0.0 33780 3568 - S Wed08 0:15.35 (pinger) (pinger) root 41017 0.0 0.0 23684 2992 - Is 11Mar18 0:17.27 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 root 41804 0.0 0.0 14868 2160 - Is 11Mar18 0:00.04 /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid squid 42479 0.0 0.0 33780 3556 - S 14Mar18 2:42.82 (pinger) (pinger) root 44112 0.0 0.0 54180 0 - IWs - 0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf root 44415 0.0 0.9 294968 66976 - S 04:00 6:22.38 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl squid 44504 0.0 22.9 3279764 1660600 - S 11Mar18 75:10.72 (squid-1) -f /usr/local/etc/squid/squid.conf (squid) squid 44630 0.0 0.0 33780 3564 - S Mon00 0:37.98 (pinger) (pinger) squid 46259 0.0 0.0 33780 3564 - S Sun11 0:39.80 (pinger) (pinger) root 47507 0.0 0.4 249912 31092 - SN Mon04 24:16.45 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 48239 0.0 0.0 33652 2472 - I 11Mar18 0:29.34 (unlinkd) (unlinkd) root 49538 0.0 0.4 249912 31228 - SN Wed04 10:13.10 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 51137 0.0 0.0 13084 2428 - IN 04:00 0:37.10 /bin/sh /var/db/rrd/updaterrd.sh squid 51284 0.0 0.0 33780 3168 - S 14Mar18 2:06.40 (pinger) (pinger) squid 51351 0.0 0.0 33780 3564 - S Tue00 0:26.57 (pinger) (pinger) dhcpd 53724 0.0 0.1 16652 7864 - Ss 04:00 0:08.08 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid vmx0 squid 54825 0.0 0.0 33780 3556 - S Fri13 1:24.08 (pinger) (pinger) root 55286 0.0 0.0 13032 2484 - Is 04:00 0:06.09 /usr/local/bin/dpinger -S -r 0 -i UNITYMEDIAIF_DHCP -B 178.200.253.238 -p /var/run/dpinger_UNITYMEDIAIF_DHCP~178.200.253.238~192.5.5.241.pid -u /var/run/dpinger_UNITYMEDIAIF_DHCP~178.200.253.238~192.5.5.241.sock -C /etc/rc.gateway_alarm -d 0 -s 1500 -l 10000 -t 70000 -A 10000 -D 500 -L 10 192.5.5.241 root 55430 0.0 0.0 10984 2440 - Is 04:00 0:18.96 /usr/local/bin/dpinger -S -r 0 -i DSL1UND1_PPPOE -B 87.123.174.173 -p /var/run/dpinger_DSL1UND1_PPPOE~87.123.174.173~62.214.63.83.pid -u /var/run/dpinger_DSL1UND1_PPPOE~87.123.174.173~62.214.63.83.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 62.214.63.83 squid 55550 0.0 0.0 33780 3556 - S 14Mar18 2:05.34 (pinger) (pinger) squid 60461 0.0 0.0 33780 3564 - S Sun13 0:42.52 (pinger) (pinger) squid 60782 0.0 0.0 33780 2560 - S 11Mar18 2:33.34 (pinger) (pinger) root 63088 0.0 0.0 6172 1928 - IN 21:46 0:00.00 sleep 60 root 65089 0.0 0.4 249912 31092 - SN Tue04 16:42.40 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 66465 0.0 0.0 12500 1232 - Is 11Mar18 0:21.07 /usr/sbin/cron -s root 66744 0.0 0.2 24612 12432 - Ss 11Mar18 2:57.69 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid squid 66857 0.0 0.0 33780 3556 - S Sat13 1:05.56 (pinger) (pinger) nobody 67383 0.0 0.1 14924 4660 - Ss 04:00 0:42.12 /usr/local/sbin/darkstat -i vmx0 -p 666 nobody 67917 0.0 0.0 12872 2440 - Is 04:00 0:00.00 darkstat: DNS child (darkstat) squid 67997 0.0 0.0 33780 3556 - S 15Mar18 1:47.31 (pinger) (pinger) root 69001 0.0 0.4 247864 30916 - SN Tue11 14:58.84 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 69042 0.0 0.0 33780 3568 - S 00:00 0:08.25 (pinger) (pinger) root 71828 0.0 0.0 6172 1928 - I 21:46 0:00.00 sleep 55 squid 74098 0.0 0.0 33780 3556 - S Sat00 1:57.78 (pinger) (pinger) root 74681 0.0 0.0 21108 2744 - R 21:46 0:00.00 ps uxawww root 74911 0.0 0.0 6188 1956 - RN 21:46 0:00.00 /sbin/sysctl -i hw.physmem root 78236 0.0 0.1 36780 7684 - S 14Mar18 89:46.94 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf root 79920 0.0 0.4 249912 31420 - SN Mon04 24:34.65 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 80877 0.0 0.1 28632 4552 - I Sun16 0:16.26 /usr/local/bin/iperf -s root 81405 0.0 9.4 1123500 681364 - SNs 15:06 4:59.09 /usr/local/bin/snort -R 55581 -D -l /var/log/snort/snort_vmx255581 --pid-path /var/run --nolock-pidfile -G 55581 -c /usr/local/etc/snort/snort_55581_vmx2/snort.conf -i vmx2 root 82355 0.0 0.4 247864 30736 - SN Mon04 24:16.20 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 83151 0.0 8.4 1072300 613040 - SNs 15:06 11:11.74 /usr/local/bin/snort -R 27439 -D -l /var/log/snort/snort_pppoe127439 --pid-path /var/run --nolock-pidfile -G 27439 -c /usr/local/etc/snort/snort_27439_pppoe1/snort.conf -i pppoe1 root 83567 0.0 0.4 247864 31640 - SN 04:00 4:30.16 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 83675 0.0 0.4 249912 31424 - SN Tue11 14:49.14 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 85383 0.0 0.0 33780 3564 - S Wed04 0:16.27 (pinger) (pinger) root 87300 0.0 0.4 249912 31248 - SN Tue04 16:44.02 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 89320 0.0 0.4 249912 31248 - SN Wed08 9:00.58 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php squid 89386 0.0 0.0 33780 3168 - S 13Mar18 2:41.51 (pinger) (pinger) root 89777 0.0 0.0 10476 2460 - Ss Sat15 1:06.95 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf squid 90052 0.0 0.0 33780 3564 - S Mon04 0:35.89 (pinger) (pinger) clamav 91899 0.0 0.0 30892 3596 - S 21:27 0:00.17 /usr/local/bin/c-icap -f /usr/local/etc/c-icap/c-icap.conf root 92165 0.0 0.5 881836 38308 - Is 04:00 0:00.03 /usr/local/sbin/radiusd squid 94641 0.0 0.0 33780 3556 - S 15Mar18 2:08.40 (pinger) (pinger) clamav 94770 0.0 0.1 43180 4320 - S 21:24 0:00.12 /usr/local/bin/c-icap -f /usr/local/etc/c-icap/c-icap.conf root 96283 0.0 0.4 247864 31716 - SN 04:00 4:24.99 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php root 96948 0.0 0.0 8224 0 - IWs - 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 97230 0.0 0.0 8224 336 - I 11Mar18 0:01.52 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) squid 97501 0.0 0.0 33780 2596 - S 13Mar18 2:04.40 (pinger) (pinger) root 98781 0.0 0.0 8224 0 - IWs - 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts root 99163 0.0 0.0 8224 0 - IWs - 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data squid 99806 0.0 0.0 33780 3564 - S Tue04 0:25.53 (pinger) (pinger) root 29341 0.0 0.0 39436 0 v0 IWs - 0:00.00 login [pam] (login) root 38020 0.0 0.0 13084 0 v0 IW - 0:00.00 -sh (sh) root 39963 0.0 0.0 13084 0 v0 IW - 0:00.00 /bin/sh /etc/rc.initial root 57267 0.0 0.0 13396 2332 v0 S+ 11Mar18 0:04.11 /bin/tcsh root 29603 0.0 0.0 10388 1716 v1 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv1 root 29683 0.0 0.0 10388 1716 v2 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv2 root 29849 0.0 0.0 10388 1716 v3 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv3 root 29911 0.0 0.0 10388 1716 v4 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv4 root 30123 0.0 0.0 10388 1716 v5 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv5 root 30240 0.0 0.0 10388 1716 v6 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv6 root 30250 0.0 0.0 10388 1716 v7 Is+ 11Mar18 0:00.00 /usr/libexec/getty Pc ttyv7
top
last pid: 25780; load averages: 2.41, 2.90, 2.82 up 11+08:37:42 21:48:09 125 processes: 3 running, 122 sleeping Mem: 1224M Active, 2506M Inact, 1827M Laundry, 978M Wired, 628M Buf, 350M Free Swap: 16G Total, 1453M Used, 15G Free, 8% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 44415 root 1 52 0 288M 67060K nanslp 0 6:23 1.95% php 78236 root 1 21 0 36780K 7684K kqread 3 89:48 0.98% lighttpd_pfb 31875 clamav 3 20 0 941M 850M select 3 173:38 0.00% clamd 44504 squid 1 20 0 3203M 1622M kqread 1 75:11 0.00% squid 11502 root 1 20 0 74576K 13308K select 1 31:08 0.00% vmtoolsd 79920 root 1 52 20 244M 31420K nanslp 0 24:35 0.00% php 30058 root 1 52 20 244M 31428K nanslp 3 24:33 0.00% php 47507 root 1 52 20 244M 31092K piperd 1 24:17 0.00% php 82355 root 1 52 20 242M 30736K nanslp 0 24:16 0.00% php 87300 root 1 52 20 244M 31248K nanslp 3 16:44 0.00% php 65089 root 1 52 20 244M 31092K nanslp 1 16:43 0.00% php 82217 root 3 41 20 1063M 680M bpf 0 16:20 0.00% snort 69001 root 1 52 20 242M 30916K nanslp 3 14:59 0.00% php 83675 root 1 52 20 244M 31424K nanslp 3 14:49 0.00% php 23291 root 1 52 20 242M 30740K nanslp 0 13:56 0.00% php 19536 root 1 52 20 242M 30696K nanslp 1 13:49 0.00% php 14268 nobody 1 -22 r30 19064K 2988K nanslp 2 11:23 0.00% LCDd 83151 root 3 40 20 1047M 599M bpf 0 11:12 0.00% snort
See pictures for top on console (two with high cpu usage, one in normal state)
![Pfsense 04_17-2018-03-17-15-16-21.png](/public/imported_attachments/1/Pfsense 04_17-2018-03-17-15-16-21.png)
![Pfsense 04_17-2018-03-17-15-16-21.png_thumb](/public/imported_attachments/1/Pfsense 04_17-2018-03-17-15-16-21.png_thumb)
![Pfsense 04_17-2018-03-22-21-33-47.png](/public/imported_attachments/1/Pfsense 04_17-2018-03-22-21-33-47.png)
![Pfsense 04_17-2018-03-22-21-33-47.png_thumb](/public/imported_attachments/1/Pfsense 04_17-2018-03-22-21-33-47.png_thumb)
![AfterresetPfsense 04_17-2018-03-22-21-50-40.png](/public/imported_attachments/1/AfterresetPfsense 04_17-2018-03-22-21-50-40.png)
![AfterresetPfsense 04_17-2018-03-22-21-50-40.png_thumb](/public/imported_attachments/1/AfterresetPfsense 04_17-2018-03-22-21-50-40.png_thumb) -
I think it has something to do with the LCDPROc Packages. Will testing this now