High CPU usage from /usr/local/bin/php-cgi -f /usr/local/sbin/
-
Recently /usr/local/bin/php-cgi -f /usr/local/sbin/ started eating CPU, rebooting the system has no effect.
2.3.4-RELEASE-p1 (amd64) on a Dell OptiPlex 780 with a Core 2 Duo 3.2GHz with powerd enabled, this system would run at 1600MHz (lowest speed), now it sits at 3.2GHz all the time.
last pid: 33932; load averages: 1.03, 1.02, 0.94 up 1+02:53:07 11:31:30 167 processes: 4 running, 137 sleeping, 26 waiting Mem: 26M Active, 107M Inact, 526M Wired, 948K Cache, 196M Buf, 3185M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 32K RUN 0 24.2H 91.26% [idle{idle: cpu0}] 28063 root 98 0 267M 28136K CPU0 0 0:18 81.49% /usr/local/bin/php-cgi -f /usr/local/sbin/ 11 root 155 ki31 0K 32K RUN 1 678:11 6.88% [idle{idle: cpu1}]vmstat -i interrupt total rate irq17: uhci1 uhci4 15 0 irq22: uhci2 ehci0 2 0 irq23: uhci3 ehci1 261031 2 irq256: hpet0:t0 110097865 1126 irq257: hpet0:t1 64988184 665 irq264: em0 21899843 224 irq265: em1 35272520 360 irq266: em2 393276 4 irq269: ahci0:ch0 103075 1 Total 233015811 2384I've searched and looked at the high load troubleshooting guide, not sure what is causing this.
The last two things I changed were enabling DDNS and OpenVPN, disabling those service has no effect.
-
Not useful, try
ps auxwwinstead.
-
ps auxww USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 197.5 0.0 0 32 - RL Sat08AM 2217:07.60 [idle] dhcpd 38499 1.1 2.3 106892 94660 - Ss 11:37AM 0:45.03 /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em1 root 78492 0.6 0.1 14512 2312 - Ss Sat08AM 13:59.17 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf root 0 0.0 0.0 0 320 - DLs Sat08AM 7:36.83 [kernel] root 1 0.0 0.0 9136 848 - ILs Sat08AM 0:00.01 /sbin/init -- root 2 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [crypto] root 3 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [crypto returns] root 4 0.0 0.0 0 32 - DL Sat08AM 0:00.59 [cam] root 5 0.0 0.0 0 16 - DL Sat08AM 0:20.62 [pf purge] root 6 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [sctp_iterator] root 7 0.0 0.0 0 16 - DL Sat08AM 0:00.04 [enc_daemon0] root 8 0.0 0.0 0 32 - DL Sat08AM 0:00.96 [pagedaemon] root 9 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [vmdaemon] root 10 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [audit] root 12 0.0 0.0 0 416 - WL Sat08AM 0:38.45 [intr] root 13 0.0 0.0 0 32 - DL Sat08AM 0:00.00 [ng_queue] root 14 0.0 0.0 0 48 - DL Sat08AM 0:00.06 [geom] root 15 0.0 0.0 0 16 - DL Sat08AM 0:06.29 [rand_harvestq] root 16 0.0 0.0 0 640 - DL Sat08AM 0:03.97 [usb] root 17 0.0 0.0 0 16 - DL Sat08AM 0:00.06 [idlepoll] root 18 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [pagezero] root 19 0.0 0.0 0 32 - DL Sat08AM 0:00.82 [bufdaemon] root 20 0.0 0.0 0 16 - DL Sat08AM 0:00.21 [vnlru] root 21 0.0 0.0 0 16 - DL Sat08AM 0:03.66 [syncer] root 48 0.0 0.0 0 16 - DL Sat08AM 0:00.83 [md0] root 53 0.0 0.0 0 16 - DL Sat08AM 0:06.02 [md1] root 325 0.0 0.6 273980 25580 - Ss Sat08AM 0:01.47 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) root 363 0.0 0.1 22996 4088 - INs Sat08AM 0:00.01 /usr/local/sbin/check_reload_status root 365 0.0 0.1 22996 3920 - IN Sat08AM 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 377 0.0 0.1 13624 4836 - Is Sat08AM 0:00.02 /sbin/devd -q dhcpd 4047 0.0 0.3 24844 10960 - Ss 11:37AM 0:00.13 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1 root 5670 0.0 0.1 14564 2184 - Is Sat08AM 0:00.01 dhclient: em0 [priv] (dhclient) root 8852 0.0 0.2 82268 7488 - Ss 12:48PM 0:00.06 sshd: admin@pts/0 (sshd) _dhcp 12106 0.0 0.1 14564 2212 - Ss Sat08AM 0:00.02 dhclient: em0 (dhclient) root 14041 0.0 0.1 12428 2172 - Is Sat08AM 0:00.01 /usr/local/sbin/dhcp6c -d -c /var/etc/dhcp6c_wan.conf -p /var/run/dhcp6c_em0.pid em0 root 17048 0.0 0.1 16676 2256 - Ss Sat08AM 0:01.74 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid root 20071 0.0 0.1 15012 2292 - Ss 4:39AM 0:01.68 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B 24.19.152.121 -p /var/run/dpinger_WAN_DHCP~24.19.152.121~24.19.152.1.pid -u /var/run/dpinger_WAN_DHCP~24.19.152.121~24.19.152.1.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 24.19.152.1 root 20473 0.0 0.1 15012 2284 - Ss 4:39AM 0:01.75 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP6 -B fe80::21b:21ff:fe45:89c9%em0 -p /var/run/dpinger_WAN_DHCP6~fe80::21b:21ff:fe45:89c9%em0~fe80::259:dcff:fe79:6022%em0.pid -u /var/run/dpinger_WAN_DHCP6~fe80::21b:21ff:fe45:89c9%em0~fe80::259:dcff:fe79:6022%em0.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 fe80::259:dcff:fe79:6022%em0 root 38793 0.0 0.0 8172 1844 - Is 11:37AM 0:00.02 /usr/local/sbin/dhcpleases6 -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php|/bin/sh -l /var/dhcpd/var/db/dhcpd6.leases root 38909 0.0 1.0 273980 39648 - I 11:55AM 0:00.46 php-fpm: pool nginx (php-fpm) root 39490 0.0 0.2 39164 6156 - Is Sat08AM 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 39735 0.0 0.2 39164 7168 - I Sat08AM 0:00.18 nginx: worker process (nginx) root 39856 0.0 0.2 39164 7044 - I Sat08AM 0:00.01 nginx: worker process (nginx) root 40374 0.0 0.1 16532 2220 - Ss Sat08AM 0:00.06 /usr/sbin/cron -s root 40946 0.0 0.4 30152 17980 - Ss Sat08AM 0:03.92 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 41200 0.0 0.2 59068 6900 - Is 11:37AM 0:00.00 /usr/sbin/sshd root 41356 0.0 0.1 17000 2564 - SN Sat09AM 0:14.13 /bin/sh /var/db/rrd/updaterrd.sh root 43419 0.0 0.0 8168 1824 - SN 12:49PM 0:00.00 sleep 60 root 49891 0.0 0.1 14400 2204 - S Sat08AM 0:01.16 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog root 50240 0.0 0.1 21632 5568 - Ss 11:56AM 0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 67225 0.0 0.0 14408 1948 - Ss Sat08AM 0:01.81 /usr/sbin/powerd -b adp -a adp -n adp root 67799 0.0 0.1 21040 4752 - Ss Sat08AM 0:03.03 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid root 79654 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 79972 0.0 0.0 12268 1884 - I Sat08AM 0:00.02 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) root 80142 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts root 80442 0.0 0.0 12268 1884 - I Sat08AM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) root 80565 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data root 80867 0.0 0.0 12268 1884 - I Sat08AM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron) root 96518 0.0 0.1 12272 2032 - Is 11:37AM 0:00.00 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d localdomain -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /var/etc/hosts unbound 98313 0.0 0.6 47928 25208 - Ss 11:37AM 0:01.22 /usr/local/sbin/unbound -c /var/unbound/unbound.conf root 99481 0.0 0.1 14612 2180 - Is Sat08AM 0:00.00 /usr/local/sbin/sshlockout_pf 15 root 99295 0.0 0.1 43440 2668 v0 Is Sat08AM 0:00.00 login [pam] (login) root 99564 0.0 0.1 17000 2636 v0 I Sat08AM 0:00.00 -sh (sh) root 99622 0.0 0.1 17000 2524 v0 I+ Sat08AM 0:00.00 /bin/sh /etc/rc.initial root 8985 0.0 0.1 17000 2528 0 Is 12:48PM 0:00.00 /bin/sh /etc/rc.initial root 10007 0.0 0.1 17340 3664 0 S 12:48PM 0:00.02 /bin/tcsh root 48620 0.0 0.1 18676 2256 0 R+ 12:49PM 0:00.00 ps auxww
-
Weird, now for no apparent reason, the issue has stopped - are noticing it consistently over the last several weeks.
-
/usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php ```^ has started to pick up at around 30%, and the load averages are back up around 1\. Not sure if it's related.ps auxww
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 106.7 0.0 0 32 - RL Sat08AM 2492:01.28 [idle]
root 44876 13.6 0.7 273892 28136 - R 3:43PM 0:01.54 /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php
dhcpd 38499 0.6 2.3 106892 94664 - Ss 11:37AM 3:07.48 /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em1
root 78492 0.2 0.1 14512 2312 - Ss Sat08AM 15:39.90 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf
root 0 0.0 0.0 0 320 - DLs Sat08AM 8:39.56 [kernel]
root 1 0.0 0.0 9136 848 - ILs Sat08AM 0:00.01 /sbin/init --
root 2 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [crypto]
root 3 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [crypto returns]
root 4 0.0 0.0 0 32 - DL Sat08AM 0:00.65 [cam]
root 5 0.0 0.0 0 16 - DL Sat08AM 0:23.11 [pf purge]
root 6 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [sctp_iterator]
root 7 0.0 0.0 0 16 - DL Sat08AM 0:00.05 [enc_daemon0]
root 8 0.0 0.0 0 32 - DL Sat08AM 0:01.06 [pagedaemon]
root 9 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [vmdaemon]
root 10 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [audit]
root 12 0.0 0.0 0 416 - WL Sat08AM 0:43.48 [intr]
root 13 0.0 0.0 0 32 - DL Sat08AM 0:00.00 [ng_queue]
root 14 0.0 0.0 0 48 - DL Sat08AM 0:00.26 [geom]
root 15 0.0 0.0 0 16 - DL Sat08AM 0:07.04 [rand_harvestq]
root 16 0.0 0.0 0 640 - DL Sat08AM 0:04.41 [usb]
root 17 0.0 0.0 0 16 - DL Sat08AM 0:00.06 [idlepoll]
root 18 0.0 0.0 0 16 - DL Sat08AM 0:00.00 [pagezero]
root 19 0.0 0.0 0 32 - DL Sat08AM 0:00.92 [bufdaemon]
root 20 0.0 0.0 0 16 - DL Sat08AM 0:00.24 [vnlru]
root 21 0.0 0.0 0 16 - DL Sat08AM 0:04.10 [syncer]
root 48 0.0 0.0 0 16 - DL Sat08AM 0:00.92 [md0]
root 53 0.0 0.0 0 16 - DL Sat08AM 0:06.70 [md1]
root 325 0.0 0.6 273980 25580 - Ss Sat08AM 0:01.82 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root 363 0.0 0.1 22996 4088 - INs Sat08AM 0:00.01 /usr/local/sbin/check_reload_status
root 365 0.0 0.1 22996 3920 - IN Sat08AM 0:00.00 check_reload_status: Monitoring daemon of check_reload_status
root 377 0.0 0.1 13624 4836 - Ss Sat08AM 0:00.02 /sbin/devd -q
dhcpd 4047 0.0 0.3 24844 10988 - Ss 11:37AM 0:00.51 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
root 5670 0.0 0.1 14564 2184 - Is Sat08AM 0:00.01 dhclient: em0 [priv] (dhclient)
_dhcp 12106 0.0 0.1 14564 2212 - Is Sat08AM 0:00.02 dhclient: em0 (dhclient)
root 14041 0.0 0.1 12428 2172 - Is Sat08AM 0:00.01 /usr/local/sbin/dhcp6c -d -c /var/etc/dhcp6c_wan.conf -p /var/run/dhcp6c_em0.pid em0
root 17048 0.0 0.1 16676 2256 - Ss Sat08AM 0:01.93 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root 20071 0.0 0.1 15012 2292 - Is 4:39AM 0:02.33 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B 24.19.152.121 -p /var/run/dpinger_WAN_DHCP~24.19.152.121~24.19.152.1.pid -u /var/run/dpinger_WAN_DHCP~24.19.152.121~24.19.152.1.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 24.19.152.1
root 20473 0.0 0.1 15012 2284 - Is 4:39AM 0:02.44 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP6 -B fe80::21b:21ff:fe45:89c9%em0 -p /var/run/dpinger_WAN_DHCP6~fe80::21b:21ff:fe45:89c9%em0~fe80::259:dcff:fe79:6022%em0.pid -u /var/run/dpinger_WAN_DHCP6~fe80::21b:21ff:fe45:89c9%em0~fe80::259:dcff:fe79:6022%em0.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 fe80::259:dcff:fe79:6022%em0
root 37768 0.0 0.0 8168 1824 - IN 3:43PM 0:00.00 sleep 60
root 38793 0.0 0.0 8172 1844 - Ss 11:37AM 0:00.10 /usr/local/sbin/dhcpleases6 -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php|/bin/sh -l /var/dhcpd/var/db/dhcpd6.leases
root 39490 0.0 0.2 39164 6156 - Is Sat08AM 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root 39735 0.0 0.2 39164 7192 - I Sat08AM 0:00.58 nginx: worker process (nginx)
root 39856 0.0 0.2 39164 7140 - I Sat08AM 0:00.07 nginx: worker process (nginx)
root 40374 0.0 0.1 16532 2220 - Is Sat08AM 0:00.07 /usr/sbin/cron -s
root 40946 0.0 0.4 30152 17980 - Ss Sat08AM 0:04.37 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
root 41200 0.0 0.2 59068 6900 - Is 11:37AM 0:00.00 /usr/sbin/sshd
root 41356 0.0 0.1 17000 2564 - IN Sat09AM 0:15.81 /bin/sh /var/db/rrd/updaterrd.sh
root 44701 0.0 0.1 17000 2512 - S 3:43PM 0:00.00 sh -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php|/bin/sh
root 45060 0.0 0.1 17000 2508 - S 3:43PM 0:00.00 /bin/sh
root 49891 0.0 0.1 14400 2204 - S Sat08AM 0:01.29 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
root 50240 0.0 0.1 21632 5568 - Ss 11:56AM 0:00.06 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
root 61878 0.0 0.8 273980 32540 - I 2:39PM 0:00.03 php-fpm: pool nginx (php-fpm)
root 66513 0.0 0.2 82268 7460 - Ss 2:42PM 0:00.21 sshd: admin@pts/0 (sshd)
root 67225 0.0 0.0 14408 1948 - Ss Sat08AM 0:02.02 /usr/sbin/powerd -b adp -a adp -n adp
root 67799 0.0 0.1 21040 4752 - Ss Sat08AM 0:03.39 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
root 79654 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root 79972 0.0 0.0 12268 1884 - I Sat08AM 0:00.02 minicron: helper /usr/local/bin/ping_hosts.sh (minicron)
root 80142 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root 80442 0.0 0.0 12268 1884 - I Sat08AM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron)
root 80565 0.0 0.0 12268 1872 - Is Sat08AM 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
root 80867 0.0 0.0 12268 1884 - I Sat08AM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron)
root 96518 0.0 0.1 12272 2032 - Is 11:37AM 0:00.01 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d localdomain -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /var/etc/hosts
unbound 98313 0.0 0.7 52152 29448 - Ss 11:37AM 0:04.37 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root 99481 0.0 0.1 14612 2180 - Is Sat08AM 0:00.00 /usr/local/sbin/sshlockout_pf 15
root 99295 0.0 0.1 43440 2668 v0 Is Sat08AM 0:00.00 login [pam] (login)
root 99564 0.0 0.1 17000 2636 v0 I Sat08AM 0:00.00 -sh (sh)
root 99622 0.0 0.1 17000 2524 v0 I+ Sat08AM 0:00.00 /bin/sh /etc/rc.initial
root 45375 0.0 0.1 18676 2256 0 R+ 3:43PM 0:00.00 ps auxww
root 67281 0.0 0.1 17000 2528 0 Is 2:42PM 0:00.00 /bin/sh /etc/rc.initial
root 68371 0.0 0.1 17340 3648 0 S 2:42PM 0:00.01 /bin/tcsh
-
Perhaps disable DHCPv6 altogether. Or you might delete /var/dhcpd/var/db/dhcpd6.leases and start over. (Copy it somewhere if you want to debug it.)
-
Thanks for the advice.
I turned off dhcp 6 server, moved the lease files, re-enabled. Issue came straight back. Tried the same again but with a reboot before re-enabling, issue came back.
Left dhcp 6 server disabled, but with Router Advertisements set to assisted (this was how it was previously set), IPv6 appears to work fine, and CPU usage is back to normal.
-
With DHCPv6 off, you can set RA to Unmanaged. (Doesn't matter much though.)
Other than that, the bunch of whacky expensive preg_match() calls in /usr/local/sbin/prefixes.php trying to parse the leases file is not something I'd be exactly interested in debugging/tuning, maybe someone else. Best to leave DHCPv6 off, not really needed for IPv6.
-
Awesome, thank you for helping solve this!