CPU loaded at 100% and hangs pfsense
-
Very difficult to say with just that info.
Some other things we'll need to know:
1. How much bandwidth do you have on you WAN?
2. Are you using any kind of traffic shaping/limiting/layer7?It would appear from the top output that you're hitting the limits of that hardware, which could certainly be possible if you have a lot of bandwidth.
Also those NICs are not the greatest, and low-grade NICs will take up more CPU and other system resources to run. Better quality NICs will push more traffic with less resource usage.
-
Looks like you have too many processes running, including 62 zombies.
Perhaps due to your many log entries of:Nov 28 18:57:19 root: l2tp is down - killing mpd5
Can we get a more complete output from top -SH? Preferably sorted by processor time.
Steve
-
jimp, 1)WAN: 80 mbit/s
2)Normal View Traffic -
stephenw10 ps uxawww Full List
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 15.0 0.0 0 8 ?? RL 1:31PM 236:59.29 idle root 12 0.9 0.0 0 128 ?? WL 1:31PM 17:15.84 intr root 60142 0.9 1.5 53596 15552 ?? S 6:15PM 0:18.32 /usr/local/bin/php root 13 70.8 0.0 0 8 ?? DL 1:31PM 14:29.18 ng_queue root 1490 0.7 1.5 53596 15552 ?? S 6:13PM 0:18.99 /usr/local/bin/php root 0 0.0 0.0 0 64 ?? DLs 1:31PM 0:10.28 kernel root 1 0.0 0.0 1888 460 ?? ILs 1:31PM 0:00.01 /sbin/init -- root 2 0.0 0.0 0 8 ?? DL 1:31PM 0:01.18 g_event root 3 0.0 0.0 0 8 ?? DL 1:31PM 0:01.55 g_up root 4 0.0 0.0 0 8 ?? DL 1:31PM 0:01.06 g_down root 5 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 crypto root 6 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 crypto returns root 7 0.0 0.0 0 8 ?? DL 1:31PM 0:00.15 fdc0 root 8 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 sctp_iterator root 9 0.0 0.0 0 8 ?? DL 1:31PM 0:03.69 pfpurge root 10 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 audit root 14 0.0 0.0 0 8 ?? DL 1:31PM 0:33.35 yarrow root 15 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 xpt_thrd root 16 0.0 0.0 0 8 ?? DL 1:31PM 0:00.02 pagedaemon root 17 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 vmdaemon root 18 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 pagezero root 19 0.0 0.0 0 8 ?? DL 1:31PM 0:00.03 idlepoll root 20 0.0 0.0 0 8 ?? DL 1:31PM 0:00.13 bufdaemon root 21 0.0 0.0 0 8 ?? DL 1:31PM 0:00.10 vnlru root 22 0.0 0.0 0 8 ?? DL 1:31PM 0:01.35 syncer root 23 0.0 0.0 0 8 ?? DL 1:31PM 0:00.11 softdepflush root 39 0.0 0.0 0 8 ?? DL 1:31PM 0:00.11 md0 root 254 0.0 0.1 3408 1156 ?? INs 1:31PM 0:00.03 /usr/local/sbin/check_reload_status root 256 0.0 0.1 3408 1032 ?? IN 1:31PM 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 267 0.0 0.1 1888 540 ?? Is 1:31PM 0:00.00 /sbin/devd root 1712 0.0 0.2 4948 2544 ?? Ss 4:39PM 0:00.85 /usr/sbin/syslogd -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf root 2712 0.0 1.5 53596 15432 ?? S 5:54PM 0:13.80 /usr/local/bin/php root 4825 0.0 0.1 3316 1356 ?? SNs 1:33PM 0:11.50 /usr/local/sbin/apinger -c /var/etc/apinger.conf root 9353 0.0 0.1 3316 1240 ?? Is 1:31PM 0:00.00 dhclient: rl0 priv (dhclient) root 9853 0.0 1.5 53596 15436 ?? S 5:53PM 0:14.68 /usr/local/bin/php root 12729 0.0 0.1 3404 1348 ?? I 1:33PM 0:00.00 cron: running job (cron) root 12921 0.0 0.1 3656 1340 ?? Is 1:33PM 0:00.01 /bin/sh /usr/scripts/start-keeper.sh root 14221 0.0 0.1 3656 1440 ?? SN 1:33PM 0:04.79 sh /usr/scripts/mpd-keeper _dhcp 15600 0.0 0.1 3316 1376 ?? Is 1:31PM 0:00.00 dhclient: rl0 (dhclient) root 16742 0.0 0.3 7992 3520 ?? RNs 4:50PM 0:00.71 sshd: admin@pts/0 (sshd) nobody 18036 0.0 0.3 5556 2636 ?? S 4:50PM 0:05.32 /usr/local/sbin/dnsmasq --local-ttl 1 --all-servers --dns-forward-max=5000 --cache-size=10000 root 18250 0.0 0.4 9488 4300 ?? SNs 1:33PM 0:00.49 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/l2tp_opt1.pid -s ppp l2tpclient root 22166 0.0 0.3 5176 2628 ?? Ss 1:31PM 0:00.48 /usr/sbin/hostapd -B /var/etc/hostapd_ath0_wlan0.conf dhcpd 24292 0.0 0.6 8436 6152 ?? Ss 4:50PM 0:01.01 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf ste0 ath0_wlan0 root 24977 0.0 0.1 3532 1208 ?? Is 4:50PM 0:00.02 /usr/local/sbin/sshlockout_pf 15 root 32707 0.0 0.2 4496 1932 ?? SN 4:51PM 0:00.10 /usr/local/bin/rrdtool - root 37039 0.0 0.1 3436 1540 ?? Is 1:31PM 0:01.23 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf root 43281 0.0 0.1 3404 1360 ?? Ss 1:32PM 0:00.18 /usr/sbin/cron -s root 43753 0.0 0.1 3316 992 ?? Is 1:32PM 0:00.03 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 43904 0.0 0.1 3316 992 ?? Is 1:32PM 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts root 43930 0.0 0.3 5272 3216 ?? INs 4:50PM 0:00.00 /usr/sbin/sshd root 44525 0.0 0.1 3316 960 ?? Is 1:32PM 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data root 45349 0.0 0.3 6588 3440 ?? SN 4:50PM 0:05.36 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf nobody 45618 0.0 0.1 3344 992 ?? Is 2:01PM 0:00.05 nc -w 2000 192.168.1.43 6667 root 47162 0.0 0.1 3532 1196 ?? Is 1:32PM 0:00.02 /usr/local/sbin/sshlockout_pf 15 root 47882 0.0 1.0 52572 10544 ?? Is 1:31PM 0:00.14 /usr/local/bin/php root 48714 0.0 1.0 52572 10544 ?? Is 1:31PM 0:00.14 /usr/local/bin/php root 51899 0.0 0.2 3656 1572 ?? IN 4:50PM 0:03.40 /bin/sh /var/db/rrd/updaterrd.sh root 53185 0.0 0.1 1564 592 ?? IN 6:46PM 0:00.00 sleep 60 root 53472 0.0 0.1 3316 1340 ?? Is 1:32PM 0:00.00 ntpd: priv (ntpd) root 53572 0.0 0.1 1564 592 ?? SN 6:47PM 0:00.00 sleep 5 _ntp 6388 0.0 0.1 3316 1344 v0- I 1:31PM 0:00.20 ntpd: ntp engine (ntpd) root 27567 0.0 0.3 5912 2612 v0- S 1:31PM 0:01.01 /usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 root 27609 0.0 0.1 3316 928 v0- S 1:31PM 0:01.12 logger -t pf -p local0.info root 46972 0.0 0.1 3684 1500 v0 Is 1:32PM 0:00.03 login pam (login) root 47163 0.0 0.1 3656 1392 v0 I 1:32PM 0:00.01 -sh (sh) root 48549 0.0 0.1 3656 1392 v0 I+ 1:32PM 0:00.01 /bin/sh /etc/rc.initial root 18234 0.0 0.2 3656 1556 0 Is 4:50PM 0:00.01 /bin/sh /etc/rc.initial root 24340 0.0 0.2 3672 2364 0 S 4:50PM 0:00.08 /bin/tcsh root 53844 0.0 0.1 3456 1248 0 R+ 6:47PM 0:00.00 ps uxawww -
I'd say it's whatever mpd-keeper and start-keeper.sh are.
Those aren't part of pfSense that I can see.
-
Looks like it's this: http://forum.pfsense.org/index.php?topic=50127.0
Steve
-
jimp,mpd-keeper and start-keeper.sh I deleted, memory is not overloaded
But the processor will ship still at 100%, what to do? -
stephenw10, this manual and did
-
You're going to have to reboot to clean up the mess of processes that modified code created.
-
сmb, rebooted, still the CPU when downloading files loaded at 100%
-
Have you enabled device polling?
People seem to do this misunderstanding how it works. You almost certainly don't want this if you have enabled it. Also it seems to have 'staying power'! In my brief test of it's worth I found it was tricky to remove, have you ever enabled it?Steve
-
stephenw10, device polling me off. If I turn it on it will also be loaded 100% CPU
-
Do you still have so many processes running after the reboot?
Also if you could edit your post above to include the close code tag it would make this thread much easier to read. ;)
Steve
-
Do you still have so many processes running after the reboot?
Also if you could edit your post above to include the close code tag it would make this thread much easier to read. ;)
Steve
Processes become less, after I removed the unnecessary script №1.2 and №1.3 from NCAT.
But that when the file is loaded CPU at 100%, and blame ng_queueUSER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 13 70.8 0.0 0 8 ?? DL 1:31PM 14:29.18 ng_queue root 11 15.0 0.0 0 8 ?? RL 1:31PM 236:59.29 idle root 12 0.9 0.0 0 128 ?? WL 1:31PM 17:15.84 intr root 60142 0.9 1.5 53596 15552 ?? S 6:15PM 0:18.32 /usr/local/bin/php root 1490 0.7 1.5 53596 15552 ?? S 6:13PM 0:18.99 /usr/local/bin/php root 0 0.0 0.0 0 64 ?? DLs 1:31PM 0:10.28 kernel root 1 0.0 0.0 1888 460 ?? ILs 1:31PM 0:00.01 /sbin/init -- root 2 0.0 0.0 0 8 ?? DL 1:31PM 0:01.18 g_event root 3 0.0 0.0 0 8 ?? DL 1:31PM 0:01.55 g_up root 4 0.0 0.0 0 8 ?? DL 1:31PM 0:01.06 g_down root 5 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 crypto root 6 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 crypto returns root 7 0.0 0.0 0 8 ?? DL 1:31PM 0:00.15 fdc0 root 8 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 sctp_iterator root 9 0.0 0.0 0 8 ?? DL 1:31PM 0:03.69 pfpurge root 10 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 audit root 14 0.0 0.0 0 8 ?? DL 1:31PM 0:33.35 yarrow root 15 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 xpt_thrd root 16 0.0 0.0 0 8 ?? DL 1:31PM 0:00.02 pagedaemon root 17 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 vmdaemon root 18 0.0 0.0 0 8 ?? DL 1:31PM 0:00.00 pagezero root 19 0.0 0.0 0 8 ?? DL 1:31PM 0:00.03 idlepoll root 20 0.0 0.0 0 8 ?? DL 1:31PM 0:00.13 bufdaemon root 21 0.0 0.0 0 8 ?? DL 1:31PM 0:00.10 vnlru root 22 0.0 0.0 0 8 ?? DL 1:31PM 0:01.35 syncer root 23 0.0 0.0 0 8 ?? DL 1:31PM 0:00.11 softdepflush root 39 0.0 0.0 0 8 ?? DL 1:31PM 0:00.11 md0 root 254 0.0 0.1 3408 1156 ?? INs 1:31PM 0:00.03 /usr/local/sbin/check_reload_status root 256 0.0 0.1 3408 1032 ?? IN 1:31PM 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 267 0.0 0.1 1888 540 ?? Is 1:31PM 0:00.00 /sbin/devd root 1712 0.0 0.2 4948 2544 ?? Ss 4:39PM 0:00.85 /usr/sbin/syslogd -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf root 2712 0.0 1.5 53596 15432 ?? S 5:54PM 0:13.80 /usr/local/bin/php root 4825 0.0 0.1 3316 1356 ?? SNs 1:33PM 0:11.50 /usr/local/sbin/apinger -c /var/etc/apinger.conf root 9353 0.0 0.1 3316 1240 ?? Is 1:31PM 0:00.00 dhclient: rl0 priv (dhclient) root 9853 0.0 1.5 53596 15436 ?? S 5:53PM 0:14.68 /usr/local/bin/php root 12729 0.0 0.1 3404 1348 ?? I 1:33PM 0:00.00 cron: running job (cron) root 12921 0.0 0.1 3656 1340 ?? Is 1:33PM 0:00.01 /bin/sh /usr/scripts/start-keeper.sh root 14221 0.0 0.1 3656 1440 ?? SN 1:33PM 0:04.79 sh /usr/scripts/mpd-keeper _dhcp 15600 0.0 0.1 3316 1376 ?? Is 1:31PM 0:00.00 dhclient: rl0 (dhclient) root 16742 0.0 0.3 7992 3520 ?? RNs 4:50PM 0:00.71 sshd: admin@pts/0 (sshd) nobody 18036 0.0 0.3 5556 2636 ?? S 4:50PM 0:05.32 /usr/local/sbin/dnsmasq --local-ttl 1 --all-servers --dns-forward-max=5000 --cache-size=10000 root 18250 0.0 0.4 9488 4300 ?? SNs 1:33PM 0:00.49 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/l2tp_opt1.pid -s ppp l2tpclient root 22166 0.0 0.3 5176 2628 ?? Ss 1:31PM 0:00.48 /usr/sbin/hostapd -B /var/etc/hostapd_ath0_wlan0.conf dhcpd 24292 0.0 0.6 8436 6152 ?? Ss 4:50PM 0:01.01 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf ste0 ath0_wlan0 root 24977 0.0 0.1 3532 1208 ?? Is 4:50PM 0:00.02 /usr/local/sbin/sshlockout_pf 15 root 32707 0.0 0.2 4496 1932 ?? SN 4:51PM 0:00.10 /usr/local/bin/rrdtool - root 37039 0.0 0.1 3436 1540 ?? Is 1:31PM 0:01.23 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf root 43281 0.0 0.1 3404 1360 ?? Ss 1:32PM 0:00.18 /usr/sbin/cron -s root 43753 0.0 0.1 3316 992 ?? Is 1:32PM 0:00.03 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 43904 0.0 0.1 3316 992 ?? Is 1:32PM 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts root 43930 0.0 0.3 5272 3216 ?? INs 4:50PM 0:00.00 /usr/sbin/sshd root 44525 0.0 0.1 3316 960 ?? Is 1:32PM 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data root 45349 0.0 0.3 6588 3440 ?? SN 4:50PM 0:05.36 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf nobody 45618 0.0 0.1 3344 992 ?? Is 2:01PM 0:00.05 nc -w 2000 192.168.1.43 6667 root 47162 0.0 0.1 3532 1196 ?? Is 1:32PM 0:00.02 /usr/local/sbin/sshlockout_pf 15 root 47882 0.0 1.0 52572 10544 ?? Is 1:31PM 0:00.14 /usr/local/bin/php root 48714 0.0 1.0 52572 10544 ?? Is 1:31PM 0:00.14 /usr/local/bin/php root 51899 0.0 0.2 3656 1572 ?? IN 4:50PM 0:03.40 /bin/sh /var/db/rrd/updaterrd.sh root 53185 0.0 0.1 1564 592 ?? IN 6:46PM 0:00.00 sleep 60 root 53472 0.0 0.1 3316 1340 ?? Is 1:32PM 0:00.00 ntpd: priv (ntpd) root 53572 0.0 0.1 1564 592 ?? SN 6:47PM 0:00.00 sleep 5 _ntp 6388 0.0 0.1 3316 1344 v0- I 1:31PM 0:00.20 ntpd: ntp engine (ntpd) root 27567 0.0 0.3 5912 2612 v0- S 1:31PM 0:01.01 /usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 root 27609 0.0 0.1 3316 928 v0- S 1:31PM 0:01.12 logger -t pf -p local0.info root 46972 0.0 0.1 3684 1500 v0 Is 1:32PM 0:00.03 login pam (login) root 47163 0.0 0.1 3656 1392 v0 I 1:32PM 0:00.01 -sh (sh) root 48549 0.0 0.1 3656 1392 v0 I+ 1:32PM 0:00.01 /bin/sh /etc/rc.initial root 18234 0.0 0.2 3656 1556 0 Is 4:50PM 0:00.01 /bin/sh /etc/rc.initial root 24340 0.0 0.2 3672 2364 0 S 4:50PM 0:00.08 /bin/tcsh -
Now that you got rid of whatever modified source was there, I'm wondering if the remaining issues are just a fact that you're running possibly the worst NICs ever created, and an old Celeron CPU (the lack of cache hits network throughput performance in a firewall scenario hard, huge diff between a Celeron and P4 of the same clock speed for firewall purposes). 80 Mbps through crap NICs and an old Celeron proc may just be tops of what your hardware can accomplish. Using a P4 proc of the same clock speed would be drastically faster for firewall purposes. Better NICs would reduce CPU usage, but not sure if by enough to make much diff.
-
cmb, Replace the network card on the TP-LINK TG-3269 and still loaded processor at 100%. Perhaps you are right, we have to change the CPU.
Or is there another way?
-
Is this really a P4 era Celeron?
http://en.wikipedia.org/wiki/List_of_Intel_Celeron_microprocessors#.22Willamette-128.22_.28180_nm.29I am running a P4-M at 1.2GHz. It can pass >300Mbps. Yes it has 512KB cache vs 128KB in the Celeron but I find it hard to believe you couldn't pass 80Mbps. :-
Interesting information about cache being so important though.
Do you have hundreds of firewall rules? What is using the CPU time in top -SH?Replacing the Celeron with a P4 should be easy though, they are very cheap. I have several here you could have for free if you were near enough. ;)
Steve
-
There have been a couple instances of people on here running really old Celerons that got horrid performance, just slapping a really old P4 with the same clock speed into the same box quadrupled throughput in one case. Way more than I would have expected, the cache makes a massive difference.
-
Replaced the processor Intel (R) Pentium (R) 4 CPU 3.00GHz, speed now works fine on all the 80-90 Mbps)))
But the CPU loading is still a lot, which is 35-85% when downloading a file, which can then be wrong?last pid: 31994; load averages: 0.86, 0.64, 0.37 up 0+00:10:32 10:02:44 94 processes: 5 running, 74 sleeping, 15 waiting CPU: 1.7% user, 0.0% nice, 38.8% system, 21.3% interrupt, 38.2% idle Mem: 45M Active, 15M Inact, 40M Wired, 108K Cache, 23M Buf, 884M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 16K RUN 0 7:41 45.75% {idle: cpu0} 11 root 171 ki31 0K 16K RUN 1 5:40 40.58% {idle: cpu1} 0 root -68 0 0K 64K - 1 3:40 36.96% {ath0 taskq} 12 root -28 - 0K 120K WAIT 0 0:32 34.86% {swi5: +} 13 root 55 - 0K 16K sleep 1 0:16 15.48% {ng_queue0} 13 root 55 - 0K 16K RUN 0 0:16 15.38% {ng_queue1} 55418 root 47 0 54620K 20792K piperd 0 0:06 0.68% php 12 root -32 - 0K 120K WAIT 0 0:11 0.39% {swi4: clock} 55510 root 76 0 53596K 20132K accept 0 0:06 0.29% php 0 root 76 0 0K 64K sched 1 1:02 0.00% {swapper} 53409 root 47 0 54620K 20524K accept 1 0:03 0.00% php 53607 root 47 0 53596K 16348K accept 1 0:02 0.00% php 14 root -16 - 0K 8K - 1 0:01 0.00% yarrow 0 root -68 0 0K 64K - 1 0:01 0.00% {ath0 taskq} 33903 root 44 0 4948K 2516K select 0 0:00 0.00% syslogd 20429 root 64 20 3316K 1356K select 1 0:00 0.00% apinger 62838 root 64 20 5564K 3256K kqread 0 0:00 0.00% lighttpd 4 root -8 - 0K 8K - 1 0:00 0.00% g_down 2937 root 76 20 3656K 1440K wait 0 0:00 0.00% sh 49585 root 44 0 3712K 2012K CPU0 0 0:00 0.00% top 3 root -8 - 0K 8K - 1 0:00 0.00% g_up 12 root -68 - 0K 120K WAIT 0 0:00 0.00% {irq18: ath0} -
Help me, Please!
