CP enabled = CPU usage goes to 100% and seemingly beyond with
-
This is the latest, this time it didnt' kill my ssh session, but it won't respond to additionall ssh or web gui requests, it looks like something is causing the swap partition to overflow, 856M used isn't what I would call normal anyways…
last pid: 39981; load averages: 0.20, 2.09, 2.40 up 0+01:57:48 12:50:25
206 processes: 103 running, 93 sleeping, 6 zombie, 4 lock
CPU states: 1.9% user, 0.0% nice, 2.6% system, 0.7% interrupt, 94.8% idle
Mem: 123M Active, 16M Inact, 40M Wired, 448K Cache, 28M Buf, 564K Free
Swap: 1024M Total, 856M Used, 168M Free, 83% Inuse, 600K In, 700K OutPID USERNAME PRI NICE SIZE RES STATE TIME CPU COMMAND
I tried doing a control c to break the top session and gain access to the command line again, it worked but its about a full minute delay for the commands I type to appear on the screen. My best guess is the swap is the problem, but I'm not sure what about the captive portal is filling it up, I did see that before it crashed a whole bunch of php commands were executed and the res state changed after about 2 sec to pfault. I tried and garner more information on the next run w/ it.
Update: before I rebooted the box, I got a ps -A off and here is what I got (deleted several entries that are irrevalant from the top to shorten the post):
14088 ?? RL 0:15.10 /usr/local/sbin/lighttpd -f /var/etc/lighty-CaptivePort
39303 ?? S 0:00.39 /usr/local/bin/php -f /etc/rc.prunecaptiveportal
39505 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39521 ?? Z 0:00.54 <defunct>39542 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39557 ?? RL 0:00.48 /usr/local/bin/php /usr/local/captiveportal/index.php
39572 ?? I 0:00.00 cron: running job (cron)
39578 ?? IWs 0:00.00 /bin/sh -c /usr/bin/getRadiusLists.sh
39600 ?? RL 0:00.53 /usr/local/bin/php /usr/local/captiveportal/index.php
39601 ?? RL 0:00.53 /usr/local/bin/php /usr/local/captiveportal/index.php
39611 ?? RL 0:00.52 /usr/local/bin/php /usr/local/captiveportal/index.php
39617 ?? IW 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
39634 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39635 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39640 ?? RL 0:00.49 /usr/local/bin/php /usr/local/captiveportal/index.php
39644 ?? I 0:00.50 /usr/local/bin/php -f /etc/rc.banner
39645 ?? I 0:00.01 grep WAN
39652 ?? RL 0:00.49 /usr/local/bin/php /usr/local/captiveportal/index.php
39655 ?? RL 0:00.52 /usr/local/bin/php /usr/local/captiveportal/index.php
39669 ?? RL 0:00.52 /usr/local/bin/php /usr/local/captiveportal/index.php
39671 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39673 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39677 ?? RL 0:00.52 /usr/local/bin/php /usr/local/captiveportal/index.php
39678 ?? RL 0:00.52 /usr/local/bin/php /usr/local/captiveportal/index.php
39689 ?? Z 0:00.52 <defunct>39692 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39695 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39703 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39709 ?? RL 0:00.44 /usr/local/bin/php /usr/local/captiveportal/index.php
39713 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39718 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php
39723 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39731 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39734 ?? RL 0:00.48 /usr/local/bin/php /usr/local/captiveportal/index.php
39736 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39738 ?? I 0:00.00 cron: running job (cron)
39739 ?? I 0:00.00 cron: running job (cron)
39740 ?? IWs 0:00.00 /bin/sh -c /usr/bin/nice -n20 /etc/ping_hosts.sh
39741 ?? IWs 0:00.00 /bin/sh -c /usr/bin/getRadiusLists.sh
39745 ?? S 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39746 ?? RL 0:00.48 /usr/local/bin/php /usr/local/captiveportal/index.php
39747 ?? IWN 0:00.00 sh /usr/local/etc/rc.d/proxy_monitor.sh start
39748 ?? RL 0:00.47 /usr/local/bin/php /usr/local/captiveportal/index.php
39749 ?? RL 0:00.49 /usr/local/bin/php /usr/local/captiveportal/index.php
39750 ?? RL 0:00.50 /usr/local/bin/php /usr/local/captiveportal/index.php
39751 ?? IN 0:00.01 /bin/sh /etc/ping_hosts.sh
39752 ?? RL 0:00.44 /usr/local/bin/php /usr/local/captiveportal/index.php
39753 ?? RL 0:00.44 /usr/local/bin/php /usr/local/captiveportal/index.php
39754 ?? S 0:00.46 /usr/local/bin/php /usr/local/captiveportal/index.php
39756 ?? RL 0:00.41 /usr/local/bin/php /usr/local/captiveportal/index.php
39759 ?? RL 0:00.36 /usr/local/bin/php /usr/local/captiveportal/index.php
39761 ?? DL 0:00.32 /usr/local/bin/php /usr/local/captiveportal/index.php
39765 ?? IW 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
39771 ?? RL 0:00.37 /usr/local/bin/php /usr/local/captiveportal/index.php
39775 ?? RNL 0:00.01 awk { print $1 }
39787 ?? RL 0:00.31 /usr/local/bin/php /usr/local/captiveportal/index.php
39789 ?? RL 0:00.27 /usr/local/bin/php /usr/local/captiveportal/index.php
39790 ?? RL 0:00.45 /usr/local/bin/php -f /etc/rc.banner
39791 ?? I 0:00.01 grep WAN
39794 ?? RL 0:00.27 /usr/local/bin/php /usr/local/captiveportal/index.php
39800 ?? IWN 0:00.00 /bin/sh /etc/ping_hosts.sh
39805 ?? L 0:00.40 /usr/local/bin/php /usr/local/captiveportal/index.php
39809 ?? RL 0:00.30 /usr/local/bin/php /usr/local/captiveportal/index.php
39811 ?? RL 0:00.43 /usr/local/bin/php /usr/local/captiveportal/index.php
39816 ?? RL 0:00.29 /usr/local/bin/php /usr/local/captiveportal/index.php
39818 ?? I 0:00.00 cron: running job (cron)
39819 ?? I 0:00.00 cron: running job (cron)
39820 ?? RNL 0:00.03 /bin/ps awwux
39821 ?? RL 0:00.30 /usr/local/bin/php /usr/local/captiveportal/index.php
39823 ?? RL 0:00.29 /usr/local/bin/php /usr/local/captiveportal/index.php
39824 ?? RL 0:00.29 /usr/local/bin/php /usr/local/captiveportal/index.php
39825 ?? RL 0:00.26 /usr/local/bin/php /usr/local/captiveportal/index.php
39826 ?? IWs 0:00.00 /bin/sh -c /usr/bin/nice -n20 /etc/ping_hosts.sh
39828 ?? IWs 0:00.00 /bin/sh -c /usr/bin/getRadiusLists.sh
39829 ?? RNL 0:00.01 grep ping_hosts.sh
39830 ?? IN 0:00.01 grep -v grep
39831 ?? RL 0:00.30 /usr/local/bin/php /usr/local/captiveportal/index.php
39833 ?? IN 0:00.00 wc -l
39845 ?? RL 0:00.29 /usr/local/bin/php /usr/local/captiveportal/index.php
39847 ?? RL 0:00.40 /usr/local/bin/php /usr/local/captiveportal/index.php
39849 ?? IW 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
39852 ?? RL 0:00.21 /usr/local/bin/php /usr/local/captiveportal/index.php
39854 ?? IN 0:00.01 /bin/sh /etc/ping_hosts.sh
39855 ?? RL 0:00.33 /usr/local/bin/php /usr/local/captiveportal/index.php
39858 ?? RL 0:00.19 /usr/local/bin/php /usr/local/captiveportal/index.php
39861 ?? RL 0:00.27 /usr/local/bin/php /usr/local/captiveportal/index.php
39863 ?? RL 0:00.30 /usr/local/bin/php /usr/local/captiveportal/index.php
39864 ?? RL 0:00.27 /usr/local/bin/php /usr/local/captiveportal/index.php
39868 ?? RL 0:00.26 /usr/local/bin/php -f /etc/rc.banner
39869 ?? I 0:00.01 grep WAN
39882 ?? RL 0:00.28 /usr/local/bin/php /usr/local/captiveportal/index.php
39886 ?? I 0:00.00 cron: running job (cron)
39887 ?? I 0:00.00 cron: running job (cron)
39889 ?? RL 0:00.17 /usr/local/bin/php /usr/local/captiveportal/index.php
39891 ?? IWN 0:00.00 /bin/sh /etc/ping_hosts.sh
39892 ?? RL 0:00.18 /usr/local/bin/php /usr/local/captiveportal/index.php
39894 ?? RL 0:00.17 /usr/local/bin/php /usr/local/captiveportal/index.php
39897 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39902 ?? IWs 0:00.00 /bin/sh -c /usr/bin/nice -n20 /etc/ping_hosts.sh
39903 ?? IWs 0:00.00 /bin/sh -c /usr/bin/getRadiusLists.sh
39904 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39908 ?? RL 0:00.15 /usr/local/bin/php /usr/local/captiveportal/index.php
39909 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39912 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39913 ?? RL 0:00.17 /usr/local/bin/php /usr/local/captiveportal/index.php
39916 ?? RNL 0:00.01 grep -v grep
39918 ?? IN 0:00.00 wc -l
39919 ?? RL 0:00.21 /usr/local/bin/php /usr/local/captiveportal/index.php
39920 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39924 ?? RL 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39927 ?? RL 0:00.13 /usr/local/bin/php /usr/local/captiveportal/index.php
39928 ?? RL 0:00.15 /usr/local/bin/php /usr/local/captiveportal/index.php
39931 ?? IN 0:00.01 /bin/sh /etc/ping_hosts.sh
39936 ?? IW 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
39949 ?? RL 0:00.17 /usr/local/bin/php /usr/local/captiveportal/index.php
39952 ?? RL 0:00.17 /usr/local/bin/php /usr/local/captiveportal/index.php
39954 ?? RNL 0:00.01 sh -c echo sleeping > /tmp/check_reload_status
39957 ?? L 0:00.16 /usr/local/bin/php /usr/local/captiveportal/index.php
39968 ?? RL 0:00.15 /usr/local/bin/php /usr/local/captiveportal/index.php
39969 ?? RNL 0:00.00 /bin/sh /var/db/rrd/updaterrd.sh
39974 ?? RL 0:00.09 /usr/local/bin/php /usr/local/captiveportal/index.php
39978 ?? I 0:00.00 cron: running job (cron)
39979 ?? I 0:00.00 cron: running job (cron)
39984 ?? RL 0:00.10 /usr/local/bin/php -f /etc/rc.banner
39985 ?? I 0:00.01 grep WAN
39986 ?? IWs 0:00.00 /bin/sh -c /usr/bin/getRadiusLists.sh
39987 ?? IWs 0:00.00 /bin/sh -c /usr/bin/nice -n20 /etc/ping_hosts.sh
39988 ?? RL 0:00.08 /usr/local/bin/php /usr/local/captiveportal/index.php
40002 ?? IW 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
40004 ?? RLs 0:00.01 /usr/sbin/sshd -R
40005 ?? IWN 0:00.00 /bin/sh /etc/ping_hosts.sh
40006 ?? RNL 0:00.01 /bin/sh /etc/ping_hosts.sh
40007 ?? RNL 0:00.00 sh /usr/local/etc/rc.d/proxy_monitor.sh start
40008 ?? RLs 0:00.01 /usr/sbin/sshd -R
40009 ?? RLs 0:00.01 /usr/sbin/sshd -R
40010 ?? RLs 0:00.01 /usr/sbin/sshd -R
40012 ?? RLs 0:00.01 /usr/sbin/sshd -R
40013 ?? RNL 0:00.00 /usr/local/sbin/check_reload_status
40014 ?? IW 0:00.00 sh /usr/local/etc/rc.d/proxy_monitor.sh start
40015 ?? RNL 0:00.00 /bin/ps awwux
40016 ?? RL 0:00.01 ps awux
40017 ?? DL 0:00.00 grep squid -D
40018 ?? DL 0:00.00 grep -v grep
40019 ?? RNL 0:00.00 /bin/sh /etc/ping_hosts.sh
40020 ?? RNL 0:00.00 /bin/sh /etc/ping_hosts.sh
40021 ?? I 0:00.00 wc -l
40022 ?? RL 0:00.01 sh -c /sbin/ping -c 1 -t 1 192.168.104.178 > /dev/null
40023 ?? RL 0:00.00 sh -c /sbin/ifconfig -l
40024 ?? RNL 0:00.00 wc -l
40025 ?? RL 0:00.00 awk { print $1 }
40027 ?? LLs 0:00.00 sshd: [accepted] (sshd)
40028 ?? RL 0:00.00 sh -c /sbin/ipfw -T list 10022 2>/dev/null
40029 ?? RL 0:00.01 /usr/local/bin/php -f /etc/rc.banner
40030 ?? D 0:00.00 /bin/sh /usr/bin/getRadiusLists.sh
40031 ?? RL 0:00.00 sshd: [accepted] (sshd)
40032 ?? RL 0:00.00 sshd: [accepted] (sshd)
40033 ?? RL 0:00.00 /usr/sbin/sshd
40034 ?? R 0:00.00 /usr/local/sbin/lighttpd -f /var/etc/lighty-CaptivePort
40035 ?? RL 0:00.00 /usr/sbin/sshd
1005 v0 IWs 0:00.00 login [pam] (login)
1007 v0 IW 0:00.00 -sh (sh)
1009 v0 IW+ 0:00.00 /bin/sh /etc/rc.initial
211 con- IW 0:00.00 dhclient: vr0 [priv] (dhclient)
291 con- S 0:00.26 /usr/sbin/tcpdump -l -n -e -ttt -i pflog0
292 con- S 0:00.08 logger -t pf -p local0.info
795 con- SW 0:00.00 ntpd: ntp engine (ntpd)
909 con- I 0:00.17 sh /usr/local/etc/rc.d/proxy_monitor.sh start
932 con- IWN 0:00.00 /usr/local/sbin/check_reload_status
938 con- IWN 0:00.00 /usr/local/sbin/check_reload_status
3591 p0 IWs 0:00.00 -sh (sh)
3592 p0 IW 0:00.00 /bin/sh /etc/rc.initial
3607 p0 IW+ 0:00.00 /usr/local/bin/bash
7565 p1 IWs 0:00.00 -sh (sh)
7567 p1 IW 0:00.00 /bin/sh /etc/rc.initial
7585 p1 IW 0:00.00 /usr/local/bin/bash
40026 p1 R+ 0:00.01 ps -A
16411 p2 IWs 0:00.00 -sh (sh)
16412 p2 IW+ 0:00.00 /bin/sh /etc/rc.initial
39876 p2 RL+ 0:00.26 /usr/local/bin/php -f /etc/rc.banner</defunct></defunct>So that is what is filling up the swap, why so many are starting, I'm not sure, shouldn't that be prevented by the max connections?
Is there possibly something wrong with the index.php that is causing it to spam itself a billion times?Something is pounding your CP with new sessions… May want to locate the client in question. Either that or limit the maximum connections per client further.
-
I had just the same problem some time ago. The solution was to close down Captive portal. After that CPU usage returned back to normal. (5-10%). The box was a firewall for a lan party with about 110 attendants. i can't post any info about the processes since the box is not online anymore but I had the same problem as Justinw.
Everything was deadly slow and there were loads of these "39542 ?? RL 0:00.51 /usr/local/bin/php /usr/local/captiveportal/index.php" php processes.
The cpu usage also went 100%Edit: The version I am using is 1.0.1
-
1.0.1 is old use a snapshot like 06-02-2007
-
When I made this post, the snapshot I was using was only 1-2 days old, I haven't been able to try it with more recent snapshots, but should be able to soon, let you know then what happens.
-
Same problem occurs on a WRAP box when capitive portal is enabled on the OPT1 wireless interface without using RADIUS only the local user manager.
-
Anybody please test with the latest snapshot. Fastcgi was enabled for the CP webserver wich should speed things up.
-
I've tested it, I also added an additional 512 ram to the server as well. The usage still hits 100, but it hasn't crashed the server yet, and watching top, the 100% times are just spikes, not holding steady anymore. No more than 5 requests or so get lined up in top before they all get cleared out. Still pretty intense but seems to be working better. Thanks for the fix
-
Sullrich mentioned in the past also that he runs a setup with 5,000 users using the CP. I'm running a lot less users (about 10-15 times less). Anyone else that could advise on hardware specs for 250-500 users with a cp would be helpful. Right now it may be somewhat skimpy 1.5 ghz duron with 768 ram. The CPU is pegged with 3-4 users hitting the CP. Right now max concurrent connections is actually set to 1. If I understand right this means a single user can only open 1 thread, and the max is open at once is either 4 or 16, not sure which? If there are ways to optimize the portal that would be helpful to know about as well. Thanks in advance.
-
I am having the same issue with a 3G Pentium with 2G of memory and 2G swap space. I have not had issues with the CP until today. CPU was 100% and swap space was totally consumed and the the system completely locked. After reboot the system would work for approximately 5 minutes and then reinitiate the same process. After man frustrating recycles, I disable CP completely. Any assistance will be greatly appreciated.
Thanks,
-
Any abnormal traffic coming in through the CP enabled interface that is flooding the CP page? Also what does top report when it starts to freak out?
-
Also check the state tables. How many states are active to the captive portal.
I have found a lot of environments that have viruses and such that are making outbound port 80 connections that end up getting sent to the captive portal. What this does is overwhelm the machine due to the amount of requests a second that some viruses attempt.
If you find that clients are making too many requests a second limit the client connection count in the Captive Portal settings screen.