[Duplicate] Multiple sldb process full the CPU load…

SurfceS · Nov 19, 2007, 3:50 PM

My boxes are really slow after 1 day running, and are completelly unbrowsable after few days.
Load CPU is 100 % after only 20 minutes.

Hardware config :

DELL PowerEdge 860

Intel(R) Xeon(R) CPU 3040 @ 1.86GHz (Dual Core)
1GB 667Mhz DDR2 SDRAM

ad10: Seagate ST3750640NS 3.BKE (Capacity: 732.87 GB)

em0: Intel(R) PRO/1000 Network Connection Version - 6.2.9
em1: Intel(R) PRO/1000 Network Connection Version - 6.2.9
bge0: Broadcom BCM5750 B1, ASIC rev. 0x4101
bge1: Broadcom BCM5750 B1, ASIC rev. 0x4101

Here is a time to startup process load (all in normal usage with clients on it):

After about 20 minutes :

top command result :
last pid: 9245; load averages: 1.00, 0.97, 0.74 up 0+00:19:09 14:59:16
43 processes: 2 running, 41 sleeping

Mem: 12M Active, 9188K Inact, 31M Wired, 4K Cache, 19M Buf, 941M Free
Swap: 2048M Total, 2048M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1026 root 1 116 0 2068K 1112K CPU0 0 18:03 99.02% slbd
439 root 7 20 0 2068K 1160K kserel 1 0:00 0.00% slbd
1226 root 1 8 20 1744K 1184K wait 0 0:00 0.00% sh
958 root 1 96 0 2848K 2304K select 1 0:00 0.00% racoon
199 root 1 96 0 1440K 1072K select 0 0:00 0.00% syslogd
1454 root 1 8 20 1272K 720K nanslp 1 0:00 0.00% check_reload_
9188 root 1 96 0 5744K 2744K select 1 0:00 0.00% sshd
1283 dhcpd 1 96 0 2288K 1992K select 1 0:00 0.00% dhcpd
1375 root 1 8 0 1720K 1156K wait 0 0:00 0.00% sh
804 root 1 4 0 3108K 2148K kqread 0 0:00 0.00% lighttpd
892 root 1 114 0 2960K 2060K select 0 0:00 0.00% mpd
381 root 1 -58 0 3712K 1844K bpf 0 0:00 0.00% tcpdump
1483 root 1 8 0 1712K 1360K wait 1 0:00 0.00% login
120 root 1 96 0 504K 360K select 1 0:00 0.00% devd
805 root 1 8 0 14200K 4672K wait 1 0:00 0.00% php
844 root 1 8 0 14200K 4728K wait 0 0:00 0.00% php
1366 _ntp 1 96 0 1344K 1056K select 1 0:00 0.00% ntpd
245 root 1 -8 0 1268K 664K piperd 1 0:00 0.00% sshlockout_pf

ps -U root | grep slbd result :
439 ?? Ss 0:00.31 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
1026 ?? R 18:34.94 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000

After about one day :

top command result :
last pid: 793; load averages: 16.39, 16.29, 15.80 up 0+22:41:30 14:30:11
71 processes: 16 running, 51 sleeping, 4 zombie

Mem: 32M Active, 11M Inact, 36M Wired, 22M Buf, 913M Free
Swap: 2048M Total, 2048M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
17555 root 1 132 0 2068K 1124K RUN 1 489:09 13.72% slbd
98808 root 1 132 0 2068K 1124K RUN 0 48:09 13.62% slbd
59948 root 1 132 0 2064K 1120K RUN 0 185:45 13.53% slbd
36821 root 1 131 0 2064K 1120K RUN 0 14:01 13.53% slbd
66294 root 1 131 0 2068K 1124K CPU1 0 264:44 13.43% slbd
1099 root 1 132 0 2068K 1124K RUN 1 508:16 13.13% slbd
90934 root 1 132 0 2068K 1124K RUN 0 1:58 13.13% slbd
88097 root 1 131 0 2064K 1120K RUN 0 111:49 13.04% slbd
4366 root 1 131 0 2060K 1116K RUN 0 75:03 13.04% slbd
14495 root 1 131 0 2072K 1128K RUN 0 368:57 12.99% slbd
51364 root 1 131 0 2064K 1120K RUN 0 338:03 12.94% slbd
62537 root 1 131 0 2064K 1120K RUN 0 8:00 12.89% slbd
56257 root 1 131 0 2064K 1120K RUN 1 223:23 12.84% slbd
86809 root 1 131 0 2068K 1136K RUN 0 25:30 12.65% slbd
99904 root 1 8 0 1732K 1272K wait 0 0:00 0.23% sh
1451 root 1 131 0 1264K 624K select 0 0:28 0.00% supervise
1453 root 1 131 0 1264K 624K select 0 0:28 0.00% supervise
771 root 1 4 0 3636K 2740K kqread 0 0:12 0.00% lighttpd

ps -U root | grep slbd result :
1099 ?? R 511:49.55 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
4366 ?? R 78:37.20 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
14495 ?? R 372:30.23 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
17555 ?? R 492:42.82 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
36821 ?? R 17:34.82 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
51364 ?? R 341:35.70 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
56257 ?? R 226:58.66 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
59948 ?? R 189:18.77 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
62537 ?? R 11:34.98 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
66294 ?? R 268:17.05 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
77459 ?? Ss 0:01.07 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
86809 ?? R 29:06.40 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
88097 ?? R 115:22.61 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
90934 ?? R 5:36.69 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
98808 ?? R 51:40.87 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000

And few days after, I need to restart the box if I want it to be in consistent state.

Any others have the same problem, or I'm really not lucky, and both servers are misconfigured/buggy ?

Best regards,

SurfceS · Nov 18, 2007, 3:52 PM

After about 3 days :

top command result :
last pid: 72938; load averages: 54.94, 54.74, 54.68 up 3+00:40:52 16:29:33
106 processes: 56 running, 47 sleeping, 3 zombie

Mem: 46M Active, 15M Inact, 38M Wired, 22M Buf, 894M Free
Swap: 2048M Total, 2048M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
98808 root 1 131 0 2068K 1124K RUN 1 278:08 4.15% slbd
88216 root 1 131 0 2068K 1124K RUN 0 142:53 4.00% slbd
56891 root 1 131 0 2064K 1120K RUN 0 44:15 4.00% slbd
71302 root 1 131 0 2064K 1120K RUN 0 33:11 3.86% slbd
59948 root 1 131 0 2064K 1120K RUN 0 415:32 3.81% slbd
55772 root 1 131 0 2064K 1120K RUN 0 83:31 3.81% slbd
57529 root 1 131 0 2068K 1124K RUN 0 107:13 3.76% slbd
51364 root 1 131 0 2064K 1120K RUN 0 568:02 3.71% slbd
23343 root 1 130 0 2064K 1120K RUN 0 204:37 3.66% slbd
25540 root 1 131 0 2064K 1120K RUN 0 75:45 3.66% slbd
50631 root 1 130 0 2068K 1124K RUN 0 72:52 3.66% slbd
80611 root 1 131 0 2064K 1120K RUN 0 59:41 3.66% slbd
94387 root 1 131 0 2068K 1136K RUN 0 190:24 3.61% slbd
46864 root 1 131 0 2064K 1120K RUN 0 148:51 3.61% slbd
79473 root 1 130 0 2064K 1120K RUN 0 69:26 3.61% slbd
51904 root 1 130 0 2068K 1124K RUN 0 120:52 3.56% slbd
62537 root 1 130 0 2064K 1120K RUN 0 237:58 3.52% slbd
56257 root 1 130 0 2064K 1120K RUN 0 453:11 3.47% slbd

ps -U root | grep slbd result :
1099 ?? R 738:03.40 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
4340 ?? R 39:18.79 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
4366 ?? R 304:59.16 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
4839 ?? R 5:41.31 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
6441 ?? R 101:08.86 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
14495 ?? R 598:49.97 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
17555 ?? R 718:57.58 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
18072 ?? R 168:34.07 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
19919 ?? R 37:43.92 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
20869 ?? R 29:01.77 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
23343 ?? R 204:36.77 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
25167 ?? R 12:34.92 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
25540 ?? R 75:45.05 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
26674 ?? R 64:41.28 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
36821 ?? R 244:02.94 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
45183 ?? R 45:29.78 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
46864 ?? R 148:50.26 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
48269 ?? R 73:13.86 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
49522 ?? R 1:58.44 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
50631 ?? R 72:51.93 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
51364 ?? R 568:01.87 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
51376 ?? R 163:29.06 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
51904 ?? R 120:52.03 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
54306 ?? R 18:33.78 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
55192 ?? R 133:31.05 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
55772 ?? R 83:31.08 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
56257 ?? R 453:10.87 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
56891 ?? R 44:14.27 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
56983 ?? R 72:02.94 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
57529 ?? R 107:12.30 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
57824 ?? Ss 0:00.70 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
59948 ?? R 415:31.69 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
60532 ?? R 178:46.31 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
62537 ?? R 237:57.89 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
66294 ?? R 494:23.43 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
71302 ?? R 33:11.12 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
75171 ?? R 60:13.80 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
79473 ?? R 69:25.31 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
80059 ?? R 7:51.84 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
80611 ?? R 59:40.89 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
82278 ?? R 69:05.82 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
86809 ?? R 255:41.68 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
87345 ?? R 174:14.46 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
88097 ?? R 341:47.91 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
88216 ?? R 142:52.34 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
89463 ?? R 91:04.83 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
94387 ?? R 190:23.74 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
96688 ?? R 30:48.59 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
98808 ?? R 278:07.37 /usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000

I've attached the http://pfSenseBox/phpSysInfo/ result.
The other strange thing, is the buffer line, on the memory usage tab… It full up the memory.

FYI, the internet line is not really used the WE (as you can see on the page).

Best Regards,

SurfceS · Nov 18, 2007, 4:08 PM

After connecting to the box thru ssh, and run :

/usr/bin/killall -9 slbd
/usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000

After that, everything become fine as if I've restart the box.

But it still overload my box few by few, fulling it with slbd process.

Strange case I have…

sai · Nov 19, 2007, 1:17 PM

see http://forum.pfsense.org/index.php/topic,6805.msg38804.html#msg38804 also. same problem there.