SLBD using entire CPU

wjs

I've been trying to execute the killall slbd command. In its defaul state (sending the TERM command) the processes don't exit. The killall -9 slbd command does seem to work (it is kill -9 !!! )

Anyway, anyone have any advice on changing the script to killall -9 slbd?

i'm not sure that this is a good idea…

sai

maybe you should reinstall using the latest image? probably some update went wrong which is why the script you have is not working.

wjs

This install was done to a clean hard drive and configured from scratch. I currently have RC3 installed also.

I'm not sure what you're recommending. Would you like me to put a newer snapshot on?

sai

I was recommending a clean install from scratch

:)

Superman

I have to say that my system is a complete clean install from the released 1.2RC3, and I'm seeing the same problems. I tried the killall -9 slbd and that worked on my system as well. But the regular script doesn't work at all.

wjs

I think i'm going to try modifying that scrpt and see what happens.

I'm not to happy about using kill -9 every 5 houts to fix a problem though…
any other ideas?

wjs

I changed the script so that it kill -9's
I ran it by hand and it worked. now its time to wait a few hours and see if the problem is "fixed".

$ cat /usr/local/sbin/reset_slbd.sh

#!/bin/sh

if [ `ps awux | grep slbd | wc -l` -gt 0 ]; then
	killall slbd
	killall -9 slbd
	/usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
fi

SurfceS

I have the same problem here…

http://forum.pfsense.org/index.php/topic,6852.0.html

I have two boxes, I change the script on the main one, and let the other one with the old script.

After few hour, I can see only 2 slbd processes on the main one, and 14 one the second one... So, that did the trick.

Script was changed as this, as the second killall command is not needed.

#!/bin/sh

if [ ps awux | grep slbd | wc -l -gt 0 ]; then
killall -9 slbd
~~killall slbd~~
/usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
fi

Regards,

wjs

Things appear stable although I have very little traffic being routed through the pool.
It looks like this 'fix' works.

I'm definatly not a developer but should something like this be considered for integration into the source tree?

I'm not sure who to talk to even to mention this…

wjs

I've started moving more and more traffic back into the load balancing pools…
SLBD is getting stuck at full usage again! even with the modified script

I think the script isn't being run often enough.
This is rapidly turning into less of a fix and more of a workaround. I want to make this right.

Is anyone having this problem still? or am I going nuts??

I currently have the majority of my traffic going into my primary wan port without going through a pool. The rest (light web browsing from a few users) goes into a pool which has its own two wan ports.

cmb

It's not a fix, it's a work around until we can properly test and implement an alternative to slbd. We know what the problem is, unfortunately it's pretty much impossible to solve. The solution is ditching slbd for hoststated, which will be done in a future version.

cmb

Also, this work around does seem to work for the vast majority of people.

wjs: how much load are you pushing to cause it to break down so easily?

wjs

cmb,
Thanks very much for pushing that change, I saw it on the cvs track.

right now its only my web browsing that going into the wan pool. the primary wan port, which is not part of the pool at the moment, has a good bit of traffic. last night we had about 1MB/s continuous sometimes going up to about 10MB/s when someone would pull down something big.

The cpu load hovers under 15% or 20% but i think most of that is because i've got the whole dashboard open.
I am only getting one process at a time maxing out before the script kicks in so the system never goes full load. (dual cpu system)

I'm not sure this answered your question…

If there is anything I can do to help get "hoststated" working for the next version let me know.

sullrich

@wjs:

If there is anything I can do to help get "hoststated" working for the next version let me know.

Basically translate the .conf file from slbd -> hostsated. I really want to get this in here and its on my gigantic whiteboard now but if you want to do the work please do so as my gigantic whiteboard has many entries now :)

wjs

i'm not an expert but i'll take a shot at it

Superman

Any steps we can take to install and test hostated, maybe having it alongside slbd just in case?

The change in the script is working, but in between several processes get started and start to chew up 100% CPU again. It would be nice to try out the newer service, but have the other to fall-back on just in case…

Juve

Sorry to pull that topic up again but I am suffering the same problem with a clean and fresh 1.2 install. The pool is a failover pool with two WANs. One of the line is currently down so one gateway can't be pinged.

Am I unlucky or is it something people still encounter ?

wjs

I was able to 'mitigate' the problem by switching to a new machine and doing a fresh install. It seems to me like a fairly random problem.

does slbd go to full load as soon as the pool is activated or after some time?

Juve

It is after some time or just after saving changes, it is really a random behaviour.

I'll monitor it to identify when it forks and goes to 99%. I have other machines where it goes very well.

Juve

21 hours without any problem an then 100% cpu usage on one CPU.
Where can we adjust the time the killall script is executed ?

Thanks