SLBD using entire CPU

SurfceS

I have the same problem here…

http://forum.pfsense.org/index.php/topic,6852.0.html

I have two boxes, I change the script on the main one, and let the other one with the old script.

After few hour, I can see only 2 slbd processes on the main one, and 14 one the second one... So, that did the trick.

Script was changed as this, as the second killall command is not needed.

#!/bin/sh

if [ ps awux | grep slbd | wc -l -gt 0 ]; then
killall -9 slbd
~~killall slbd~~
/usr/local/sbin/slbd -c/var/etc/slbd.conf -r5000
fi

Regards,

wjs

Things appear stable although I have very little traffic being routed through the pool.
It looks like this 'fix' works.

I'm definatly not a developer but should something like this be considered for integration into the source tree?

I'm not sure who to talk to even to mention this…

wjs

I've started moving more and more traffic back into the load balancing pools…
SLBD is getting stuck at full usage again! even with the modified script

I think the script isn't being run often enough.
This is rapidly turning into less of a fix and more of a workaround. I want to make this right.

Is anyone having this problem still? or am I going nuts??

I currently have the majority of my traffic going into my primary wan port without going through a pool. The rest (light web browsing from a few users) goes into a pool which has its own two wan ports.

cmb

It's not a fix, it's a work around until we can properly test and implement an alternative to slbd. We know what the problem is, unfortunately it's pretty much impossible to solve. The solution is ditching slbd for hoststated, which will be done in a future version.

cmb

Also, this work around does seem to work for the vast majority of people.

wjs: how much load are you pushing to cause it to break down so easily?

wjs

cmb,
Thanks very much for pushing that change, I saw it on the cvs track.

right now its only my web browsing that going into the wan pool. the primary wan port, which is not part of the pool at the moment, has a good bit of traffic. last night we had about 1MB/s continuous sometimes going up to about 10MB/s when someone would pull down something big.

The cpu load hovers under 15% or 20% but i think most of that is because i've got the whole dashboard open.
I am only getting one process at a time maxing out before the script kicks in so the system never goes full load. (dual cpu system)

I'm not sure this answered your question…

If there is anything I can do to help get "hoststated" working for the next version let me know.

sullrich

@wjs:

If there is anything I can do to help get "hoststated" working for the next version let me know.

Basically translate the .conf file from slbd -> hostsated. I really want to get this in here and its on my gigantic whiteboard now but if you want to do the work please do so as my gigantic whiteboard has many entries now :)

wjs

i'm not an expert but i'll take a shot at it

Superman

Any steps we can take to install and test hostated, maybe having it alongside slbd just in case?

The change in the script is working, but in between several processes get started and start to chew up 100% CPU again. It would be nice to try out the newer service, but have the other to fall-back on just in case…

Juve

Sorry to pull that topic up again but I am suffering the same problem with a clean and fresh 1.2 install. The pool is a failover pool with two WANs. One of the line is currently down so one gateway can't be pinged.

Am I unlucky or is it something people still encounter ?

wjs

I was able to 'mitigate' the problem by switching to a new machine and doing a fresh install. It seems to me like a fairly random problem.

does slbd go to full load as soon as the pool is activated or after some time?

Juve

It is after some time or just after saving changes, it is really a random behaviour.

I'll monitor it to identify when it forks and goes to 99%. I have other machines where it goes very well.

Juve

21 hours without any problem an then 100% cpu usage on one CPU.
Where can we adjust the time the killall script is executed ?

Thanks

cmb

It's a known issue, using the uniprocessor kernel resolves it.

wjs

@cmb:

It's a known issue, using the uniprocessor kernel resolves it.

That would explain why the problem resolved itself on the new machine, its single core.

Any reason why that is? I had been planning to build a new dual core router.

Juve

Thanks cmb,

When we do the killall -9 on slbd what effects does it have on connections ? does it kills openned connections ?

eri--

1.2.1 should not suffer from it.

Juve

OK fine.
But what about connections when killing slbd ?

Thanks

cmb

slbd has no effects on your connections, its only function is to update the ruleset.

Juve

Ok thanks !
This mean I can my own crontab script to kill/restart it more often.