Haproxy : kern.maxfiles limit exceeded by uid 80

th3penguinwhisperer

Hi all,

This is my first post here. Not sure if this is the right place to put it.

We have haproxy running for loadbalancing and for some use its SSL offloading capabilities too.
We already seen a few times that haproxy doesn't seem to function anymore while all backend nodes do.
I noticed in the logs the message "kern.maxfiles limit exceeded by uid 80".
I restarted haproxy to resolve the issue. And haproxy is also running as uid 80 so I'm pretty sure this is an issue with HAproxy.

Anyone an idea what might be causing this or a direction in which I should look?

I changed the value of kern.maxfiles to 32000. However that probably wont handle the root cause. Or maybe there were just too many open connections?

Thanks in advance!

PiBa

On the settings tab of haproxy-devel you can read that these 2 sysctl settings need to be 2*maxconn 'kern.maxfiles', 'kern.maxfilesperproc' and then some more.. If thats not the case you could indeed see strange issues.

Check on the stats page (if its still working when you have that issue) what kind of connection counts are used.

th3penguinwhisperer

Thanks for the response.

I got 500 connections per backend configured.
1 process running.

I see this listed:
Current memory usage: 19772 19772 19772 27964 kB.
Current 'System Tunables' settings.
'kern.maxfiles': 12328
'kern.maxfilesperproc': 11095

First line looks like a bug. I'm pretty sure I highered maxfiles and maxfilesperproc. But it should still be plenty enough as 500*2 + 31 = 1031

PiBa

The first line shows that 4 haproxy processes are running, if you have long living sessions, and a few applied config changes that could be fine.. It could also mean a few did not shutdown properly.. You might want to check the pfsense systemlog it should show what pid was running and what gets started..(if package was recently re- installed)

Either way it seems indeed maxproc is high enough for haproxy…

What you could try is to install lsof, and check for open handles. Not sure if that will work..

lsof | awk '{print $2}' | sort | uniq -c | sort -n

then check if the pid of haproxy indeed has a high number of handles.