HAProxy "Too many sockets on the process"
-
All of a sudden i am having serious problems with HAProxy. I'm on 2.4.5-RELEASE-p1 . HAProxy is 1.8.25.
The issues is, all of a sudden all my backends are having L4 issues. The error messages are as below. Restarting HAproxy resolves the issue, but it comes back. Rebooting does the same trick. I had an uptime of 90 or so days without any issues. I have less that 20 frontends and backends. Some backends are clustered with 2-3 servers.
This is the only reference, and it does not help me in resolving this issue. I have tried to shorten my keep alive checks to 5-10-20 seconds, but makes no difference. The issue may be gone for a week, and then comes back. The log excerpts are examples, in reality I get thousands of log events.
Help, ideas - anything?
===group
Health check for server critters-01.mintsecurity.fi_ipvANY/critters-01.ipa.mintsecurity.fi failed, reason: Layer4 connection problem, info: "Too many sockets on the process", check duration: 0ms, status: 2/3 UP. Health check for server code-01.ipa.mintsecurity.fi_ipvANY/code-01.ipa.mintsecurity.fi failed, reason: Layer4 connection problem, info: "Too many sockets on the process", check duration: 0ms, status: 0/2 DOWN.===
-
This is the only reference (trying to paste an URL again): https://discourse.haproxy.org/t/freeze-sockets-in-1-8-1-no-http-2/1912
-
Anyone? Anything? This seems completely random and I have no clue where to even start debugging this.
-
Would this be a solution or just moving the issue into the future?
https://forums.freebsd.org/threads/cannot-increase-file-descriptor-limits.67013/
Can I just add these kern.maxfiles and kern.maxfilesperproc statements to system tunables without restarting?
-
sysctl kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 521187
kern.maxfilesperproc: 469062These seem like quite high values already for an internal HAProxy. Without any knowledge on how these are used in any context of course. I'm not a kernel-person.