High swap usage
-
Hi guy's,
I would be interested to know if when this happens again, and you restart any other service if that would help as well..?As haproxy only seems to take 22 MB of memory, it seems illogical that it would account for 350 MB swap usage..
p.s. how much connections do you allow in the 'settings' tab of haproxy? If thats like 10.000 then that might explain it. -
Hi Piba,
I checked, but the settings is on "1000' connections, not really much…
-
Late but perhaps not to late,
I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.
And what would be if you insert in each box 4 GB or 8 GB RAM?
This would be sufficient enough as I see it right. -
might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
That would simply increase the "790M Free" counter higher wouldn't it.? -
I was able to Google that HAProxy can use a substantial amount of memory if you have a lot of connections. Say a burst of connections come in, causing HAProxy to swap, but a large number of those connections are idle TCP or are otherwise not properly terminated. Then the memory containing their state will be swapped out, but very rarely referenced. These connections could effectively live in swap. Then real memory frees up, but the pages that were swapped out will not swap back in until referenced. This could leave you with free memory and high swap usage.
I would watch for spikes in VM usage.
-
Hi Harvy,
What would be considered a high number of connections in those online results?(do you have a link?) From what is calculated on the haproxy settings example for active connections it could be 1.000 = 48 MB and 10.000 = 488 MB.. However the poster only allows 1000 connections. So in my opinion this does not account for the swapped out memory. I have not actually run a load test to check those example calculations.. -
might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
That would simply increase the "790M Free" counter higher wouldn't it.?If this is a connection based problem, so what you want to do? Shorten the connections? And then
nobody is able to connect anymore? But with 8 GB RAM the connections count can be as it is and
you have enough free RAM for the system in spare. -
I just googled HAProxy and swap usage. Not to say that this is the issue, but closing HAProxy did result in a substantial reduction in swap. I would still be curious to what happens if you increase the memory on your systems. That would be the easiest fix to test for, in case my idea of a spike of Virtual Memory is causing this issues.
-
Not sure if my google foo is the same as yours, but the first two results google shows me are: maxconn 300000 and maxconn 262144, which would be in a completely different ballpark than maxconn 1000..
-
Sounds about right. While the connection numbers don't match up, HAProxy did free up swap once you closed it and swap is handled by the OS. To me that means the OS ran low on physical memory at some point in time, paged out data to swap, but then never paged the data back in. The only reason it would not page back in is because the data has not been referenced since.
Anyway, still sounds like there was a lack of memory at some point, even if it only lasted for a brief moment.