High swap usage
-
Hi there,
I have several PFsense firewalls running and have the following issue:
Some of those firewalls are having an high swap usage. How to find out what is using the swap? The memory usage isn't high at all, around ~60 - 70%. I can restart the whole firewall, but eventually the swap will increase again. I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.I try to take a look at the /proc folders, grep on 'swap', but nothing is there.
I'm running 2.1.5 on those firewalls.
Kind regards,
Werner
-
run```
top -o sizeit'll tell you what processes are using up your memory
-
Which packages are you running? Vanilla pfSense without packages shouldn't use that much memory.
-
Hi,
Thank you for making time to answer me.
Had to post this information in the original post: this swap usage happens after an certain amount of time (Weeks or months), not immediately or directly after an reboot.I did an top -o size:
last pid: 50177; load averages: 0.06, 0.07, 0.02 up 107+00:49:14 13:38:33 57 processes: 3 running, 54 sleeping CPU: % user, % nice, % system, % interrupt, % idle Mem: 60M Active, 18M Inact, 104M Wired, 208K Cache, 57M Buf, 790M Free Swap: 512M Total, 45M Used, 467M Free, 8% Inuse PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 13907 root 1 76 0 150M 24792K accept 0:10 0.00% php 2424 root 1 44 0 146M 19572K accept 0:05 0.00% php 47382 root 1 76 0 144M 10520K wait 0:00 0.00% php 43189 root 1 44 0 144M 10436K wait 0:00 0.00% php 59379 root 1 44 0 33540K 11064K select 0:12 0.00% bsnmpd 42778 root 1 44 0 28328K 5680K kqread 1:57 0.00% lighttpd 48670 root 1 44 0 26164K 4168K RUN 0:00 0.00% sshd 66673 www 1 44 0 21816K 9252K kqread 0:06 0.00% haproxy 11817 root 1 76 0 19480K 1300K wait 0:00 0.00% login 96315 nobody 1 44 0 16040K 4672K select 0:00 0.00% dnsmasq 9441 root 1 44 0 15264K 2064K select 0:00 0.00% sshd 10624 root 1 44 0 14392K 2240K piperd 0:00 0.00% rrdtool 21435 root 1 44 0 12160K 7120K select 0:01 0.00% ntpd 67230 root 1 44 0 11744K 3852K bpf 0:25 0.00% tcpdump 50177 root 1 44 0 9324K 2084K RUN 0:00 0.00% top 32229 root 1 44 0 8984K 1344K select 0:00 0.00% inetdI can see the php processes, but are they using the swap?
I only have the following services running:
-
apinger
-
bsnmpd
-
dnsmasq
-
haproxy
-
ntpd
Restarting HAproxy did help a lot. From 80% Swap usage, it went to ~ 10% now. Restarting the rest of the services didn't help much.
Kind regards,
Werner
-
-
so now you know … it's haproxy :)
-
so now you know … it's haproxy :)
True 8)
But there isn't an check to see what the other 10% is being used by?
-
Hi guy's,
I would be interested to know if when this happens again, and you restart any other service if that would help as well..?As haproxy only seems to take 22 MB of memory, it seems illogical that it would account for 350 MB swap usage..
p.s. how much connections do you allow in the 'settings' tab of haproxy? If thats like 10.000 then that might explain it.
-
Hi Piba,
I checked, but the settings is on "1000' connections, not really much…
-
Late but perhaps not to late,
I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.
And what would be if you insert in each box 4 GB or 8 GB RAM?
This would be sufficient enough as I see it right.
-
might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
That would simply increase the "790M Free" counter higher wouldn't it.?
-
I was able to Google that HAProxy can use a substantial amount of memory if you have a lot of connections. Say a burst of connections come in, causing HAProxy to swap, but a large number of those connections are idle TCP or are otherwise not properly terminated. Then the memory containing their state will be swapped out, but very rarely referenced. These connections could effectively live in swap. Then real memory frees up, but the pages that were swapped out will not swap back in until referenced. This could leave you with free memory and high swap usage.
I would watch for spikes in VM usage.
-
Hi Harvy,
What would be considered a high number of connections in those online results?(do you have a link?) From what is calculated on the haproxy settings example for active connections it could be 1.000 = 48 MB and 10.000 = 488 MB.. However the poster only allows 1000 connections. So in my opinion this does not account for the swapped out memory. I have not actually run a load test to check those example calculations..
-
might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
That would simply increase the "790M Free" counter higher wouldn't it.?If this is a connection based problem, so what you want to do? Shorten the connections? And then
nobody is able to connect anymore? But with 8 GB RAM the connections count can be as it is and
you have enough free RAM for the system in spare.
-
I just googled HAProxy and swap usage. Not to say that this is the issue, but closing HAProxy did result in a substantial reduction in swap. I would still be curious to what happens if you increase the memory on your systems. That would be the easiest fix to test for, in case my idea of a spike of Virtual Memory is causing this issues.
-
Not sure if my google foo is the same as yours, but the first two results google shows me are: maxconn 300000 and maxconn 262144, which would be in a completely different ballpark than maxconn 1000..
-
Sounds about right. While the connection numbers don't match up, HAProxy did free up swap once you closed it and swap is handled by the OS. To me that means the OS ran low on physical memory at some point in time, paged out data to swap, but then never paged the data back in. The only reason it would not page back in is because the data has not been referenced since.
Anyway, still sounds like there was a lack of memory at some point, even if it only lasted for a brief moment.