High swap usage



  • Hi there,

    I have several PFsense firewalls running and have the following issue:
    Some of those firewalls are having an high swap usage. How to find out what is using the swap? The memory usage isn't high at all, around ~60 - 70%. I can restart the whole firewall, but eventually the swap will increase again. I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.

    I try to take a look at the /proc folders, grep on 'swap', but nothing is there.

    I'm running 2.1.5 on those firewalls.

    Kind regards,
    Werner



  • run```
    top -o size

    it'll tell you what processes are using up your memory


  • Which packages are you running? Vanilla pfSense without packages shouldn't use that much memory.



  • Hi,

    Thank you for making time to answer me.
    Had to post this information in the original post: this swap usage happens after an certain amount of time (Weeks or months), not immediately or directly after an reboot.

    I did an top -o size:

    last pid: 50177;  load averages:  0.06,  0.07,  0.02                                                                                up 107+00:49:14 13:38:33
    57 processes:  3 running, 54 sleeping
    CPU:     % user,     % nice,     % system,     % interrupt,     % idle
    Mem: 60M Active, 18M Inact, 104M Wired, 208K Cache, 57M Buf, 790M Free
    Swap: 512M Total, 45M Used, 467M Free, 8% Inuse
    
      PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    13907 root        1  76    0   150M 24792K accept   0:10  0.00% php
     2424 root        1  44    0   146M 19572K accept   0:05  0.00% php
    47382 root        1  76    0   144M 10520K wait     0:00  0.00% php
    43189 root        1  44    0   144M 10436K wait     0:00  0.00% php
    59379 root        1  44    0 33540K 11064K select   0:12  0.00% bsnmpd
    42778 root        1  44    0 28328K  5680K kqread   1:57  0.00% lighttpd
    48670 root        1  44    0 26164K  4168K RUN      0:00  0.00% sshd
    66673 www         1  44    0 21816K  9252K kqread   0:06  0.00% haproxy
    11817 root        1  76    0 19480K  1300K wait     0:00  0.00% login
    96315 nobody      1  44    0 16040K  4672K select   0:00  0.00% dnsmasq
     9441 root        1  44    0 15264K  2064K select   0:00  0.00% sshd
    10624 root        1  44    0 14392K  2240K piperd   0:00  0.00% rrdtool
    21435 root        1  44    0 12160K  7120K select   0:01  0.00% ntpd
    67230 root        1  44    0 11744K  3852K bpf      0:25  0.00% tcpdump
    50177 root        1  44    0  9324K  2084K RUN      0:00  0.00% top
    32229 root        1  44    0  8984K  1344K select   0:00  0.00% inetd
    

    I can see the php processes, but are they using the swap?

    I only have the following services running:

    • apinger

    • bsnmpd

    • dnsmasq

    • haproxy

    • ntpd

    Restarting HAproxy did help a lot. From 80% Swap usage, it went to ~ 10% now. Restarting the rest of the services didn't help much.

    Kind regards,
    Werner



  • so now you know … it's haproxy :)



  • @heper:

    so now you know … it's haproxy :)

    True  8)
    But there isn't an check to see what the other 10% is being used by?



  • Hi guy's,
    I would be interested to know if when this happens again, and you restart any other service if that would help as well..?

    As haproxy only seems to take 22 MB of memory, it seems illogical that it would account for 350 MB swap usage..
    p.s. how much connections do you allow in the 'settings' tab of haproxy? If thats like 10.000 then that might explain it.



  • Hi Piba,

    I checked, but the settings is on "1000' connections, not really much…



  • Late but perhaps not to late,

    I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.

    And what would be if you insert in each box 4 GB or 8 GB RAM?
    This would be sufficient enough as I see it right.



  • might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
    That would simply increase the "790M Free" counter higher wouldn't it.?



  • I was able to Google that HAProxy can use a substantial amount of memory if you have a lot of connections. Say a burst of connections come in, causing HAProxy to swap, but a large number of those connections are idle TCP or are otherwise not properly terminated. Then the memory containing their state will be swapped out, but very rarely referenced. These connections could effectively live in swap. Then real memory frees up, but the pages that were swapped out will not swap back in until referenced. This could leave you with free memory and high swap usage.

    I would watch for spikes in VM usage.



  • Hi Harvy,
    What would be considered a high number of connections in those online results?(do you have a link?) From what is calculated on the haproxy settings example for active connections it could be 1.000 = 48 MB and 10.000 = 488 MB.. However the poster only allows 1000 connections. So in my opinion this does not account for the swapped out memory. I have not actually run a load test to check those example calculations..



  • @PiBa:

    might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
    That would simply increase the "790M Free" counter higher wouldn't it.?

    If this is a connection based problem, so what you want to do? Shorten the connections? And then
    nobody is able to connect anymore? But with 8 GB RAM the connections count can be as it is and
    you have enough free RAM for the system in spare.



  • I just googled HAProxy and swap usage. Not to say that this is the issue, but closing HAProxy did result in a substantial reduction in swap. I would still be curious to what happens if you increase the memory on your systems. That would be the easiest fix to test for, in case my idea of a spike of Virtual Memory is causing this issues.



  • Not sure if my google foo is the same as yours, but the first two results google shows me are: maxconn 300000 and maxconn 262144, which would be in a completely different ballpark than maxconn 1000..



  • Sounds about right. While the connection numbers don't match up, HAProxy did free up swap once you closed it and swap is handled by the OS. To me that means the OS ran low on physical memory at some point in time, paged out data to swap, but then never paged the data back in. The only reason it would not page back in is because the data has not been referenced since.

    Anyway, still sounds like there was a lack of memory at some point, even if it only lasted for a brief moment.


Log in to reply