Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    High swap usage

    Scheduled Pinned Locked Moved General pfSense Questions
    16 Posts 6 Posters 17.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      wdijkerman
      last edited by

      Hi there,

      I have several PFsense firewalls running and have the following issue:
      Some of those firewalls are having an high swap usage. How to find out what is using the swap? The memory usage isn't high at all, around ~60 - 70%. I can restart the whole firewall, but eventually the swap will increase again. I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.

      I try to take a look at the /proc folders, grep on 'swap', but nothing is there.

      I'm running 2.1.5 on those firewalls.

      Kind regards,
      Werner

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        run```
        top -o size

        it'll tell you what processes are using up your memory
        1 Reply Last reply Reply Quote 0
        • F
          fragged
          last edited by

          Which packages are you running? Vanilla pfSense without packages shouldn't use that much memory.

          1 Reply Last reply Reply Quote 0
          • W
            wdijkerman
            last edited by

            Hi,

            Thank you for making time to answer me.
            Had to post this information in the original post: this swap usage happens after an certain amount of time (Weeks or months), not immediately or directly after an reboot.

            I did an top -o size:

            last pid: 50177;  load averages:  0.06,  0.07,  0.02                                                                                up 107+00:49:14 13:38:33
            57 processes:  3 running, 54 sleeping
            CPU:     % user,     % nice,     % system,     % interrupt,     % idle
            Mem: 60M Active, 18M Inact, 104M Wired, 208K Cache, 57M Buf, 790M Free
            Swap: 512M Total, 45M Used, 467M Free, 8% Inuse
            
              PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
            13907 root        1  76    0   150M 24792K accept   0:10  0.00% php
             2424 root        1  44    0   146M 19572K accept   0:05  0.00% php
            47382 root        1  76    0   144M 10520K wait     0:00  0.00% php
            43189 root        1  44    0   144M 10436K wait     0:00  0.00% php
            59379 root        1  44    0 33540K 11064K select   0:12  0.00% bsnmpd
            42778 root        1  44    0 28328K  5680K kqread   1:57  0.00% lighttpd
            48670 root        1  44    0 26164K  4168K RUN      0:00  0.00% sshd
            66673 www         1  44    0 21816K  9252K kqread   0:06  0.00% haproxy
            11817 root        1  76    0 19480K  1300K wait     0:00  0.00% login
            96315 nobody      1  44    0 16040K  4672K select   0:00  0.00% dnsmasq
             9441 root        1  44    0 15264K  2064K select   0:00  0.00% sshd
            10624 root        1  44    0 14392K  2240K piperd   0:00  0.00% rrdtool
            21435 root        1  44    0 12160K  7120K select   0:01  0.00% ntpd
            67230 root        1  44    0 11744K  3852K bpf      0:25  0.00% tcpdump
            50177 root        1  44    0  9324K  2084K RUN      0:00  0.00% top
            32229 root        1  44    0  8984K  1344K select   0:00  0.00% inetd
            

            I can see the php processes, but are they using the swap?

            I only have the following services running:

            • apinger

            • bsnmpd

            • dnsmasq

            • haproxy

            • ntpd

            Restarting HAproxy did help a lot. From 80% Swap usage, it went to ~ 10% now. Restarting the rest of the services didn't help much.

            Kind regards,
            Werner

            1 Reply Last reply Reply Quote 0
            • H
              heper
              last edited by

              so now you know … it's haproxy :)

              1 Reply Last reply Reply Quote 0
              • W
                wdijkerman
                last edited by

                @heper:

                so now you know … it's haproxy :)

                True  8)
                But there isn't an check to see what the other 10% is being used by?

                1 Reply Last reply Reply Quote 0
                • P
                  PiBa
                  last edited by

                  Hi guy's,
                  I would be interested to know if when this happens again, and you restart any other service if that would help as well..?

                  As haproxy only seems to take 22 MB of memory, it seems illogical that it would account for 350 MB swap usage..
                  p.s. how much connections do you allow in the 'settings' tab of haproxy? If thats like 10.000 then that might explain it.

                  1 Reply Last reply Reply Quote 0
                  • W
                    wdijkerman
                    last edited by

                    Hi Piba,

                    I checked, but the settings is on "1000' connections, not really much…

                    1 Reply Last reply Reply Quote 0
                    • ?
                      Guest
                      last edited by

                      Late but perhaps not to late,

                      I have boxes with 1GB of ram and 2GB of ram, but both have the swap issues.

                      And what would be if you insert in each box 4 GB or 8 GB RAM?
                      This would be sufficient enough as I see it right.

                      1 Reply Last reply Reply Quote 0
                      • P
                        PiBa
                        last edited by

                        might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
                        That would simply increase the "790M Free" counter higher wouldn't it.?

                        1 Reply Last reply Reply Quote 0
                        • H
                          Harvy66
                          last edited by

                          I was able to Google that HAProxy can use a substantial amount of memory if you have a lot of connections. Say a burst of connections come in, causing HAProxy to swap, but a large number of those connections are idle TCP or are otherwise not properly terminated. Then the memory containing their state will be swapped out, but very rarely referenced. These connections could effectively live in swap. Then real memory frees up, but the pages that were swapped out will not swap back in until referenced. This could leave you with free memory and high swap usage.

                          I would watch for spikes in VM usage.

                          1 Reply Last reply Reply Quote 0
                          • P
                            PiBa
                            last edited by

                            Hi Harvy,
                            What would be considered a high number of connections in those online results?(do you have a link?) From what is calculated on the haproxy settings example for active connections it could be 1.000 = 48 MB and 10.000 = 488 MB.. However the poster only allows 1000 connections. So in my opinion this does not account for the swapped out memory. I have not actually run a load test to check those example calculations..

                            1 Reply Last reply Reply Quote 0
                            • ?
                              Guest
                              last edited by

                              @PiBa:

                              might as well put 256 GB memory in.. :o if we aren't going to try and find a explanation for the usage..
                              That would simply increase the "790M Free" counter higher wouldn't it.?

                              If this is a connection based problem, so what you want to do? Shorten the connections? And then
                              nobody is able to connect anymore? But with 8 GB RAM the connections count can be as it is and
                              you have enough free RAM for the system in spare.

                              1 Reply Last reply Reply Quote 0
                              • H
                                Harvy66
                                last edited by

                                I just googled HAProxy and swap usage. Not to say that this is the issue, but closing HAProxy did result in a substantial reduction in swap. I would still be curious to what happens if you increase the memory on your systems. That would be the easiest fix to test for, in case my idea of a spike of Virtual Memory is causing this issues.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  PiBa
                                  last edited by

                                  Not sure if my google foo is the same as yours, but the first two results google shows me are: maxconn 300000 and maxconn 262144, which would be in a completely different ballpark than maxconn 1000..

                                  1 Reply Last reply Reply Quote 0
                                  • H
                                    Harvy66
                                    last edited by

                                    Sounds about right. While the connection numbers don't match up, HAProxy did free up swap once you closed it and swap is handled by the OS. To me that means the OS ran low on physical memory at some point in time, paged out data to swap, but then never paged the data back in. The only reason it would not page back in is because the data has not been referenced since.

                                    Anyway, still sounds like there was a lack of memory at some point, even if it only lasted for a brief moment.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.