Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    swap usage shows 100%

    General pfSense Questions
    6
    23
    782
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      posix
      last edited by posix

      Hello,

      I just noticed today my SWAP USAGE graphic shows 100%

      Below is my setup and information:

      Netgate 5100
      
      24.11-RELEASE (amd64)
      built on Wed Nov 27 13:22:00 EST 2024
      FreeBSD 15.0-CURRENT
      
      Memory usage	
      21% of 16287 MiB
      SWAP usage	
      100% of 372 MiB
      

      Current packages:

      acme	 				0.9_1	  
      arpwatch	 			0.2.3	 
      Avahi	 	                        2.2_6	  
      Cron	 				0.3.8_5	 
      darkstat	 			3.1.4	 
      lldpd	 				0.9.11_3	  
      pfBlockerNG-devel	 	        3.2.1_20	  
      snort	 				4.1.6_22	  
      Status_Traffic_Totals	                2.3.2_7	 
      System_Patches	 		        2.2.20_1
      
      [/root: top -n -o res
      last pid: 17785;  load averages:    0.66,    1.02,    0.72  up 42+23:25:55    11:59:31
      92 processes:  1 running, 91 sleeping
      CPU:  0.3% user,  0.6% nice,  0.6% system,  0.0% interrupt, 98.5% idle
      Mem: 1330M Active, 10G Inact, 1140M Laundry, 1903M Wired, 1410M Buf, 807M Free
      Swap: 373M Total, 373M Used, 300K Free, 99% Inuse
      
        PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
      89646 root          2  20    0   841M   717M bpf      1   0:00   0.00% snort
      81560 root          2  20    0   697M   574M bpf      2   0:05   0.00% snort
      30883 root          1  58   20   849M   501M urdlck   1   5:53   0.00% snort
      29178 root          1  60   20   753M   396M urdlck   0  23:52   0.00% snort
      13971 root          1  20    0   320M   257M select   3  11:55   0.00% bsnmpd
      93985 unbound       4  20    0   284M   215M kqread   0   0:31   0.00% unbound
      69369 root          1  52   20   159M    94M urdlck   2   0:10   0.00% snort
      90761 root          2  20    0   123M    72M bpf      1   0:00   0.00% snort
      14847 root          1  20    0    91M    52M piperd   3   1:10   0.00% php_pfb
      38346 root          1  68    0   142M    43M accept   0   0:05   0.00% php-fpm
      11883 root          1  31    0   142M    43M accept   1   0:03   0.00% php-fpm
       7116 root          1  68    0   142M    43M accept   2   0:05   0.00% php-fpm
      24106 root          1  68    0   142M    42M accept   1   0:03   0.00% php-fpm
      82301 root          1  68    0   142M    42M accept   3   0:04   0.00% php-fpm
      10822 root          1  68    0   113M    36M accept   3   0:01   0.00% php-fpm
      17728 root          1  20    0    45M    35M bpf      1   0:25   0.00% arpwatch
      18125 root          1  20    0    45M    35M bpf      2   0:26   0.00% arpwatch
      17177 root          1  20    0    45M    35M bpf      3   0:26   0.00% arpwatch]
      

      Researching some posts and checking memory it shows SNORT, my configuation should* be basic:

      HOME_VLAN (igb1.10)	    AC-BNFA	DISABLED	Home vlan interface	  
      IOT_VLAN (igb1.30)	    AC-BNFA	DISABLED	IoT vlan interface	  
      MGMT (igb1.100)	    	    AC-BNFA	DISABLED	LAN interface
      

      Updates Tab/Installed Rule sets:

      Snort Subscriber Ruleset	MD5_HAS removed	Tuesday, 04-Mar-25 12:32:41 EST
      Snort GPLv2 Community Rules	Not Enabled	Not Enabled
      Emerging Threats Open Rules	Not Enabled	Not Enabled
      Snort OpenAppID Detectors	MD5_HAS removed	Thursday, 28-Nov-24 19:42:42 EST
      Snort AppID Open Text Rules	MD5_HAS removed	Thursday, 28-Nov-24 19:42:42 EST
      
      bmeeksB 1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Check the Status > Monitoring graphs for previous memory usage. Was it ever close to 100%?

        M P 3 Replies Last reply Reply Quote 0
        • M
          michmoor LAYER 8 Rebel Alliance @stephenw10
          last edited by

          @stephenw10

          Running into the same issue. The IPS packages take up swap for some reason.

          26f53b55-2fe4-4634-a6a4-5900f47749ba-image.png

           top -n -o res
          last pid: 35679;  load averages:    1.17,    1.31,    1.00  up 19+10:22:01    18:03:00
          122 processes: 3 running, 119 sleeping
          CPU:  4.5% user,  1.3% nice,  2.1% system,  0.2% interrupt, 91.9% idle
          Mem: 971M Active, 2698M Inact, 833M Laundry, 1572M Wired, 56K Buf, 1713M Free
          ARC: 341M Total, 168M MFU, 152M MRU, 1613K Anon, 2712K Header, 16M Other
               273M Compressed, 790M Uncompressed, 2.89:1 Ratio
          Swap: 1024M Total, 597M Used, 426M Free, 58% Inuse
          
            PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
          61670 root         12  20    0  4166M  2951M nanslp   3  26.8H   3.56% suricata
          53138 root         12  20    0  1064M   491M nanslp   3 464:38   0.10% suricata
          34063 unbound       4  20    0   167M   132M kqread   1  14:38   0.00% unbound
          59476 root         17  48    0  5659M   112M uwait    0 171:29   1.46% telegraf
          98055 root         14  60    0  1517M    87M uwait    3  84:44   0.29% tailscaled
          14741 root         28  68    0  1388M    86M uwait    1  16.8H  10.25% pfnet-controller
          78215 root          1  68    0   143M    53M accept   1   0:03   0.00% php-fpm
          82015 root          1  68    0   130M    45M accept   1   0:03   0.00% php-fpm
          31737 root          1  58    0   120M    45M CPU2     2   0:00   0.00% php-cgi
          72711 root          1  68    0   159M    45M accept   0   0:02   0.00% php-fpm
          88350 root          1  68    0   155M    45M accept   1   0:02   0.00% php-fpm
          20810 root          1  68    0   142M    43M accept   2   0:03   0.00% php-fpm
          43359 root          1  68    0   113M    43M accept   1   0:02   0.00% php-fpm
          32399 root          1  68    0   113M    43M accept   1   0:02   0.00% php-fpm
          31411 root          1  20    0    48M    34M bpf      1   0:36   0.00% arpwatch
          33286 root          1  20    0    48M    34M bpf      2   0:36   0.00% arpwatch
          33707 root          1  20    0    48M    34M bpf      1   0:36   0.00% arpwatch
          31622 root          1  20    0    48M    34M bpf      3   0:38   0.00% arpwatch
          
          

          Firewall: NetGate,Palo Alto-VM,Juniper SRX
          Routing: Juniper, Arista, Cisco
          Switching: Juniper, Arista, Cisco
          Wireless: Unifi, Aruba IAP
          JNCIP,CCNP Enterprise

          bmeeksB GertjanG 2 Replies Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            How do you have suricata configured? Is it using that when it updates it's signatures?

            M 1 Reply Last reply Reply Quote 0
            • M
              michmoor LAYER 8 Rebel Alliance @stephenw10
              last edited by

              @stephenw10

              Configured pretty standard I would think. Certain rules are enabled. Alerting only. Updates are on a scheduled. So far since it’s been installed it’s been no fuss.

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                So what happened at that first spike in the graph, ~07:00? Did Suricata update?

                M 1 Reply Last reply Reply Quote 0
                • M
                  michmoor LAYER 8 Rebel Alliance @stephenw10
                  last edited by

                  @stephenw10 you mean the “Laundry” line?
                  Updates take place at 8am with an update interval of 4 days.

                  Graph time doesn’t seem to correlate

                  Firewall: NetGate,Palo Alto-VM,Juniper SRX
                  Routing: Juniper, Arista, Cisco
                  Switching: Juniper, Arista, Cisco
                  Wireless: Unifi, Aruba IAP
                  JNCIP,CCNP Enterprise

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks @michmoor
                    last edited by bmeeks

                    @michmoor said in swap usage shows 100%:

                    Running into the same issue. The IPS packages take up swap for some reason.

                    Allocation of swap is an operating system decision. It is not under the control of the package. All the package binary would do is request an allocation of memory. The operating system (FreeBSD in this case) decides how to fulfill the request. If there is not enough physical RAM present, the OS may decide to move certain inactive processes out to swap and reallocate that RAM to the new request. Then, when the requesting process sleeps and it's time for the swapped out process to be awakened, the data is moved back to active RAM. This is why using swap is not desired as it slows things down considerably.

                    Memory usage for the IDS/IPS packages increases during rule updates. Ditto for a package such as pfBlockerNG. If both collide on an update, it is certainly possible to stress the system and cause swap to become necessary.

                    1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks @posix
                      last edited by

                      @posix said in swap usage shows 100%:

                      Updates Tab/Installed Rule sets:

                      Snort Subscriber Ruleset MD5_HAS removed Tuesday, 04-Mar-25 12:32:41 EST
                      Snort GPLv2 Community Rules Not Enabled Not Enabled
                      Emerging Threats Open Rules Not Enabled Not Enabled
                      Snort OpenAppID Detectors MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST
                      Snort AppID Open Text Rules MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST

                      Why did you obfuscate the MD5 hashes of the rules archive? Those are public. It simply shows the calculated MD5 hash of the gzip archive you downloaded from the AWS CDN node. There is absolutely no private information disclosed in that hash.

                      P 1 Reply Last reply Reply Quote 0
                      • GertjanG
                        Gertjan @michmoor
                        last edited by

                        @michmoor said in swap usage shows 100%:

                        The IPS packages take up ...

                        nothing when you install them.
                        Some memory when you activated them, if there are any 'default' (suggested) rules or conditions or lists.
                        Then the admin start to add a 'max' of other conditions, rules etc ....
                        At this precise moment, the admin has a new job : monitoring the system resources very frequently.
                        As IPS, also pfBlockerng, will take what it needs to do the job.
                        Suddenly, "4 Gbytes of RAM" becomes a very limited resource. 16 or 32 Gbytes of disk space is even worse.

                        In your case, the packages took what RAM was avaible. real RAM ran out, the system starts to use the spare 'virtual' RAM : starts to use the extremely slow swap space, and when that's filled up also, it, the OS, will start to kill randomly (the biggest resource users first) processes.
                        Things will go downhill fast from this point.
                        Using swap on a desktop PC, phone, or any other end user deice is probably more ok, but not for routers and that kind of equipment.

                        IPS tools are like toddlers : you keep an eye om them, even when they sleep.

                        So, choices to make :
                        Start doing some (more) serious resource management.
                        and/or
                        Use less 'IPS' (and other) functionalities.

                        Btw : normally, it's not the RAM that goes low, but its the disk space filled up to 100 % as massive log files are created, and that will trigger a kernel panic for sure, with a chance that it can't reboot anymore.

                        No "help me" PM's please. Use the forum, the community will thank you.
                        Edit : and where are the logs ??

                        1 Reply Last reply Reply Quote 1
                        • P
                          posix @stephenw10
                          last edited by

                          This post is deleted!
                          1 Reply Last reply Reply Quote 0
                          • P
                            posix @bmeeks
                            last edited by posix

                            Hello @bmeeks
                            Edited message to show MD5 hash:

                            Rule Set Name/Publisher	MD5 Signature Hash	MD5 Signature Date
                            Snort Subscriber Ruleset	dbc140f3f6c01f0cff5e53d63e462df7	Thursday, 06-Mar-25 12:32:47 EST
                            Snort GPLv2 Community Rules	Not Enabled	Not Enabled
                            Emerging Threats Open Rules	Not Enabled	Not Enabled
                            Snort OpenAppID Detectors	c726cf937d84c651a20f2ac7c528384e	Thursday, 28-Nov-24 19:42:42 EST
                            Snort AppID Open Text Rules	2c26cb4f6a3bc03ab9c8e02befcf6fe1	Thursday, 28-Nov-24 19:42:42 EST
                            Feodo Tracker Botnet C2 IP Rules	Not Enabled	Not Enabled
                            
                            1 Reply Last reply Reply Quote 0
                            • P
                              posix @stephenw10
                              last edited by

                              Hello @stephenw10

                              I updated the graph

                              Screenshot 2025-03-07 at 3.49.07 PM.png

                              1 Reply Last reply Reply Quote 0
                              • P
                                posix
                                last edited by

                                Don't know if this helps since it was mentioned about disk space:

                                /root: df -h
                                Filesystem                     Size    Used   Avail Capacity  Mounted on
                                /dev/ufsid/61bf5ededf06b426    227G     70G    138G    34%    /
                                devfs                          1.0K      0B    1.0K     0%    /dev
                                tmpfs                          4.0M    204K    3.8M     5%    /var/run
                                /lib                           227G     70G    138G    34%    /var/unbound/lib
                                devfs                          1.0K      0B    1.0K     0%    /var/unbound/dev
                                /var/log/pfblockerng           227G     70G    138G    34%    /var/unbound/var/log/pfblockerng
                                /usr/local/share/GeoIP         227G     70G    138G    34%    /var/unbound/usr/local/share/GeoIP
                                /usr/local/bin                 227G     70G    138G    34%    /var/unbound/usr/local/bin
                                /usr/local/lib                 227G     70G    138G    34%    /var/unbound/usr/local/lib
                                devfs                          1.0K      0B    1.0K     0%    /var/dhcpd/dev
                                
                                GertjanG 1 Reply Last reply Reply Quote 0
                                • GertjanG
                                  Gertjan @posix
                                  last edited by

                                  Just checking :

                                  @posix said in swap usage shows 100%:

                                  /var/log/pfblockerng 227G 70G 138G 34% /var/unbound/var/log/pfblockerng

                                  I use the default pfBlockerng log file 'size' setup :

                                  06d507f9-45c9-482b-9ae2-2626bc4b2736-image.png

                                  and right now that about 110 Mbytes for me:

                                  18c92cb1-0117-4e0c-94a1-8b37c4cc2c46-image.png

                                  You : really 70 Gbytes ?

                                  pfBlockerng use these files to generate these pages on demand :

                                  5361f144-cc3b-4bcb-83a5-7529ff90b414-image.png

                                  If your files are that big, it would take ages to do that .... nginx/PHP will probably time out.

                                  No "help me" PM's please. Use the forum, the community will thank you.
                                  Edit : and where are the logs ??

                                  P 2 Replies Last reply Reply Quote 0
                                  • P
                                    posix @Gertjan
                                    last edited by

                                    This post is deleted!
                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      posix @Gertjan
                                      last edited by

                                      @Gertjan

                                      Edit I see the pfblockerlogs increasing:

                                      Thanks for the response, the log size I have configured are the same

                                      Screenshot 2025-03-10 4.14.32 PM.png

                                      /var/log/pfblockerng: ls -l
                                      total 20376
                                      -rw-------  1 unbound unbound 2012634 Mar 10 16:16 dns_reply.log
                                      -rw-------  1 unbound unbound 2474917 Mar 10 16:16 dnsbl.log
                                      -rw-------  1 root    wheel   5258413 Mar 10 16:00 dnsbl_parsed_error.log
                                      -rw-------  1 root    wheel    426448 Mar 10 16:00 error.log
                                      -rw-------  1 root    wheel    274484 Mar 10 16:00 extras.log
                                      -rw-------  1 root    wheel   3481641 Mar 10 16:11 ip_block.log
                                      -rw-------  1 root    wheel   3649640 Mar 10 16:00 ip_permit.log
                                      -rw-r--r--  1 root    unbound     120 Mar 10 12:00 maxmind_ver
                                      -rw-------  1 root    wheel    624287 Mar 10 16:00 pfblockerng.log
                                      -rw-r--r--  1 unbound unbound     229 Feb 21  2023 py_error.log
                                      -rw-------  1 unbound unbound 2208509 Mar 10 16:16 unified.log
                                      

                                      second capture:

                                      -rw-------  1 unbound unbound 2054793 Mar 10 16:23 dns_reply.log
                                      -rw-------  1 unbound unbound 2484058 Mar 10 16:22 dnsbl.log
                                      -rw-------  1 root    wheel   5258413 Mar 10 16:00 dnsbl_parsed_error.log
                                      -rw-------  1 root    wheel    426448 Mar 10 16:00 error.log
                                      -rw-------  1 root    wheel    274484 Mar 10 16:00 extras.log
                                      -rw-------  1 root    wheel   3483020 Mar 10 16:18 ip_block.log
                                      -rw-------  1 root    wheel   3649640 Mar 10 16:00 ip_permit.log
                                      -rw-r--r--  1 root    unbound     120 Mar 10 12:00 maxmind_ver
                                      -rw-------  1 root    wheel    624287 Mar 10 16:00 pfblockerng.log
                                      -rw-r--r--  1 unbound unbound     229 Feb 21  2023 py_error.log
                                      -rw-------  1 unbound unbound 2260172 Mar 10 16:23 unified.log
                                      

                                      right now up at 75G

                                      /var/log/pfblockerng: df -h
                                      Filesystem                     Size    Used   Avail Capacity  Mounted on
                                      /dev/ufsid/61bf5ededf06b426    227G     75G    133G    36%    /
                                      devfs                          1.0K      0B    1.0K     0%    /dev
                                      tmpfs                          4.0M    204K    3.8M     5%    /var/run
                                      /lib                           227G     75G    133G    36%    /var/unbound/lib
                                      devfs                          1.0K      0B    1.0K     0%    /var/unbound/dev
                                      /var/log/pfblockerng           227G     75G    133G    36%    /var/unbound/var/log/pfblockerng
                                      /usr/local/share/GeoIP         227G     75G    133G    36%    /var/unbound/usr/local/share/GeoIP
                                      /usr/local/bin                 227G     75G    133G    36%    /var/unbound/usr/local/bin
                                      /usr/local/lib                 227G     75G    133G    36%    /var/unbound/usr/local/lib
                                      devfs                          1.0K      0B    1.0K     0%    /var/dhcpd/dev
                                      
                                      P 1 Reply Last reply Reply Quote 0
                                      • P
                                        posix @posix
                                        last edited by

                                        Hello, can anyone provide further guidance?

                                        1 Reply Last reply Reply Quote 0
                                        • P
                                          posix
                                          last edited by

                                          @bmeeks @michmoor @stephenw10

                                          Sorry to blast, but looking for further guidance and next steps.

                                          S 1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            I assume it resets if you reboot?

                                            How long does it take to refill?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.