swap usage shows 100%

posix

Hello,

I just noticed today my SWAP USAGE graphic shows 100%

Below is my setup and information:

Netgate 5100

24.11-RELEASE (amd64)
built on Wed Nov 27 13:22:00 EST 2024
FreeBSD 15.0-CURRENT

Memory usage	
21% of 16287 MiB
SWAP usage	
100% of 372 MiB

Current packages:

acme	 				0.9_1	  
arpwatch	 			0.2.3	 
Avahi	 	                        2.2_6	  
Cron	 				0.3.8_5	 
darkstat	 			3.1.4	 
lldpd	 				0.9.11_3	  
pfBlockerNG-devel	 	        3.2.1_20	  
snort	 				4.1.6_22	  
Status_Traffic_Totals	                2.3.2_7	 
System_Patches	 		        2.2.20_1

[/root: top -n -o res
last pid: 17785;  load averages:    0.66,    1.02,    0.72  up 42+23:25:55    11:59:31
92 processes:  1 running, 91 sleeping
CPU:  0.3% user,  0.6% nice,  0.6% system,  0.0% interrupt, 98.5% idle
Mem: 1330M Active, 10G Inact, 1140M Laundry, 1903M Wired, 1410M Buf, 807M Free
Swap: 373M Total, 373M Used, 300K Free, 99% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
89646 root          2  20    0   841M   717M bpf      1   0:00   0.00% snort
81560 root          2  20    0   697M   574M bpf      2   0:05   0.00% snort
30883 root          1  58   20   849M   501M urdlck   1   5:53   0.00% snort
29178 root          1  60   20   753M   396M urdlck   0  23:52   0.00% snort
13971 root          1  20    0   320M   257M select   3  11:55   0.00% bsnmpd
93985 unbound       4  20    0   284M   215M kqread   0   0:31   0.00% unbound
69369 root          1  52   20   159M    94M urdlck   2   0:10   0.00% snort
90761 root          2  20    0   123M    72M bpf      1   0:00   0.00% snort
14847 root          1  20    0    91M    52M piperd   3   1:10   0.00% php_pfb
38346 root          1  68    0   142M    43M accept   0   0:05   0.00% php-fpm
11883 root          1  31    0   142M    43M accept   1   0:03   0.00% php-fpm
 7116 root          1  68    0   142M    43M accept   2   0:05   0.00% php-fpm
24106 root          1  68    0   142M    42M accept   1   0:03   0.00% php-fpm
82301 root          1  68    0   142M    42M accept   3   0:04   0.00% php-fpm
10822 root          1  68    0   113M    36M accept   3   0:01   0.00% php-fpm
17728 root          1  20    0    45M    35M bpf      1   0:25   0.00% arpwatch
18125 root          1  20    0    45M    35M bpf      2   0:26   0.00% arpwatch
17177 root          1  20    0    45M    35M bpf      3   0:26   0.00% arpwatch]

Researching some posts and checking memory it shows SNORT, my configuation should* be basic:

HOME_VLAN (igb1.10)	    AC-BNFA	DISABLED	Home vlan interface	  
IOT_VLAN (igb1.30)	    AC-BNFA	DISABLED	IoT vlan interface	  
MGMT (igb1.100)	    	    AC-BNFA	DISABLED	LAN interface

Updates Tab/Installed Rule sets:

Snort Subscriber Ruleset	MD5_HAS removed	Tuesday, 04-Mar-25 12:32:41 EST
Snort GPLv2 Community Rules	Not Enabled	Not Enabled
Emerging Threats Open Rules	Not Enabled	Not Enabled
Snort OpenAppID Detectors	MD5_HAS removed	Thursday, 28-Nov-24 19:42:42 EST
Snort AppID Open Text Rules	MD5_HAS removed	Thursday, 28-Nov-24 19:42:42 EST

stephenw10

Check the Status > Monitoring graphs for previous memory usage. Was it ever close to 100%?

michmoor

@stephenw10

Running into the same issue. The IPS packages take up swap for some reason.

 top -n -o res
last pid: 35679;  load averages:    1.17,    1.31,    1.00  up 19+10:22:01    18:03:00
122 processes: 3 running, 119 sleeping
CPU:  4.5% user,  1.3% nice,  2.1% system,  0.2% interrupt, 91.9% idle
Mem: 971M Active, 2698M Inact, 833M Laundry, 1572M Wired, 56K Buf, 1713M Free
ARC: 341M Total, 168M MFU, 152M MRU, 1613K Anon, 2712K Header, 16M Other
     273M Compressed, 790M Uncompressed, 2.89:1 Ratio
Swap: 1024M Total, 597M Used, 426M Free, 58% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
61670 root         12  20    0  4166M  2951M nanslp   3  26.8H   3.56% suricata
53138 root         12  20    0  1064M   491M nanslp   3 464:38   0.10% suricata
34063 unbound       4  20    0   167M   132M kqread   1  14:38   0.00% unbound
59476 root         17  48    0  5659M   112M uwait    0 171:29   1.46% telegraf
98055 root         14  60    0  1517M    87M uwait    3  84:44   0.29% tailscaled
14741 root         28  68    0  1388M    86M uwait    1  16.8H  10.25% pfnet-controller
78215 root          1  68    0   143M    53M accept   1   0:03   0.00% php-fpm
82015 root          1  68    0   130M    45M accept   1   0:03   0.00% php-fpm
31737 root          1  58    0   120M    45M CPU2     2   0:00   0.00% php-cgi
72711 root          1  68    0   159M    45M accept   0   0:02   0.00% php-fpm
88350 root          1  68    0   155M    45M accept   1   0:02   0.00% php-fpm
20810 root          1  68    0   142M    43M accept   2   0:03   0.00% php-fpm
43359 root          1  68    0   113M    43M accept   1   0:02   0.00% php-fpm
32399 root          1  68    0   113M    43M accept   1   0:02   0.00% php-fpm
31411 root          1  20    0    48M    34M bpf      1   0:36   0.00% arpwatch
33286 root          1  20    0    48M    34M bpf      2   0:36   0.00% arpwatch
33707 root          1  20    0    48M    34M bpf      1   0:36   0.00% arpwatch
31622 root          1  20    0    48M    34M bpf      3   0:38   0.00% arpwatch

stephenw10

How do you have suricata configured? Is it using that when it updates it's signatures?

michmoor

@stephenw10

Configured pretty standard I would think. Certain rules are enabled. Alerting only. Updates are on a scheduled. So far since it’s been installed it’s been no fuss.

stephenw10

So what happened at that first spike in the graph, ~07:00? Did Suricata update?

michmoor

@stephenw10 you mean the “Laundry” line?
Updates take place at 8am with an update interval of 4 days.

Graph time doesn’t seem to correlate

bmeeks

@michmoor said in swap usage shows 100%:

Running into the same issue. The IPS packages take up swap for some reason.

Allocation of swap is an operating system decision. It is not under the control of the package. All the package binary would do is request an allocation of memory. The operating system (FreeBSD in this case) decides how to fulfill the request. If there is not enough physical RAM present, the OS may decide to move certain inactive processes out to swap and reallocate that RAM to the new request. Then, when the requesting process sleeps and it's time for the swapped out process to be awakened, the data is moved back to active RAM. This is why using swap is not desired as it slows things down considerably.

Memory usage for the IDS/IPS packages increases during rule updates. Ditto for a package such as pfBlockerNG. If both collide on an update, it is certainly possible to stress the system and cause swap to become necessary.

bmeeks

@posix said in swap usage shows 100%:

Updates Tab/Installed Rule sets:

Snort Subscriber Ruleset MD5_HAS removed Tuesday, 04-Mar-25 12:32:41 EST
Snort GPLv2 Community Rules Not Enabled Not Enabled
Emerging Threats Open Rules Not Enabled Not Enabled
Snort OpenAppID Detectors MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST
Snort AppID Open Text Rules MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST

Why did you obfuscate the MD5 hashes of the rules archive? Those are public. It simply shows the calculated MD5 hash of the gzip archive you downloaded from the AWS CDN node. There is absolutely no private information disclosed in that hash.

Gertjan

@michmoor said in swap usage shows 100%:

The IPS packages take up ...

nothing when you install them.
Some memory when you activated them, if there are any 'default' (suggested) rules or conditions or lists.
Then the admin start to add a 'max' of other conditions, rules etc ....
At this precise moment, the admin has a new job : monitoring the system resources very frequently.
As IPS, also pfBlockerng, will take what it needs to do the job.
Suddenly, "4 Gbytes of RAM" becomes a very limited resource. 16 or 32 Gbytes of disk space is even worse.

In your case, the packages took what RAM was avaible. real RAM ran out, the system starts to use the spare 'virtual' RAM : starts to use the extremely slow swap space, and when that's filled up also, it, the OS, will start to kill randomly (the biggest resource users first) processes.
Things will go downhill fast from this point.
Using swap on a desktop PC, phone, or any other end user deice is probably more ok, but not for routers and that kind of equipment.

IPS tools are like toddlers : you keep an eye om them, even when they sleep.

So, choices to make :
Start doing some (more) serious resource management.
and/or
Use less 'IPS' (and other) functionalities.

Btw : normally, it's not the RAM that goes low, but its the disk space filled up to 100 % as massive log files are created, and that will trigger a kernel panic for sure, with a chance that it can't reboot anymore.

posix

This post is deleted!

posix

Hello @bmeeks
Edited message to show MD5 hash:

Rule Set Name/Publisher	MD5 Signature Hash	MD5 Signature Date
Snort Subscriber Ruleset	dbc140f3f6c01f0cff5e53d63e462df7	Thursday, 06-Mar-25 12:32:47 EST
Snort GPLv2 Community Rules	Not Enabled	Not Enabled
Emerging Threats Open Rules	Not Enabled	Not Enabled
Snort OpenAppID Detectors	c726cf937d84c651a20f2ac7c528384e	Thursday, 28-Nov-24 19:42:42 EST
Snort AppID Open Text Rules	2c26cb4f6a3bc03ab9c8e02befcf6fe1	Thursday, 28-Nov-24 19:42:42 EST
Feodo Tracker Botnet C2 IP Rules	Not Enabled	Not Enabled

posix

Hello @stephenw10

I updated the graph

Screenshot 2025-03-07 at 3.49.07 PM.png

posix

Don't know if this helps since it was mentioned about disk space:

/root: df -h
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/61bf5ededf06b426    227G     70G    138G    34%    /
devfs                          1.0K      0B    1.0K     0%    /dev
tmpfs                          4.0M    204K    3.8M     5%    /var/run
/lib                           227G     70G    138G    34%    /var/unbound/lib
devfs                          1.0K      0B    1.0K     0%    /var/unbound/dev
/var/log/pfblockerng           227G     70G    138G    34%    /var/unbound/var/log/pfblockerng
/usr/local/share/GeoIP         227G     70G    138G    34%    /var/unbound/usr/local/share/GeoIP
/usr/local/bin                 227G     70G    138G    34%    /var/unbound/usr/local/bin
/usr/local/lib                 227G     70G    138G    34%    /var/unbound/usr/local/lib
devfs                          1.0K      0B    1.0K     0%    /var/dhcpd/dev

Gertjan

Just checking :

@posix said in swap usage shows 100%:

/var/log/pfblockerng 227G 70G 138G 34% /var/unbound/var/log/pfblockerng

I use the default pfBlockerng log file 'size' setup :

and right now that about 110 Mbytes for me:

You : really 70 Gbytes ?

pfBlockerng use these files to generate these pages on demand :

If your files are that big, it would take ages to do that .... nginx/PHP will probably time out.

posix

This post is deleted!

posix

@Gertjan

Edit I see the pfblockerlogs increasing:

Thanks for the response, the log size I have configured are the same

Screenshot 2025-03-10 4.14.32 PM.png

/var/log/pfblockerng: ls -l
total 20376
-rw-------  1 unbound unbound 2012634 Mar 10 16:16 dns_reply.log
-rw-------  1 unbound unbound 2474917 Mar 10 16:16 dnsbl.log
-rw-------  1 root    wheel   5258413 Mar 10 16:00 dnsbl_parsed_error.log
-rw-------  1 root    wheel    426448 Mar 10 16:00 error.log
-rw-------  1 root    wheel    274484 Mar 10 16:00 extras.log
-rw-------  1 root    wheel   3481641 Mar 10 16:11 ip_block.log
-rw-------  1 root    wheel   3649640 Mar 10 16:00 ip_permit.log
-rw-r--r--  1 root    unbound     120 Mar 10 12:00 maxmind_ver
-rw-------  1 root    wheel    624287 Mar 10 16:00 pfblockerng.log
-rw-r--r--  1 unbound unbound     229 Feb 21  2023 py_error.log
-rw-------  1 unbound unbound 2208509 Mar 10 16:16 unified.log

second capture:

-rw-------  1 unbound unbound 2054793 Mar 10 16:23 dns_reply.log
-rw-------  1 unbound unbound 2484058 Mar 10 16:22 dnsbl.log
-rw-------  1 root    wheel   5258413 Mar 10 16:00 dnsbl_parsed_error.log
-rw-------  1 root    wheel    426448 Mar 10 16:00 error.log
-rw-------  1 root    wheel    274484 Mar 10 16:00 extras.log
-rw-------  1 root    wheel   3483020 Mar 10 16:18 ip_block.log
-rw-------  1 root    wheel   3649640 Mar 10 16:00 ip_permit.log
-rw-r--r--  1 root    unbound     120 Mar 10 12:00 maxmind_ver
-rw-------  1 root    wheel    624287 Mar 10 16:00 pfblockerng.log
-rw-r--r--  1 unbound unbound     229 Feb 21  2023 py_error.log
-rw-------  1 unbound unbound 2260172 Mar 10 16:23 unified.log

right now up at 75G

/var/log/pfblockerng: df -h
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/61bf5ededf06b426    227G     75G    133G    36%    /
devfs                          1.0K      0B    1.0K     0%    /dev
tmpfs                          4.0M    204K    3.8M     5%    /var/run
/lib                           227G     75G    133G    36%    /var/unbound/lib
devfs                          1.0K      0B    1.0K     0%    /var/unbound/dev
/var/log/pfblockerng           227G     75G    133G    36%    /var/unbound/var/log/pfblockerng
/usr/local/share/GeoIP         227G     75G    133G    36%    /var/unbound/usr/local/share/GeoIP
/usr/local/bin                 227G     75G    133G    36%    /var/unbound/usr/local/bin
/usr/local/lib                 227G     75G    133G    36%    /var/unbound/usr/local/lib
devfs                          1.0K      0B    1.0K     0%    /var/dhcpd/dev

posix

Hello, can anyone provide further guidance?

posix

@bmeeks @michmoor @stephenw10

Sorry to blast, but looking for further guidance and next steps.

stephenw10

I assume it resets if you reboot?

How long does it take to refill?