swap usage shows 100%
-
Hello,
I just noticed today my SWAP USAGE graphic shows 100%
Below is my setup and information:
Netgate 5100 24.11-RELEASE (amd64) built on Wed Nov 27 13:22:00 EST 2024 FreeBSD 15.0-CURRENTMemory usage 21% of 16287 MiB SWAP usage 100% of 372 MiBCurrent packages:
acme 0.9_1 arpwatch 0.2.3 Avahi 2.2_6 Cron 0.3.8_5 darkstat 3.1.4 lldpd 0.9.11_3 pfBlockerNG-devel 3.2.1_20 snort 4.1.6_22 Status_Traffic_Totals 2.3.2_7 System_Patches 2.2.20_1[/root: top -n -o res last pid: 17785; load averages: 0.66, 1.02, 0.72 up 42+23:25:55 11:59:31 92 processes: 1 running, 91 sleeping CPU: 0.3% user, 0.6% nice, 0.6% system, 0.0% interrupt, 98.5% idle Mem: 1330M Active, 10G Inact, 1140M Laundry, 1903M Wired, 1410M Buf, 807M Free Swap: 373M Total, 373M Used, 300K Free, 99% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 89646 root 2 20 0 841M 717M bpf 1 0:00 0.00% snort 81560 root 2 20 0 697M 574M bpf 2 0:05 0.00% snort 30883 root 1 58 20 849M 501M urdlck 1 5:53 0.00% snort 29178 root 1 60 20 753M 396M urdlck 0 23:52 0.00% snort 13971 root 1 20 0 320M 257M select 3 11:55 0.00% bsnmpd 93985 unbound 4 20 0 284M 215M kqread 0 0:31 0.00% unbound 69369 root 1 52 20 159M 94M urdlck 2 0:10 0.00% snort 90761 root 2 20 0 123M 72M bpf 1 0:00 0.00% snort 14847 root 1 20 0 91M 52M piperd 3 1:10 0.00% php_pfb 38346 root 1 68 0 142M 43M accept 0 0:05 0.00% php-fpm 11883 root 1 31 0 142M 43M accept 1 0:03 0.00% php-fpm 7116 root 1 68 0 142M 43M accept 2 0:05 0.00% php-fpm 24106 root 1 68 0 142M 42M accept 1 0:03 0.00% php-fpm 82301 root 1 68 0 142M 42M accept 3 0:04 0.00% php-fpm 10822 root 1 68 0 113M 36M accept 3 0:01 0.00% php-fpm 17728 root 1 20 0 45M 35M bpf 1 0:25 0.00% arpwatch 18125 root 1 20 0 45M 35M bpf 2 0:26 0.00% arpwatch 17177 root 1 20 0 45M 35M bpf 3 0:26 0.00% arpwatch]Researching some posts and checking memory it shows SNORT, my configuation should* be basic:
HOME_VLAN (igb1.10) AC-BNFA DISABLED Home vlan interface IOT_VLAN (igb1.30) AC-BNFA DISABLED IoT vlan interface MGMT (igb1.100) AC-BNFA DISABLED LAN interfaceUpdates Tab/Installed Rule sets:
Snort Subscriber Ruleset MD5_HAS removed Tuesday, 04-Mar-25 12:32:41 EST Snort GPLv2 Community Rules Not Enabled Not Enabled Emerging Threats Open Rules Not Enabled Not Enabled Snort OpenAppID Detectors MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST Snort AppID Open Text Rules MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST -
Check the Status > Monitoring graphs for previous memory usage. Was it ever close to 100%?
-
Running into the same issue. The IPS packages take up swap for some reason.
top -n -o res last pid: 35679; load averages: 1.17, 1.31, 1.00 up 19+10:22:01 18:03:00 122 processes: 3 running, 119 sleeping CPU: 4.5% user, 1.3% nice, 2.1% system, 0.2% interrupt, 91.9% idle Mem: 971M Active, 2698M Inact, 833M Laundry, 1572M Wired, 56K Buf, 1713M Free ARC: 341M Total, 168M MFU, 152M MRU, 1613K Anon, 2712K Header, 16M Other 273M Compressed, 790M Uncompressed, 2.89:1 Ratio Swap: 1024M Total, 597M Used, 426M Free, 58% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 61670 root 12 20 0 4166M 2951M nanslp 3 26.8H 3.56% suricata 53138 root 12 20 0 1064M 491M nanslp 3 464:38 0.10% suricata 34063 unbound 4 20 0 167M 132M kqread 1 14:38 0.00% unbound 59476 root 17 48 0 5659M 112M uwait 0 171:29 1.46% telegraf 98055 root 14 60 0 1517M 87M uwait 3 84:44 0.29% tailscaled 14741 root 28 68 0 1388M 86M uwait 1 16.8H 10.25% pfnet-controller 78215 root 1 68 0 143M 53M accept 1 0:03 0.00% php-fpm 82015 root 1 68 0 130M 45M accept 1 0:03 0.00% php-fpm 31737 root 1 58 0 120M 45M CPU2 2 0:00 0.00% php-cgi 72711 root 1 68 0 159M 45M accept 0 0:02 0.00% php-fpm 88350 root 1 68 0 155M 45M accept 1 0:02 0.00% php-fpm 20810 root 1 68 0 142M 43M accept 2 0:03 0.00% php-fpm 43359 root 1 68 0 113M 43M accept 1 0:02 0.00% php-fpm 32399 root 1 68 0 113M 43M accept 1 0:02 0.00% php-fpm 31411 root 1 20 0 48M 34M bpf 1 0:36 0.00% arpwatch 33286 root 1 20 0 48M 34M bpf 2 0:36 0.00% arpwatch 33707 root 1 20 0 48M 34M bpf 1 0:36 0.00% arpwatch 31622 root 1 20 0 48M 34M bpf 3 0:38 0.00% arpwatch -
How do you have suricata configured? Is it using that when it updates it's signatures?
-
Configured pretty standard I would think. Certain rules are enabled. Alerting only. Updates are on a scheduled. So far since it’s been installed it’s been no fuss.
-
So what happened at that first spike in the graph, ~07:00? Did Suricata update?
-
@stephenw10 you mean the “Laundry” line?
Updates take place at 8am with an update interval of 4 days.Graph time doesn’t seem to correlate
-
@michmoor said in swap usage shows 100%:
Running into the same issue. The IPS packages take up swap for some reason.
Allocation of swap is an operating system decision. It is not under the control of the package. All the package binary would do is request an allocation of memory. The operating system (FreeBSD in this case) decides how to fulfill the request. If there is not enough physical RAM present, the OS may decide to move certain inactive processes out to swap and reallocate that RAM to the new request. Then, when the requesting process sleeps and it's time for the swapped out process to be awakened, the data is moved back to active RAM. This is why using swap is not desired as it slows things down considerably.
Memory usage for the IDS/IPS packages increases during rule updates. Ditto for a package such as pfBlockerNG. If both collide on an update, it is certainly possible to stress the system and cause swap to become necessary.
-
@posix said in swap usage shows 100%:
Updates Tab/Installed Rule sets:
Snort Subscriber Ruleset MD5_HAS removed Tuesday, 04-Mar-25 12:32:41 EST
Snort GPLv2 Community Rules Not Enabled Not Enabled
Emerging Threats Open Rules Not Enabled Not Enabled
Snort OpenAppID Detectors MD5_HAS removed Thursday, 28-Nov-24 19:42:42 EST
Snort AppID Open Text Rules MD5_HAS removed Thursday, 28-Nov-24 19:42:42 ESTWhy did you obfuscate the MD5 hashes of the rules archive? Those are public. It simply shows the calculated MD5 hash of the gzip archive you downloaded from the AWS CDN node. There is absolutely no private information disclosed in that hash.
-
@michmoor said in swap usage shows 100%:
The IPS packages take up ...
nothing when you install them.
Some memory when you activated them, if there are any 'default' (suggested) rules or conditions or lists.
Then the admin start to add a 'max' of other conditions, rules etc ....
At this precise moment, the admin has a new job : monitoring the system resources very frequently.
As IPS, also pfBlockerng, will take what it needs to do the job.
Suddenly, "4 Gbytes of RAM" becomes a very limited resource. 16 or 32 Gbytes of disk space is even worse.In your case, the packages took what RAM was avaible. real RAM ran out, the system starts to use the spare 'virtual' RAM : starts to use the extremely slow swap space, and when that's filled up also, it, the OS, will start to kill randomly (the biggest resource users first) processes.
Things will go downhill fast from this point.
Using swap on a desktop PC, phone, or any other end user deice is probably more ok, but not for routers and that kind of equipment.IPS tools are like toddlers : you keep an eye om them, even when they sleep.
So, choices to make :
Start doing some (more) serious resource management.
and/or
Use less 'IPS' (and other) functionalities.Btw : normally, it's not the RAM that goes low, but its the disk space filled up to 100 % as massive log files are created, and that will trigger a kernel panic for sure, with a chance that it can't reboot anymore.
-
This post is deleted! -
Hello @bmeeks
Edited message to show MD5 hash:Rule Set Name/Publisher MD5 Signature Hash MD5 Signature Date Snort Subscriber Ruleset dbc140f3f6c01f0cff5e53d63e462df7 Thursday, 06-Mar-25 12:32:47 EST Snort GPLv2 Community Rules Not Enabled Not Enabled Emerging Threats Open Rules Not Enabled Not Enabled Snort OpenAppID Detectors c726cf937d84c651a20f2ac7c528384e Thursday, 28-Nov-24 19:42:42 EST Snort AppID Open Text Rules 2c26cb4f6a3bc03ab9c8e02befcf6fe1 Thursday, 28-Nov-24 19:42:42 EST Feodo Tracker Botnet C2 IP Rules Not Enabled Not Enabled -
-
Don't know if this helps since it was mentioned about disk space:
/root: df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/61bf5ededf06b426 227G 70G 138G 34% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 4.0M 204K 3.8M 5% /var/run /lib 227G 70G 138G 34% /var/unbound/lib devfs 1.0K 0B 1.0K 0% /var/unbound/dev /var/log/pfblockerng 227G 70G 138G 34% /var/unbound/var/log/pfblockerng /usr/local/share/GeoIP 227G 70G 138G 34% /var/unbound/usr/local/share/GeoIP /usr/local/bin 227G 70G 138G 34% /var/unbound/usr/local/bin /usr/local/lib 227G 70G 138G 34% /var/unbound/usr/local/lib devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev -
Just checking :
@posix said in swap usage shows 100%:
/var/log/pfblockerng 227G 70G 138G 34% /var/unbound/var/log/pfblockerng
I use the default pfBlockerng log file 'size' setup :
and right now that about 110 Mbytes for me:
You : really 70 Gbytes ?
pfBlockerng use these files to generate these pages on demand :
If your files are that big, it would take ages to do that .... nginx/PHP will probably time out.
-
This post is deleted! -
Edit I see the pfblockerlogs increasing:
Thanks for the response, the log size I have configured are the same
/var/log/pfblockerng: ls -l total 20376 -rw------- 1 unbound unbound 2012634 Mar 10 16:16 dns_reply.log -rw------- 1 unbound unbound 2474917 Mar 10 16:16 dnsbl.log -rw------- 1 root wheel 5258413 Mar 10 16:00 dnsbl_parsed_error.log -rw------- 1 root wheel 426448 Mar 10 16:00 error.log -rw------- 1 root wheel 274484 Mar 10 16:00 extras.log -rw------- 1 root wheel 3481641 Mar 10 16:11 ip_block.log -rw------- 1 root wheel 3649640 Mar 10 16:00 ip_permit.log -rw-r--r-- 1 root unbound 120 Mar 10 12:00 maxmind_ver -rw------- 1 root wheel 624287 Mar 10 16:00 pfblockerng.log -rw-r--r-- 1 unbound unbound 229 Feb 21 2023 py_error.log -rw------- 1 unbound unbound 2208509 Mar 10 16:16 unified.logsecond capture:
-rw------- 1 unbound unbound 2054793 Mar 10 16:23 dns_reply.log -rw------- 1 unbound unbound 2484058 Mar 10 16:22 dnsbl.log -rw------- 1 root wheel 5258413 Mar 10 16:00 dnsbl_parsed_error.log -rw------- 1 root wheel 426448 Mar 10 16:00 error.log -rw------- 1 root wheel 274484 Mar 10 16:00 extras.log -rw------- 1 root wheel 3483020 Mar 10 16:18 ip_block.log -rw------- 1 root wheel 3649640 Mar 10 16:00 ip_permit.log -rw-r--r-- 1 root unbound 120 Mar 10 12:00 maxmind_ver -rw------- 1 root wheel 624287 Mar 10 16:00 pfblockerng.log -rw-r--r-- 1 unbound unbound 229 Feb 21 2023 py_error.log -rw------- 1 unbound unbound 2260172 Mar 10 16:23 unified.logright now up at 75G
/var/log/pfblockerng: df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/61bf5ededf06b426 227G 75G 133G 36% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 4.0M 204K 3.8M 5% /var/run /lib 227G 75G 133G 36% /var/unbound/lib devfs 1.0K 0B 1.0K 0% /var/unbound/dev /var/log/pfblockerng 227G 75G 133G 36% /var/unbound/var/log/pfblockerng /usr/local/share/GeoIP 227G 75G 133G 36% /var/unbound/usr/local/share/GeoIP /usr/local/bin 227G 75G 133G 36% /var/unbound/usr/local/bin /usr/local/lib 227G 75G 133G 36% /var/unbound/usr/local/lib devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev -
Hello, can anyone provide further guidance?
-
Sorry to blast, but looking for further guidance and next steps.
-
I assume it resets if you reboot?
How long does it take to refill?