Change in memory management/usage?

pfsjap

23.05RC seems to use some swap space in Netgate 6100MAX while 23.01 with the same configuration did not. It may not be aggressive usage, but I would rather not have it use swap at all.

Is this swap usage a symptom of some memory management problem in some service? This is after upgrading to 23.05.r.20230519.0600:

Dobby_

When a FreeBSD based system runs out of memory, the kernel can move sleeping or inactive processes into swap area. A dedicated Swap partition goes a long way to avoid system freeze but if you notice you are running out of RAM or your applications are consuming too much of it then you may want to own a swap partition that is let us call it buffering down that behaviour.

Many years ago, the rule of thumb for the amount of swap space that should be allocated was 2X the amount of RAM installed in the computer.

I have on board soldered RAM of 4 GB and pfSense was
using nearly 67 % to over 90 % of it, now I am on something around 22% RAM / 24% swap on 23.05 RC
and 18% RAM / 24% Swap on 2.7 Devel now it can be
used (the free RAM) for other things, such mbuf size,
queues amount and size and others.

pfsjap

@dobby_ So have you noticed any change in swap usage between 23.01 and 23.05 RC?

Are there any kernel parameters in FreeBSD, that control the "sensitivity" to start swapping?

Dobby_

@pfsjap said in Change in memory management/usage?:

@dobby_ So have you noticed any change in swap usage between 23.01 and 23.05 RC?

Something about my situation, I have two boxes with
onboard soldered RAM, so I am not able to change it
it if needed to a greater size by installing a greater RAM
module with let us say 8 GB or 16 GB like other users
will be able to do. And so it comes that I am watching
the RAM and Swap usage a little bit more then others.

In former days together with pfSense CE 2.6 and pfSense+ 23.01 if I updated or upgraded something and must reboot the hardware I saw at a fresh boot up where
all the snort rules, pfBlocker-NG feeds and the ClamAV
av signatures were fresh loaded (common) that the RAM
were spiking out at something between 60% to over 90%
and the entire swap partition was not in usage. But now on
pfSense 23.05 RC (APU6B4) and pfSense 2.7 Devel
(APU4D4) the entire system will be not like before in 2 - 4 days needing to level off, so that swap and ram will be used together much faster and more let us call it "weighted" and this in or after only some hours.
Please see the added pictures below

pfSense+ (Plus) 23.05 RC
Fresh upgraded and rules, feeds and signatures will be fresh loaded
23.05RC swap ram.jpg

pfSense 2.7 Devel
After some hours it is really fast level off to use both
Swap and RAM like shown below
2.7 Devel latest swap ram.jpg

Are there any kernel parameters in FreeBSD, that control the "sensitivity" to start swapping?

I can`t anything about that, but the reaction time and
also the art and wise of taking advantage from the
available swap in the system is going more fast and moderate as I was recognizing. And for sure with
onboard soldered ram I will be much more happy
then perhaps others with that behaviour!

mvikman

I have 8GB RAM and so far have never seen my system use swap, though not running pfblocker etc.
After upgrades, the memory usage jumps to somewhere between 25-35%, but after a second reboot it goes back to the normal 6-7%

RobbieTT

@pfsjap I agree with the above and that swap usage like you are experiencing is not normal, especially with 8GB on Netgate hardware.

I also run a 6100 Max and I guess you are more familiar with the memory use shown on mine:

Mem: 125M Active, 398M Inact, 622M Wired, 40K Buf, 6635M Free
ARC: 223M Total, 48M MFU, 167M MRU, 1490K Header, 6081K Other
     191M Compressed, 518M Uncompressed, 2.72:1 Ratio
Swap: 1024M Total, 1024M Free

There has to be a ripple in that particular load, which is good data for Netgate. Did you try a full restart from cold and log those details too?

️

pfsjap

@robbiett Swap usage starts at 0% after restart, but increases later on. I haven't taken up any figures, but after restart memory usage might be 40-50%, but then goes down. Upgraded earlier today to 23.05.r.20230521.0305 and now memory usage is 23% and swap usage 2%.

I have these packages installed:

pfBlocker has 23 IP and DNSBL groups configured. IPS policy is set to Security in Snort with 22 categories enabled in ET Open Rules.

Patch

@pfsjap said in Change in memory management/usage?:

parameters in FreeBSD

See https://redmine.pfsense.org/issues/14030 and
https://forum.netgate.com/topic/177886/23-1-using-more-ram

When disk intensive activities occur (such as software update), the zfs file system cache increases. The cache will be released if required but not immediately forcing use of swap. The solution is to limit the cache size down from the very generous default.

pfsjap

@patch In 23.01 I had set vfs.zfs.arc.max to 419430400. With 23.05 RC I had already decreased that and at the moment it's 400000.

RobbieTT

@pfsjap Could you run top -aS -o res for us so we can look at how the memory is being used?

️

pfsjap

@robbiett Sure, looks like Snort is using a lot of memory:

last pid: 85727;  load averages:  0.25,  0.55,  0.47                                                                                                            up 0+09:12:12  18:28:07
93 processes:  2 running, 89 sleeping, 2 waiting
CPU:  0.1% user,  0.2% nice,  0.1% system,  0.0% interrupt, 99.6% idle
Mem: 1520M Active, 2240M Inact, 124M Laundry, 756M Wired, 3148M Free
ARC: 257M Total, 164M MFU, 83M MRU, 920K Anon, 2009K Header, 7051K Other
     223M Compressed, 546M Uncompressed, 2.44:1 Ratio
Swap: 2048M Total, 43M Used, 2005M Free, 2% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
83927 root          3  52   20  1421M  1079M bpf      3   1:16   0.23% /usr/local/bin/snort -R _34675 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -
13140 root          2  52   20  1421M  1063M bpf      0   0:23   0.05% /usr/local/bin/snort -R _8486 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l
 6934 unbound       4  20    0   822M   756M kqread   3   0:11   0.26% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
28523 root          1  68    0   147M    69M accept   0   0:26   0.00% php-fpm: pool nginx (php-fpm)
72038 root          1  44    0   153M    65M accept   3   0:43   0.00% php-fpm: pool nginx (php-fpm)
 1040 root          1  68    0   147M    62M accept   0   0:48   0.00% php-fpm: pool nginx (php-fpm)
 1041 root          1  68    0   147M    55M accept   0   0:38   0.00% php-fpm: pool nginx (php-fpm)
85623 root          1  20    0    47M    37M bpf      0   0:00   0.00% /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_igc2.dat -i igc2 -w digger909@hotmail.com
85048 root          1  20    0    47M    37M bpf      2   0:00   0.00% /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_igc0.dat -i igc0 -w digger909@hotmail.com
93605 root          1  20    0    72M    36M piperd   3   0:01   0.01% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
 1039 root          1  20    0   112M    17M kqread   1   0:01   0.00% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
 3716 dhcpd         1  20    0    25M    10M select   0   0:01   0.01% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igc

Dobby_

@mvikman said in Change in memory management/usage?:

I have 8GB RAM and so far have never seen my system use swap, though not running pfblocker etc.

I run something nearly a fully UTM and you run a
pure firewall there is for sure a difference as I see it.

Squid & SquidGuard (DNSBL)
pfBlocker-NG (Feeds)
ClamAV (AV Signatures)
Snort (Rules)

This will be fresh load after a reboot, and in my eyes there
is nothing wrong with and so I am happy that swap will be used in such cases.

After upgrades, the memory usage jumps to somewhere between 25-35%,

But from your 8 GB will 35% are 2.8 GB and this is nearly
70% of my RAM you have forgotten in your counting.

but after a second reboot it goes back to the normal 6-7%

Only after several hours here, look at the picture from today morning (11 hours) and from now (below) all is balanced fine.

pfSense 23.05 RC swap ram 2.jpg

mvikman

@dobby_

The original poster asked about swap usage / change in swap usage from 23.01 to 23.05RC, also didn't specify what packages were running in system.

I just provided my experience with my system, which is that I have never seen swap usage in it.
I haven't bothered to change the ZFS ARC cache limits because it hasn't been a problem in my system.

Of course you can directly compare systems with 4GB and 8GB RAM.

jimp

All of this (and more) is covered in recent docs updates:

https://docs.netgate.com/pfsense/en/latest/hardware/memory.html