23.1 using more RAM

[SHoover80]

@jimp kk, I had just restart it before reading this and posting my last screenshots. I'll let it run a bit and when I see memory usage start to increases I'll follow your recomendation.

[SHoover80]

@jimp This is about 2 hours later, So I guess I was wrong earlier when I stated it appears memory increasing after about 8 hours of uptime.

[SHoover80]

This post is deleted!

[SHoover80]

@jimp Here's about 6 hours after the last two screenshots I posted.

[SHoover80]

@jimp
here's after 15 hours of uptime.

[jimp]

At least from that top output it appears the ARC usage is still going up, probably any time there is a scheduled process that hits the disk a lot it will go up (e.g. updating rules for snort/suricata/whatever). But that ARC memory will be released as needed so it looks worse than it really is.

[stephenw10]

Mmm, 100% SWAP usage is... not great though.

[DefenderLLC]

OK I’m experiencing something similar. My 6100 MAX memory utilization was always between 18% and 20% on 22.05. After installing 23.01 on release day, the memory will start in the mid-teens and slowly creep its way back up to nearly 40% with the exact same configuration on my 6100 MAX within about 18 hours. Once I get to that point, I just reboot it and start the cycle all over again which has been every morning since release day.

What can I give you to help troubleshoot my issue?

[rcoleman-netgate]

@stephenw10 said in 23.1 using more RAM:

Mmm, 100% SWAP usage is... not great though.

Neither is running both Suricata and Snort, though.

[SHoover80]

@rcoleman-netgate, @stephenw10 Yea, I realized that. running snort and suricata after doing some research over the weekend so I stopped using Snort. I've done a complete fresh install, started all my setup from scratch only maintaining my DNS and DHCP settings. I installed all the plugins one by one, (testing one at a time for a period of time watching the results of RAM usage) to see if it only happened with certain plugins over the weekend. I started to notice the increase of RAM usage over time when pfBlockerNG or when Suricata was installed. None of the other plugins I'm running caused a increase of memory and only used about 7% of my memory with idle traffic only going up to about 17% with increased traffic. @jimp claiming it's not as bad as it looks, well when it's slowing down my network as i experienced while testing over the weekend and causing streaming videos to buffer and slowing down my data transfers over network I'd say it is exactly as bad as it looks.

[stephenw10]

@defenderllc said in 23.1 using more RAM:

the memory will start in the mid-teens and slowly creep its way back up to nearly 40% with the exact same configuration on my 6100 MAX within about 18 hours.

Does it just keep climbing if you do not reboot?

Is it actually causing a problem or just seems unexpected?

Steve

[SHoover80]

@stephenw10 Yes, for me it keeps climbing the longer it's running. and after SWAP fills up or gets around 70% I start to experience slow network issues. Streaming video from local plex server I start to experience buffering, Same with Youtube and transfering files are slowed down till I restart PFSense.

[DefenderLLC]

@stephenw10 said in 23.1 using more RAM:

@defenderllc said in 23.1 using more RAM:

the memory will start in the mid-teens and slowly creep its way back up to nearly 40% with the exact same configuration on my 6100 MAX within about 18 hours.

Does it just keep climbing if you do not reboot?

Is it actually causing a problem or just seems unexpected?

Steve

Yes it does exactly that, but it is very slow increment. Once it gets close to 40%, it never seems to go down much. It might go down 1% or 2% here and there, but never returns to the original state of 18% to 20%. It’s worth noting it’s not causing any problems, but when the memory utilization doubles overnight, it just concerns me that there might be a memory leak. I’m going to let it run for a few days without rebooting it to see what it does.

I’m using my 6100 strictly as a DMZ firewall only and for primary DNS with pfBlocker. I have a UDM behind it that manages the clients, but the 6100 is the network-wide primary DNS. I do have Suricata installed, but it’s not really doing anything at the moment. Unbound and ntop seem to always be using a lot of RAM since upgrading to 23.01.

[Dobby_]

@defenderllc

Me too. For sure I use a slower hardware then you, but after upgrading and rebooting several times (setting backup playing in, ....) It went "normal" for my setup, but that said with a small higher CPU usage RAM usage and Swap usage too.

running on APU4D4

• with tuned CPU from 600MHz - 1000MHz to
  1000MHz - 1400MHz
  running as UTM
• Snort, Squid, SquidGuard, ClamAV and pfBlocker-NG

After a while, it becomes more stable and using "less" ram, cpu and swap but more as before together with 22.05!

[DefenderLLC]

Looks like my spikes begin at 3AM and never return to normal. A few others shared this exact same issue (and at the exact same time). This is my memory utilization for the past 2 days and is the same each morning after a reboot the night before:

[stephenw10]

Check the cron table for what is triggered at that time?

I'd guess it's a pfBlocker or Snort update.

[manicmoose]

@shoover80

FWIW (I realise the memory change is only since you upgraded) I run telegraf & pfblocker (not Suricata), but my telegraf config differs from yours only by:

from_beginning = false

You could try changing that as I haven't noticed any memory difference since upgrading.
Maybe telegraf is keeping more of that data in memory than it used to and not releasing it.

[DefenderLLC]

@stephenw10 said in 23.1 using more RAM:

Check the cron table for what is triggered at that time?

I'd guess it's a pfBlocker or Snort update.

I don’t run snort and my pfBlocker lists update every hour. I’ll take a look. What’s the best way for me to figure out what’s running at 3AM without having to stay up to watch it? Thanks.

FYI, I rebooted at 4 PM today. I will share tomorrow morning’s graph where I expect the behavior will be exactly the same.

[stephenw10]

Check the cron table. You can install the cron package to do that easily.

Check the system logs at that time, what was logged?

Steve

[SteveITS]

@stephenw10 Now that I've looked I see it too, on a 2100. However it only happened once and not the second day.

1. 3:00 am
2. 5:15 am next day, pfBlocker update
   

No pfSense system log entries between 01:01 and 04:30.

# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD$
#
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
#
#minute	hour	mday	month	wday	who	command
#
# Save some entropy so that /dev/random can re-seed on boot.
*/11	*	*	*	*	operator /usr/libexec/save-entropy
#
# Rotate log files every hour, if necessary.
0	*	*	*	*	root	newsyslog
#
# Perform daily/weekly/monthly maintenance.
1	3	*	*	*	root	periodic daily
15	4	*	*	6	root	periodic weekly
30	5	1	*	*	root	periodic monthly
#
# Adjust the time zone if the CMOS clock keeps local time, as opposed to
# UTC time.  See adjkerntz(8) for details.
1,31	0-5	*	*	*	root	adjkerntz -a
#
# pfSense specific crontab entries
# Created: February 18, 2023, 8:19 pm
#

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata
*/1	*	*	*	*	root	/usr/sbin/newsyslog
1	3	*	*	*	root	/etc/rc.periodic daily
15	4	*	*	6	root	/etc/rc.periodic weekly
30	5	1	*	*	root	/etc/rc.periodic monthly
0	*/4	*	*	*	root	/etc/rc.backup_rrd.sh
0	*/4	*	*	*	root	/etc/rc.backup_dhcpleases.sh
0	*/1	*	*	*	root	/etc/rc.backup_logs.sh
0	6	*	*	5	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1
15	5	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1
#
# If possible do not add items to this file manually.
# If done so, this file must be terminated with a blank line (e.g. new line)
#

No particular memory hogs though?

Mem: 157M Active, 235M Inact, 1129M Wired, 1780M Free
ARC: 781M Total, 638M MFU, 71M MRU, 280K Anon, 9429K Header, 61M Other
     650M Compressed, 1401M Uncompressed, 2.16:1 Ratio

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
16983 unbound      21    0   141M   120M kqread   1   0:11   2.04% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
16983 unbound      20    0   141M   120M kqread   0   0:06   0.01% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
26241 root         68    0   148M    52M accept   0   0:29   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
  495 root         68    0   148M    52M accept   0   0:23   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
15289 root         68    0   148M    52M accept   0   0:37   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
  757 root         68    0   148M    51M accept   1   0:36   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
  494 root         29    0   149M    51M accept   0   0:37   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
90123 root         68    0   144M    51M accept   1   0:22   0.00% php-fpm: pool nginx (php-fpm)
  752 root         68    0   144M    51M accept   0   0:23   0.00% php-fpm: pool nginx (php-fpm)
  493 root         20    0   111M    29M kqread   0   0:08   0.00% php-fpm: master process (/usr/local/lib/php-fpm.conf
47310 dhcpd        20    0    24M    13M select   1   0:00   0.01% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chro
49371 dhcpd        20    0    22M    10M select   0   0:00   0.01% /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -c
60576 root         20    0    30M    10M kqread   1   0:06   0.00% nginx: worker process (nginx)
66552 root         20    0    20M  9840K select   1   0:00   0.02% sshd: admin@pts/0 (sshd)
60427 root         20    0    30M  9768K kqread   0   0:00   0.00% nginx: worker process (nginx)
59743 root         20    0    20M  8796K kqread   0  13:58   0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dns
18148 root         21    0    20M  8404K select   1   0:00   0.00% sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
60229 root         48    0    28M  7808K pause    0   0:00   0.00% nginx: master process /usr/local/sbin/nginx -c /var/
59735 root         20    0    21M  7000K select   1   0:29   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /va

Edit: Would "/etc/rc.periodic daily" be doing something to trigger ZFS memory usage, that wouldn't release? (since the release notes say it won't after upgrading but that's normal and it will eventually release as needed)