pfSense 2.7.2 RAM leak (wired memory pool)

[zeroflow]

Hi,(information text)

I'm running two pfSense boxes at my house & flat. One J4125/i225v3 based and one N100/i226 based.
The J4125 is running pfSense+ 24.11 while the N100 box is running CE 2.7.2

Now I've had some strange problems with the N100 box, where the connections would slowly degrade. One device after the other would stop routing / receive an IP until no traffic was passed. A reboot always fixed the issue. Now after the latest hangup, I've seen in my metrics, that RAM was slowly filling up until it reached 100% and the issues started. It took roughly 68 days to fill up all 8 GB of RAM. In the metrics, I see that the memory used up by the wired portion (kernel) is constantly increasing. I have installed a VM with the same config that is behaving normal (details below).

Now my question is: Where and how can I look further to find the culprit?

Currently, I only see the following options:

• Reinstall 2.7.2 to rule out any config issues
• Update to 2.8.0 and hope it was a config/driver bug

Anything major I'm missing?

Hardware Specs:

• N100
• 4x i226-V NICs
• 8 GB RAM
• 128 GB M.2 NVMe
• EATON UPS connected via USB

Software

• pfSense 2.7.2-RELEASE
• Installed Packages
  • acme 0.9_1
  • Avahi 2.2_4
  • Cron 0.3.8_3
  • haproxy 0.63_2
  • iperf 3.0.3
  • lldpd 0.9.11_2
  • nmap 1.4.4_7
  • ntopng 0.8.13_10 (but not enabled in settings)
  • nut 2.8.2_1
  • pfBlockerNG 3.2.0_8
  • System_Patches 2.2.11_17
  • Tailscale 0.1.4
  • Telegraf 0.9_6
  • WireGuard 0.2.1

Setup

• Interfaces
  • WAN with DCHP from Modem (Bridge Mode)
  • Main LAN
  • IoT VLAN with some rule restrictions
  • Guest Net routed over OpenVPN
• Services
  • OpenVPN Client to VPN Provider
  • Wireguard S2S connection to pfSense+ Box
  • pfBlocker for IP Blacklisting and DNS filtering
  • haproxy for accessing hosted services

Now when looking at the RAM measurements, I can see that the portion for wired (kernel) is constantly increasing.

After that, I made some measurements to compare to:

• "pfsense.home" in Green
  • pfSense+ on the J4125 box
• "pfsense272.home.arpa" in orange
  • Fresh CE 2.7.2 in Proxmox VM
• "pfsense280.home.arpa"
  • Fresh CE 2.8.0 Beta in Proxmox VM
• "pfsense-272copy.home.arpa"
  • Config of affected pfSense box in Proxmox VM
• "pfsense.<blank>.dev" in purple
  • Affected device

All other devices show a steady RAM use. There are some peaks where pfBlockerNG does it update. The perfect ramps in the green line are from one of my scripts that ran at 1:15 in the morning which killed Telegraf, but otherwise it's nice, steady and constant.

Thanks in advance!

[Gertjan]

@zeroflow

This is easy to solve. Or at least, easy to know what is happening.

First :
Remove all packages.

You can keep :
acme - Avahi - Cron - nut - pfBlockerng-devel (I don't know about the non dev version) System_Patches.
Why ? Because I use these myself and look here : no memory eaters on my 4100.

I use 25.03 beta 2, very comparable to 2.8.0 beta.

I put my bets on haproxy - iperf and ntopng so it's a package issue.

[zeroflow]

@Gertjan

Thank you, I will try that.

I have ruled out those packages at first, because the pfsense+ box running at my flat (green line) has the same packages & config and works fine.

But I'l never know until I try.

[Gertjan]

@zeroflow

It can't be 'pfSense' (bare bone, no packages install). If it was, every pfSense being used out there would run out of memory every week. That's something that would edit : not happen without some one talking about it here on the forum.

[zeroflow]

@Gertjan

Thanks.
I would have figured as much, since that would be a much bigger issue affecting everyone.
I still kept wireguard, since that's needed for the S2S connection.

[stephenw10]

Check the Diag > System Activity page. What's using the RAM?

Or at the CLI check ps -auxwd or top -HaSP.

[zeroflow]

@stephenw10

I don't have all the details since I had to reboot yesterday.
But from the general stats right before the reboot:

Mem: 101M Active, 558M Inact, 2790M Wired, 56k Buf, 4550M Free

From previous investigations, I've been blaming unbound since that process has increased by a few MB.
But looking at it for a longer timeframe, the wired memory keeps increasing by ~100 MB per day. Everything else stays pretty much constant.

[stephenw10]

I would expect to see some specific process using it.

Are you able to test 2.8-beta? There's a good chance it's already fixed.

[shoulders]

@zeroflow What do you use to make those RAM graphs. I am running pfsense 2.7.2 and for a while it has been suffering similiar to you, runs for a while, then network start slowing right down until off, the GUI is inaccessible. I tnink also the CLI becomes unresponsivce and I have to use a restart command for the VM in TrueNAS.

I wrongly assumed it was a extension such as snort, did not think it might be pfsense.

[stephenw10]

He's polling it with some external service but you can get that data from Status > Monitoring:

[shoulders]

@stephenw10 Thanks for the heads up. I will monitor my pfsense and if it goes funny I will check ram usage etc...

[zeroflow]

@shoulders
In my setup, data is pushed via the Telegraf Plugin to an Influxdb time series DB. From there, I can create the visualizations in Grafana.

[zeroflow]

@stephenw10
I've now been able to test 2.8.0-BETA
(build date Mon Apr 28 1:42:00 CEST 2025)

For now, this seems to have fixed the issue.

The dotted line is the installation of 2.8.0.

1 - Data from the original post.
2- Data after uninstalling plugins
3 - 2.8.0 Beta

The different form of spikes comes from the pfBlocker update. There is a new cron job at 3pm fetching some data.