MBUF slowly increasing over time

TheCyberfreak

Hello all,

since the update to pfSense 22.05 I have problems with a rising MBUF.
The value increases according to monitoring about 500 every 5 minutes. I had, because I also had problems with the RAM usage, restarted the firewall yesterday, before that the MBUF rose to just under 90%. Since the restart, it rises again continuously. Graph Screenshot

Actually, the load is not higher than usual. The problems I have only since the update to 22.05. Is there a solution here to avoid this increase?

Briefly about the hardware:

Intel(R) Pentium(R) CPU 6405U @ 2.40GHz
RAM 16GB

stephenw10

Hmm, not seeing that anywhere here.

What NICs do you have? What sort of traffic levels are you seeing?

The increase rate looks surprisingly linear...

Steve

TheCyberfreak

6x Intel I211-AT NICs

What do you mean by traffic levels? I haven't changed anything in the configuration since the update or before. There are also no new devices or similar that cause large traffic. Both externally and within the network.
So the traffic, I would say, the same as before.

I appreciate any help and will be happy to provide any information needed!

stephenw10

I mean like constant 500Mbps? What do the traffic graphs show for the same period?

TheCyberfreak

Not much activity..
Here is a picture of MBUF and WAN-Interface.. internally there was also hardly any traffic.
Bildschirmfoto 2022-07-07 um 01.05.30.png

stephenw10

Hmm. So I assume igb interfaces from i211 NICs?

Do you have any non-default sysctls or loader values in play?

What does netstat -m show?

Steve

TheCyberfreak

Yes, igb..
No, I have not made any changes.

here the excerpt:

302445/2175/304620 mbufs in use (current/cache/total)
221226/816/222042/1000000 mbuf clusters in use (current/cache/total/max)
217112/721 mbuf+clusters out of packet secondary zone in use (current/cache)
1/148/149/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/524288 9k jumbo clusters in use (current/cache/total/max)
0/0/0/83968 16k jumbo clusters in use (current/cache/total/max)
518069K/2767K/520837K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
2 sendfile syscalls
2 sendfile syscalls completed without I/O request
2 requests for I/O initiated by sendfile
3 pages read by sendfile as part of a request
6 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
3 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

stephenw10

Ok. And in 22.01 and previously the mbuf usage was always low? Do you still have RRD data for that in 22.01?

Steve

TheCyberfreak

Yes, under 22.01 it was always low...
Unfortunately, I do not know how I can prepare the data. But I have you a CSV file with the readings from pfSense of the last 3 months. There the value was permanently constant.

MBUF.xls

stephenw10

Must be some config you have or something you're running.

What packages do you have installed?

Steve

TheCyberfreak

These are my installed packages

stephenw10

Are you running OpenVPN with DCO enabled perhaps?

Are you using traffic shaping at all? Captive portal?

TheCyberfreak

I’ve enabled DCO but in the last week there wasn’t any incoming connection.
I’ve disabled DCO right now, but I dont believe that’s the reason..

I’m not using traffic shaping or CP

stephenw10

You were definitely seeing it before you enabled DCO?

TheCyberfreak

I have now tested a bit over the day. In fact, the cause was the DCO setting of the OpenVPN tunnels. After I fixed this setting on two tunnels, the problem is gone and the MBUF stays constant again!

Thanks a lot for your help!

stephenw10

Ooo, that sounds like something we need to address. You just disabled DCO and the mbuf leak stopped? Made some other change?

Steve

TheCyberfreak

I only disabled DCO on two existing tunnels, since then it has remained constantly at that value, after a restart it is constantly low again.

stephenw10

Are you able to share the settings you were using that created the leak? None of my DCO test systems appear to be leaking.

TheCyberfreak

Yes, what settings do you need?

stephenw10

Ideally the full server config with DCO enabled and whatever redacted you need to.

So either the OpenVPN config from:
/var/etc/openvpn/server1/config.ovpn

Or the server section from the main pfSense config file.

But anything you can provide to help us replicate it would be very helpful.

Steve