MBUF usage at 87%

antillie

stephenw10

Ah, OK. You will see a lot then. Cores X NICs X mbuf allocation = big. :)

Steve

almabes

Can someone knowledgeable post some guidelines for mbuf configuration? There's incomplete and conflicting information out there which is confusing folks. Some information as to what might cause mbuf utilzation to climb would be useful too.

Thanks

tattinger

I'm running a Supermicro MBD-A1SRM-LN7F-2758 with 16gb memory with pfSense 2.2.2. My initial MBUF was 73% (19496/26584) memory usage 4%. I edited the /boot/loader.conf.local using Diagnostics-Edit file and added kern.ipc.nmbclusters="1000000"

Now , after reboot my MBUF is 2% (19750/1000000) Memory usage is 2%.

Sir Loin

I have the Supermicro A1SRi-2758F with 16GB RAM. I ran into the same MBUF problem initially. I had to up kern.ipc.nmbclusters to 1000000 and now all is good.

robi

@stephenw10:

Ah, OK. You will see a lot then. Cores X NICs X mbuf allocation = big. :)

Steve

I've got a system with Atom D525 (4 cores - 1 package(s) x 2 core(s) x 2 HTT threads) and 5 Intel Gigabit NICs, all use the em driver, MBUF is at 2%, no tweak.
I've also got a new A1SRi-2758F with Atom C2758 (8 cores - 1 package(s) x 2 core(s) x 2 HTT threads) and 4 Intel Gigabit NICs, all use the igb driver, MBUF is at 14%, no tweak.

Both in the same place using exactly the same config (the 5th NIC on the D525 not connected to keep machines interchangeable).

Don't see why the MBUFs are so much higher on the C2758. According to your math it shouldn't be more than 5%…
Couldn't we somehow force to use the em driver instead of the igb driver on the intel nics of A1SRi-2758F?

stephenw10

I guess there are more variables in play than I'm aware of. Most likely the usage scales with traffic throughput. Though I'm guessing now…. ::)
There's no way to use the em driver with newer Intel NICs as far as I know.

Steve

robi

There could be some automatic detection at boot which would pre-set the correct value based on the specific hardware, using some math like you suggested.

got0

I own a new SG-4860 and only did some basic configuration and testing so far. However, the usage of MBUF is causing issues:

MBUF Usage: 81% (21516/26584) <- just booted
…
MBUF Usage: 100% (26584/26584) <- climbing without anything really happening on the box
...
kernel: [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached
Uptime ~18h

As suggested by Steve before, I start now the game of doubling the nmbclusters until the box it not freezing anymore. But is it just my box, or is that a general issue with the SG-*? Aren't they already tuned?

stephenw10

We were discussing that internally just recently. In testing the limit was not reached apparently but as always the real world can be different to the test bench. You're not the first person to query that setting.
It's likely that value will be set higher by default in future releases for the SG series. If you run real world tests and come to a conclusion about a suitable value we'd love to hear it.

Steve

Edit: managed to leave out an entire word there!

got0

I'm still in playing and testing mode. Currently, it's 28090 and raising step for step with my changes. But I'll tell you once the value has settled.

almabes

From what I have learned working with supermule on DDosing pfSense, there are major architectural differences between the two Intel gigE drivers, em and igb.

The long and the short of it is Intel cards that use the igb driver set up more queues, which require more mbufs than the older em driver. I have also read somewhere, that there is talk of Intel rewriting the em driver to make it more scalable, and more like the igb driver.

jimp

For 2.2.3 we bumped up the default mbuf allocation on the 4860 and its relatives.

You can adjust the nmbclusters value as described at https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards – though one change: We have found that it's also able to be adjusted "live" by adding it as a system tunable. Used to be it only worked as a loader.conf(.local) value but not any more.

pfcode

@jimp:

though one change: We have found that it's also able to be adjusted "live" by adding it as a system tunable. Used to be it only worked as a loader.conf(.local) value but not any more.

Hi,

I couldn't find it (kern.ipc.nmbclusters) in System->Advantage->System tunable list, using 2.2.3

jimp

It's not there by default, that list isn't limited though. Click + to add it in and set whatever value you want.

pfcode

@jimp:

It's not there by default, that list isn't limited though. Click + to add it in and set whatever value you want.

Oh, I c, Thanks.

got0

For SG-4860, you bumped nmbclusters to 26584 in 2.2.3? That's still too close to the max values I witnessed while playing on my box:

max current = 26.60K
max total= 28.09K

I go with 32K to have a stable box.

jimp

It's not corrected on upgrade, new install only. The 4860 units should probably have 2-4x that value, give or take, minimum.

dennypage

So, what is the value that is being chosen on new installs for the SG-4860?

[I've just upgraded my SG-2400 to a SG-4860, but I'm am using the same drive & essentially same config that was in the SG-2440]

Thanks.

@jimp:

It's not corrected on upgrade, new install only. The 4860 units should probably have 2-4x that value, give or take, minimum.

got0

@jimp:

It's not corrected on upgrade, new install only.

What is the recommended upgrade method to get such tuning parameters after an install? Re-installing an SG-* box for each new version of pfsense is not the way, is it?