kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached

w0w

"kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached"
I've received this in syslog server, after that pfsense does not accessible via network.
https://lists.pfsense.org/pipermail/list/2015-January/007916.html Seems to be already reported.
And some similar problem here https://forum.pfsense.org/index.php?topic=84788.msg465221#msg465221

Since my hardware is quite different from what other reporters have, what exactly I must tune in freebsd settings?

My hardware is D2500CC and I believe that it have 82574L onboard NICs. Pfsense 2.2

jimp

Since the error message specifically said kern.ipc.nmbjumbo9 limit reached, I'll go out on a limb and say you need to increase kern.ipc.nmbjumbo9 :-)

Or at least kern.ipc.nmbjumbop (See https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_ix.284.29_Cards for an example)

w0w

Recently I've tuned kern.ipc.nmbclusters="1000000"

$ netstat -m
2048/3772/5820 mbufs in use (current/cache/total)
1024/1512/2536/1000000 mbuf clusters in use (current/cache/total/max)
1024/1506 mbuf+clusters out of packet secondary zone in use (current/cache)
0/8/8/13291 4k (page size) jumbo clusters in use (current/cache/total/max)
1023/1940/2963/3938 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2215 16k jumbo clusters in use (current/cache/total/max)
11767K/21459K/33226K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/9/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

w0w

#sysctl kern.ipc.nmbjumbo9=11815
kern.ipc.nmbjumbo9: 11814
sysctl: kern.ipc.nmbjumbo9=11815: Invalid argument

Any number that differs from 11814 causes "Invalid argument", so I am not sure what is going on.

EDIT: Applied just for test

kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"

2047/2003/4050 mbufs in use (current/cache/total)
1023/1513/2536/1000000 mbuf clusters in use (current/cache/total/max)
1023/1507 mbuf+clusters out of packet secondary zone in use (current/cache)
0/3/3/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
1023/8/1031/524288 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2215 16k jumbo clusters in use (current/cache/total/max)
11764K/3610K/15375K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/6/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

How can I define appropriate numbers for stable work? I have only 2GB of RAM installed.

jimp

It's a loader tunable, so it must be set in /boot/loader.conf.local and only activates during a reboot.

w0w

Which values are best for me? Near the maximum detected or twice or any mathematics based on stat else?
kern.ipc.nmbjumbop="?"
kern.ipc.nmbjumbo9="?"

jimp

The link I posted above has a suggested value.

w0w

Yes, it is, but only for nmbjumbop and Intel "ix" cards and mine is "em".
Thank you for your patience, jimp. I know that I am asking dumb questions.
Unfortunately I can not find any good guide for freebsd 10 about tuning those values and old guides are too old for 10.x version, I think. The only one good thing I've found is https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning but there is no any explanation… and it is mostly for 10 gigabit throughput, mine is 1 gigabit and hardware is not such powerful as described in article.

jimp

When in doubt, try 2x what you have now for a max.

Harvy66

fyi, 524288 9k jumbo frames is over 4GB and you only have 2GB. you're a bit over-subscribed. It won't be an issue unless your box actually attempts to use all of them, then bad things may happen when your kernel runs out of memory.

I think, anyway.

w0w

Ok, thanks.
Tuned a little bit differently
kern.ipc.nmbclusters="131072"
kern.ipc.nmbjumbo9="20000"