Kern.ipc.nmbufs limit reached
-
Hey folks,
I've got this recurring crash that I can't figure out; about once a week or so my pfSense box just drops off the network and stops passing traffic. When I look at the console, it says:
[zone: mbuf] kern.ipc.nmbufs limit reachedEverything I could find on the forum here and around the web suggested tweaking kern.ipc.nmbclusters upwards. I've done that, maybe a couple times. I don't remember what the default was but it's currently set to 262144.
This is on 2.2.6, it also happened on the previous release and maybe the one before that as well.
Any ideas?
-
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
-
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
Alright, yeah. That's the document where I got the idea to increase kern.ipc.nmbclusters.
I suppose I'll try the recommended Broadcom bge tweaks and see what happens. It's a right pain to troubleshoot when there's such a long interval between crashes.
-
you can use the mbuf rrd graphs to see the frequency, or find out if you tweaks seem to solve your issue
-
you can use the mbuf rrd graphs to see the frequency, or find out if you tweaks seem to solve your issue
That's another thing that seems kind of odd. According to my RRD graphs, my mbuf usage never changes much at all, and never gets anywhere close to the maximum.
24 hour data, including the most recent crash; you can see at the tail end where I just tweaked nmbclusters back down to the recommended bge settings:
1 year data, since I brought this firewall up last April:
Also, the y-axis label says 'percent', that can't be right. UI bug?
-
So, here we are exactly a month later, and I had another nmbufs crash. I was able to get a local shell, but I could really tell was that the system load average was about 2.5, with pfctl and dhcpd each using around 50% CPU.
I checked dmesg and saw nothing but the nmbufs error. /var/log/system.log didn't have any entries at all since Feb. 21, which was when it crashed last.
I probably should spend a bit more time learning what files and commands on a BSD system will give me better info, for the next time this happens.
-
Hello,
on some 10 GbE cards it was helping out to adjust this mbuf sizes to something around 65000
and in other cases pending on the installed RAM or system memory it was better to go in some
steps upwards, likes 250000, 500000 and 1000000. So you will see it is not like something to
type it in and then al is going fine, it is more a try and error game to get the best results matching
your system hardware and according to your RAM size. Please be careful to set it too low! Then you can
be easily ending up in a so called booting loop.Even to tune one thing and then you might be getting over a smaller time a closer look on this system
if it was this exactly is better then turning on 50 things and you will never find out what was it exactly
in your situation! -
I didn't notice any mention in your previous posts,
What hardware is this running on and are you using Nano or a full install, 32 or 63 bit? -
This is a 32-bit Intel system running a full install. 1 gig of RAM. The really weird thing is that the RRD graphs still show a completely steady low mbuf usage, much less than the limit set by kern.ipc.nmbufs.