MBUF usage
-
Hello,
I'm seeing on the WebGUI MBUF Usage: 18006/25600, this machine is in stand-by, there is just 131 State entries and there is no network traffic. Why is that so high while in others pfsense with 4297 State entries(don't know if it is related), MBUF usage:2106/25600.
I'm paranoid because in another implementation with same NIC there have been some weird problems like "No Buffer Space Available" while there are plenty mbuf space when I tried to ping some domains and unfortunatly I had to switch back to the old firewall.Thanks!lspci -lv
igb0@pci0:4:0:0: class=0x020000 card=0xa04c8086 chip=0x10c98086 rev=0x01 hdr=0x00
class = network
subclass = ethernet
igb1@pci0:4:0:1: class=0x020000 card=0xa04c8086 chip=0x10c98086 rev=0x01 hdr=0x00
class = network
subclass = ethernet
bce0@pci0:2:0:0: class=0x020000 card=0x02a31028 chip=0x163b14e4 rev=0x20 hdr=0x00
class = network
subclass = ethernet
bce1@pci0:2:0:1: class=0x020000 card=0x02a31028 chip=0x163b14e4 rev=0x20 hdr=0x00
class = network
subclass = ethernetnetstat -m
16328/1851/18179 mbufs in use (current/cache/total)
16327/1695/18022/25600 mbuf clusters in use (current/cache/total/max)
16326/954 mbuf+clusters out of packet secondary zone in use (current/cache)
0/104/104/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
40818K/4731K/45549K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines -
There is no direct correlation between mbufs, states, and so on.
Certain NICs or configurations will use more mbufs than others.
Just crank up the nmbclusters value and it'll be fine.
http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
If you run out of mbufs, the system will panic+reboot, not just toss an error, so that isn't likely to be related. It can be, but it's not necessarily related.
-
When I first started dealing with a similar issue I found that pfsense tuning page thanks to the forum. I was able to get it all working with that tuning page.
I did some searching a few months ago to find out why the kernel/driver doesn't self adjust itself to an appropriate nmbcluster setting or queue setting and I found an insightful email discussion explaining the issue from what seems like freebsd igb driver developers. Having the driver automatically adjust itself to handle more situations was discussed but I didn't see if a solution was planned or not. Does anyone else know?
http://lists.freebsd.org/pipermail/freebsd-stable/2012-February/066372.html
The total used seems to depend on how many cores are enabled which influences how many queues the network card uses which each use a certain amount of nmbclusters per queue.
I had an 8 core system that would panic because of the issue. After finding out why it happened I was rather surprised at the time that a driver would allow that to happen.
Another thread I found discussed queues and mentioned that since pf is single threaded that having more than 1 or 2 queues when using pfsense can actually cause problems if you are using pf a lot because all the packets using pf are funneled through a single pf thread. This also surprised me a lot coming from Linux. I assumed it was multi-threaded so that multiple irqs could handle pf code on different processors concurrently. All OSs have their issues though. I haven't experimented with the queues enough to know either way. This is just what I have read. I just set it to 1 for now but plan to increase it to at least 2.
I am relatively new to freebsd though so I probably should stay quieter than I am :).
Problems like this seem to make me wonder about freebsd at times. pfSense is still my favorite firewall though :).
-
Some more intelligent tuning code recently went into FreeBSD 10-CURRENT
-
Sounds like the next version after the next upcoming release of 2.1 (pfsense 2.2) will be a nice upgrade eventually.
The thing that scared me the most when I did my searching on the mbuf and single threaded pf was the statement from the freebsd wiki…
https://wiki.freebsd.org/NetworkPerformanceTuning
Under the Firewalls section...
PF: Simply say NO (2012-02-23). Global lock kills all performanceI remember reading that freebsd 10 has some improvements of that though.
-
Yes, 10.x has an SMP-capable pf.
