MBUF spike



  • Hi all,

    Currently running :
    2.1.4-RELEASE (i386)
    built on Fri Jun 20 12:59:29 EDT 2014

    Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
    8 CPUs: 2 package(s) x 4 core(s)

    Yesterday, my system had a couple of massive spikes in MBUF usage. These spikes were bad enough to disconnect clients, and caused all kinds of hate and discontent to be aimed at me….it of course happened at a pretty critical time.....please see attached JPG's for the RRD graph.

    Any advise as to how to track down what caused these spikes ??





  • Netgate Administrator

    So you hit the limit at 25,600?
    How much ram do you have? What NICs are you running? You could simply increase the mbuf limit to make sure it doesn't happen again, or as easily at least.

    Steve



  • Currently running 2 bce (Broadcom) and 1 em0 (Intel) total of three NIC's

    Mem: 315M Active, 2270M Inact, 257M Wired, 60M Cache, 112M Buf, 344M Free

    I looked at the NIC tuning page but wasn't sure If I needed to set the settings "per card" ??

    ie:
    kern.ipc.nmbclusters="131072"
    hw.bce.tso_enable=0
    hw.pci.enable_msix=0

    Does the following change affect the bce driver (so as to have the setting applied to both bce cards)?
    or do I need to add specific card info ie: hw.bce0.tso_enable=0  and    hw.bce1.tso_enable=0

    and are the nmbclusters applied "system wide" or would I need to add a setting for the Intel NIC as well. ?

    kern.ipc.nmbclusters="131072"
    hw.igb.num_queues=1

    Thanks for any advise….

    -S



  • kern.ipc.nmbclusters="131072"

    is a global setting. This can be set at boot time in /boot/loader.conf as well as being set at runtime with the command:

    sysctl -w kern.ipc.nmbclusters="131072"

    This is the only setting that would affect your problem. The other two may or may not cause your system to behave strangely (ie hang).

    But, your processor indicates that you are using a dual socket Xeon motherboard, yet you only have approximately 4GB of RAM. Why not just buy more ram so you can increase the buffers so that you never experience this problem again?



  • Thanks, I already decided to just go with the

    kern.ipc.nmbclusters="131072"

    line in the  /boot/loader.conf.local

    Mem: 128M Active, 27M Inact, 133M Wired, 164K Cache, 75M Buf, 2958M Free

    Still almost 3G Free.

    Will more RAM have any effect on (i386) ???


  • Netgate Administrator

    Those settings are not per card, you only need one entry.
    nmbclusters is a single setting for the system so only one entry for that also.

    The only setting that might affect MBUFs there is that for Intel nmb_queues which for your card would be:
    hw.em.num_queues=1

    The only time this usually seems to give problems is if you have, as it says in the docs, multiple em or igb NICs. Odd to see it with one, perhaps it's something different entirely. I suggest you just add the line to increase the available nmbclusters and see what happens.

    Typed too slow! Posting anyway.  ;) 4GB shouldn't limit you here. Any reason you're running 32bit?

    Steve



  • "Any reason you're running 32bit?"

    It's what I upgraded from.

    Earlier this year I migrated my 1.2.3 install over to new hardware, then upgraded to 2.1.3

    I believe that is was highly discouraged to try and upgrade from 32 to 64 ??

    Either way….for the most part, I'm happy as can be with the 32bit version.  :)


  • Netgate Administrator

    Fair enough. As you can see 4GB is way more than pfSense uses without any packages running. I have a test box running 64bit that I upgraded from 32bit, the process was painless. You should check the firmware update location is set correctly if you do try this. I would probably go for a full re-install on a production box to be safe. There have been several hints from the devs that 32bit will eventually be phased out so that's one reason to be running 64bit. I hope it's not for a while though since I have several boxes that aren't 64bit.  ;)

    Steve