MBUFs are not freing in build 09 sep

[wladikz]

Hi,

i installed new firewall with latest snapshot (09/09/2013). it installed on HP DL360 G6 with two dual port intel 10G cards (x520-DA2 and x540-T2) 8GB RAM and 2xquad core CPU. ports of each card are in LAGG. the firewall routes 40 subnets on 20 VLANs connected to X540 LAGG. firewall configured with 512000 MBUFs. 512000 MBUFs are filled in around 2 hours and after that firewall reboot itself with crash dump. I'll have access to firewall to get its configuration and any additional required info at Sunday. please help how to solve this issue. the firewall serve datacenter that will run 162 class C subnets over 132 VLANs in two weeks.

[tomj]

PfSense (all 2.x versions I have tested - both 32-&-64-Bit) MBUF only increases then PfSense quits working.
I found a fix (for how I use PfSense)

I have many PfSense boxes running on ESXi with the vmware-tools installed and using the ESXi network cards vxn0 and vxn1 and up…

What happens is that my MBUFs only go higher and never lower.  This can be verified using the following command in the Shell:
netstat -m | grep "mbuf clusters"

After more than a year of dozens of PfSense boxes constantly locking up every few days - I decided to find out what was going on and here is what I found...
The problem is either the PfSense package vmware-tools and/or the network-adaptor vxn0 driver provided with vmware tools.

For the fix - I changed back to the original em0 network adaptor under wmware ESXi and I removed the vmware-tools in PfSense.Now it works good and the MBUFs goes up and down for usage (instead of just growing only).
I suspect there is a bug somewhere in the vmware-tools package -or- the network adaptor driver for vxn0 nics in PfSense when running on ESXi.

North Idaho Tom

[jimp]

There is a known issue with the Intel 10Gbit NICs using the ixgbe driver, one of the problems with the cards is an mbuf leak. We're working on a fix for the driver already.

[wladikz]

@tomj:

PfSense (all 2.x versions I have tested - both 32-&-64-Bit) MBUF only increases then PfSense quits working.
I found a fix (for how I use PfSense)

I have many PfSense boxes running on ESXi with the vmware-tools installed and using the ESXi network cards vxn0 and vxn1 and up…

What happens is that my MBUFs only go higher and never lower.  This can be verified using the following command in the Shell:
netstat -m | grep "mbuf clusters"

After more than a year of dozens of PfSense boxes constantly locking up every few days - I decided to find out what was going on and here is what I found...
The problem is either the PfSense package vmware-tools and/or the network-adaptor vxn0 driver provided with vmware tools.

For the fix - I changed back to the original em0 network adaptor under wmware ESXi and I removed the vmware-tools in PfSense.Now it works good and the MBUFs goes up and down for usage (instead of just growing only).
I suspect there is a bug somewhere in the vmware-tools package -or- the network adaptor driver for vxn0 nics in PfSense when running on ESXi.

North Idaho Tom

i use pfsense 2.1 on esxi 5.5 with vxnet3 adapters without any problems. you should use vmware original tools and not pfsense provided. uptime for my firewall cluster is 3 month for now.

[MillerDan]

This seems to be fixed with 2.1.1 and 2.1.2 running on ESXi 5.1 with the vmxnet3 drivers loaded from the vmware tools installed from the host.

[burnsl]

I experienced this today across 3 complete re-installs using the following setup:
mBUFF filled to 99% and stopped working or outright crashed

Physical Server Hardware (ESXi Host):

• HP DL360 G4p

• 12GB RAM

• Dual 72Gb U320 drives in RAID 0+1

• 2 tgz3 NICs onboard the server

• 2 INTEL Dual GB 82546GB NICs installed - (bringing total interfaces to 6.)

ESXi Host :

• 4.1.0 u1 (fully patched) - BUILD: 1682698

Guest OS Configuration for PFsense 2.1.4 i386:

PF NIC:                    ESXi NIC:
0: WAN1  DHCP  –------->  ESXi_NIC1
1: LAN  192.168.1.1  -->  ESXi_NIC2
2: WAP  192.168.2.1  -->  ESXi_NIC3
3: DMZ  192.168.3.1  -->  ESXi_NIC4
4: WAN2  PPPoE  -------->  ESXi_NIC5
5: LAN  192.168.5.1  -->  ESXi_NIC6
6: PFL  192.168.6.1  -->  ESXi_BLIND_SWITCH (PFlink to other PFsense FW VM on SAME ESXi host)

Using official VMware Tools drivers and install.
(NOT Open Vmware Tools Driver Package)

This guest OS continuously has driver issues or something because i cannot keep the guest running correctly.
I lose network connectivity constantly and/or the PFsense firewall hangs.