Enabling CARP on pfSense ESXi VMs causes large amounts of promiscuous to pf VMs
Over and above the normal issues with ESXi and CARP, I was experiencing large amounts of promiscuous traffic showing up on pfsense VMs. This was causing a lot of overhead in my environment just to run CARP/promiscuous mode. I have since found a solution using a MAC learning VIB installed on each ESXi. I have linked my originally Reddit post below for others to search for.
Has anyone else experienced this problem?
My problem post from Reddit:
"I'm setting up CARP with two ESXi 5.5 host, with two pfsense VM, one on each host.
I've created a vDS Port Group specifically for pfSense. This port group has promiscuous mode, forged transmits, and MAC address changes enabled. I've also enabled Net.ReversePathFwdCheckPromisc on each host and rebooted.
When I enable CARP large amounts of promiscuous traffic is delivered to the pfsense VMs vNICs. For example on my storage network, random amounts of about 5-100mbps of traffic are consistently hitting its interface.
None of this traffic is routed through pfsense. Though when I run a packet capture, it shows all packets on the network like the capture is running in promiscuous mode. This is annoying because ntop and the bandwidth graphs log all this traffic even though its not being routed through pfSense. Not to mention the overhead of the traffic being delivered to pfsense.
When CARP is disabled, no traffic is seen on the packet capture or vNICs.
Is this correct behavior of the pfSense VM?
Environment and Detailed Issue
Here is my setup as follows
ESXi 5.5 Update 2 on both hosts, running a vDS. Specific vDS port groups created just for pfSense. pfSense VMs are 2.2.x.
Here some screenshots of the interface on my iSCSi network:
- With CARP IP deleted from pfSense traffic is as expected
-vCenter shows that vNIC with no packets (other than normal broadcast traffic) being presented to it
-Tcpdump/Packet capture (not in promiscuous mode) shows not traffic other than normal broadcast traffic
-Bandwidth graphs show not traffic
-State table shows nothing being routed on that interface
- With CARP IP added and active on pfSense, weird traffic shows up as if pfSense is doing a promiscuous packet capture
-vCenter shows that vNIC with a lot of traffic presented to it
-Tcpdump/Packet capture (not in promiscuous mode) shows iSCSi traffic
-Bandwidth graphs pickup and graph traffic
-State table still shows nothing being routed on that interface
The solution was provided by /u/Mr_That_Guy, this included installing a MAC learning VIB on the ESXi hosts. I no longer saw the issues with all traffic being presented to the pfSense VMs.
I have been struggling with the same problem recently in a similar setup, and I also have found the mentioned MAC learning filter VIB, but I wanted to ask few questions before I go for it.
- It is not clear to me, that after I install the VIB, on which vm network ports should I enable the dvfilter? I suppose those VM-ports have to be filter enabled which are connected to a port group with promiscuous mode enabled, right?
- Another point is, that this MAC filter never forgets. Once a MAC is stored in the MAC table, it remains there for good(!?), which a little bit worries me, since if the MAC table is full, the filter would no longer be capable to prevent the packet flood. But on the other side, I personally do not know how many MACs can be stored for a given port in the MAC table.
Do you happen to have any experience with that, or a workaround whether the MAC table can be emptied somehow manually once it gets full?