Anyway to use CARP on Vmware without putting vswitchs/VDS into promiscuous mode?

jedblack

Hi all,

I have two ESXi hosts running Vsphere 4.1 U2 and Vcenter 5.0 managing them

I have tons of VMs on each host and a pfsense instance on each host setup in a HA config.

everything works great except for CARP. Any machine outside of the ESXi hosts can use/ping the CARP interfaces. However, if its any VM that is on the ESXi hosts, they cannot use/ping the CARP IP's. The only way i can get a ping/echo is by setting "promiscuous mode" to on withing the virtual distributed switch(same as vswitchX) or the actual port groups.

Needless to say this is a no-no, since promiscuous mode creates a huge security hole….

is there any way to do it without enabling promiscous mode?

Thanks for all the help!

biggsy

I have no experience of using CARP but perhaps you could use PCI pass-through to provide dedicated NICs for CARP.

You would sacrifice a bit of functionality though - VM suspend and snapshots for starters.

biggsy

Using a VLAN might be another possibility for working around the promiscuity problem.

Hopefully someone with a better understanding of CARP can help.

jimp

You must use promiscuous mode on the vswitch in order to use CARP properly. Without that, they probably aren't seeing backup/master status properly either, they both probably believe they are master.

You might be able to get away with something like this:

One port group for the firewall interfaces on that segment with promisc on
All other nodes on that segment on a non-promisc port group

I haven't tried that, I've only seen it work with promisc on.

jedblack

@jimp:

You must use promiscuous mode on the vswitch in order to use CARP properly. Without that, they probably aren't seeing backup/master status properly either, they both probably believe they are master.

You might be able to get away with something like this:

One port group for the firewall interfaces on that segment with promisc on

All other nodes on that segment on a non-promisc port group

I haven't tried that, I've only seen it work with promisc on.

Jimp,

Sir, you are awesome….

It never even dawned on me to try this....

So, within a single VDS(virtual distributed switch)[which is just like a vswitch0/1/2..etc]. I crated two port groups.

1. infra-fe-PF
2. infra-fe

The vmnic interface(name is also "infra-fe") on the pfsense VM instance got assigned the 'infra-fe-PF' port group (this port group has promiscuous mode enabled)

All VM's that I want to using pfsense interface "infra-fe" as the gateway got assigned to the port group "infra-fe" (this port group has promiscuous mode disabled)

At first I was able to ping the carp interface and have external WAN access, GREAT/BINGO!! – then realized I could also ping other CARP interface and other subnets.... DAMN/NO-BINGO. However, looked at the FW rules and realized I had a stale ANY/ANY rule for the pfsense interface 'infra-fe' -- once I adjust the rule set to block most traffic and reloaded the FW state tables....then.... BINGO!, REALLY!!

TL:DR;
My external/physical machines can ping the CARP IP's, and now any of my VM's can ping/use the CARP IP's. You just have to make sure the VM's and the PFsense vmnic interfaces are on separate port groups within your VDS/Vswitch and "promiscuous mode" is enabled on any of the port groups that pfsense interfaces are assigned too. Make sure you have proper firewall rules in place for each pfsense interface....

jimp

Good to hear that worked, I added a note to the doc wiki for carp on esx that mentions it.

Taras_

Hi and thanks for info & discussion!

In fact, this configuration (2 interconnected ESXi hosts + 2 FreeBSDs with CARP) works even without VDS (ESXi 4.1). But there is one problem: packets sent to "internal" CARPed IP or other IPs on "internal" (VM's) interfaces of FreeBSD's are transmitted to both machines. I.e. we converted our switch into hub. It's problem if network bandwidth matters (I don't know about performance impact of this behavior in case of transmitting internally in one ESXi, but we use ESXi-ESXi HW network to send this packets). For external network it's not big problem as external HW switch are still works as switch.