Simple CARP VIP doesnt work in sphere - promiscuous mode doent help - help!
-
LAN side CARP VIPS dont work at all in pfsense and vsphere, but work on physical swtiches, help!
In production, we have a 3 tier architecture, with 3 firewalls:
topFW (pfsense)
:
web servers
:
App FW (pfsense)
:
tomcat etc.
:
DB FW (pfsense)
:We have virtualised this into vsphere so we can perform testing and staging. Between each FW is a vswitch.
Only the top FW is connected to a physical switch. The other 2 switches (app and DB layer) are just vswitches, not connected to any physical swtich except through pfsense firewalls (same as production)
The firewalls are just pfsense virtual appliances with haproxy module.
In pfsense, we defined CARPed VIPS, the same as in production. HAProxy then load banances these on the LAN or WAN depending on the FW. We dont have a backup FW, so there is no CARP slave, only master (HAProxy needs CARP VIPs)
In the top FW, none of the external (WAN) VIPS worked (you could not reach them from the WAN like you should be able to), untill we changed "promiscuous mode" to Accept in the top vswitch Security tab in the vsphere admin app. Then the VIPS worked. Problem solved we thought.
The problem is the app layer FW also has LAN VIPS for app layer services which are load balacned with haproxy. These do not work - they are unreachable (connection times out) for the hosts on the same (LAN) network. It is the same problem as we had in the top FW. however, when we enable permiscuous mode on these switches, it gives you a warning that there is no physical adapter attached (which is correct), and doesnt work. However, you can ping them - but you cant hit them on the proxied ports.
The FW rules are set to all all traffic on all ports on all protocols on all interfaces, and has been treble checked.
So we are stuck - if vsphere doesnt support basic VIPs on a device connected to their vswitch, we cant use it for any of our apps.
Is the problem that we have to manually add additional IPs in shell in BSD (multihome?). does anyone know how to do this (we know nothing about BSD, only solaris).
Any ideas?
-
Well this is embarassing. I had not enabled haproxy.
But the VIPS still do not work on the LAN (which is where they are defined), the only work on the WAN!
I cant even ping the VIPs on the LAN now.
any ideas?
-
Check the doc wiki. There are several things you need to do on vswitch/VDS setups for ESX to make CARP work properly.
-
I have read all the wiki and docs on carp. There are a lot of posts about getting carp replication in esxi to work, but we only have one pfsense box and one switch so none of that is relevant, unfortunately. It looks like it is just not possible to use pfsense CARPs on a LAN in vsphere 5.
-
They work fine, you still need to use the same settings/workarounds regardless of whether or not you have a cluster. It's all still relevant.
We have many, many customers using CARP in ESX, and they are working fine, so long as you have the vswitch/vds/portgroup settings right as documented on the wiki.
-
@ace:
It looks like it is just not possible to use pfsense CARPs on a LAN in vsphere 5.
If that were true, this website wouldn't work, amongst a ton of other production systems. This site and all our sites are on a CARP IP on VMs in vsphere 5.
info here on the ESX settings that will break multicast or multiple MACs on a single VM (and hence break CARP):
http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#VMware_ESX.2FESXi_Users
