Carp IPs on both PFSense stay master - 2.0 Beta4, 27 oct build on xenserver
I'm in need of updating a setup that works under VMWare to run under XenServer 5.6 (all other servers migrated, all that's left is the firewall infrastructure). Installation worked perfectly, started with 1.2.3, and started replicating the config manually (I don't need the exact same config)
This is a multi-wan setup, so I want to use CARP addresses on lan and both wans for NAT, so that the sync interface can work it's magic for failover if one firewall goes down. The config works, and the synchronization too, but all CARP IPs on both hosts stay as Master. They do seem to work, and I can close down a firewall and the other takes over the traffic and do it the other way around, but I'm not sure this is expected behaviour.
I thought at first XenServer might not permit multicast, but tcpdump does show traffic, albeit weird to my eyes:
On host 1:
tcpdump -i re3
20:45:54.429546 IP 172.30.43.2 > 18.104.22.168: pfsync 196
20:45:54.429690 IP 172.30.43.1 > 172.30.43.2: pfsync 372
20:45:54.429726 IP 172.30.43.1 > 172.30.43.2: pfsync 52
20:45:54.430494 IP 172.30.43.2 > 22.214.171.124: pfsync 500
20:45:54.631724 IP 172.30.43.1 > 172.30.43.2: pfsync 108
20:45:54.731767 IP 172.30.43.1 > 172.30.43.2: pfsync 260
20:45:54.732297 IP 172.30.43.1 > 172.30.43.2: pfsync 108
20:45:54.751935 IP 172.30.43.1 > 172.30.43.2: pfsync 260
20:45:54.752362 IP 172.30.43.1 > 172.30.43.2: pfsync 108
20:45:54.752617 IP 172.30.43.1 > 172.30.43.2: pfsync 260
On host 2:
tcpdump -i re3
20:48:18.326372 IP 172.30.43.1 > 172.30.43.2: pfsync 284
20:48:18.748090 IP 172.30.43.2 > 126.96.36.199: pfsync 372
20:48:18.850681 IP 172.30.43.2 > 188.8.131.52: pfsync 260
20:48:18.850954 IP 172.30.43.2 > 184.108.40.206: pfsync 108
20:48:18.870663 IP 172.30.43.2 > 220.127.116.11: pfsync 260
20:48:18.871270 IP 172.30.43.2 > 18.104.22.168: pfsync 196
20:48:18.897234 IP 172.30.43.2 > 22.214.171.124: pfsync 260
20:48:19.327332 IP 172.30.43.1 > 172.30.43.2: pfsync 284
20:48:19.897083 IP 172.30.43.2 > 126.96.36.199: pfsync 460
20:48:20.260541 IP 172.30.43.2 > 188.8.131.52: pfsync 284
I know that multicat is sent to 184.108.40.206, and replies seem to get sent between the ips, but the master is the .1, slave .2.
The two XenServers are connected directly with a cable, and no other VMs share that network.
I am not well versed enough on BSD and carp to diagnose this further, and no posts I found on xen or xenserver mentions CARP not working, so I am looking for well versed pfSense users or devs to look if this rings any bells for them!
Thanks for your time!
More details as I tested a bit more and found some more info.
I figured that maybe all packets were not sent back to the VM even if it entered promiscuous mode, since the VIF and PIF in Xen were not set up that way by default. So I found an article explaining how to do it, and set the other-config:promiscuous="true" settings on both pifs and vifs, which didn't work. I see with tcpdump VRRP traffic being sent, from both VMs, but not being received by the other.
A post on the xenserver forums stated that they had problems with promiscuous settings on XenServer and it worked once they set the promiscuous="on" instead of "true". Which I tried, to no avail.
Anyone using CARP on XenServer 5.5 or 5.6 care to tell me if they had to play around with VIF or PIF settings to make CARP work?
I have the same problem. My enviroment is pfSense 1.2.3 running inside a XenServer 5.6 + pfSense 1.2.3 running on a physical machine.
From my tests, I think the problem is bridging in XenServer not able to forward multicast traffic. Here is what i did:
1 - In XenCenter, click on you VM, Network tab. Take note of the Device column and the related Network.
2 - SSH to your XenServer
3 - xe vm-list name-label= <name-of-your-vm>params=dom-id
Now, you have the running ID of your VM.
4 - brctl show
You will be able to see that the interfaces of you VM (vif<id>.Device) are attached to different bridges.
5 - Do tcpdumps
Now, do a tcpdump on the eth interface attached. You will be able to see multicast traffic between your 2 pfSense nodes.
Do a tcpdump on the bridge (xapi<n>). You won't be able to see the multicast traffic from one of the peers.
Do a tcpdump on the vif<id>.Device. Also, you won't be able to see the multicast traffic from one of the peers. This is expected as the multicast traffic even comes to bridge interface.
I wasn't able to solve this problem, but I think it is a bit closer to the real problem. Please, repeat the test I wrote and post the results.
I have the same problem, too. The problem is related to the MAC address which CARP/VRRP uses. It is not exactly a multicast MAC Address, but a special class of MAC designated to CARP/VRRP implementations.
Packets from Windows NLB Ips works properly, because they are going to really multicast MAC Addresses.
I guess the Linux Bridge don´t manage CARP packets properly.
I´m thinking in migrate my virtualization servers to VMWare, where I need this feature.