Mirrored traffic from Cluster Member 2 on cluster member 1
Have what I think is an odd issue, and my google kung fu is failing me today.
I have 4 firewalls sharing an interconnect network, setup in 2 clusters. One cluster is external/front end, the other is backend/private. the are using different vhid's on the shared subnet. The vlan on the DVS is set to promiscuous, and also has the advanced net.reverse… flags configured (also disabled/reenabled the port group security features after setting the flags). I've tried disabling the LAG group in favor of the 10G interfaces I have on each host, etc... all to no avail.
Here's the long and short.
I was working on cleaning up firewall logs so I only log "interesting" traffic, when I noticed something odd. on both external and internal active carp members, I see mirrored traffic from the backup carp firewalls. By that I mean, for example, I see all the ICMP echo-request and echo-replies (gateway monitoring) for the backup firewall on the primary's interface. I see the NTP and DNS requests that the secondary members are sourcing as well. Being a network analyst of some years, this strikes me as wrong, almost as if a SPAN port was configured.
Having had experience with VRRP/HSRP/GLBP/etc... I've never seen this behavior either. I've checked arp entries (which is how I found I had the external and internal clusters using the same vhid, which has been corrected)... tcpdumps show that it's not doing something funky like fw1 masquerading as fw2 (the traffic I'm seeing is in fact appearing as being from the mac address and IP of fw2...)
I guess I have a couple of question:
- is this normal/expected behavior? I can't imagine it is
- any suggestions on how I might resolve/pinpoint the issue?
Update. I've also tried vmotioning the guests to different hosts in the cluster, and the behavior is the same.
Anybody have any ideas? Or can anyone even confirm the behavior I'm seeing is expected? (i.e. tcpdump -i <int>host <carp-neighbor>– and look for ping/response to from that member on the primary firewall)
I've gone as far as to damn near tear out my entire networking stack in esxi... Still the problem persists. I've upgraded to the latest patch levels, VM hardware levels, etc... I've tried manually setting MAC Addresses. Nothing working. My next step is to try swapping E100E interfaces for VMXNET3 ones...
But I'd love to at least know if I'm completely wasting my time.
/ <------ \ -------- Traffic from INT Member 2 to EXT Gateway is seen on this interface
INT Member1 INT Member 2</carp-neighbor></int>