Outbound NAT to Virt. IP issues. Maybe it's the config, maybe it's VMWare ESXi?

[jakbeatz]

I have an ESXi server installed with the 1.2.1-RC2 VM up and running.  Everything was working as expected, but then I tried to setup outbound NAT to a virtual IP and everything stopped:

I've configured a Virtual IP on the WAN side which is on the same subnet as the WAN interface itself.  I have an outbound NAT rule set up to nat all outbound connections to the Virtual IP.  I also have the outbound NAT set for Manual Outbound NAT rule generation (Advanced Outbound NAT (AON)).

From the WAN side, I see the MAC for both the virtual IP and the physical WAN interface IP but I can't ping the Virtual IP however I can ping the physical WAN interface IP, no problem.  As soon as I set outbound NAT to Automatic Outbound NAT rule generation, traffic works again (albeit I still can't ping the virtual IP, but at that point, it's moot).

I checked the pfSense firewall rules and verified that it's configured to pass ICMP from any to any on the WAN interface and the LAN interface has a rule to allow IP from any to any, so by all accounts this should be working.

I'm not sure if it's something in pfSense that I'm doing wrong, or if it's a VMWare issue.  The fact that I can see the MAC Address on the WAN side seems to indicate that ESXi is doing what it's supposed to.  I haven't seen any indication that ESXi doesn't want to pass traffic for a virtual MAC address while I've been looking over it's configuration, so I'm at a loss and I'm wondering if anyone has any insight.

Just for completeness, here's the ARP table from a 3550 I have on the WAN side to verify it sees the MAC address and ARP, etc.  I've also included the ifconfig from the pfSense shell.

switch>show arp | i Vlan5
Internet  aaa.bbb.ccc.215          -  000b.5f33.6100  ARPA  Vlan5
Internet  aaa.bbb.ccc.209          0  0013.5f1e.93c0  ARPA  Vlan5
Internet  aaa.bbb.ccc.211          16  000c.291b.3c6f  ARPA  Vlan5
Internet  aaa.bbb.ccc.210          17  0000.5e00.0101  ARPA  Vlan5

switch>show mac-address-table | i Fa0/1
  5    0000.5e00.0101    DYNAMIC    Fa0/1
  5    000c.291b.3c6f    DYNAMIC    Fa0/1

.215 is the 3550 I'm using to verify the WAN side.
.209 is the default gateway for the pfSense box that leads to the intermaweb.
.210 is the virtual IP.
.211 is the physical IP.

switch>ping aaa.bbb.ccc.209

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to aaa.bbb.ccc.209, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
switch>ping aaa.bbb.ccc.211

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to aaa.bbb.ccc.211, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
switch>ping aaa.bbb.ccc.210

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to aaa.bbb.ccc.210, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
switch>

ifconfig

le0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
  options=8 <vlan_mtu>ether 00:0c:29:1b:3c:65
  inet 10.1.11.1 netmask 0xffffff00 broadcast 10.1.11.255
  inet6 fe80::20c:29ff:fe1b:3c65%le0 prefixlen 64 scopeid 0x1
  media: Ethernet autoselect
  status: active
le1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
  options=8 <vlan_mtu>ether 00:0c:29:1b:3c:6f
  inet6 fe80::20c:29ff:fe1b:3c6f%le1 prefixlen 64 scopeid 0x2
  inet aaa.bbb.ccc.211 netmask 0xfffffff0 broadcast aaa.bbb.ccc.223
  media: Ethernet autoselect
  status: active
le2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
  options=8 <vlan_mtu>ether 00:0c:29:1b:3c:79
  inet 10.255.255.1 netmask 0xffffff00 broadcast 10.255.255.255
  inet6 fe80::20c:29ff:fe1b:3c79%le2 prefixlen 64 scopeid 0x3
  media: Ethernet autoselect
  status: active
plip0: flags=108810 <pointopoint,simplex,multicast,needsgiant>metric 0 mtu 1500
pfsync0: flags=41 <up,running>metric 0 mtu 1460
  pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
  inet 127.0.0.1 netmask 0xff000000
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33204
tun0: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
  inet6 fe80::20c:29ff:fe1b:3c65%tun0 prefixlen 64 scopeid 0x9
  inet 192.0.2.1 --> 192.0.2.2 netmask 0xffffffff
  Opened by PID 334
carp0: flags=49 <up,loopback,running>metric 0 mtu 1500
  inet aaa.bbb.ccc.210 netmask 0xfffffff0
  carp: MASTER vhid 1 advbase 1 advskew 0

Thanks in advance!</up,loopback,running></up,pointopoint,running,multicast></promisc></up,loopback,running,multicast></up,running></pointopoint,simplex,multicast,needsgiant></vlan_mtu></up,broadcast,running,simplex,multicast></vlan_mtu></up,broadcast,running,promisc,simplex,multicast></vlan_mtu></up,broadcast,running,simplex,multicast>

[jakbeatz]

After doing some more digging, I figured it out.  It was a VMWare thing. I had to set the virtual adapter with a security policy exception to allow promiscuous mode.

There seems to be another issue though - it seems as though there is another client out there on the WAN (albeit, on a different VLAN) using a pfSense box, because I see the same MAC address as what my pfSense box is using for my CARP MAC Address.

Is there a way to change the CARP MAC address so I can differentiate my MAC address from this other person's?