Multi-WAN, IP alias, and indbound connections…

[echoranger]

So I'm starting to tear my hair out wondering why this does not work, hoping someone here can help.

I have 2 pfSense firewalls in an active/passive configuration via CARP with 2 WAN connections, both on the same physical interface (both pfSense boxes and both WAN connections are all attached to a single switch). The primary connection is Comcast on 50.243.x.144/28 and the other connection is Level3 on 4.35.x.128/26. As these are both on the same physical interface I used an IP Alias on each pfSense box for this Level3 connection.

As we are migrating from Comcast to Level3 in a phased approach I'm slowly moving various 1-1 NATs to the Level3 IP space and have noticed that inbound connections to anything on the Level3 space is not working. Outbound works just fine– once I move a NAT from the Comcast space to the Level3 space I can see traffic going out the new Level3 NAT just fine. Anything inbound to that same IP however simply times-out.

I've been running tcpdump on both the LAN and WAN interfaces on pfSense and when I initiate a connection from the outside to one of these 1-1 Level3 NATs I do see the traffic come into pfSense, pfSense passes it to the LAN IP of the host in question, the host does indeed reply, and I even see the response on the WAN interface of pfSense with the correct Level3 source IP. This traffic never gets to its destination.

In doing further research I'm even unable to ping pfSense's WAN CARP address or either WAN IP alias dedicated to each pfSense box. Again I see the traffic hit the WAN interface and it generate a reply which never gets to its destination.

On the flip side, inbound connectivity to the Comcast IPs on either box respond normally. I don't believe routing to be the issue as I can generate traffic out from my LAN and it gets where it needs to go on Level3, its just responses to indbound traffic that seem to disappear.

Here's a sample tcpdump to show what I'm seeing:

Ping from my home server to the primary pfSense WAN IP on Level3 (again using an IP Alias on pfSense here, the .131 address).

tcpdump on my home server (initiating the connection):
21:39:52.399237 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 669, length 64
21:39:53.409291 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 670, length 64
21:39:54.419679 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 671, length 64

tcpdump on the WAN interface of the primary pfSense box:
21:39:52.426579 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 669, length 64
21:39:52.426609 IP 4.35.x.131 > 173.14.x.129: ICMP echo reply, id 42267, seq 669, length 64
21:39:53.436994 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 670, length 64
21:39:53.437034 IP 4.35.x.131 > 173.14.x.129: ICMP echo reply, id 42267, seq 670, length 64
21:39:54.445073 IP 173.14.x.129 > 4.35.x.131: ICMP echo request, id 42267, seq 671, length 64
21:39:54.445104 IP 4.35.x.131 > 173.14.x.129: ICMP echo reply, id 42267, seq 671, length 64

As you can see, pfSense does see the traffic and generate an echo-reply but that never gets back to the originating host.

Alternatively, when I do the ping in the other direction everything looks fine...

tcpdump on my home server when I ping it from pfSense:
21:42:08.682057 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 0, length 64
21:42:08.682438 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 0, length 64
21:42:09.684646 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 1, length 64
21:42:09.684742 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 1, length 64
21:42:10.683965 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 2, length 64
21:42:10.684062 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 2, length 64

and the tcpdump on pfsense:
21:42:08.665361 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 0, length 64
21:42:08.711274 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 0, length 64
21:42:09.666727 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 1, length 64
21:42:09.711784 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 1, length 64
21:42:10.667699 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 2, length 64
21:42:10.710260 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 2, length 64
21:42:11.668680 IP 4.35.x.131 > 173.14.x.129: ICMP echo request, id 55871, seq 3, length 64
21:42:11.711277 IP 173.14.x.129 > 4.35.x.131: ICMP echo reply, id 55871, seq 3, length 64

I believe I'm supposed to use an IP Alias for this as its the only way I know to set 2 IP addresses on a single physical interface. Should I be using something else? As I said, pings to CARP addresses seem to be the same way (they work to the Comcast WAN address and CARP, but not to the Level3 ones).

I suspect I could try reversing the interfaces-- move the Comcast IPs to the IP Aliases on each pfSense and use Level3 as the primary interface IP, but this would potentially break stuff on our Comcast connection and makes a phased approach almost a non-starter. I also am not planning to do any load-balancing or anything of that nature, I'm just trying to transition from Comcast to our Level3 connection.

Help?

[echoranger]

Just in case anyone else gets bitten by this…

My solution was to move the 2nd WAN to its own VLAN and create another interface on pfSense to handle it. Now both WANs coexist in harmony.