Need a static route to take precedence over the NAT and NOT go to the default WAN-GW? Possible?

[steve1399]

I run a nested lab under my desk with two physical hosts. Both hosts connect to a netgear switch before heading to the corporate network. Each host is considered a site. Both sites use PFsense appliance to NAT traffic out to the internet. This is all working well, until I needed to create a couple routed networks that connect the two sides. Starting with the management network. I tried to do the following.

Mgmt = Site1 - 192.168.1.0/24 and Site2 - 192.168.2.0/24.

Site1 PFSense
WAN 10.33.72.65/22
LAN 192.168.1.1/24

Site2 PFsense
WAN 10.33.72.66/22
LAN 192.168.2.1/24

WAN_GW=10.33.75.253/22

On Site1 I added a GW (the other corp WAN interface on Site2)

Added a Static route

Show’s up in the routing table.

It’s still trying to go to the gateway first…

Is there a way to do this where the static route takes precedence over the NAT and doesn't need to go to the WAN default GW?

[dotdash]

You've probably got a rule to NAT anything going out the WAN. Turn on AON and put in a no nat rule for traffic between the subnets. Checking the box to bypass rules for static routes may fix it too, I forget.

[steve1399]

I've switched over to AON NAT now, I didn't originally see all of the rules. Turned out that I didn't have an upstream gateway set. Once Set I see the rules automatically generated. I added my rule to disable NAT to create a routed network from my two servers.

I still see an issue however. Can anyone suggest a fix or maybe a better way to achieve what I'm trying to do? I'll try and detail what the issue is...

So it seems like it's an L2 issue...

Here are the test details... Ping from Site1 Nested ESXi VM fails...

The path is: Request: "n-esx1" -> "p-esx1" -> "p-esx2" -> "n-esx7" Reply: "n-esx7" -> "p-esx2" -> dfGW.... where I need it to go back to "p-esx1"
n=Nested
p=Physical

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

For reference...
Site1 WAN: 60:ac:a6
Site2 WAN: 7b:81:88
WAN GW: ff:fd:90

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

Destination Nested ESXi sees the request and replies to the request.

[root@stevelab-n-esx7:/vmfs/volumes/3a3b5bc8-88bd4760] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:23:35.839598 00:0c:29:7b:81:9c > 00:0c:29:35:9b:0d, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 0, length 64
22:23:35.839758 00:0c:29:35:9b:0d > 00:0c:29:7b:81:9c, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 0, length 64

Then the Physical host is sending the reply to the WAN GW. It doesn't send it back from where it came...

[root@stevelab-p-esx2:/vmfs/volumes] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr –
22:24:09.239994 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 18, length 64
22:24:09.240589 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 18, length 64
22:24:10.241698 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 19, length 64
22:24:10.242331 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 19, length 64

[root@stevelab-p-esx2:/vmfs/volumes] esxcli network ip neighbor list
Neighbor Mac Address Vmknic Expiry State Type
10.33.72.65 00:0c:29:60:ac:a6 vmk0 928 sec Unknown
10.33.72.64 00:0c:29:8c:15:73 vmk0 1196 sec Unknown
10.33.72.62 00:11:32:a6:9a:3f vmk0 1020 sec Unknown
10.33.75.253 00:08:e3:ff:fd:90 vmk0 1198 sec Unknown

Reply never arrives back at Site1 (Of course, because the packet went to the WAN GW of Site2.

[root@stevelab-p-esx1:~] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:24:24.270369 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 33, length 64
22:24:25.271133 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 34, length 64
22:24:26.271396 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 35, length 64

Willing to try just about anything...

Thanks.