Need a static route to take precedence over the NAT and NOT go to the default WAN-GW? Possible?



  • I run a nested lab under my desk with two physical hosts. Both hosts connect to a netgear switch before heading to the corporate network. Each host is considered a site. Both sites use PFsense appliance to NAT traffic out to the internet. This is all working well, until I needed to create a couple routed networks that connect the two sides. Starting with the management network. I tried to do the following.

    Mgmt = Site1 - 192.168.1.0/24 and Site2 - 192.168.2.0/24.

    Site1 PFSense
    WAN 10.33.72.65/22
    LAN 192.168.1.1/24

    Site2 PFsense
    WAN 10.33.72.66/22
    LAN 192.168.2.1/24

    WAN_GW=10.33.75.253/22

    On Site1 I added a GW (the other corp WAN interface on Site2)
    Screen Shot 2020-02-25 at 2.26.28 PM.png
    Added a Static route
    Screen Shot 2020-02-25 at 2.26.37 PM.png

    Show’s up in the routing table.
    Screen Shot 2020-02-25 at 2.25.36 PM.png

    It’s still trying to go to the gateway first…
    Screen Shot 2020-02-26 at 10.22.09 AM.png

    Is there a way to do this where the static route takes precedence over the NAT and doesn't need to go to the WAN default GW?



  • You've probably got a rule to NAT anything going out the WAN. Turn on AON and put in a no nat rule for traffic between the subnets. Checking the box to bypass rules for static routes may fix it too, I forget.



  • I've switched over to AON NAT now, I didn't originally see all of the rules. Turned out that I didn't have an upstream gateway set. Once Set I see the rules automatically generated. I added my rule to disable NAT to create a routed network from my two servers.

    I still see an issue however. Can anyone suggest a fix or maybe a better way to achieve what I'm trying to do? I'll try and detail what the issue is...

    Screen Shot 2020-03-25 at 3.41.24 PM.png

    So it seems like it's an L2 issue...

    Here are the test details... Ping from Site1 Nested ESXi VM fails...

    The path is: Request: "n-esx1" -> "p-esx1" -> "p-esx2" -> "n-esx7" Reply: "n-esx7" -> "p-esx2" -> dfGW.... where I need it to go back to "p-esx1"
    n=Nested
    p=Physical

    [root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
    PING 192.168.2.17 (192.168.2.17): 56 data bytes

    For reference...
    Site1 WAN: 60:ac:a6
    Site2 WAN: 7b:81:88
    WAN GW: ff:fd:90

    [root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
    PING 192.168.2.17 (192.168.2.17): 56 data bytes

    Destination Nested ESXi sees the request and replies to the request.

    [root@stevelab-n-esx7:/vmfs/volumes/3a3b5bc8-88bd4760] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
    22:23:35.839598 00:0c:29:7b:81:9c > 00:0c:29:35:9b:0d, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 0, length 64
    22:23:35.839758 00:0c:29:35:9b:0d > 00:0c:29:7b:81:9c, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 0, length 64

    Then the Physical host is sending the reply to the WAN GW. It doesn't send it back from where it came...

    [root@stevelab-p-esx2:/vmfs/volumes] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr –
    22:24:09.239994 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 18, length 64
    22:24:09.240589 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 18, length 64
    22:24:10.241698 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 19, length 64
    22:24:10.242331 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 19, length 64

    [root@stevelab-p-esx2:/vmfs/volumes] esxcli network ip neighbor list
    Neighbor Mac Address Vmknic Expiry State Type
    10.33.72.65 00:0c:29:60:ac:a6 vmk0 928 sec Unknown
    10.33.72.64 00:0c:29:8c:15:73 vmk0 1196 sec Unknown
    10.33.72.62 00:11:32:a6:9a:3f vmk0 1020 sec Unknown
    10.33.75.253 00:08:e3:ff:fd:90 vmk0 1198 sec Unknown

    Reply never arrives back at Site1 (Of course, because the packet went to the WAN GW of Site2.

    [root@stevelab-p-esx1:~] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
    22:24:24.270369 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 33, length 64
    22:24:25.271133 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 34, length 64
    22:24:26.271396 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 35, length 64

    Willing to try just about anything...

    Thanks.


Log in to reply