Routing + Virtual Networks = Headaches



  • Hi all.

    I'm having some issues with my current setup. Now, I'll be straight up, I may be doing something completely wrong here, networking is something I am trying to improve on. My issue is quite complicated but I will do my best to explain.

    Summary:

    I have a physical network. It has a pfSense physical box. I have an ESX server with what I would like to be several different vSwitches representing 'fake' sites, like London, New York etc. I want to be able to route between my LAN and my virtual networks.

    The long story…

    The issues I'm having are between my physical home LAN (with physical pfSense router) and a VMware ESX box.

    So basically, I have four pNICs in my ESX host:

    • Vmnic0 – Dedicated management network.
      Vmnic1 – Meant to be for linking my VM network to my physical LAN (this is the one playing up I think)
      Vmnic2 – Dedicated iSCSI (works well)
      Vmnic3 – Dedicated iSCSI (works well)

    So my physical LAN is 10.0.0.0/24. I have everything connected via a managed gigabit switch. iSCSI ports are designated (no VLAN but the pNICs connected for iSCSI are on a separate subnet and have jumbo frames).

    In ESX, I have some vSwitches for some made up sites (AD primarily) I use. London and New York. These vSwitches do not have any pNICs attached. I then also have a vSwitch called External which has a pNIC connected which is plugged directly to my switch.

    In ESX, I also have a virtual router running pfSense. I have 3 vNICs attached to the server, one in the ‘External’ vSwitch and one in each fake network vSwitch. The interfaces are:

    LAN – 10.0.0.60/24
    LON – 172.16.0.254/16
    NYC – 172.17.0.254/16

    On my physical LAN, I have a hardware firewall/router, which also runs pfSense and handles my internet connection and stuff. Here, I have a static route to both of the 172 networks, this allows me to RDP/PING etc the VMs in the fake sites on my ESX host.

    I can RDP to the servers, but they drop out and re-connect all the time, like every 15 seconds. If I ping –t I see the network reply a bunch of times, then it drops for a while. This happens when pinging and RDP’ing to any of the fake networks.

    Nothing seems to be working. Have I done something fundamentally wrong with the networking in pfSense? Perhaps I have overcomplicated things and there may be an easier way to achieve this.

    Thanks again for taking the time everyone. Really appreciate it.



  • Sounds like you will have asymmetric routing - an ordinary LAN client to LON packet goes:
    LAN client->relpfSense->vpfSense->London
    a reply packet goes:
    London->vpfSense->LAN client
    on the reply, the vpfSense is on LAN so can deliver the packet directly.
    realpfSense does not see the replies, and so the state dies after some seconds.
    Try:
    a) System: Advanced: Firewall and NAT - Bypass firewall rules for traffic on the same interface
    or
    b) Switch to Manual Outbound NAT and add NAT rule/s on LAN to NAT traffic with source LAN destination LON+NY to the LAN IP - then vpfSense will see realpfSense LAN IP as the source address of the packets and send the replies back there, to be unNATed and delivered back to the LAN client, removing the asymmetric routing.



  • Thanks for the input Phil.

    I have tried setting bypass firewall rules option on both the physical and the virtual pfSense devices to no avail, but the NAT setting seems like it may be something good to try. Your explanation of asymmetric routing sounds like it could be the cause!

    With regard to option b, should I configure the NAT settings on the virtual router or the physical?

    At the moment, I have manual NAT enabled with all rules removed in the virtual pfSense. On the physical, it is set to manual NAT with only rules for my WAN and VPN connection.

    Regards.

    Tom.



  • With regard to option b, should I configure the NAT settings on the virtual router or the physical?

    The LAN clients are using the physical router as their default gateway to the ordinary internet. When they access things on the virtual subnets, that traffic is turned back around across the LAN again to get to the virtual router.
    You want to configure NAT at the physical router as it turns the traffic around.



  • @phil.davis:

    Sounds like you will have asymmetric routing - an ordinary LAN client to LON packet goes:
    LAN client->relpfSense->vpfSense->London
    a reply packet goes:
    London->vpfSense->LAN client
    on the reply, the vpfSense is on LAN so can deliver the packet directly.
    realpfSense does not see the replies, and so the state dies after some seconds.
    Try:
    a) System: Advanced: Firewall and NAT - Bypass firewall rules for traffic on the same interface
    or
    b) Switch to Manual Outbound NAT and add NAT rule/s on LAN to NAT traffic with source LAN destination LON+NY to the LAN IP - then vpfSense will see realpfSense LAN IP as the source address of the packets and send the replies back there, to be unNATed and delivered back to the LAN client, removing the asymmetric routing.

    Phil - this seems to be spot on! I've just added the NAT rule to my physical pfsense and so far no RDP drop out! To say I am chuffed is an understatement. I've been putting up with RDP drops for months, if not a year. Thank you so much for your help!

    t.