PfSense OVA in ESXi. Performance issues over GRE tunnel across sites.



  • Site A - 1Gbps network connection
    pfSense gateway appliance (purchased from Amazon) WAN -> internet uplink (216.x.x.x/28), LAN -> local interface (10.0.1.1/24), OPT1 (not used), OPT2 HA uplink to another pfSense
    GRE0 1.1.1.2/30 (parent interface WAN)
    static route route 10.99.0.0/24 next-hop 1.1.1.1
    outbound NAT - manual only on WAN interface, NoNAT on LAN and GRE0 interface (we don't want to NAT traffic between sites)

    Site B - 1Gbps network connection
    pfSense virtual appliance under ESXi vmx0 -> WAN uplink (214.x.x.x/28), vmx1 -> LAN interface (10.99.0.1/24)
    GRE0 1.1.1.1/30 (parent interface WAN)
    static route 10.0.1.0/24 next-hop 1.1.1.2
    outbound NAT - manual only on WAN interface, NoNAT on LAN and GRE0 interface (we don't want to NAT traffic between sites)

    Observation:

    setup works great. machines behind the pfSense can access internet at high speeds without a problem. We have an existing MPLS network with 50Mbps links on each end but we want to configure our storage replication to use the higher bandwidth routes

    • WAN and LAN interfaces have MTU 1500
    • GRE tunnel has a MTU 1472 (default)

    Problem:

    When a machine in site A tries to access a machine in site B across the GRE tunnel the performance drops considerably (3-5Mbps). We have checked the networks are not choked and the same machines when put on MPLS network drive throughput of 50Mbps

    I have tried my best to figure out the problem but cannot get it to work. Any suggestions/direction would be much appreciated.

    Thank you