PfSense OVA in ESXi. Performance issues over GRE tunnel across sites.

[gnapartners]

Site A - 1Gbps network connection
pfSense gateway appliance (purchased from Amazon) WAN -> internet uplink (216.x.x.x/28), LAN -> local interface (10.0.1.1/24), OPT1 (not used), OPT2 HA uplink to another pfSense
GRE0 1.1.1.2/30 (parent interface WAN)
static route route 10.99.0.0/24 next-hop 1.1.1.1
outbound NAT - manual only on WAN interface, NoNAT on LAN and GRE0 interface (we don't want to NAT traffic between sites)

Site B - 1Gbps network connection
pfSense virtual appliance under ESXi vmx0 -> WAN uplink (214.x.x.x/28), vmx1 -> LAN interface (10.99.0.1/24)
GRE0 1.1.1.1/30 (parent interface WAN)
static route 10.0.1.0/24 next-hop 1.1.1.2
outbound NAT - manual only on WAN interface, NoNAT on LAN and GRE0 interface (we don't want to NAT traffic between sites)

Observation:

setup works great. machines behind the pfSense can access internet at high speeds without a problem. We have an existing MPLS network with 50Mbps links on each end but we want to configure our storage replication to use the higher bandwidth routes

• WAN and LAN interfaces have MTU 1500
• GRE tunnel has a MTU 1472 (default)

Problem:

When a machine in site A tries to access a machine in site B across the GRE tunnel the performance drops considerably (3-5Mbps). We have checked the networks are not choked and the same machines when put on MPLS network drive throughput of 50Mbps

I have tried my best to figure out the problem but cannot get it to work. Any suggestions/direction would be much appreciated.

Thank you