IPsec site to site performance not great

[5mall5nail5]

Hey all - trying to improve the performance between two locations.  My one location is 85/85 Mbps, the other is 120 Mbps/12 Mbps however if I access a windows share and transfer a large file I get varying speeds around 350 - 750 KB/sec.  I went and set the Enable MSS clamping on VPN traffic  to on, and set it to 1300 on both sides, and got to about 1.0 MB/sec - 1.1 MB/sec.  I tried with FTP also, same results.

How do I get this to be more like my actual upload speed?  CPU use was very low, 1-2%.  Both pfsense boxes are on ESXi Virtual Machines.

Thanks!

[Hugh]

Can you confirm that you push the traffic levels you are hoping for without the VPN involved?

If you SSH in or look in the console, run ifconfig, what do your options look like:

options=60009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6>Have a look at:

https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

I'm suggesting that you might be having issues with the TSO and LRO areas.

What sort of network cards are you using under ESXi, what have you setup under FreeBSD?</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6>

[5mall5nail5]

@Hugh:

Can you confirm that you push the traffic levels you are hoping for without the VPN involved?

If you SSH in or look in the console, run ifconfig, what do your options look like:

options=60009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6>Have a look at:

https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

I'm suggesting that you might be having issues with the TSO and LRO areas.  Here is the full output of ifconfig:

[2.1.5-RELEASE][admin@pfSense.conway.local]/root(1): ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:88:5d:36
        inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255
        inet6 fe80::250:56ff:fe88:5d36%em0 prefixlen 64 scopeid 0x1
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether f8:e4:fb:22:40:ee
        inet 72.92.54.39 netmask 0xffffff00 broadcast 72.92.54.255
        inet6 fe80::fae4:fbff:fe22:40ee%em1 prefixlen 64 scopeid 0x2
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:88:08:18
        inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
        inet6 fe80::250:56ff:fe88:818%em2 prefixlen 64 scopeid 0x3
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
pfsync0: flags=0<> metric 0 mtu 1460
        syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        nd6 options=3 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33144
enc0: flags=41 <up,running>metric 0 mtu 1536
ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fe88:5d36%ovpns1 prefixlen 64 scopeid 0x9
        inet 10.8.0.1 –> 10.8.0.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 81705
ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fe88:5d36%ovpns2 prefixlen 64 scopeid 0xa
        inet 10.0.2.1 --> 10.0.2.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 86563
ovpns3: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fe88:5d36%ovpns3 prefixlen 64 scopeid 0xb
        inet 10.8.8.1 --> 10.8.8.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 90177
ovpns4: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::250:56ff:fe88:5d36%ovpns4 prefixlen 64 scopeid 0xc
        inet 10.8.1.1 --> 10.8.1.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 94308

Right now none of the OpenVPN servers are actively used.

Thanks!

What sort of network cards are you using under ESXi, what have you setup under FreeBSD?</performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></up,running></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6>

Thanks Hugh for the reply.  Without the VPN tunnel I can download via web server at 4.1 MB/s from one location to the other.  I am using Intel Pro/1000 VT quad port nics in each ESXi host.

Both TSO and LRO boxes are checked on each side