Why aren't my end to end vpn speeds keeping up? (site to site)

  • pfSense 2.4.1 on two Dell Optiplex 7010's (i7-3770's) ipsec VPN'd using 2 FIOS gigabit connections.

    Looking at my diagram, I have iperf tested each link along the way between two sites on identical hardware running pfSense.

    NON VPN site to site interface speed is good (1gb connection).  That's public ip to public ip
    Ipsec vpn LAN to LAN interface speed is good (thought this would be the bottleneck) usually over 800+ mbps, a little slow tonight.
    PC to LAN on one side and LAN to NAS on the other side is good

    But then you look at the PC to REMOTE LAN OR PC TO ANYTHING REMOTE ACTUALLY, and it's 1/3rd the speed of all the other links. Same results on any device on either side, in either direction.

    pfSense utilization on the hardware is very low during these PC to NAS tests (7-9%) during a 30 second iperf run.

    With such good link speeds, and such good vpn site to site speeds I can't figure out why I'm not seeing better end to end speed.  Every device I look at to try and find fault has great speeds to other components so I can't call it a bad device or bad config, but big picture it's 1/3rd of the other links.

    Any ideas on what's happening here?

  • I was looking at some other websites and came across a iperf syntax that I tried.  The result is windows pc at home to windows pc at work (across the vpn)

    iperf command line was: iperf -c -u -b 1000m

    Results are pretty telling:  I'm not sure what these swithes do (-u says use UDP not TCP and I'm not understanding -b much at all) but I'm getting full line speed.  Hopefully this can tell us something which in turn I can tune on my firewalls.  If I lower the -b to 900 800 700 the speed starts to decrease.

    Client connecting to, UDP port 5001
    Sending 1470 byte datagrams, IPG target: 11.76 us (kalman adjust)
    UDP buffer size:  208 KByte (default)

    [  3] local port 58746 connected with port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0-10.0 sec  1.11 GBytes  953 Mbits/sec
    [  3] Sent 810345 datagrams

    I just did the same test except -b 3000m from lan interface to lan interface (on each router), and got 1.5gbps throughput.  What's going on here.  How do I unleash this beast? (no graph) bad command line, it never sent any data across the network.

  • After reading a few more iperf thread I tried using the -P option which will open multiple streams to send data. So from my computer at work I did iperf -c -P 3 (101 being my NAS on the other side of the vpn), and it fully saturated the line, 890 mbps.

    So what's that telling me? My windows file copies are single stream and 280+ mbps is the most I'm going to get out of one stream? (as one post suggests). Are their copy programs that will do multiple streams? I've been searching and haven't come across anything.

    My eventual need would be to be able to move data from the computer at work to the NAS on the other side of the vpn at line speeds. iperf just showed I can do it from machine to NAS, now I just have to find a program that can make it happen.


  • At this point I've been having a conversation with myself on this topic but I'm determined to provide some valuable information to someone who will inevitably come across the same dilemma that I have.

    So the past few nights I've been doing a lot of reading.  WAN Accelerators, alternate protocols etc.  Tonight I came across an article about transferring data across ipsec tunnels.  One of the items the author mentioned was different speeds using different protocols.  One of the protocols was http.  Hmm.  My NAS at home has a http front end and I remembered that it did some form of file transfer.  I gave it a shot, uploading a 17.7 gig rar archive in 3 minutes and 11 seconds.  Here's the tail end of the transfer:  As you can see, it achieved full line rate 100+ MBps

    I see there are a number of windows programs out there allowing for http transfer.  Hopefully I can find a command line version or better yet some that might actually map a drive or at least allow me to send files to my NAS.  That would be super.  This could be just what I'm looking for to finally saturate my ipsec vpn for file transfer.  Sure beats a four thousand dollar WAN Accelerator.


Log in to reply