IPsec throughput issues…



  • I have two Pfsense 1.2 boxes, one on each side of the IPSEC link.

    Side1:
    P3 1GHz with 512SDRAM
    Running multi-wan. 2 WANs and 1 LAN. Loadbalanced and failover. 
    Machine is using all 10/100 Intel Pro network cards.
    WAN 1 = Commercial Cable modem, 8mbit down / 1mbit up.
    WAN 2 = T1 1.5mbit down / 1.5mbit up
    IPSEC Interface: WAN1

    Side2:
    Single Xeon 2.0GHz with 1GB DDR
    Single WAN and Single LAN.
    Machine is using all 10/100 Intel Pro network cards.
    WAN 1 = FIOS 20mbit down / 20mbit up
    IPSEC Interface: WAN 1

    IPSEC Config:
                        Phase 1:
    Negotiation: Aggressive
    Encryption: Blowfish
    Hash: SH1
    DH Key Group: 2 (1024bit)
    Lifetime: 28800
    Authentication method: Pre-Shared Key

    Phase 2:
    Protocol: ESP
    Encryption: BlowFish
    Hash: SHA1
    PFS Key: 2 (1024bit)
    Lifetime: 43200

    Problem: When I pull from side2, 20mbitupstream/downstream, I can only pull around 1.5mbit through the tunnel.  I am using the Traffic Graph to see the throughput on each side.  CPU Utilization on side1 hovers around 10%, on side2, its around 3%.  I need to be able to pull around 6-8mbit from side2, to side1.  What could be causing this throughput problem?



  • your wan on side 1 is rate limited for wan 1 = 1 mbit up and wan 2 = 1,5 up, this is the max throughput for "ack" i think



  • I am PULLING from side2

    Side2 has 20mbit upstream and 20mbit downstream.

    Side1 has 8mbit downstream…I should be able to pull from side2 at a much faster rate than 1.5mbit/s



  • 1,5 Mbit is the shortest ack, my opinion!

    Ipsec/racoon is not rate limited!

    Hm, i think if you pull (down) from SIDE2 to SIDE1, for SIDE1 the traffic to SIDE2 is upstream and for my understandings you have only 1.5 MBit on this side.., maybe i´m incorrect!



  • What does that mean though? Because the upstream is only 1.5, the system rate limits the downstream to 1.5?  I am just confused as to how it works.

    Thank you for taking the time to read this, heiko!  :D

    P.S.  When I first noticed this problem, I thought it might have something to do with the encryption algorithms, but from what you see; none of that can really be the slow down?

    P.S.S. Can you tell me really new to IPSEC/VPN?  :(



  • Are you able to pull 8Mbps without the VPN? Have you also tried OpenVPN for comparison?



  • @drees:

    Are you able to pull 8Mbps without the VPN? Have you also tried OpenVPN for comparison?

    #1
    Yes, when I go straight out to the net and pull something down, I get 8mbit down.

    #2
    I was under the impression you couldn't do the filtering like you could with IPSEC, in regards to OpenVPN.  I need to be able to firewall off lan traffic etc…  I could probably test this over a weekend.  My two PFsense boxes are in production at the moment.



  • @stratos:

    Yes, when I go straight out to the net and pull something down, I get 8mbit down.

    Let me clarify - are you able to pull something from site to site at 8Mbps?

    @stratos:

    I was under the impression you couldn't do the filtering like you could with IPSEC, in regards to OpenVPN.  I need to be able to firewall off lan traffic etc…  I could probably test this over a weekend.  My two PFsense boxes are in production at the moment.

    I believe you are right, but I have not tried it. If you had two pfSense boxes at each site then you should be able to filter even with OpenVPN (if you can't already).



  • You got me thinking, Drees.  So i setup an FTP server on a server @ side2.  I pulled from it over the IPSEC tunnel, and i was able to get around 4mbit, so I am assuming its a protocol issue.  The overhead involved maybe?  I still wish I could get closer to 8mbit, because of the amount of data I need to be able to move from side2 to side1.

    I am going to try going over the internet and skipping the tunnel to see what kind of speeds i can get.



  • Ok so I have tried going over the internet directly, skipping the IPSEC tunnel, and I am still only able to pull around 4 mbit.  I guess thats going to be it, but sadly I wish I could get these speeds with every protocol I pass over the tunnel.  I have tested windows file copy and ftp, and windows file copy only gets around 1.5 mbit. 
    sighs

    I need a better pipe…



  • Yep, if you can only do 4Mbps from site to site using FTP without the VPN, you can't expect to get more than that when going over the VPN.

    Windows file copy must have more overhead than FTP.

    What kind of latency do you have between the sites? Have you looked at TCP Window Scaling settings on the client and server?



  • I get around 45ms latency between both sites, thats over a 1000 icmp packet average.

    I am not really sure what TCP Window Scaling is…



  • @stratos:

    I am not really sure what TCP Window Scaling is…

    Google it…



  • Hi Stratos,

    Several things going on here.  You mention that you have the IPSEC VPN going from side1's WAN (which is a cable modem) to side2's WAN (which is FIOS).  You will not be able to push more than 1 MBit from side1 to side2 in this setup.  Next, would your cable ISP happen to be Comcast or from another ISP?  I ask this because I have a similar setup to yours with a 16/2 Comcast cable line and a 15/15 FIOS line.  The issue I have is that I cannot steadily get more than 5 MB's out of the 15 Mbit upload that my Fios has to offer.  In fact, no non-Verizon based ISP can pull down that much of the Fios' upload bandwidth.  They've all be anywhere from 3-5 MBit/sec when pulling data from the Fios line.  This includes all sorts of speed tests and file transfers including HTTP, FTP, and netio transfers (to name a few).  Verizon still hasn't been able to answer this for me.  If I am running with a 7 MBit Verizon DSL line, then I have no issues.  If I am at a site with 15 Mbit or 30 Mbit Fios, I can easily get the full 15 MBit and same goes for a few sites where I have a Verizon based T3/DS3 line.

    The fact that you cannot pull 4 MBit/sec via you FTP test shows that you won't get more than that via your IPSEC tunnel.  I don't know if you're in the NY/NJ or Boston/New-England area, but what type of results do you get from speed-test websites regarding your Fios setup?  I'd be interested if you get full bandwidth out of the Fios line from multiple servers.  Last year, I read an article about users in the Florida area who had 20/20 Fios lines who were having issues filling up their bandwidth in both directions (download and upload) when doing single file transfers.  When performing multiple transfers from different sites, only then were they able to get their 20 MBit/sec connections fully going.

    Hope this helps a little and makes sense… :)



  • Hey razor,

    Just to clarify, I am not trying to push anything from side1(cable modem) to side2(fios).  I am trying to pull from side2(fios).  Yes, Comcast Business is the ISP of the cable modem.  On the FIOS line, I can max out the bandwidth at speedtest.net and in multi-threaded downloads (usenet,downloadmanagers,etc).  I guess Ill have to figure out a work around until I can get FIOS at my side1 location.

    Thanks for taking the time to reply!


Log in to reply