SG-3100 IPsec Performance Issues



  • Greetings everyone!
    I recently purchased an SG-3100 for a satellite office but have been having issues with the speed through the IPsec tunnel connected to our main office. I currently am only able to get about 12-14mb down/up, our internet connection is 400/20 down/up. Because the tunnel speed is synchronous it makes me think I have reached the limit of the processor or I have made a mistake on the configuration. I played around with various crypto/hashing combinations and haven't seen much difference, I landed on AES128-SHA256-ECP256 since elliptic curve is normally faster than RSA.

    Am I missing something? I have also played around with various hardware crypto settings, but I haven’t found an easy way to tell what crypto devices and algorithms are compatible with IPsec, reading in the forums I see conflicting data about what is supported, and it seems that in the last few versions there have been some significant changes. I was considering trying OpenVPN to see if that has different results.

    Setup:
    Main Office:
    Xeon E31230 custom box HA Master
    SG-8860 HA Backup
    Internet 100mb x 100mb (speed test 90mb x 86mb)

    Satellite Office:
    SG-3100
    Internet connection 400mb x 20mb (speed test 450mb x 22mb)

    Things I have tried:
    Swapping Main/Backup Firewalls
    Enabling and Disabling hardware crypto
    Multiple ciphers/hash combos
    MSS Clamping down to 1400



  • I was able to increase the speed slightly by setting the MSS Clamping to 1360 although the ping tests I performed showed that it shouldn't be required. I am still scratching my head as to why it isn't performing faster, I attached iperf3 results below.

    (Server Main Office)
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 1.50 MBytes 12.6 Mbits/sec
    [ 4] 1.00-2.00 sec 1.75 MBytes 14.7 Mbits/sec
    [ 4] 2.00-3.00 sec 2.12 MBytes 17.8 Mbits/sec
    [ 4] 3.00-4.00 sec 1.88 MBytes 15.8 Mbits/sec
    [ 4] 4.00-5.00 sec 2.12 MBytes 17.8 Mbits/sec
    [ 4] 5.00-6.00 sec 2.12 MBytes 17.8 Mbits/sec
    [ 4] 6.00-7.00 sec 1.75 MBytes 14.7 Mbits/sec
    [ 4] 7.00-8.00 sec 2.12 MBytes 17.8 Mbits/sec
    [ 4] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec
    [ 4] 9.00-10.00 sec 2.00 MBytes 16.8 Mbits/sec


    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-10.00 sec 19.2 MBytes 16.1 Mbits/sec sender
    [ 4] 0.00-10.00 sec 19.2 MBytes 16.1 Mbits/sec receiver

    (Server Satellite Office)
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 640 KBytes 5.24 Mbits/sec
    [ 4] 1.00-2.00 sec 1.38 MBytes 11.5 Mbits/sec
    [ 4] 2.00-3.00 sec 1.50 MBytes 12.6 Mbits/sec
    [ 4] 3.00-4.02 sec 1.88 MBytes 15.5 Mbits/sec
    [ 4] 4.02-5.00 sec 1.62 MBytes 13.9 Mbits/sec
    [ 4] 5.00-6.01 sec 1.25 MBytes 10.4 Mbits/sec
    [ 4] 6.01-7.01 sec 1.38 MBytes 11.5 Mbits/sec
    [ 4] 7.01-8.01 sec 1.38 MBytes 11.5 Mbits/sec
    [ 4] 8.01-9.00 sec 1.75 MBytes 14.8 Mbits/sec
    [ 4] 9.00-10.01 sec 1.88 MBytes 15.6 Mbits/sec


    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-10.01 sec 14.6 MBytes 12.3 Mbits/sec sender
    [ 4] 0.00-10.01 sec 14.5 MBytes 12.2 Mbits/sec receiver


  • Netgate Administrator

    To get the best IPSec performance from the SG-3100 you need to be sure the CESA crypto hardware is being used.
    To do that you need to two things:
    Set 'Cryptographic Hardware' to BSD Crypto Device in System > Advanced > Misc.

    Make sure you;re using a cryto algorithm that CESA supports. That means AES-CBC, 128, 192 or 256 and sha1 or sha256.
    AES-CBC-128 and SHA1 will give you best performance.

    I expect it to fill that 86/22Mbps connection easily based purely on processing.

    Steve



  • @stephenw10 would the dh group prevent hardware offloading?


  • Netgate Administrator

    Hmm, you are using PFS key 28? I'm using 14 and can see connections close to my line rate at 70+Mbps even between London and Austin. I'm not sure what gets off-loaded there. Nothing obvious shown in the driver:
    https://github.com/pfsense/FreeBSD-src/blob/ff7d4801f1b88de656e028209818ff005e8a1353/sys/dev/cesa/cesa.c#L1229

    Are you able to test group14?

    I'm assuming you're using IKEv2. The phase 2 settings are where the speed is decided.

    Steve



  • I just tested it no real change I am seeing some more variability in the results though, I am not seeing dropped packets through the link so I wonder if there is something between the route on these two, they are just down the road a few miles but are using different providers (Main is using Timewarner/Spectrum fiber provided by a third party and the remote office is Timewarner/Spectrum Cable).

    I will contact my providers and see if I can track something down. I have been testing everything I can think of for the last two weeks and am about ready to price out running a wireless point-to-point link from Ubiquiti if I can't figure it out haha.

    Phase 1
    AES_CBC
    HMAC_SHA2_256_128
    PRF_HMAC_SHA2_256
    MODP_2048

    Phase 2
    AES_CBC
    HMAC_SHA2_256_128
    IPComp: none



  • I ran some speed tests yesterday between the two sites outside of the tunnel and was seeing similar results the issue must be the connections. I am reaching out to my providers today to see if they can resolve the problem.


  • LAYER 8 Netgate

    We have, on occasion, seen paths that are decidedly unfriendly to ESP but work well using NAT-T (UDP 4500).

    Since you are seeing similar issues outside the tunnel this is probably moot for you but I wanted to put it out there.

    Note that there is no way for force NAT-T using IKEv2 but you can using IKEv1.



  • Hi, the ipsec speed is limited by upload speed at remote site due to ACKs.

    Having said that I have issue with SG-3100 - I cannot get greater than 45 Mbps on a physical 100Mbps fiber link to my home pfsense which is 940Mbps fiber, whereas I can 380Mbps+ to a SG-5100 on physical 500Mbps link, and 180Mbps to a SG-4860 on a 200Mbps physical link. My home is a home-built pfsense running i7. I will open another thread on this.


Log in to reply