Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Slow IPSec Site-to-Site Speeds

    IPsec
    1
    2
    90
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      TheStormsOfFury
      last edited by TheStormsOfFury

      Good evening, everyone. I am running into an issue with IPSec Site-to-Site and slow speeds between two sites with nearly identical setups.

      We basically run (3) things through this tunnel:
      1.) VoIP Phone System (Avaya IP Office 500v2)
      2.) Database Connections for Store Software (Site A has connections to Site B's DB and Site B has connections to Site A's DB)
      3.) Domain Controller Synchronization, Late night backups, and other miscellaneous utilities running off hours (read after Midnight local time)

      I would assume that we should be able to get at least 100-160 Mbps across the tunnel; however, that is not the case.

      When I use Speedtest.net, the sites have the following real-time speeds
      Site A: Download - 201.09 Mbps; Upload - 201.09 Mbps
      Site B: Download - 620.15 Mbps; Upload - 628 Mbps

      Before I started all of this, we were getting about 24 Mbps Upload and 26 Mbps Download across the OpenVPN tunnel. I then upgraded to pfSense+ as I was told this would enable OpenVPN Data Channel Offloading, which makes OpenVPN. Even WITH that enabled, I was still only able to get, at max 35Mbps Upload and 38Mbps Download across the OpenVPN tunnel.

      These slow speeds are causing some serious issues with accessing the software at the opposite sites (read 3+ minutes to load some of the data, and to be fair I know that there is serious database design/query issues involved; however, it shouldn't be that long given how "small" the system is), and it's even causing the tunnels to be overloaded during this access which during the day is causing phones calls to drop randomly when both sides are access the others systems.

      I decided to switch to IPSec on Saturday as I know several people who have said they get faster speeds, and now I am getting somewhat faster speeds, but they still are falling short of what these connections should be getting.

      Here is my current configuration at each site with IPSec enabled:

      Site A:
      (2) Dell 1U servers w/dual E5-2620 (24 cores@ 2.00GHz)
      32GB Memory
      HA Setup with (1) WAN, (1) LAN, and (1) HASYNC connection
      pfSense+ v 24.11
      200Mbps by 200 Mbps COX Fiber Connection
      WAN Connection direct to internet (No NAT)
      LAN Network is CAT6 with Gigabit Switches (max run 220')
      IPSec policy setup and connected to Site B
      P1 Tunnel Settings:

      • IKE version: IKEv2
      • Mutual PSK w/ My IP Address & Peer IP Address w/ 128 character PSK
      • Encryption Algorithm: AES, 256b key, SHA256 Hash, 14 DH Group

      P2 Settings:

      • Protocol: ESP
      • Encryption: AES256-GCM, 128b
      • Hash: None
      • PFS Key: 14

      Site B:
      (2) Dell 1U servers w/dual E5-2620 (24 cores@ 2.00GHz)
      32GB Memory
      HA Setup with (1) WAN, (1) LAN, and (1) HASYNC connection
      pfSense+ v 24.11
      500Mbps by 500 Mbps COX Fiber Connection
      WAN Connection direct to internet (No NAT)
      Network is CAT6 with Gigabit Switches (max run 115')
      IPSec policy setup and connected to Site A
      P1 Tunnel Settings:

      • IKE version: IKEv2
      • Mutual PSK w/ My IP Address & Peer IP Address w/ 128 character PSK
      • Encryption Algorithm: AES, 256b key, SHA256 Hash, 14 DH Group

      P2 Settings:

      • Protocol: ESP
      • Encryption: AES256-GCM, 128b
      • Hash: None
      • PFS Key: 14

      As mentioned, IPSec is up and running, and now when I do a simple LAN speed test with a 50MB file, these are the results:

      Site A to Site B:
      Upload - 43.67Mbps
      Download - 86.6 Mbps

      Site B to Site A:
      Upload - 61.57 Mbps
      Download - 42.12 Mbps

      Does anyone have any ideas to speed this up (outside of fixing the database querying (which I cannot do))?

      Thanks,

      TSoF

      Edit: Updated to confirm that these devices are connected directly to the internet and are not being NAT
      Edit 2: Updated to include the memory installed (4 @ 8GB p/Module)

      T 1 Reply Last reply Reply Quote 0
      • T
        TheStormsOfFury @TheStormsOfFury
        last edited by

        I am working on troubleshooting through iperf and here is what I have going public IP to public IP (read not through IPSec Tunnel) and private IP to private IP (read through the IPSec Tunnel)

        Public IP to Public IP - Site B to Site A
        iperf 3.17.1
        FreeBSD ngr2-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
        Control connection MSS 1460
        Time: Mon, 03 Mar 2025 16:13:38 UTC
        Connecting to host 207.162.137.152, port 5201
        Cookie: g3oi22k7ksrx6tkmxub52glxdgoc7l2xocks
        TCP MSS: 1460 (default)
        [ 5] local 75.61.85.194 port 15753 connected to 207.162.137.152 port 5201
        Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
        [ ID] Interval Transfer Bitrate Retr Cwnd
        [ 5] 0.00-1.00 sec 10.9 MBytes 90.8 Mbits/sec 0 399 KBytes
        [ 5] 1.00-2.06 sec 12.9 MBytes 102 Mbits/sec 0 399 KBytes
        [ 5] 2.06-3.06 sec 12.2 MBytes 103 Mbits/sec 0 399 KBytes
        [ 5] 3.06-4.00 sec 11.5 MBytes 102 Mbits/sec 0 399 KBytes
        [ 5] 4.00-5.01 sec 12.2 MBytes 102 Mbits/sec 0 399 KBytes
        [ 5] 5.01-6.03 sec 12.9 MBytes 106 Mbits/sec 0 444 KBytes
        [ 5] 6.03-7.06 sec 15.5 MBytes 126 Mbits/sec 0 522 KBytes
        [ 5] 7.06-8.05 sec 15.4 MBytes 131 Mbits/sec 42 181 KBytes
        [ 5] 8.05-9.00 sec 5.75 MBytes 50.4 Mbits/sec 0 213 KBytes
        [ 5] 9.00-10.00 sec 6.88 MBytes 57.7 Mbits/sec 0 233 KBytes

        Test Complete. Summary Results:
        [ ID] Interval Transfer Bitrate Retr
        [ 5] 0.00-10.00 sec 116 MBytes 97.4 Mbits/sec 42 sender
        [ 5] 0.00-10.04 sec 115 MBytes 96.2 Mbits/sec receiver
        CPU Utilization: local/sender 12.4% (0.0%u/12.4%s), remote/receiver 9.3% (2.0%u/7.3%s)
        snd_tcp_congestion cubic
        rcv_tcp_congestion cubic

        iperf Done.

        Private IP to Private IP THROUGH IPSec - Site B to Site A
        Connecting to host 10.1.0.2, port 5201
        [ 5] local 10.2.0.2 port 31524 connected to 10.1.0.2 port 5201
        [ ID] Interval Transfer Bitrate Retr Cwnd
        [ 5] 0.00-1.00 sec 16.0 MBytes 134 Mbits/sec 0 1.07 MBytes
        [ 5] 1.00-2.01 sec 16.0 MBytes 133 Mbits/sec 411 218 KBytes
        [ 5] 2.01-3.00 sec 7.12 MBytes 60.2 Mbits/sec 0 257 KBytes
        [ 5] 3.00-4.04 sec 8.50 MBytes 68.9 Mbits/sec 0 281 KBytes
        [ 5] 4.04-5.00 sec 8.50 MBytes 73.8 Mbits/sec 0 293 KBytes
        [ 5] 5.00-6.06 sec 9.50 MBytes 75.0 Mbits/sec 0 306 KBytes
        [ 5] 6.06-7.00 sec 9.12 MBytes 81.6 Mbits/sec 0 327 KBytes
        [ 5] 7.00-8.00 sec 10.4 MBytes 86.9 Mbits/sec 0 351 KBytes
        [ 5] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec 0 374 KBytes
        [ 5] 9.00-10.05 sec 12.4 MBytes 99.3 Mbits/sec 0 398 KBytes

        [ ID] Interval Transfer Bitrate Retr
        [ 5] 0.00-10.05 sec 108 MBytes 90.5 Mbits/sec 411 sender
        [ 5] 0.00-10.08 sec 107 MBytes 89.4 Mbits/sec receiver

        iperf Done.

        Public IP to Public IP - Site A to Site B
        iperf 3.17.1
        FreeBSD ngr1-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
        Control connection MSS 1460
        Time: Mon, 03 Mar 2025 16:53:33 UTC
        Connecting to host 75.61.85.194, port 5201
        Cookie: fediws4o47ceehawmnp3zegzg4zch3oa2wn3
        TCP MSS: 1460 (default)
        [ 5] local 207.162.137.152 port 33788 connected to 75.61.85.194 port 5201
        Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
        [ ID] Interval Transfer Bitrate Retr Cwnd
        [ 5] 0.00-1.04 sec 11.8 MBytes 94.8 Mbits/sec 299 367 KBytes
        [ 5] 1.04-2.06 sec 12.1 MBytes 99.6 Mbits/sec 0 412 KBytes
        [ 5] 2.06-3.05 sec 12.9 MBytes 109 Mbits/sec 0 441 KBytes
        [ 5] 3.05-4.06 sec 13.8 MBytes 114 Mbits/sec 0 458 KBytes
        [ 5] 4.06-5.06 sec 14.0 MBytes 118 Mbits/sec 0 466 KBytes
        [ 5] 5.06-6.05 sec 14.0 MBytes 118 Mbits/sec 0 468 KBytes
        [ 5] 6.05-7.00 sec 13.5 MBytes 119 Mbits/sec 0 469 KBytes
        [ 5] 7.00-8.01 sec 14.2 MBytes 119 Mbits/sec 0 482 KBytes
        [ 5] 8.01-9.01 sec 15.0 MBytes 126 Mbits/sec 0 503 KBytes
        [ 5] 9.01-10.00 sec 15.1 MBytes 128 Mbits/sec 0 523 KBytes

        Test Complete. Summary Results:
        [ ID] Interval Transfer Bitrate Retr
        [ 5] 0.00-10.00 sec 136 MBytes 114 Mbits/sec 299 sender
        [ 5] 0.00-10.03 sec 135 MBytes 113 Mbits/sec receiver
        CPU Utilization: local/sender 11.2% (0.0%u/11.1%s), remote/receiver 17.8% (1.2%u/16.7%s)
        snd_tcp_congestion cubic
        rcv_tcp_congestion cubic

        iperf Done.

        Private IP to Private IP THROUGH IPSec - Site A to Site B
        iperf 3.17.1
        FreeBSD ngr1-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
        Control connection MSS 1460
        Time: Mon, 03 Mar 2025 16:54:25 UTC
        Connecting to host 10.2.0.2, port 5201
        Cookie: aji4w5n3rzsobdygb2zxvqydl7mrciv3gum7
        TCP MSS: 1460 (default)
        [ 5] local 10.1.0.2 port 30411 connected to 10.2.0.2 port 5201
        Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
        [ ID] Interval Transfer Bitrate Retr Cwnd
        [ 5] 0.00-1.06 sec 11.1 MBytes 87.8 Mbits/sec 210 362 KBytes
        [ 5] 1.06-2.06 sec 11.6 MBytes 97.9 Mbits/sec 0 406 KBytes
        [ 5] 2.06-3.02 sec 12.2 MBytes 107 Mbits/sec 0 434 KBytes
        [ 5] 3.02-4.06 sec 14.1 MBytes 113 Mbits/sec 0 453 KBytes
        [ 5] 4.06-5.01 sec 13.0 MBytes 115 Mbits/sec 0 462 KBytes
        [ 5] 5.01-6.03 sec 14.4 MBytes 118 Mbits/sec 0 463 KBytes
        [ 5] 6.03-7.06 sec 14.4 MBytes 117 Mbits/sec 0 464 KBytes
        [ 5] 7.06-8.02 sec 13.6 MBytes 120 Mbits/sec 0 464 KBytes
        [ 5] 8.02-9.06 sec 14.9 MBytes 119 Mbits/sec 0 482 KBytes
        [ 5] 9.06-10.01 sec 14.1 MBytes 126 Mbits/sec 0 499 KBytes

        Test Complete. Summary Results:
        [ ID] Interval Transfer Bitrate Retr
        [ 5] 0.00-10.01 sec 134 MBytes 112 Mbits/sec 210 sender
        [ 5] 0.00-10.04 sec 133 MBytes 111 Mbits/sec receiver
        CPU Utilization: local/sender 63.1% (0.1%u/63.0%s), remote/receiver 40.2% (2.2%u/38.0%s)
        snd_tcp_congestion cubic
        rcv_tcp_congestion cubic

        iperf Done.

        Speeds seem better using iperf thought they are still somewhat slower than it should be.

        1 Reply Last reply Reply Quote 0
        • First post
          Last post
        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.