SG-3100 ipSec slower than expected performance

[brians]

Have issue with SG-3100. One is a site that has 100Mbps fiber upstream and downstream.
I have another site with SG-4860 that is 200Mbps fiber upstream and downstream.
I cannot get higher speed than around 40-45Mbps when making ipsec tunnel between both locations.

I test from home so I don't disturb customer's live connection.
My Home pfSense is a home built i7 on 940Mbps fiber upstream and downstream.

ipSec tunnel to the SG-4860 and test around 170-180Mbps both directions which is acceptable for 200Mbps link.
ipSec tunnel to the SG-3100 and test only around 45Mbps both directions which is below expected rate of 80-90Mbps.

I have tried all sorts of different combinations between my home and the SG-3100 eg. AES, AES-GCM etc, and some even run slower than the 45Mbps. Verified crypto is on with SG-3100 settings. MSS clamping set to 1400 both ends.

What gives with the SG-3100? I have put a lot of these in customers' sites and fortunately only one or two customer so far would needs speeds greater than this. What am I doing wrong with configuring ipsec? So far a free i7 I recycled outperforms SG-3100 drastically but would rather support Netgate by purchasing official hardware.

What I am looking for is someone who has an ipSec tunnel between an SG-3100 and a SG-5100 and is able to provide me with their settings.

Thank you

Brian

[stephenw10]

To get best speed from the SG-3100 you need to use a cipher set that is supported by the CESA hardware crypto it has.

I am using AES-CBC 256 and SHA1 but SHA256 is also supported:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4_4/sys/dev/cesa/cesa.c#L1229

Enabling Asynchronous Cryptography in the IPSec advanced settings can also provide far greater throughput. That should be enabled by default on the 3100 but check in case you imported an old config etc. We do not enable it by default on CE as some systems stop passing traffic entirely.

Steve

[brians]

Hi, I tried all those settings, reset states after and is still slow.
login-to-view
login-to-view
login-to-view

Output from my home to SG-3100 is:

C:\iperf>iperf3.exe -c 192.168.0.254 -R
Connecting to host 192.168.0.254, port 5201
Reverse mode, remote host 192.168.0.254 is sending
[ 4] local 192.168.10.56 port 55655 connected to 192.168.0.254 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 659 KBytes 5.40 Mbits/sec
[ 4] 1.00-2.00 sec 1.87 MBytes 15.7 Mbits/sec
[ 4] 2.00-3.00 sec 3.06 MBytes 25.7 Mbits/sec
[ 4] 3.00-4.00 sec 4.44 MBytes 37.3 Mbits/sec
[ 4] 4.00-5.00 sec 5.73 MBytes 48.1 Mbits/sec
[ 4] 5.00-6.00 sec 6.61 MBytes 55.5 Mbits/sec
[ 4] 6.00-7.00 sec 5.99 MBytes 50.2 Mbits/sec
[ 4] 7.00-8.00 sec 6.74 MBytes 56.5 Mbits/sec
[ 4] 8.00-9.00 sec 6.45 MBytes 54.1 Mbits/sec
[ 4] 9.00-10.00 sec 6.71 MBytes 56.4 Mbits/sec

---

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 48.5 MBytes 40.7 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 48.5 MBytes 40.7 Mbits/sec receiver

iperf Done.

To another site with SG-4860
C:\iperf>iperf3.exe -c 192.168.221.2
Connecting to host 192.168.221.2, port 5201
[ 4] local 192.168.10.56 port 55618 connected to 192.168.221.2 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 21.0 MBytes 176 Mbits/sec
[ 4] 1.00-2.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 2.00-3.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 3.00-4.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 4.00-5.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 5.00-6.00 sec 21.0 MBytes 176 Mbits/sec
[ 4] 6.00-7.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 7.00-8.00 sec 21.1 MBytes 177 Mbits/sec
[ 4] 8.00-9.00 sec 21.2 MBytes 178 Mbits/sec
[ 4] 9.00-10.00 sec 21.0 MBytes 176 Mbits/sec

---

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 211 MBytes 177 Mbits/sec sender
[ 4] 0.00-10.00 sec 211 MBytes 177 Mbits/sec receiver

[stephenw10]

What is the latency?

What speed do you see if you test outside the tunnel?

Steve

[brians]

Speedtest.net
26ms
94 Mbps down
89 Mbps up

login-to-view
Latency does seem little higher at 26ms but shouldn't matter.

Here is other site for comparison.
login-to-view

Also ping tests across to servers from my house across VPN

Pinging 192.168.0.5 with 32 bytes of data:
Reply from 192.168.0.5: bytes=32 time=30ms TTL=126
Reply from 192.168.0.5: bytes=32 time=30ms TTL=126
Reply from 192.168.0.5: bytes=32 time=30ms TTL=126
Reply from 192.168.0.5: bytes=32 time=30ms TTL=126

Ping statistics for 192.168.0.5:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 30ms, Maximum = 30ms, Average = 30ms

C:\Users\Administrator>ping 192.168.221.2

Pinging 192.168.221.2 with 32 bytes of data:
Reply from 192.168.221.2: bytes=32 time=6ms TTL=126
Reply from 192.168.221.2: bytes=32 time=4ms TTL=126
Reply from 192.168.221.2: bytes=32 time=4ms TTL=126
Reply from 192.168.221.2: bytes=32 time=4ms TTL=126

Ping statistics for 192.168.221.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 4ms, Maximum = 6ms, Average = 4ms

Latency does seem worse... could this make a difference?

[stephenw10]

Yes, it will make a difference.

However what I meant was test using iperf between those sites directly, outside the tunnel. Just open the port locally for the test. Limit it to the source IP of the other site.
That will give you the possible speed. There may be something in the route limiting the speed independently of whatever IPSec is doing.
No point chasing something in the encryption if you can't see close to 100Mbps unencrypted.

Steve

Steve

[brians]

Oh ok I will try that.

[stephenw10]

Just for reference I was able to see >300Mbps though a 3100 using iperf3 in local testing, so very low latency. That was using AES-CBC 128 and SHA1.