APU --> SG-1100, Faster at IPSec; Slower at Everything Else

TheWaterbug

I have an ancient APU stuck at 2.4.4, and Netgate was kind enough to modify its config for compatibility with my SG-1100/22.05.

I have 1000/1000 service at my house, and my APU was doing ~450/450 via speedtest.net. My IPSec tunnel (AES256-GCM (128 bits)) to my MBT-2220/2.4.4 at the office (also 1000/1000) was iperfing at around 20/20.

Today I restored the APU's converted config to the SG-1100 and put the SG-1100 in the APU's place in my home. It took about 5 minutes for the ONT to recognize the SG-1100, but then everything worked just like it used to.

But it's not as fast as I would have expected. speedtest.net is reporting 250/350. This is via a Gigabit Ethernet, there's no WiFi in the way. It's slower than the APU.

The good news is that the IPSec tunnel is a bit faster, iperfing at 30/30. I turned on SafeXcel Crypto and rebooted it, but that didn't make a difference.

Is this expected? Or am I doing this wrong?

stephenw10

Both those figures look low for IPSec. What is the latency between the sites?

How are you running the iperf test?

Both the MBT and the APU are capable of running the current pfSense CE version, 2.6.

Steve

TheWaterbug

@stephenw10

Ah, PEBCAK. I just looked at my encryption settings on the tunnel, and apparently I was using settings that were neither secure nor accelerated. Now the tunnel is both, and the performance is dramatically improved:

./iperf3 -c 192.168.0.13
Connecting to host 192.168.0.13, port 5201
[  4] local 192.168.1.100 port 56795 connected to 192.168.0.13 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  14.8 MBytes   124 Mbits/sec                  
[  4]   1.00-2.00   sec  16.1 MBytes   135 Mbits/sec                  
[  4]   2.00-3.00   sec  14.4 MBytes   121 Mbits/sec                  
[  4]   3.00-4.00   sec  12.1 MBytes   102 Mbits/sec                  
[  4]   4.00-5.00   sec  14.9 MBytes   125 Mbits/sec                  
[  4]   5.00-6.00   sec  16.0 MBytes   135 Mbits/sec                  
[  4]   6.00-7.00   sec  16.3 MBytes   136 Mbits/sec                  
[  4]   7.00-8.00   sec  14.9 MBytes   125 Mbits/sec                  
[  4]   8.00-9.00   sec  14.2 MBytes   119 Mbits/sec                  
[  4]   9.00-10.00  sec  15.6 MBytes   131 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   149 MBytes   125 Mbits/sec                  sender
[  4]   0.00-10.00  sec   149 MBytes   125 Mbits/sec                  receiver

I hadn't even thought to optimize this previously, because my line speeds were 20/20 and 50/20, so I thought IPSec throughput of 10/10 was reasonable.

125/125 is quite nice!

Do you know what IPSec throughput the original APU unit (AMD G-T40E Processor, 2 CPUs: 1 package(s) x 2 core(s)) would be capable of, with all the correct settings?

What the MBT-2220 top out at?

Thanks!

stephenw10

Not sure we have any directly comparable numbers. The MBT-2220 can probably do ~300Mbps IPSec given the correct conditions. The APU is probably somewhere in the 100-150Mbps range.

TheWaterbug

@stephenw10

Very interesting! I had no idea those old APUs were so performant. 100 is very respectable.

I also didn't know the APU could run 2.6CE. I had it in my mind from several years ago that they ran out of life after 2.4.x.

Do you know if I can back up the config from an MBT-2220/2.4.4. and restore it to an APU/2.4.4? This would allow me to swap in the APU while I update the MBT to 22.05, then put the MBT back in place once I know it's running properly.

I'm always leery of doing an upgrade in place of a device that's the single point of failure for my office, with no way to swap back quickly.

Thanks!

stephenw10

Yes, you could import the MBT config into the APU. The interfaces are different so it will ask you to re-assign WAN and LAN before rebooting but that's quite straight forward.

Steve

TheWaterbug

@stephenw10

Thanks! I'll work on it this weekend.

BTW, what is the most secure encryption that the APU is capable of accelerating in hw?

stephenw10

The APU doesn't have any specific hardware for crypto off-loading so really it's jut about speed vs relative security. I would consider AES-GCM 256 more than sufficiently secure and fast enough.

Steve

A Former User

The APU is only sorted with a 1GHz cpu w 2 cores and
and it is suggested to own a 2GHz CPU to reach ~500
MBit/s and you got 450 MBit/s.

stephenw10

Those are very old references, they need to be updated. Badly!

I'd guess that was true in the Pentium 4 era.

Steve

TheWaterbug

@stephenw10 said in APU --> SG-1100, Faster at IPSec; Slower at Everything Else:

The APU doesn't have any specific hardware for crypto off-loading so really it's jut about speed vs relative security. I would consider AES-GCM 256 more than sufficiently secure and fast enough.

Do you think the APU would do 100 Mbps IPSec with AES-GCM 256?

stephenw10

Probably is about the best I can say. There are a lot of variables.

TheWaterbug

I don't know if this is of interest to anyone but me, but I fiddled with the encryption/hashing settings between the MBT-2220 at the office (2.4.4/Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM) and the SG-1100 at my home (22.05/Hardware crypto AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256,SHA384,SHA512).

Line speed is >>>> 250/250 at both ends. It's actually 1000/1000 nominal, but I'm router limited.

iperf reports:

AES256-GCM (256 bits) SHA256 14 (2048 bit)

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 157 MBytes 132 Mbits/sec sender
[ 4] 0.00-10.00 sec 157 MBytes 132 Mbits/sec receiver

AES128-GCM (128 bits) SHA256 14 (2048 bit)

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 165 MBytes 139 Mbits/sec sender
[ 4] 0.00-10.00 sec 165 MBytes 139 Mbits/sec receiver

AES CBC (256 bits) SHA256 14 (2048 bit)

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 146 MBytes 123 Mbits/sec sender
[ 4] 0.00-10.00 sec 146 MBytes 123 Mbits/sec receiver

AES CBC (128 bits) SHA256 14 (2048 bit)

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 149 MBytes 125 Mbits/sec sender
[ 4] 0.00-10.00 sec 149 MBytes 125 Mbits/sec receiver

AES CBC (128 bits) SHA1 14 (2048 bit)

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 168 MBytes 141 Mbits/sec sender
[ 4] 0.00-10.00 sec 168 MBytes 141 Mbits/sec receiver

Sampling error is ±10 Mbps for any particular configuration.

Given the modest differences, I'm going to stick with AES256-GCM (256 bits) SHA256 14 (2048 bit).

What other knobs can I turn to improve IPSec throughput between these two boxes?

TheWaterbug

I put the APU back in place of the SG-1100, temporarily, to repeat the experiment, and I got the same speed regardless of what encryption settings I chose:

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 67.0 MBytes 56.2 Mbits/sec sender
[ 4] 0.00-10.00 sec 66.9 MBytes 56.2 Mbits/sec receiver

I've got the SG-1100 back in place, and the 125/125 is a very nice upgrade from the 30/30 I was getting just a few days ago.

stephenw10

You have safexcel enabled at the 1100 end? If so use any of the ciphers it supports:
https://www.freebsd.org/cgi/man.cgi?query=safexcel

AES-GCM is inherently faster as it doesn't require a separate authentication step. So I would have expected AES-GCM 128 to be the fastest.

Enabling asynchronous crypto can make a huge improvement on systems that support it. Thats in the IPSec Advanced settings. It's probably enabled in the 1100 but may not be in the MBT.

Steve

TheWaterbug

@stephenw10

Yes, SafeXcel is on. That link says it "implements SHA1 and SHA2 transforms," but does not specifically list SHA256. I compared SHA1 vs. SHA256 and didn't see any difference in performance.

Async was turned on in the SG-1100 and off in the MBT-2220, so I turned it on in the MBT-2220, but that didn't make any difference in performance, either. In fact it iperfs 5-10 Mbps slower, but that could easily be sampling error.

Thanks!

TheWaterbug

@thewaterbug said in APU --> SG-1100, Faster at IPSec; Slower at Everything Else:

Async was turned on in the SG-1100 and off in the MBT-2220, so I turned it on in the MBT-2220, but that didn't make any difference in performance, either. In fact it iperfs 5-10 Mbps slower, but that could easily be sampling error.

Actually I just turned async off on both ends, and now it iperfs at 143/143, vs. 120/20 when async is on at both ends. I don't think it's sampling error, because it's repeatable (n=3 trials).

And now there's no measurable difference in throughput between AES-GCM128 and AES-GCM256. They both test right around ~135-140 Mbps.

stephenw10

Hmm, interesting. I wonder if you're hitting some other limit there then.

140Mbps is about what I expect from the 1100 though.

TheWaterbug

@stephenw10 said in APU --> SG-1100, Faster at IPSec; Slower at Everything Else:

Hmm, interesting. I wonder if you're hitting some other limit there then.

140Mbps is about what I expect from the 1100 though.

Thanks! That's confirmation that I'm not doing anything grossly incorrect.

TheWaterbug

@stephenw10 said in APU --> SG-1100, Faster at IPSec; Slower at Everything Else:

Both the MBT and the APU are capable of running the current pfSense CE version, 2.6.

I found my null modem adapter, so I now have one of my APU units up and running 2.6.

I need to run over to my 3rd site and swap it into place of the other APU, and then upgrade that one to 2.6, and then all of my devices will be at the latest release.

Thanks!