Upgraded to pfSense 2.4 & switched from AES-CBC-256 to AES-GCM-256, now slower

gertty · Oct 26, 2017, 1:43 PM

I'm running pfSense on a Netgate SG-4860. my WAN connection is cable Internet with 20Mbps down/5Mbps up. I'm running two OpenVPN clients on the router and most of my traffic goes out one or the other VPN tunnel depending on which VLAN is comes from. I run one of the OVPN servers myself, the other is a paid service. I upgraded to pfSense 2.4 last week and it looks like both VPN connections are slow.

Prior to last week, I was using the latest pfSense 2.3.x release, with AES-CBC-256 on both tunnels. For the ovpn server I control, it has been running OpenVPN v2.4.x for a while. I don't know what version of OpenVPN the paid service is running. Doing several bandwidth tests over many months showed me that I got basically my line speed (20Mpbs/5Mbps) nearly all of the time over either VPN tunnel, as well as when just routed out over the WAN.

When I upgraded to pfSense 2.4.0, I was hoping for the AES-NI extensions in the CPU would help with the GCM cipher, although I wasn't seeing really huge CPU usage before when saturating the connection, but I thought I could try it. So, I changed the VPN settings on my own tunnel to AES-GCM-256 in the negotiated ciphers, and confirmed in the logs that it is using that cipher when connected to the other end. The paid service remains at AES-CBC-256.

Ever since the upgrade, I notice sluggish performance and several speed tests on different days and times tell me I'm getting 2-5Mbps in both directions. This is true for both VPN tunnels, but when for traffic routed directly out of the WAN, speed tests show 20Mbps/5Mbps.

Any help on where to begin debugging this would be appreciated. To repeat, it is happening with both AES-CBC and AES-GCM, and not at all when router directly over the WAN interfaces. This is consistent.

gertty · Oct 27, 2017, 1:05 PM

Ok, so I've had a bit more time to play with this. The following two changes have improved things:

In System -> Advanced -> Miscellaneous, set Cryptographic Hardware to "None" (I had it set to AES-NI). After reboot, this improved my paid-for VPN service, which is talking to a (I think 2.3 OpenVPN server that will only do AES-256-CBC)

After the above, the connection to my own OpenVPN server (2.4 using AES-256-GCM in ncp-chiphers) showed no improvement. To fix that I did:

On both the client and server configs, force the cipher to be AES-256-CBC.

With both of these in place, I get roughly line speed both up and down thru both VPN tunnels. BTW, even when the throughput was slow, the CPU on the pfSense router didn't looked very busy.

kejianshi · Oct 27, 2017, 3:40 AM

You say you switch your crypto to "none" and saw improvement?

What was the setting there before you changed it?

gertty · Oct 27, 2017, 1:04 PM

@kejianshi:

You say you switch your crypto to "none" and saw improvement?

What was the setting there before you changed it?

System -> Advanced -> Miscellaneous -> Cryptographic Hardware has three options for me, "None", "AES-NI", "BSD Crypto Device", "AES-NI and BSD Crypto Device"

Previously (pFSense 2.3.x, when the tunnels were fast) I has it set to AES-NI. Immediately after upgrade it was set to AES-NI, but then the tunnels seem limited to 5Mbps. Changing it to None, helped the paid-for VPN tunnel that was always using CBC. It did not help my own tunnel until I switched it to CBC.

kejianshi · Oct 27, 2017, 1:20 PM

Thing is, with or without AES-NI, you should easily be able to get 20 or more.

I get up to 40 and sometimes 45 or so through the vpn to my laptop here from the USA and neither end has AES-NI and both ends are old hardware.

Thats my bandwidth being saturated, not a pfsense limit. I've seen 60/60 via vpn on my private machine and its sitting on a 60/60 connection. Again. Not a pfsense limit, even without AES-NI.

Heck - I used to pull 6/6 up and down via an e2000 dd-wrt router.

It could be a hardware problem or settings issue on either end, but it is more likely an ISP issue. I've run into ISP throttling very often when using VPNs.

gertty · Oct 27, 2017, 3:06 PM

@kejianshi:

Thing is, with or without AES-NI, you should easily be able to get 20 or more.

I get up to 40 and sometimes 45 or so through the vpn to my laptop here from the USA and neither end has AES-NI and both ends are old hardware.

Thats my bandwidth being saturated, not a pfsense limit. I've seen 60/60 via vpn on my private machine and its sitting on a 60/60 connection. Again. Not a pfsense limit, even without AES-NI.

Heck - I used to pull 6/6 up and down via an e2000 dd-wrt router.

It could be a hardware problem or settings issue on either end, but it is more likely an ISP issue. I've run into ISP throttling very often when using VPNs.

I agree, and PRIOR to the upgrade to pfSense 2.4.0 (now running 2.4.1), I did see 20Mbps down/ 5Mbps up which is the limit of my connection. I've been doing this for many months on my connection with no problems.

As soon as a upgraded, I noticed the slowdown. I don't see how it can be the ISP throttling me. I didn't see throttling when I was using pfSense 2.3, and the problem appears better when I force my connection to AES-256-CBC. Although for one VPN tunnel, even CBC mode was slow when I had AES-NI enabled in the menu I mentioned above.

It appear OpenVPN 2.4 which is what is shipping with pfSense 2.4, wants to default to GCM if the other end supports it. I have yet to find a combination of settings that lets me use GCM without being limited to about 5Mbps.

kejianshi · Oct 27, 2017, 4:11 PM

How far apart is your server from your client?

I'm about 8k miles from my server, using AES-GCM 256. Getting about 30/30 tonight. Trust me. Thats isp traffic shaping in my case.

Had a terrible time getting good speed until I changed the send/receive buffers.

Stuck these lines in the pfsense openvpn config under custom options on the server side. Again, this is long haul. It can't get any longer!

keepalive 5 120;
push "sndbuf 524288";
push "rcvbuf 524288";

gertty · Oct 27, 2017, 4:39 PM

@kejianshi:

How far apart is your server from your client?

I'm about 8k miles from my server, using AES-GCM 256. Getting about 30/30 tonight. Trust me. Thats isp traffic shaping in my case.

Had a terrible time getting good speed until I changed the send/receive buffers.

Stuck these lines in the pfsense openvpn config under custom options on the server side. Again, this is long haul. It can't get any longer!

keepalive 5 120;
push "sndbuf 524288";
push "rcvbuf 524288";

In miles, I'm maybe ~10-20miles as the crow files. In ping time? I think <60ms. I don't know how how many hops. I'll check later.

I'll try those settings over the weekend.

But let's look at this as two problems (I have two OVPN clients running, so two tunnels)

On the paid-for-one, the server is enforcing AES-256-CBC. Running pfSense 2.3, with AES-NI set for Crypto. I got as much bandwidth as my connection allows for months. Right after I upgraded to pfSense 2.4. no changes on the server side, I can't control that. No intentional changes on the client config on my side and then the bandwidth drops to no more than 5Mbps in each direction. As soon as I set Hardware Crypto to "None" and reboot, this tunnel resumes the pre-upgrade behavior of delivering data about as fast as the underlying connection allows.
For the vpn server I control, I have had 2.4.x running on the server for months, on the pfSense client, under pfSense 2.3, using AES-256-CBC with the same "AES-NI" hardware crypto settings as above, I also got as much speed as my underlying WAN connection allowed. This was for months. When I upgraded to pfSense 2.4, I turned on AES-256-GCM and changed nothing else on the client. The server didn't need changes. I confirmed in the logs that this tunnel negotiated GCM. Speed dropped to 5Mbps in both directions. As soon as I forced this connection to use AES-256-CBC (required changes on both the client and server), I once again get about as fast as my underling connection allows.

In both cases, I was not seeing any throttling for months. Minutes after upgrading, I notice sluggish behavior in a browser, ran a speed test and saw about ~5Mbps instead of the 20down/5up I was used to. I confirmed it wasn't the underling WAN connection. Turning encryption to None instantly fixed the first problem. Forcing the CBC mode cipher fixed the 2nd one.

kejianshi · Oct 27, 2017, 5:09 PM

It is odd. No GCM troubles here. Not sure what to tell you.

I'm really impressed by the new features the openvpn engine in pfsense brings along.

So, lets say I have some old devices that like blowfish-cbc 128 and not much else.

I can set that as the primary cryp for openvpn and also set up the Enable Negotiable Cryptographic Parameters on the server side (pfsense)

Then my old junk will be able to connect and the newer stuff will automatically default to more secure algorithms.

Annnnnnddd. As a plus, they did it without breaking anything, which is always nice.

gertty · Nov 28, 2017, 4:23 AM

Update: MSSFIX!!!

So, after exhausting what I thought were the reasonable options around crypto, I settled into just using CBC mode. Then about a week ago, this CBC config went back to being about 2-5Mbps. After talking to a couple of friends, someone suggested this sounded like there could be some sort of retransmit thing going on. That is, something related to MTU or MSS.

So, I did some reading and ended up here: https://www.sonassi.com/help/troubleshooting/setting-correct-mtu-for-openvpn
That ping test given at that URL gave me an MSS of 1430. Setting 'mssfix 1430' on the server seems to have fixed the problem! I even went back to the OpenVPN 2.4.x default AES-256-GCM crypto and I'm still seeing the speeds I expect.

I've read enough to know this fixes only TCP streams. I'm also not sure if I can set mssfix only on the problematic client (the pfSense router) and let my other clients use their defaults.

Gil · Nov 28, 2017, 10:41 AM

Interesting post, thanks for providing an update on the MTU.

I also agree with kejianshi. NCP is working well for my array of platforms.

Nice to see pfSense providing backward compatibility whilst advancing rapidly.