PFSense and PIA - Slow download speed

[rothbard]

I've had some more time to test hardware acceleration and it's not working for PIA, but is for pfSense.

In console, openssl speed -evp -aes-256-cbc:

• aes-ni hw acceleration: 0.03-0.41 seconds
• without hw accel: 2.9 - 3 seconds
  Pia VPN with hw accel bsd cryptodev … aes-256-cbc checked OR unchecked, SAME results using top in shell (and yes, I've got the 4k cert from pia):
  -140mbps dl @ 35% cpu
  -14mbps ul @ 15% cpu

Checking the log for pia vpn:
Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'

So, I know hardware acceleration works in pfSense, but not for PIA. This lets me conclude PIA accepts it, but overrides it with the blowfish encryption…
I'd love to see some factual data to confirm or deny my thought. The internet is full of conjecture (people enabling it, but it doesn't do anything), but no proof if it works or not as what they claim or do… Enlighten me please, thanks!

EDIT:
Ok, I tested further and checked logs with verb 4 instead of 3 and see cipher is used for data channel encrypt and decrypt, which is very good. The remaining problem is, how to get hardware acceleration working for the encrypted connection for pia... will test further, but until now, there are no differences between on and off, while it does for the host machine.

[rothbard]

Ok, after reading the doc for openvpn, openssl and especially pfsense, I assume AES-NI instruction set for is automatically used by PIA OpenVPN, regardless if you have it enabled or not at the openvpn client?

The following quote from pfsense wiki seems at the least not clear (src @ https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supported#Practical_Use):
To take advantage of acceleration in OpenVPN, choose a supported cipher such as aes-128-cbc on each end of a given tunnel, then select BSD Cryptodev Engine for Hardware Crypto.
…
Nothing needs selected for OpenVPN to utilize AES-NI. The OpenSSL engine has its own code for handling AES-NI that works well without using the BSD Cryptodev Engine.

Ok, as I've assumed (it either doesn't work for pia or it's enabled by default) before I've read it, because from observations during pia speedtesting, there was no difference in cpu load per same throughput.
It would be nice to make that clear in the openvpn client config page  8), because it's very confusing when it's enabled by default in stealth when intuition tells one you have to enable it, because you have the option to, especially when the aes-256-cbc is listed (by enabling aes-ni for intel in the hardware acceleration section of System -> Advanced -> Miscellaneous).

So then… the load is still the same whether I've enabled hw accel in miscellaneous or not... rebooted and did 3x test for each mode and the results made zero differences...
I'm not in the ability to test it with vt-d off, since the host needs continuity. Maybe other people can chime in and confirm or make my argument rebuttable as to why and how. Thank you.

As for the moment, I don't know if it's enabled by default or not and the options of hardware acceleration are just obsolete, not only for aes-ni for openvpn client, but also for miscellaneous.

[rothbard]

Alright, after doing some more tests and checking the logs, I've noticed a lot of "write UDPv4: No buffer space available (code=55)" entries in openvpn log and sometimes a large filetransfer abruptly stops. I've Google a lot and there doesn't seem to be a definitive solution, let alone the cause to resolve it in the first place (many possible scenario's and theories, yes, but not definitive answer). Instead of wasting more hours on research, I've decided to test it by trial and error and increased the buffer sizes with factor 2 to a total of 1MB for both receive and send socket buffers. With that being set, the buffer error does barely come up anymore and I've got a very stable connection without drops, even at max speed file transfers.

@mauroman33 I've rechecked your settings again and I've notcied that you didn't enable "Don't pull routes". I can advice you to enable that, otherwise you'd run in to issues later on. You can verify it by checking your logfile for entries:
NOTE: unable to redirect default gateway – Cannot read current default gateway from system 
ROUTE: default_gateway=UNDEF 
Could not retrieve default gateway from route socket:: No such process (errno=3)

When you don't pull routes, you won't have those issues anymore and less write udp buffer errors.

Greetings!

[rothbard]

I've changed buffers to 1572864 (the limit seems to be 3700KB, because more openvpn will force to default values). No more buffer errors and very smooth (pfsense running as hyper-v guest with pia vpn as client and totally locked down in case gateway goes down (i.e. killswitch) + another hyper-v guest going through pfsense maxing out throughput isp) :) Also wrote a ps script which forwards pia forwarding port in my program. All is done [for now…  ;)]

EDIT:
I forgot to mention that I also changed net.inet.tcp.recvspace & net.inet.tcp.sendspace (under System -> Advanced -> System Tunables) to max 2048K (=2097152 bytes) as defined by Intel, which will also increase throughput for your entire LAN as well.
So… openvpn uses 75% of 2048K available buffer, so 512K is available at worst case scenario for the rest. Everything is working consistently without anymore errors  :)

[rothbard]

Ok, increasing the buffers only delayed the dreaded buffer error, but it will pop up after having max throughput for a certain amount of time. I've finally got rid of it by starting from scratch and is now fully working without any errors.
I've got a feeling that my routes got messed up, even though they showed up fine in Diagnostics -> Routes (even with pia connected). I had a complex setup with lots of nat, tags, vlans, etc. where this test machine was behind a double nat and triple router, but that is not of relevance to the issue that has now been resolved. Another possibility might be Snort. A good reminder for me to isolate machines to the scope they're intended to be used, instead of installing all kinds of packages and last, but definitely not least, a very good reminder why pfSense rocks! :)

For anybody who has the same issues:

• Low throughput => Increase send and receive buffers with openvpn arguments/options
• Udp buffer write errors => messed up routes c.q. snort c.q. other packages => last resort, make backup of configs and start from scratch

That's it, all is good!  :D

[joelones]

@rothbard:

Also wrote a ps script which forwards pia forwarding port in my

I was wondering if you could perhaps share this script? I'd be interested to implement something similar.

[bgbird03]

Well it seems like I've landed in the right spot. I was struggling with this for the past 48-72 hours (brand new to the VPN & pfSense world), and was so excited to get my connection going only to pull between 1-3 Mbps connecting to any PIA VPN server (under any encryption settings). I have a 50 Mbps up/down Verizon FIOS link (small compared to everyone else on the forums, but whatever), so this 1-3 Mbps was especially pathetic. I'm running pfSense as the OS on a  computer with "plenty" of hardware to take care of my negligible bandwidth requirements, and no resources are even close to maxing out.

So far I've changed the compression to Adaptive LZO, switched send/receive buffer to 2.00 MiB, and have the following settings:

tls-client;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
remote-cert-tls server;
persist-key;
persist-tun;
persist-remote-ip;
keysize 256;
reneg-sec 0;
link-mtu 1540;
fragment 0;
mssfix 0;
fast-io;
sndbuf 524288;
rcvbuf 524288;

This results in VPN speeds around 10-15 Mbps, but never more. This is awesome because my bandwidth has increased ~10,000% (10x), but still is pretty pathetic. Any other suggestions? I keep screwing around with variations of the above, and can't really break out of that bandwidth range.

Thanks!

[Rango]

@rothbard:

I've changed buffers to 1572864 (the limit seems to be 3700KB, because more openvpn will force to default values). No more buffer errors and very smooth (pfsense running as hyper-v guest with pia vpn as client and totally locked down in case gateway goes down (i.e. killswitch) + another hyper-v guest going through pfsense maxing out throughput isp) :) Also wrote a ps script which forwards pia forwarding port in my program. All is done [for now…  ;)]

EDIT:
I forgot to mention that I also changed net.inet.tcp.recvspace & net.inet.tcp.sendspace (under System -> Advanced -> System Tunables) to max 2048K (=2097152 bytes) as defined by Intel, which will also increase throughput for your entire LAN as well.
So… openvpn uses 75% of 2048K available buffer, so 512K is available at worst case scenario for the rest. Everything is working consistently without anymore errors  :)

My friend was this change turn out to be stable without connection dropping or error in log? What speed increases did you get from this alone?

 I also changed net.inet.tcp.recvspace & net.inet.tcp.sendspace (under System -> Advanced -> System Tunables) to max 2048K (=2097152 bytes)

Can you post your final working config?
I have N3150 with AES-NI extentions and my ISP line is 180Mbps. I'm hoping i can do 180Mbps less 10% so around 160Mbps with those tweeks you guys did and AES-NI extention on and hardware

acceleration in VPN client ON. Is that achievable? I noticed you guys have this option off in openvpn client (not system settings. why is that? It should help no?

[katinatez]

@mauroman33:

Here my configuration, my custom options and the speed that I reach

explicit-exit-notify 2;
ifconfig-nowarn;
tls-client;
persist-key;
persist-tun;
persist-remote-ip;
remote-cert-tls server;
auth-nocache;
keysize 256;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
fast-io;
sndbuf 524288;
rcvbuf 524288

Thank you so much!! have been struggling with PIA speeds. Tried just about every configuration but could not pass 45 Mb download with a 400/40 Mb internet plan. I have achieved  the best speeds so far with your config!!

My question is if I want to use 128 encryption would the special config change to this? Thanks for your reply.

This is running pfsense 2.4.3 for referrence for other users.

explicit-exit-notify 2;
ifconfig-nowarn;
tls-client;
persist-key;
persist-tun;
persist-remote-ip;
remote-cert-tls server;
auth-nocache;
keysize 128;
tls-cipher TLS-DHE-RSA-WITH-AES-128-CBC-SHA;
fast-io;
sndbuf 524288;
rcvbuf 524288


[mauroman33]

@katinatez:

Thank you so much!! have been struggling with PIA speeds. Tried just about every configuration but could not pass 45 Mb download with a 400/40 Mb internet plan. I have achieved  the best speeds so far with your config!!

My question is if I want to use 128 encryption would the special config change to this? Thanks for your reply.

This is running pfsense 2.4.3 for referrence for other users.

explicit-exit-notify 2;
ifconfig-nowarn;
tls-client;
persist-key;
persist-tun;
persist-remote-ip;
remote-cert-tls server;
auth-nocache;
keysize 128;
tls-cipher TLS-DHE-RSA-WITH-AES-128-CBC-SHA;
fast-io;
sndbuf 524288;
rcvbuf 524288

I have N3150 with AES-NI extensions and my ISP line is 100/20 Mbps.
I need to actually say I didn't noticed any appreciable speed difference setting down AES-128-CBC.
Your config looks good, just try it.