PFSense and PIA - Slow download speed

Wisiwyg

Looking for the hardware acceleration option under System - Advanced - Miscellaneous on version 2.3.2 Development and don't find it. I do see an option for Cryptographic Hardware. Is that what you're referring to? Are you suggesting turning that option ON has a positive impact? TIA

mauroman33

@Wisiwyg:

Looking for the hardware acceleration option under System - Advanced - Miscellaneous on version 2.3.2 Development and don't find it. I do see an option for Cryptographic Hardware. Is that what you're referring to? Are you suggesting turning that option ON has a positive impact? TIA

You're right, I referred to that one.
I read somewhere here that if the CPU supports the AES-NI instructions then that option should be enabled, but actually I don't know how much big its impact is.

lcalamar

I had set up my ASUS router so that all my internet access was running through PIA VPN via openvpn

I've been looking for a way to setup my pfsense firewall the same way (no client apps) - so that any device connected in my home will be running under PIA.

I can only find information on setting up openvpn clients - which isn't what I want.

Is there a way to configure PIA via openvpn on PFSENSE.

OH- would be GREAT if I could do 2 other things:

have an easy way to toggle on/off from my UI - so I can turn off to watch Netflix
or better yet - can I somehow route NetFlix traffic so it isn't going through VPN? (stupid Netflix)

secdeath

@lcalamar:

I had set up my ASUS router so that all my internet access was running through PIA VPN via openvpn

I've been looking for a way to setup my pfsense firewall the same way (no client apps) - so that any device connected in my home will be running under PIA.

I can only find information on setting up openvpn clients - which isn't what I want.

Is there a way to configure PIA via openvpn on PFSENSE.

OH- would be GREAT if I could do 2 other things:

have an easy way to toggle on/off from my UI - so I can turn off to watch Netflix

or better yet - can I somehow route NetFlix traffic so it isn't going through VPN? (stupid Netflix)

Oh you can do all of this if you really wanted ;-P Look at the link the original poster made and follow that. I have additional setup host based routing through each VPN tunnel i have setup. Currently have 3 different VPN tunnels. 2 through PIA and 1 through ipvanish. I'm eventually just going to move completely to PIA though. Most of my traffic goes out through the default route via my ISP. I have other traffic going out through my VPN tunnels depending on which IP i am coming from. I have one subnet routing entirely through PIA as well.

You could tech break it down even further and send certain traffic going to a specific destination and coming from a specific host and route that through you PIA VPN. IT all depends on what you eventually want to do… You really need to get the VPN setup first. I can help you after that if you want though.

lcalamar

Geez - that actually seemed to work!!! (I'm not sure exactly what it is I did - I need to study it a little more)

Here's what my NAT OUtbound looks like now (I had quite a few interfaces) (attached)… I did confirm that I"m running under a different IP now... and I'm still getting throughput.

So - how to toggle on/off and setup so NetFlix isn't blocked?

Thanks much for all the help... doh that I didn't see that setup in the first message... seemed a little intimidating - but actually not bad to do - as I said I need to get a better basic understanding of what I just did!

![Image 1.png](/public/imported_attachments/1/Image 1.png)
![Image 1.png_thumb](/public/imported_attachments/1/Image 1.png_thumb)

lcalamar

Quick note:

I was running PIA on my ASUS RTN66U router prior to getting PFSENSE on my ITX Celeron box.

The speed/performance with OPENVPN is great. Barely noticeable - whereas on the ASUS RTN66U - was considerably slower throughput.

tigs

@mauroman33:

@tigs

Glad to hear.
I honestly don't know the reasons, I've never had problems with speed.
About compression, its activation did not give me any advantage so I preferred to turn it off.
The hardware acceleration is set in System-> Advanced-> Miscellaneous.
If set also in the VPN Client settings, after a while the connection drops, and not only with PIA but also with IPVanish and PureVPN.

Hey @mauroman
I am having this problem with sustain connection. It seems to drop very frequently. When I try to use speedtest.net, I often noticed interruptions in the process of testing.

here is the warning part of the log:

Jul 8 18:07:42	openvpn	45458	WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1570', remote='link-mtu 1542'
Jul 8 18:07:42	openvpn	45458	WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'
Jul 8 18:07:42	openvpn	45458	WARNING: 'auth' is used inconsistently, local='auth SHA256', remote='auth SHA1'
Jul 8 18:07:42	openvpn	45458	WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'

any idea?

Second, how should I setup the custom config if want to try the AES128?

thanks

Pippin

@tigs:


Jul 8 18:07:42	openvpn	45458	WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1570', remote='link-mtu 1542'
Jul 8 18:07:42	openvpn	45458	WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'
Jul 8 18:07:42	openvpn	45458	WARNING: 'auth' is used inconsistently, local='auth SHA256', remote='auth SHA1'
Jul 8 18:07:42	openvpn	45458	WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'

any idea?

If you read what the snippet from the log states.
You have mismatched config/parameters.
Look at local (you the client) and remote (the server PIA).

Make them match or maybe better, download your config from PIA…..

Second, how should I setup the custom config if want to try the AES128?

Client needs matching config with server, you cannot just change it on one side (or at least not all).

AndrewZ

@tigs:

here is the warning part of the log:

Jul 8 18:07:42	openvpn	45458	WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'
Jul 8 18:07:42	openvpn	45458	WARNING: 'auth' is used inconsistently, local='auth SHA256', remote='auth SHA1'
Jul 8 18:07:42	openvpn	45458	WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'

any idea?

Please check https://forum.pfsense.org/index.php?topic=103934.msg634831

mauroman33

Ciao @tigs

in my log there are no warnings, so I think it depends on the server's configuration which you are connected to.
Not all their servers seem configured in the same way. Sometimes a service that is available from a server is not available from another one.
As you can see in my first screenshot I'm using sweden.privateinternetaccess.com. You could try using a different server, or you should match your configuration parameters to the chosen server.
About the dropping, if you have bandwidth enough you may try to disable the compression. Maybe this can help the connection's stability.

Wisiwyg

Firstly, really appreciate the OP and guide and all of the helpful input on this. I see multiple posts in other threads about slow throughput on PIA. As my signature states, I"m running hardware that should breeze through this. The best throughput I am able to get is 3.5mbps down, 3.3mbps up. I've tried multiple endpoints, too. I'm in Denver - us-west is the closest, then us-midwest, and I've tried us-east, others. None of them make a difference in throughput.

I used the tutorial here to set everything up: https://forum.pfsense.org/index.php?topic=76015.0 and the tweaks in this thread to attempt to improve throughput without resolution. Everything else seems to work.

So now I'm wondering if Comcast has introduced throttling on any VPN endpoints. You see, I've tried SlickVPN and PureVPN as well. Slick because they have an endpoint in Denver and Pure because they have feedback as being very fast. None of the three VPN services get above 3.5mbps. Anyone have experience with throughput issues on Comcast? TIA

lcalamar

I'm using Comcast Cable - and not seeing and governing of speeds while I'm on VPN.

The only issue I have is NetFlix

Wisiwyg

@Wisiwyg:

Firstly, really appreciate the OP and guide and all of the helpful input on this. I see multiple posts in other threads about slow throughput on PIA. As my signature states, I"m running hardware that should breeze through this. The best throughput I am able to get is 3.5mbps down, 3.3mbps up. I've tried multiple endpoints, too. I'm in Denver - us-west is the closest, then us-midwest, and I've tried us-east, others. None of them make a difference in throughput.

I used the tutorial here to set everything up: https://forum.pfsense.org/index.php?topic=76015.0 and the tweaks in this thread to attempt to improve throughput without resolution. Everything else seems to work.

So now I'm wondering if Comcast has introduced throttling on any VPN endpoints. You see, I've tried SlickVPN and PureVPN as well. Slick because they have an endpoint in Denver and Pure because they have feedback as being very fast. None of the three VPN services get above 3.5mbps. Anyone have experience with throughput issues on Comcast? TIA

And it would be helpful to also state that without OpenVPN in place I'm getting 120 down, 12 up.

@lcalamar:

I'm using Comcast Cable - and not seeing and governing of speeds while I'm on VPN.
The only issue I have is NetFlix

Yes, have that too, but not a problem if I can only get 3.5 down - Netflix wouldn't be bearable.

rothbard

@mauroman33:

Here my configuration, my custom options and the speed that I reach

explicit-exit-notify 2;
ifconfig-nowarn;
tls-client;
persist-key;
persist-tun;
persist-remote-ip;
remote-cert-tls server;
auth-nocache;
keysize 256;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
fast-io;
sndbuf 524288;
rcvbuf 524288

Thank you so much mauroman33! I've been pulling my hair out for the last 20 hours why pia vpn as client in pfsense always capped at 40mbps, while only utilizing 7% cpu load (openvpn process) on 2 cores @ 3.5GHz each.

I thought my pfsense box was not configured right, since a vm guest getting an ip assigned from pfsense was pulling 145mbps with the pia vpn client running on it.

I then proceeded to start from scratch by installing pfsense and directly setup pia in pfsense, tested it again with speedtest.net and again 40mbps.
Then I've googled a lot today and found many useless posts and outright wrong advice and even more posts where it was never resolved and the threads died. Then I found this thread, quickly scanned over it and found your post.
I've tried the above custom options for my pia client and it actually works! From 40mpbs to 142mbps (and only 20% load on openvpn).

How do you know about those openvpn options? Did you read their entire man? Very nice, thank you so much again :-)

I'm still left with 2 pfsense issues, which many other people also experience [and which also unresolved]

Very slow webgui after the first hour, which will only get worse and never get better (I've tried Chrome and it's totally useless, since pages will never load; I'm on Edge now and it works as of now)

I did found a working solution for this though by trial and error and I haven't read it anywhere else. I've put the var on ramdisk with 256mb and browsing the webgui is now instant [for the time being]. This makes me believe the requests from chrome (or webbrowsers in general, if this applies to Edge as well, we will see, it's working now) are cached in pfsense (rather than the browser, since deleting the browser cache didn't help at all);

cpu resource usage in pfsense != cpu resource usage in vm host (hyper-v). I.e., 100% load in pfsense (watched on console via shell with cmd 'tops') is only 7% tops in hyper-v. I've allocated 2/6 cores to pfsense with 100% reserve and limit, so you'd expect a a max 33% on the host, but nope. Vt-d and virtualizations are ofourse enabled in bios.

I've also spent a lot of time digging through posts, which people also never resolved and I'm too lazy to make a new thread for it. It's no priority [yet] and always have a lack of time at my hands, so be it for now.

Greetings!

rothbard

@rothbard:

@mauroman33:

Here my configuration, my custom options and the speed that I reach

explicit-exit-notify 2;
ifconfig-nowarn;
tls-client;
persist-key;
persist-tun;
persist-remote-ip;
remote-cert-tls server;
auth-nocache;
keysize 256;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
fast-io;
sndbuf 524288;
rcvbuf 524288

Thank you so much mauroman33! I've been pulling my hair out for the last 20 hours why pia vpn as client in pfsense always capped at 40mbps, while only utilizing 7% cpu load (openvpn process) on 2 cores @ 3.5GHz each.

I thought my pfsense box was not configured right, since a vm guest getting an ip assigned from pfsense was pulling 145mbps with the pia vpn client running on it.

I then proceeded to start from scratch by installing pfsense and directly setup pia in pfsense, tested it again with speedtest.net and again 40mbps.
Then I've googled a lot today and found many useless posts and outright wrong advice and even more posts where it was never resolved and the threads died. Then I found this thread, quickly scanned over it and found your post.
I've tried the above custom options for my pia client and it actually works! From 40mpbs to 142mbps (and only 20% load on openvpn).

How do you know about those openvpn options? Did you read their entire man? Very nice, thank you so much again :-)

I'm still left with 2 pfsense issues, which many other people also experience [and which also unresolved]

Very slow webgui after the first hour, which will only get worse and never get better (I've tried Chrome and it's totally useless, since pages will never load; I'm on Edge now and it works as of now)

I did found a working solution for this though by trial and error and I haven't read it anywhere else. I've put the var on ramdisk with 256mb and browsing the webgui is now instant [for the time being]. This makes me believe the requests from chrome (or webbrowsers in general, if this applies to Edge as well, we will see, it's working now) are cached in pfsense (rather than the browser, since deleting the browser cache didn't help at all);

cpu resource usage in pfsense != cpu resource usage in vm host (hyper-v). I.e., 100% load in pfsense (watched on console via shell with cmd 'tops') is only 7% tops in hyper-v. I've allocated 2/6 cores to pfsense with 100% reserve and limit, so you'd expect a a max 33% on the host, but nope. Vt-d and virtualizations are ofourse enabled in bios.

I've also spent a lot of time digging through posts, which people also never resolved and I'm too lazy to make a new thread for it. It's no priority [yet] and always have a lack of time at my hands, so be it for now.

Greetings!

I've narrowed it down to those two options (the others did top out max 60mbps dl, the two options below maxed out my throughput @ 145mbps dl):
sndbuf 524288;
rcvbuf 524288;

I've also put in mtu option, since I've noticed in the logs "WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1558', remote='link-mtu 1542'", where client mtu > server mtu => fragmentation, which is bad. Quote from OpenVPN man: "OpenVPN requires that packets on the control or data channels be sent unfragmented." I've set it to 1540, which is good. Setting it to 1542 will make it drop to 1470, so keep it at 1540. As a precaution, I've also set fragment and mssfix option. Increasing the buffers really helped the throughput, which makes sense. If I were the vpn server, I'd do the same. I've also removed explicit-exit-notify, because I like the timeout over faster socket closure. If the gateway is really down, the openvpn daemon will automatically stop anyway and watchdog will restart it for me automatically with e-mail notification (which in the end, decreases the amount of e-mails I'll receive and know it's not a false alarm vs a [more] serious blackout).

My working config now (max speed and no dropouts so far):
tls-client;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
remote-cert-tls server;
persist-key;
persist-tun;
persist-remote-ip;
keysize 256;
reneg-sec 0;
link-mtu 1540;
fragment 0;
mssfix 0;
fast-io;
sndbuf 524288;
rcvbuf 524288;

All is working perfectly, very happy :)

rothbard

I've had some more time to test hardware acceleration and it's not working for PIA, but is for pfSense.

In console, openssl speed -evp -aes-256-cbc:

aes-ni hw acceleration: 0.03-0.41 seconds
without hw accel: 2.9 - 3 seconds
Pia VPN with hw accel bsd cryptodev … aes-256-cbc checked OR unchecked, SAME results using top in shell (and yes, I've got the 4k cert from pia):
-140mbps dl @ 35% cpu
-14mbps ul @ 15% cpu

Checking the log for pia vpn:
Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'

So, I know hardware acceleration works in pfSense, but not for PIA. ~~This lets me conclude PIA accepts it, but overrides it with the blowfish encryption…~~
I'd love to see some factual data to confirm or deny my thought. The internet is full of conjecture (people enabling it, but it doesn't do anything), but no proof if it works or not as what they claim or do… Enlighten me please, thanks!

EDIT:
Ok, I tested further and checked logs with verb 4 instead of 3 and see cipher is used for data channel encrypt and decrypt, which is very good. The remaining problem is, how to get hardware acceleration working for the encrypted connection for pia... will test further, but until now, there are no differences between on and off, while it does for the host machine.

rothbard

Ok, after reading the doc for openvpn, openssl and especially pfsense, I assume AES-NI instruction set for is automatically used by PIA OpenVPN, regardless if you have it enabled or not at the openvpn client?

The following quote from pfsense wiki seems at the least not clear (src @ https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supported#Practical_Use):
To take advantage of acceleration in OpenVPN, choose a supported cipher such as aes-128-cbc on each end of a given tunnel, then select BSD Cryptodev Engine for Hardware Crypto.
…
Nothing needs selected for OpenVPN to utilize AES-NI. The OpenSSL engine has its own code for handling AES-NI that works well without using the BSD Cryptodev Engine.

Ok, as I've assumed (it either doesn't work for pia or it's enabled by default) before I've read it, because from observations during pia speedtesting, there was no difference in cpu load per same throughput.
It would be nice to make that clear in the openvpn client config page 8), because it's very confusing when it's enabled by default in stealth when intuition tells one you have to enable it, because you have the option to, especially when the aes-256-cbc is listed (by enabling aes-ni for intel in the hardware acceleration section of System -> Advanced -> Miscellaneous).

So then… the load is still the same whether I've enabled hw accel in miscellaneous or not... rebooted and did 3x test for each mode and the results made zero differences...
I'm not in the ability to test it with vt-d off, since the host needs continuity. Maybe other people can chime in and confirm or make my argument rebuttable as to why and how. Thank you.

As for the moment, I don't know if it's enabled by default or not and the options of hardware acceleration are just obsolete, not only for aes-ni for openvpn client, but also for miscellaneous.

rothbard

Alright, after doing some more tests and checking the logs, I've noticed a lot of "write UDPv4: No buffer space available (code=55)" entries in openvpn log and sometimes a large filetransfer abruptly stops. I've Google a lot and there doesn't seem to be a definitive solution, let alone the cause to resolve it in the first place (many possible scenario's and theories, yes, but not definitive answer). Instead of wasting more hours on research, I've decided to test it by trial and error and increased the buffer sizes with factor 2 to a total of 1MB for both receive and send socket buffers. With that being set, the buffer error does barely come up anymore and I've got a very stable connection without drops, even at max speed file transfers.

@mauroman33 I've rechecked your settings again and I've notcied that you didn't enable "Don't pull routes". I can advice you to enable that, otherwise you'd run in to issues later on. You can verify it by checking your logfile for entries:
NOTE: unable to redirect default gateway – Cannot read current default gateway from system
ROUTE: default_gateway=UNDEF
Could not retrieve default gateway from route socket:: No such process (errno=3)

When you don't pull routes, you won't have those issues anymore and less write udp buffer errors.

Greetings!

rothbard

I've changed buffers to 1572864 (the limit seems to be 3700KB, because more openvpn will force to default values). No more buffer errors and very smooth (pfsense running as hyper-v guest with pia vpn as client and totally locked down in case gateway goes down (i.e. killswitch) + another hyper-v guest going through pfsense maxing out throughput isp) :) Also wrote a ps script which forwards pia forwarding port in my program. All is done [for now… ;)]

EDIT:
I forgot to mention that I also changed net.inet.tcp.recvspace & net.inet.tcp.sendspace (under System -> Advanced -> System Tunables) to max 2048K (=2097152 bytes) as defined by Intel, which will also increase throughput for your entire LAN as well.
So… openvpn uses 75% of 2048K available buffer, so 512K is available at worst case scenario for the rest. Everything is working consistently without anymore errors :)

rothbard

Ok, increasing the buffers only delayed the dreaded buffer error, but it will pop up after having max throughput for a certain amount of time. I've finally got rid of it by starting from scratch and is now fully working without any errors.
I've got a feeling that my routes got messed up, even though they showed up fine in Diagnostics -> Routes (even with pia connected). I had a complex setup with lots of nat, tags, vlans, etc. where this test machine was behind a double nat and triple router, but that is not of relevance to the issue that has now been resolved. Another possibility might be Snort. A good reminder for me to isolate machines to the scope they're intended to be used, instead of installing all kinds of packages and last, but definitely not least, a very good reminder why pfSense rocks! :)

For anybody who has the same issues:

Low throughput => Increase send and receive buffers with openvpn arguments/options
Udp buffer write errors => messed up routes c.q. snort c.q. other packages => last resort, make backup of configs and start from scratch

That's it, all is good! :D