22.05 and NordVPN tunneling
-
I have a netgate appliance 4100 running 22.05. I have FTTH at 1GBps. When I tunnel my traffic through a VPN on the appliance, my speed goes down to about 180MBps.
That seems like quite a drop. I know there is some overhead and CPU limits, but I was wondering what settings I should check to make sure I am configured as best as I can be.
I have followed many different step-throughs on NordVPN forums as well as the netgate pages and others. I have made some changes (as each step-through is a little different), but no luck making any improvement on throughput.
I am not using wireless. I'm all wired and get ~800+ when I disable the client on pfsense. I have tried different servers on Nord...closest and others.
If it matters, I'm using NordVPN.
-
What encryption settings are they using/allowing?
Enabling DSO with QAT would give you a significant increase if it's cpu limited but that can only use AES-CGM.
Check the system activity page whilst testing the speed and see if one CPU core is at 100%.
180Mbps would be a low result though so I suspect it might be limited at the server end or by route conditions.Steve
-
I'm not sure how to see each core on my 4100, but the CPU chart on the main page was at 55-61% during the test.
Supposedly this unit should be able to easily do 500+ while tunneling (was my understanding per the specs).
-
It's shows per core in Diag > System Activity.
Or you can run at the command line:
top -HaSP500Mbps would be a very good result for OpenVPN on the 4100. What settings are you using? Does Nord allow other encryption types?
Steve
-
@stephenw10 Here are the specs for the 4100. https://shop.netgate.com/products/4100-base-pfsense
It shows it can do much much much more than 50mbps.
Using the command prompt (I forgot about that), I get very little response. It's like it's not reporting correctly.
last pid: 88049; load averages: 0.54, 0.31, 0.16 up 32+23:48:49 09:56:34
603 threads: 4 running, 569 sleeping, 30 waiting
CPU 0: 0.6% user, 0.2% nice, 1.3% system, 0.4% interrupt, 97.6% idle
CPU 1: 0.3% user, 0.2% nice, 1.0% system, 0.1% interrupt, 98.4% idle
Mem: 62M Active, 175M Inact, 472M Wired, 3105M Free
ARC: 203M Total, 28M MFU, 167M MRU, 301K Anon, 903K Header, 6855K Other
81M Compressed, 206M Uncompressed, 2.53:1 Ratio -
Sorry I typo'd that, I meant 500Mbps. That's for OpenVPN, it will do more for IPSec.
You need to use the actual command line not the Command Prompt page in the gui, that is non interactive. Or use Diag > System Activity which does update.
Steve
-
If I log in directly to a shell, it shows about 20-30% for each CPU as it does a speedtest.
-
@stephenw10 So, based on the 20-30% cpu, what do you suggest at this point?
-
What is that load? What does it actually show?
It could be a limit on the server side or the route. Are you able to see more than that using a client device directly to the same Nord server?
Steve
-
@stephenw10 Sorry, I guess I don't know what you mean by the load....other that 20-30% on each cpu.
Yes, I can get 800+ mb/s without the VPN connected.
Yes, it could be the far end, but I have tried different servers on NordVPN and I get very similar results.
-
I mean if I test an OpenVPN tunnel on a 4100 here and look at the activity I see:
last pid: 6977; load averages: 0.64, 0.30, 0.16 up 71+01:47:45 20:28:48 539 threads: 4 running, 522 sleeping, 13 waiting CPU 0: 36.6% user, 0.0% nice, 43.3% system, 0.0% interrupt, 20.1% idle CPU 1: 12.2% user, 0.0% nice, 30.3% system, 0.0% interrupt, 57.5% idle Mem: 33M Active, 261M Inact, 472M Wired, 3033M Free ARC: 237M Total, 92M MFU, 138M MRU, 692K Anon, 1027K Header, 5445K Other 133M Compressed, 249M Uncompressed, 1.87:1 Ratio Swap: 1024M Total, 1024M Free Message from syslogd@4100-2 at Oct 5 20:24:41 ...C TIME WCPU COMMAND 21459pm[80344]: /i 96 0 17M 7468K CPU0 0 46:25 93.43% /usr/local/sbin/openvpn --config /var/etc/openvpn/client1/config.ovpn 11 root 155 ki31 0B 32K RUN 1 1684.1 59.69% [idle{idle: cpu1}] 11 root 155 ki31 0B 32K RUN 0 1685.2 18.61% [idle{idle: cpu0}] 96097 root 26 0 15M 6668K select 0 0:02 12.32% iperf3 -c 10.2.4.1 -t 60 0 root -76 - 0B 544K - 1 15:31 7.75% [kernel{if_io_tqg_1}] 0 root -76 - 0B 544K - 0 4:25 1.24% [kernel{if_io_tqg_0}] 74666 root 52 0 133M 58M accept 0 0:17 0.38% php-fpm: pool nginx (php-fpm) 0 root -92 - 0B 544K - 0 242:49 0.32% [kernel{dummynet}] 0 root -76 - 0B 544K - 1 165:14 0.18% [kernel{if_config_tqg_0}] 20 root -16 - 0B 16K - 0 12:35 0.17% [rand_harvestq] 71350 root 20 0 15M 5528K CPU1 1 0:00 0.12% top -HaSPSo you can see OpenVPN using most of the CPU. iperf itself is using quite a bit there because I'm testing from the firewall directly.
Try testing a nordvpn client that isn't running on the firewall and see what sort of throughput you can get to the same server. It may be that's the best you can get and we are chasing the unobtainable.
Steve
-
@stephenw10 said in 22.05 and NordVPN tunneling:
Ok, here is what when speed testing with the vpn running on the 4100...top -HaSP
last pid: 21582; load averages: 0.27, 0.20, 0.13 up 35+04:02:56 14:10:41 603 threads: 4 running, 569 sleeping, 30 waiting CPU 0: 15.7% user, 0.0% nice, 34.1% system, 0.0% interrupt, 50.2% idle CPU 1: 23.1% user, 0.0% nice, 34.9% system, 0.0% interrupt, 42.0% idle Mem: 38M Active, 185M Inact, 473M Wired, 3117M Free ARC: 207M Total, 29M MFU, 170M MRU, 432K Anon, 918K Header, 6906K Other 84M Compressed, 212M Uncompressed, 2.53:1 Ratio PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 13920 root 85 0 17M 7480K CPU0 0 124:02 85.90% /usr/local/sbin/openvpn --config /var/etc/openvpn/client6/config.ovpn 11 root 155 ki31 0B 32K RUN 0 823.4H 50.90% [idle{idle: cpu0}] 11 root 155 ki31 0B 32K RUN 1 829.0H 42.11% [idle{idle: cpu1}] 0 root -76 - 0B 528K - 0 102:18 16.04% [kernel{if_io_tqg_0}] 0 root -76 - 0B 528K - 1 114:40 2.98% [kernel{if_io_tqg_1}] 73000 root 20 0 12M 3480K bpf 1 5:38 0.85% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid 25057 root 20 0 11M 2640K select 1 4:47 0.42% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/s 12 root -60 - 0B 480K WAIT 1 155:42 0.33% [intr{swi4: clock (0)}] 20 root -16 - 0B 16K - 0 25:18 0.17% [rand_harvestq] 23700 root 20 0 14M 4856K CPU1 1 0:01 0.12% top -HaSP 0 root -76 - 0B 528K - 0 31:00 0.06% [kernel{if_config_tqg_0}] 19 root -16 - 0B 16K pftm 1 19:02 0.06% [pf purge] 21929 root 20 0 11M 2184K select 1 0:54 0.01% /usr/sbin/powerd -b hadp -a hadp -n hadp 5420 dhcpd 20 0 23M 12M select 1 0:42 0.01% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -
Ok, well neither CPU core is anywhere near 100% there so it's not CPU limiting.
I would have to assume it's either limited at the server side or as a result of something in the route.
So check what throughput you can see using a NordVPN client running off the firewall.
What VPN client settings are you using in pfSense currently?
Steve
-
@stephenw10
So, if I use UDP, it cuts the speed way down. If I use Nord Lynx, it helps a LOT (testing with the same server). Is it possible to setup Nordlynx on pfsense? -
Nord Lynx is mostly Wireguard. So...maybe! I've never tried it and it may require some extra bits pfSense cannot currently do.
OpenVPN over TCP is almost always significantly slower than UDP. So if you're seeing the opposite that's suspicious. What settings have you tried? What were you using for the results you have stated here?
Steve
-
@stephenw10
I'm sorry I meant UDP vs Nord Lynx was slower vs faster. I can't get TCP to work, so no worry there. -
Ok, well it looks like Nord Lynx can be made to work in pfSense. You just need to install their client in something else and extract the keys first because for some reason they won't give them to you directly.
Or use one of the other VPN providers that do support Wireguard dircetly.
But you still haven't said what client settings you're using in OpenVPN so you might just have very slow encryption. Or no fastio. Or too smaller buffers.
Steve
-
@stephenw10
Sorry about not getting the client config to you...dev tun persist-tun persist-key ncp-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC cipher AES-256-CBC auth SHA256 tls-client client resolv-retry infinite remote xxx.xxx.xxx.xxx 1194 tcp-client nobind auth-user-pass remote-cert-tls server -
Ok, so you have AES-128-GCM available as a cipher via ncp which should be the fastest available. Can you see if Nord is negotiating that in the connection logs?
Since you're using UDP I would enable FastIO and increase the send/recv buffers to 512K. Both of those should give you an increase in throughput.
Steve
-
@stephenw10
The fast io option was already checked in the pfsense client GUI. The buffer was set to "default", whatever that is and I updated to 512k.On a separate note...regarding wireguard...I found another page to setup the wireguard because the reddit post ws limited....and it is overwhelming. I get the interface to show as "up", but wasn't routing traffic even though I set the pass option in the new interface and set a lan firewall entry to use that gateway.
