22.05 and NordVPN tunneling

stephenw10

What encryption settings are they using/allowing?

Enabling DSO with QAT would give you a significant increase if it's cpu limited but that can only use AES-CGM.

Check the system activity page whilst testing the speed and see if one CPU core is at 100%.
180Mbps would be a low result though so I suspect it might be limited at the server end or by route conditions.

Steve

davidstoll

I'm not sure how to see each core on my 4100, but the CPU chart on the main page was at 55-61% during the test.

Supposedly this unit should be able to easily do 500+ while tunneling (was my understanding per the specs).

stephenw10

It's shows per core in Diag > System Activity.

Or you can run at the command line: top -HaSP

500Mbps would be a very good result for OpenVPN on the 4100. What settings are you using? Does Nord allow other encryption types?

Steve

davidstoll

@stephenw10 Here are the specs for the 4100. https://shop.netgate.com/products/4100-base-pfsense

It shows it can do much much much more than 50mbps.

Using the command prompt (I forgot about that), I get very little response. It's like it's not reporting correctly.

last pid: 88049; load averages: 0.54, 0.31, 0.16 up 32+23:48:49 09:56:34
603 threads: 4 running, 569 sleeping, 30 waiting
CPU 0: 0.6% user, 0.2% nice, 1.3% system, 0.4% interrupt, 97.6% idle
CPU 1: 0.3% user, 0.2% nice, 1.0% system, 0.1% interrupt, 98.4% idle
Mem: 62M Active, 175M Inact, 472M Wired, 3105M Free
ARC: 203M Total, 28M MFU, 167M MRU, 301K Anon, 903K Header, 6855K Other
81M Compressed, 206M Uncompressed, 2.53:1 Ratio

stephenw10

Sorry I typo'd that, I meant 500Mbps. That's for OpenVPN, it will do more for IPSec.

You need to use the actual command line not the Command Prompt page in the gui, that is non interactive. Or use Diag > System Activity which does update.

Steve

davidstoll

If I log in directly to a shell, it shows about 20-30% for each CPU as it does a speedtest.

davidstoll

@stephenw10 So, based on the 20-30% cpu, what do you suggest at this point?

stephenw10

What is that load? What does it actually show?

It could be a limit on the server side or the route. Are you able to see more than that using a client device directly to the same Nord server?

Steve

davidstoll

@stephenw10 Sorry, I guess I don't know what you mean by the load....other that 20-30% on each cpu.

Yes, I can get 800+ mb/s without the VPN connected.

Yes, it could be the far end, but I have tried different servers on NordVPN and I get very similar results.

stephenw10

I mean if I test an OpenVPN tunnel on a 4100 here and look at the activity I see:

last pid:  6977;  load averages:  0.64,  0.30,  0.16                                                                   up 71+01:47:45  20:28:48
539 threads:   4 running, 522 sleeping, 13 waiting
CPU 0: 36.6% user,  0.0% nice, 43.3% system,  0.0% interrupt, 20.1% idle
CPU 1: 12.2% user,  0.0% nice, 30.3% system,  0.0% interrupt, 57.5% idle
Mem: 33M Active, 261M Inact, 472M Wired, 3033M Free
ARC: 237M Total, 92M MFU, 138M MRU, 692K Anon, 1027K Header, 5445K Other
     133M Compressed, 249M Uncompressed, 1.87:1 Ratio
Swap: 1024M Total, 1024M Free

Message from syslogd@4100-2 at Oct  5 20:24:41 ...C   TIME    WCPU COMMAND
21459pm[80344]: /i 96    0    17M  7468K CPU0     0  46:25  93.43% /usr/local/sbin/openvpn --config /var/etc/openvpn/client1/config.ovpn
   11 root        155 ki31     0B    32K RUN      1 1684.1  59.69% [idle{idle: cpu1}]
   11 root        155 ki31     0B    32K RUN      0 1685.2  18.61% [idle{idle: cpu0}]
96097 root         26    0    15M  6668K select   0   0:02  12.32% iperf3 -c 10.2.4.1 -t 60
    0 root        -76    -     0B   544K -        1  15:31   7.75% [kernel{if_io_tqg_1}]
    0 root        -76    -     0B   544K -        0   4:25   1.24% [kernel{if_io_tqg_0}]
74666 root         52    0   133M    58M accept   0   0:17   0.38% php-fpm: pool nginx (php-fpm)
    0 root        -92    -     0B   544K -        0 242:49   0.32% [kernel{dummynet}]
    0 root        -76    -     0B   544K -        1 165:14   0.18% [kernel{if_config_tqg_0}]
   20 root        -16    -     0B    16K -        0  12:35   0.17% [rand_harvestq]
71350 root         20    0    15M  5528K CPU1     1   0:00   0.12% top -HaSP

So you can see OpenVPN using most of the CPU. iperf itself is using quite a bit there because I'm testing from the firewall directly.

Try testing a nordvpn client that isn't running on the firewall and see what sort of throughput you can get to the same server. It may be that's the best you can get and we are chasing the unobtainable.

Steve

davidstoll

@stephenw10 said in 22.05 and NordVPN tunneling:
Ok, here is what when speed testing with the vpn running on the 4100...

top -HaSP

last pid: 21582;  load averages:  0.27,  0.20,  0.13                                                                            up 35+04:02:56  14:10:41
603 threads:   4 running, 569 sleeping, 30 waiting
CPU 0: 15.7% user,  0.0% nice, 34.1% system,  0.0% interrupt, 50.2% idle
CPU 1: 23.1% user,  0.0% nice, 34.9% system,  0.0% interrupt, 42.0% idle
Mem: 38M Active, 185M Inact, 473M Wired, 3117M Free
ARC: 207M Total, 29M MFU, 170M MRU, 432K Anon, 918K Header, 6906K Other
     84M Compressed, 212M Uncompressed, 2.53:1 Ratio

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
13920 root         85    0    17M  7480K CPU0     0 124:02  85.90% /usr/local/sbin/openvpn --config /var/etc/openvpn/client6/config.ovpn
   11 root        155 ki31     0B    32K RUN      0 823.4H  50.90% [idle{idle: cpu0}]
   11 root        155 ki31     0B    32K RUN      1 829.0H  42.11% [idle{idle: cpu1}]
    0 root        -76    -     0B   528K -        0 102:18  16.04% [kernel{if_io_tqg_0}]
    0 root        -76    -     0B   528K -        1 114:40   2.98% [kernel{if_io_tqg_1}]
73000 root         20    0    12M  3480K bpf      1   5:38   0.85% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
25057 root         20    0    11M  2640K select   1   4:47   0.42% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/s
   12 root        -60    -     0B   480K WAIT     1 155:42   0.33% [intr{swi4: clock (0)}]
   20 root        -16    -     0B    16K -        0  25:18   0.17% [rand_harvestq]
23700 root         20    0    14M  4856K CPU1     1   0:01   0.12% top -HaSP
    0 root        -76    -     0B   528K -        0  31:00   0.06% [kernel{if_config_tqg_0}]
   19 root        -16    -     0B    16K pftm     1  19:02   0.06% [pf purge]
21929 root         20    0    11M  2184K select   1   0:54   0.01% /usr/sbin/powerd -b hadp -a hadp -n hadp
 5420 dhcpd        20    0    23M    12M select   1   0:42   0.01% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf

stephenw10

Ok, well neither CPU core is anywhere near 100% there so it's not CPU limiting.

I would have to assume it's either limited at the server side or as a result of something in the route.

So check what throughput you can see using a NordVPN client running off the firewall.

What VPN client settings are you using in pfSense currently?

Steve

davidstoll

@stephenw10
So, if I use UDP, it cuts the speed way down. If I use Nord Lynx, it helps a LOT (testing with the same server). Is it possible to setup Nordlynx on pfsense?

stephenw10

Nord Lynx is mostly Wireguard. So...maybe! I've never tried it and it may require some extra bits pfSense cannot currently do.

OpenVPN over TCP is almost always significantly slower than UDP. So if you're seeing the opposite that's suspicious. What settings have you tried? What were you using for the results you have stated here?

Steve

davidstoll

@stephenw10
I'm sorry I meant UDP vs Nord Lynx was slower vs faster. I can't get TCP to work, so no worry there.

stephenw10

Ok, well it looks like Nord Lynx can be made to work in pfSense. You just need to install their client in something else and extract the keys first because for some reason they won't give them to you directly.

Or use one of the other VPN providers that do support Wireguard dircetly.

But you still haven't said what client settings you're using in OpenVPN so you might just have very slow encryption. Or no fastio. Or too smaller buffers.

Steve

davidstoll

@stephenw10
Sorry about not getting the client config to you...

dev tun
persist-tun
persist-key
ncp-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC
cipher AES-256-CBC
auth SHA256
tls-client
client
resolv-retry infinite
remote xxx.xxx.xxx.xxx 1194 tcp-client
nobind
auth-user-pass
remote-cert-tls server

stephenw10

Ok, so you have AES-128-GCM available as a cipher via ncp which should be the fastest available. Can you see if Nord is negotiating that in the connection logs?

Since you're using UDP I would enable FastIO and increase the send/recv buffers to 512K. Both of those should give you an increase in throughput.

Steve

davidstoll

@stephenw10
The fast io option was already checked in the pfsense client GUI. The buffer was set to "default", whatever that is and I updated to 512k.

On a separate note...regarding wireguard...I found another page to setup the wireguard because the reddit post ws limited....and it is overwhelming. I get the interface to show as "up", but wasn't routing traffic even though I set the pass option in the new interface and set a lan firewall entry to use that gateway.

stephenw10

Do you see recent handshakes in the Wireguard status?

The interface status can be misleading there.

Steve