PfSense hardware for home router - OpenVPN performance
-
If it helps someone…
Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
(timings averaged over 3 runs, but they never varied more than .1 sec)
(with aesni.ko loaded): 3200/24.03 = 133.17
(with aesni.ko unloaded): 3200 / 23.58 = 135.71Dear garyd9,
thanks for your tests!
Just to get it right: This means C2558 gets a OpenVPN throughput of around 130 Mbit/s?
Right now I'm trying to make a decision between C2358, C2558, C2758, Xeon D-1518…
Basically, all I need is 500 Mbit/s WAN throughput and around 100 Mbit/s OpenVPN throughput. I would like to use snort.
But the most important point to me right now is to have a stable OpenVPN throughput of 100 Mbit/s...I'm wondering if a C2358 will accomplish this performance as well?
Since OpenVPN is single-threaded, afaik, a C2358 may have nearly the same openvpn performance as the c2558?EDIT: Well, I just see, there is a big difference in base clock rate... 1,7 GHz vs. 2,4 GHz... So I think the C2358 won't be able to run 100 Mbit/s OpenVPN throughput? According to other threads the OpenVPN throughput of C2358 is somewhere at 70 Mbit/s. Is that correct?
-
Just to get it right: This means C2558 gets a OpenVPN throughput of around 130 Mbit/s?
Theoretically. Real world performance would vary.
I'm wondering if a C2358 will accomplish this performance as well?
Since OpenVPN is single-threaded, afaik, a C2358 may have nearly the same openvpn performance as the c2558?EDIT: Well, I just see, there is a big difference in base clock rate… 1,7 GHz vs. 2,4 GHz... So I think the C2358 won't be able to run 100 Mbit/s OpenVPN throughput? According to other threads the OpenVPN throughput of C2358 is somewhere at 70 Mbit/s. Is that correct?
I don't run openvpn. I just ran the test proposed by the OP and posted the results. However, based on the differences between the C2358 and C2558, I'd guess that the 2358 would have a difficult time getting to 100 megabit/sec openvpn performance.
-
Can you report on exactly what you changed? Is the config you just posted your current config? Thanks!
The changes I made were to disable lzo compression and use mauroman33's custom options. Like he mentioned the main difference in that is the "tls-cipher" line it would seem.
I also made sure to turn off any crypto hardware "assistance" options I may have fiddled with. Most were already disabled, but I did play around with using "RDRAND", so I had to disable that again.
My overall config isn't exactly like his screen shots, though. I tick the boxes to disable addition of routes to the route table and handle those with AON.
-
Can you report on exactly what you changed? Is the config you just posted your current config? Thanks!
The changes I made were to disable lzo compression and use mauroman33's custom options. Like he mentioned the main difference in that is the "tls-cipher" line it would seem.
I also made sure to turn off any crypto hardware "assistance" options I may have fiddled with. Most were already disabled, but I did play around with using "RDRAND", so I had to disable that again.
My overall config isn't exactly like his screen shots, though. I tick the boxes to disable addition of routes to the route table and handle those with AON.
The suggestion to disable the Cryptographic Hardware comes by Pippin's messages in this thread:
https://forum.pfsense.org/index.php?topic=115627.30 -
here are my advanced server settings:
fast-io;sndbuf 0;rcvbuf 0;push "sndbuf 524288";push "rcvbuf 524288";keepalive 10 120;push "redirect-gateway def1";push "redirect-gateway-ipv6 def1";push "route-ipv6 2000::/3";
Here are my advanced client settings:
fast-io fragment 0 mssfix 0 sndbuf 524288 rcvbuf 524288 lport 0 remote-random remote-cert-tls server resolv-retry 4 key-method 2 mute 10 mute-replay-warnings keepalive 10 120 auth-retry nointeract setenv FORWARD_COMPATIBLE 1 verb 3 reneg-sec 0 script-security 2
Ultimately, i think we should push to change to Softether as the VPN client. It supports backwards compatibility to OpenVPN and is much faster than OpenVPN for the same hardware.
Here is a feature list: https://www.softether.org/1-features
-
What some people consider here to be VPN is quite different from what's VPN is in a corporate world. Hiding one's identity to be able to download stolen content is not the reson VPN was invented. When one lists VPN throughput to such a service, it's not what VPN's purpose is in pfSense.
-
What some people consider here to be VPN is quite different from what's VPN is in a corporate world. Hiding one's identity to be able to download stolen content is not the reson VPN was invented. When one lists VPN throughput to such a service, it's not what VPN's purpose is in pfSense.
While softether can be used for obsfcation, softether actually supports more corporate features than OpenVPN. It supports IPSec, VLANs, OpenVPN, etc. In addition, the code is more modern and supports multithreaded VPN. I get comparable speeds from my Raspberry Pi running Softether VPN versus my i7 pfsense box running pfSense.
Anyway, not trying to hijack the thread. Just adding my .02
-
Ultimately, i think we should push to change to Softether as the VPN client.
Well, it's already in freeBSD ports…
-
What some people consider here to be VPN is quite different from what's VPN is in a corporate world. Hiding one's identity to be able to download stolen content is not the reson VPN was invented. When one lists VPN throughput to such a service, it's not what VPN's purpose is in pfSense.
And that doesn't have any bearing on the performance numbers, which is what we're discussing here. We're talking about the technology and performance on given hardware, not the reasons we're using it. This discussion is directly relevant to anyone using OpenVPN, regardless of the use case.
-
What some people consider here to be VPN is quite different from what's VPN is in a corporate world. Hiding one's identity to be able to download stolen content is not the reson VPN was invented. When one lists VPN throughput to such a service, it's not what VPN's purpose is in pfSense.
Sorry mate, I don't understand. How do you know what is the reason why people are using a VPN here?
Are you maybe speaking of your personal experience?
Personally I have some good reasons to use it, starting from the systematic throttling performed by the ISP in my area if you don't use the services directly purchased from it …
But maybe, as rightly said whosmatt, this is not the forum for this kind of discussion. -
Test VPN from the gear you own (and whose specs you know) to your pfSense box across a reliable internet connection. That will give you the performance indication of your hardware. Better yet, test the VPN throughput in the lab to see the maximum throughput your hardware is capable of.
Stating the VPN throughput of 5 Mbps to some third-party VPN service is hardly an indication that something is wrong with your hardware or software.
-
here are my advanced server settings:
fast-io;sndbuf 0;rcvbuf 0;push "sndbuf 524288";push "rcvbuf 524288";keepalive 10 120;push "redirect-gateway def1";push "redirect-gateway-ipv6 def1";push "route-ipv6 2000::/3";
Here are my advanced client settings:
fast-io fragment 0 mssfix 0 sndbuf 524288 rcvbuf 524288 lport 0 remote-random remote-cert-tls server resolv-retry 4 key-method 2 mute 10 mute-replay-warnings keepalive 10 120 auth-retry nointeract setenv FORWARD_COMPATIBLE 1 verb 3 reneg-sec 0 script-security 2
Ultimately, I think we should push to change to Softether as the VPN client. It supports backwards compatibility to OpenVPN and is much faster than OpenVPN for the same hardware.
Here is a feature list: https://www.softether.org/1-features
I started a thread in the package sub-forum regarding my SoftEther FreeBSD package port and performance. My initial tests show I can easily push 150/150 mbps (maximum of my WAN speed) through SoftEther with very low load on my pfSense machine!
https://forum.pfsense.org/index.php?topic=117626
I will post some benchmarks and run some theoretical maximum tests to see how fast SoftEther is compared to OpenVPN on my machine (my theoretical OpenVPN max is 299 mbps).
-
ci323 nano u
Stats for you guys
hw.model: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz
(cryptodev) BSD cryptodev engine
(rsax) RSAX engine support
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading supportopenssl speed aes-128-cbc aes-192-cbc aes-256-cbc
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128 cbc 27446.63k 31065.88k 32203.01k 79808.59k 80497.95k
aes-192 cbc 23419.20k 25885.49k 26810.11k 66946.05k 67553.96k
aes-256 cbc 20250.74k 22164.16k 22912.07k 57832.11k 58552.27kopenssl speed -engine cryptodev -multi 4 aes-128-cbc aes-192-cbc aes-256-cbc
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
aes-128 cbc 104805.14k 118668.96k 128936.14k 318822.11k 321327.26k
aes-192 cbc 93729.09k 103600.58k 107154.76k 267119.27k 271302.66k
aes-256 cbc 80937.83k 88656.43k 91726.42k 226115.54k 230708.57kopenssl speed -multi 4 bf-cbc
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
blowfish cbc 164957.33k 185746.01k 191440.98k 193272.15k 193628.57kopenssl speed bf-cbc
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
blowfish cbc 41251.69k 46459.56k 47838.06k 48378.47k 48452.95k -
The measurement above using command:
time openvpn –test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbcseem to be total innacurate.
I made following tests. I connect a i5 laptop and a RK3288 based linux box to a vlan switch. The RK3288 is used as a vlan router. On Laptop I runned a speed test through this router. and I obtain 300M/150M which is what provider offer.
Running the above test command I got :
For RK3288 27 sec which mean 118.5Mbps
For I5 Laptop 6 sec which mean 533 Mbps.So I expect a throughput of around 120Mbps . However insialling openwrt on both RK3288 box and I5 laptop, performing same test I have only 32Mbps/43Mbps. Why so big difference comparative with theoeretical speed of 120Mbps
-
Very much doubt these calculations or any….., to much variables to make a good estimate that will reflect reality.
Cipher, digest, hash, compression, mtu, buffersizes, network, latency, etc. all play a role.
And also the type of data that goes through the tunnel.Intel Celeron N3150 4x1.6GHz -TDP 6W -CPU Mark 1642 -Single Thread 456
3200/27,5 = 116 Mbps OpenVPN performance (estimate)As argument, with N3150 (Gigabyte N3150N-D3V), I can tell you that in a client to client Iperf test, I was getting 160 Mbit/s throughput, I used:
No crypto hardware selected (meaning AES-NI will be used automatically if it`s supported, N3150 does)
no compression
DH 2048
AES-256-CBC
SHA512
prng SHA512 32 #(prng_hash = 'RSA-SHA512'/prng_nonce_secret_len = 32)
cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384/2048 bit RSAAs you can see, with these somewhat "heavier" settings it is higher then the calculated 116 Mbit/s.
Furthermore, keep in mind that this was client to client, meaning there is an extra round of crypto happening at server…..
I'd guess that the original benchmark was done with aesni.ko loaded, hence the low crypto performance. Without aesni.ko I benchmark just about exactly 160Mbps on that hardware…
-
I made following tests. I connect a i5 laptop and a RK3288 based linux box to a vlan switch. The RK3288 is used as a vlan router. On Laptop I runned a speed test through this router. and I obtain 300M/150M which is what provider offer.
Running the above test command I got :
For RK3288 27 sec which mean 118.5Mbps
For I5 Laptop 6 sec which mean 533 Mbps.So I expect a throughput of around 120Mbps . However insialling openwrt on both RK3288 box and I5 laptop, performing same test I have only 32Mbps/43Mbps. Why so big difference comparative with theoeretical speed of 120Mbps
Do I understand correctly that you changed the OS after running the benchmark? Try running the benchmark on the OS you're using for the test.
-
No, was the same OS . I did not change anything. Just test directly and the immediately run same test through openvpn. The again run same test directly just to be sure. The results are :
Directly 300/150, through vpn 32/43.
-
The measurement above using command:
time openvpn –test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbcseem to be total innacurate.
I made following tests. I connect a i5 laptop and a RK3288 based linux box to a vlan switch. The RK3288 is used as a vlan router. On Laptop I runned a speed test through this router. and I obtain 300M/150M which is what provider offer.
Running the above test command I got :
For RK3288 27 sec which mean 118.5Mbps
For I5 Laptop 6 sec which mean 533 Mbps.So I expect a throughput of around 120Mbps . However insialling openwrt on both RK3288 box and I5 laptop, performing same test I have only 32Mbps/43Mbps. Why so big difference comparative with theoeretical speed of 120Mbps
OpenWRT is Linux based and not BSD based! This at first. But you will be also getting total
other results if you take on both sides Intel Core i5 CPUs and or i7 CPUs. And theoretical
you could do a test on the same devices for OpenSSL likes many others are doing, but
what you get then out as a result in the real life you should know, is totally another thing!This numbers even can be and will be different pending on the;
used hardware (horse power), devices it self and topology of the network or done test.And pease don´t forget that you will need more horse power such OpenWRT is needing,
but on the other side you get then not only a small router, you might be able to set up until
a fully UTM device if needed. -
Here is my benchmark for
Intel J1900 Quad Core 4x2GHz
Network 4*Intel WG82583
Eglobal Fanless Mini PC[2.3.2-RELEASE][admin@pfSense.localdomain]/root: time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc 30.309u 0.023s 0:30.35 99.9% 742+177k 0+0io 0pf+0w
30s = 106 Mbps according to the calculation.
"Real world" performance:
I have a 100/100 connection -
No, was the same OS . I did not change anything. Just test directly and the immediately run same test through openvpn. The again run same test directly just to be sure. The results are :
Directly 300/150, through vpn 32/43.
Don't you have doubt that it could be related to your VPN provider?
-
Both i5 laptop and RK3288 box are in my location connected through a gigabit network. There is no VPN provider involved, only normal ISP that provide connection with 300Mbit/150Mbit. On laptop I have Windows 7 installed and on RK3288 Ubuntu 14.10. As I said the RK3288 act like a VLAN router and connected directly , on laptop I can successfully reach maximum speed provider offer (300/150). Now maing the same speed but through openvpn (I simulate laptop using openvpn client tried to conect to RK3288 openvpn server that connect to internet. In this case the speed was just 32/43. he issue here is not why the speed is so low. It maybe because RK3288 cannot do more (however this is also strange becasue is a capable processor having crypto hardware accelerated). The issue is why testing openvpn speed (and not openssl) gives a so big difference. And all those tests were repeated several times, just to be sure. The results were consistent. Is there any way to be sure that openvpn use hardware accelerated crypto or not. Because maybe this is the reason of so big difference.
-
Sorry mate, but I didn't understand how pfSense is involved in your test.
-
It is not yet. I intend to install it. ut what I wrote has nothing to do with pfsense, or to any operating system. It has to do with theoretical speed calculation cs real speed.
-
You should consider this thread is about the theoretical speed and the real speed obtained through a device running pfSense.
-
Ok, so on a non pFsense device there is no correlation between theoretical and real sped ?
-
Ok, so on a non pFsense device there is no correlation between theoretical and real sped ?
The test is a heuristic, and it remains to be seen how accurate it is across a wide variety of machines. In this case I'd specifically wonder whether the abnormally large blocks used to pad the test runtime give wildly inaccurate results for off-board crypto processors. (On the RK3288, IIRC, the crypto is handled by a specific module, as in the old via padlock stuff, and those generally do much better for large blocks than small blocks because of a high fixed setup cost. On systems with onboard crypto like AES-NI for which the heuristic was initially developed, there's a much lower penalty for small blocks.) I doubt that linux vs fbsd has much to do with it.
-
Ok, so on a non pFsense device there is no correlation between theoretical and real sped ?
I think there is also correlation for not pfSense devices, but I don't think you could be sure to get a definitive answer on the pfSense forum.
-
Figured I'd show my J3455 results:
Intel Celeron J3455 4x1.5GHz -TDP 10W -CPU Mark 2134 -Single Thread 782
AES-256-CBC : 267.9 Mbps
AES-256-GCM: 282.4 MbpsAES-128-CBC: 270.0 Mbps
AES-128-GCM: 284.9Mbps -
I know it's a bit of an old topic but I'm currently looking at some pfsense hardware with openvpn capabilities as well..
As I have a 1000/1000 fiber connection, I was wondering if a kaby lake celeron 3865u (1.8 GHz, dual core) would do similar or better than a j3355 for pfsense+openvpn purposes?
-
Probably pretty similar to j3355, but that's a mobile part.
-
aes256 is just needlessly throwing away performance especially on CBC, I suggest sticking to aes128-gcm guys.
-
@teh:
Figured I'd show my J3455 results:
Intel Celeron J3455 4x1.5GHz -TDP 10W -CPU Mark 2134 -Single Thread 782
AES-256-CBC : 267.9 Mbps
AES-256-GCM: 282.4 MbpsAES-128-CBC: 270.0 Mbps
AES-128-GCM: 284.9MbpsEnded up with a G4400 build myself, speed with PIA using OpenVPN and AES-128-CBC is near 500 Mbps. I never see the CPU taxed above 30% though with AES-NI enabled.
Also, the idle power consumption for the total system is about 13 watts, which is still fine for me.
-
Nice result. I would assume one core is at 100% in those conditions?
Steve
-
While using some of the tips and tricks in this topic: https://forum.pfsense.org/index.php?topic=112877.15 I was able to increase my speed to 700-800 Mbps with a Speedtest.net test using a nearby 1000 mbit server (see attached).
That's actually a lot better than I had expected. While testing, the OpenVPN WCPU goes up to a about 50% and the max one core is used is also about 50%. Is this normal? I thought OpenVPN would be harder on my processor? (3.3 Ghz G4400 Skylake processor with AES-NI enabled) I'm quite sure my VPN is working fine as the OpenVPN process is spiking and both speedtest.net and privateinternetaccess.com report my PIA ip address. The cipher used is AES-256-CBC.
-
Hmm, that is surprisingly fast. Almost suspiciously so.
Increasing the buffers and setting fast-io can help quite a bit though. Those options are in the gui in 2.4.
Steve
-
Thanks Steve. Yeah, I'm not sure what to think of it yet (1000/1000 mbit connection)
These are the settings I've used:
tls-client;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
remote-cert-tls server;
persist-key;
persist-tun;
persist-remote-ip;
keysize 256;
reneg-sec 0;
link-mtu 1540;
fragment 0;
mssfix 0;
fast-io;
sndbuf 1572864;
rcvbuf 1572864;All seemed fine in the logs until I just received this error in the OpenVPN log: 38532 tun packet too large on write (tried=1500,max=1482).
Is that bad news? Can't seem to find anything decisive about it. -
There's no way the link-mtu is 1540, you should probably remove that.
I'm not sure what setting mssfix and fragment to 0 does. Probably nothing, it seems to be undefined.
When I tested locally I found increasing the send receive buffers above 512k made negligible difference.
Steve
-
Think you are correct, changed to this settings:
tls-client;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
remote-cert-tls server;
persist-key;
keysize 256;
reneg-sec 0;
fast-io;
sndbuf 572864;
rcvbuf 572864;No more errors in the log but the same speed (700-800 Mbps).
EDIT: a bit too soon, a new error occurs every approx. 5 minutes: PID_ERR replay-window backtrack occurred [13] [SSL-0] [000000000000__00000000000000000000000000000000000000000000000000] 0:4803054 0:4803041 t=1505773250[0] r=[-3,64,15,13,1] sl=[22,64,64,528]. Apparently this is connected to having some packet loss while using UDP instead of TCP.
Back on topic: apparently a G4400 is able to reach 700+ Mbps with PIA under favorable circumstances (server very close etc).
-
Hi denova, your results are really interesting.
It could be nice if you'll have the time to run test suggested in the first post. Just out of curiosity.
cheers -
Hi denova, your results are really interesting.
It could be nice if you'll have the time to run test suggested in the first post. Just out of curiosity.
cheers8.45 seconds, so 3200/8.45 would be around 380 Mbps. I'm getting up to 850 Mbps though, with about 50% core taxing.
I still don't really get it, is there something I'm missing?
Connection 1000/1000 mbit, Private Internet Access, OpenVPN settings:
Server mode: peer to peer
Protocol: UDP on IP4 only
Peer Certificate Authority: PIA certificate
AES-256-CBC
Compression: LZO compression
Device mode: tun layer 3 tunnel mode
Custom options:tls-client;
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;
remote-cert-tls server;
persist-key;
persist-tun;
persist-remote-ip;
keysize 256;
reneg-sec 0;
fragment 0;
mssfix 0;
fast-io;
sndbuf 572864;
rcvbuf 572864;