Hardware Configuration for Optimal VPN Connection Speeds
-
Yeah, I don't know any hard numbers to equate CPU clock to AES-xxx OpenVPN speed throughput but my understanding is that it is a single-thread affair for now and probably the foreseeable future?
If there really is no multi-core support for VPN I would think two cores with a higher clock would be better than four cores clocked lower for VPN. (Unless the OP is going to be using a lot of other CPU-intensive things).
The j3345 base is 1.5Ghz vs the 3355 @ 2.0Ghz, the 3355 also bursts 200Mhz higher than the 3455.I don't know what kind of throughput you can get on one core at 1.5Ghz with AES-NI encrypting at AES-256, but if it can do 300Mbps then that seems the way to go!
I'm pretty sure the so-dimm spec is correct, I have a J3355B mini-ITX running LibreElec for an HTPC and I'm pretty sure it has old laptop RAM in it.
-
Yeah, I don't know any hard numbers to equate CPU clock to AES-xxx OpenVPN speed throughput but my understanding is that it is a single-thread affair for now and probably the foreseeable future?
If there really is no multi-core support for VPN I would think two cores with a higher clock would be better than four cores clocked lower for VPN. (Unless the OP is going to be using a lot of other CPU-intensive things).
The j3345 base is 1.5Ghz vs the 3355 @ 2.0Ghz, the 3355 also bursts 200Mhz higher than the 3455.I don't know what kind of throughput you can get on one core at 1.5Ghz with AES-NI encrypting at AES-256, but if it can do 300Mbps then that seems the way to go!
Agree that the 30% higher single thread would be better for openvpn. As to what the actual VPN performance would end up being, I haven't seen any openssl or openvpn benchmarks on goldmont/apollo lake. Extrapolating from the older silvermont/braswell systems it wouldn't hit 300Mbps but intel specifically improved the aes-ni and pclmulqdq instructions, and made massive changes to the restrictions on instruction ordering and added a third element to the pipeline so an extrapolation from silvermont is completely bogus. (The clock is meaningless across microarchitectures, what's important is the number of cycles per byte of crypto on a given architecture. For comparison, haswell can do aes-128-gcm in a bit more than 1 cycle per byte, broadwell & skylake can do it in less than one cycle per byte, and silvermont is about 7 cycles per byte–so a skylake is more than 7 times as fast as a silvermont at the same clock speed. goldmont will probably be somewhere between 1 and 7 cycles per byte, but I have no idea where--and without that number there's no way to predict performance.)
-
Yeah, I don't know any hard numbers to equate CPU clock to AES-xxx OpenVPN speed throughput but my understanding is that it is a single-thread affair for now and probably the foreseeable future?
If there really is no multi-core support for VPN I would think two cores with a higher clock would be better than four cores clocked lower for VPN. (Unless the OP is going to be using a lot of other CPU-intensive things).
The j3345 base is 1.5Ghz vs the 3355 @ 2.0Ghz, the 3355 also bursts 200Mhz higher than the 3455.I don't know what kind of throughput you can get on one core at 1.5Ghz with AES-NI encrypting at AES-256, but if it can do 300Mbps then that seems the way to go!
Agree that the 30% higher single thread would be better for openvpn. As to what the actual VPN performance would end up being, I haven't seen any openssl or openvpn benchmarks on goldmont/apollo lake. Extrapolating from the older silvermont/braswell systems it wouldn't hit 300Mbps but intel specifically improved the aes-ni and pclmulqdq instructions, and made massive changes to the restrictions on instruction ordering and added a third element to the pipeline so an extrapolation from silvermont is completely bogus. (The clock is meaningless across microarchitectures, what's important is the number of cycles per byte of crypto on a given architecture. For comparison, haswell can do aes-128-gcm in a bit more than 1 cycle per byte, broadwell & skylake can do it in less than one cycle per byte, and silvermont is about 7 cycles per byte–so a skylake is more than 7 times as fast as a silvermont at the same clock speed. goldmont will probably be somewhere between 1 and 7 cycles per byte, but I have no idea where--and without that number there's no way to predict performance.)
That is a ton of good info! Just shows me how little I know about all of this.
Out of curiosity, how much will Intel quickAssist increase performance and are there any low power products out now that support it in a reasonable price range? I don't really fully understand quick assist, is it completely separate tech from AES-NI? Does it work along with it or is a replacement?
Also, unrelated, but how do the SoC Intel i-series NICs compare to the PCIe versions? any difference? -
Out of curiosity, how much will Intel quickAssist increase performance and are there any low power products out now that support it in a reasonable price range? I don't really fully understand quick assist, is it completely separate tech from AES-NI? Does it work along with it or is a replacement?
quickassist is an off-cpu coprocessor. It can do AES, SHA, RSA, even things like compress/decompress. Just like CPUs, there are different revisions with different performance characteristics. (The one built into the rangeley product line is different from the current add-in cards, and incompatible.) In theory it has really high performance, and the CPU doesn't have to be involved in whatever operations happen on the card, but the catch is that the CPU has to bundle up the data to be processed and send it off to the card–and there's quite a bit of overhead involved in doing that. The optimal use case is that you have a whole bunch of data bundled up, you send it off to quickassist in one operation, you go do something else, you come back and the maybe goes almost directly to the NIC to get broken up into packets and sent. If the package of data is big enough and you can do something else in the meantime, then you can get good performance and the CPU can be busy doing other things. Now, VPN: you get a little tiny packet, you set it up, send it off to quickassist, and then you have to wait for it to be done before much else happens. The CPU is waiting for quickassist, and the time it takes to send the packet off to quickassist is as much time as it would take to do the work! For this use case, a $700 quickassist add-in card performs worse than a sub $100 skylake. What's more, the skylake does AES-GCM roughly 5 times as fast as AES-CBC+SHA1 and as far as I know the quickassist doesn't do GCM at all. And goldmont and the upcoming cannonlake add SHA instructions to the existing AES-NI instructions on the CPU, making AES-CBC+SHA1 even faster relative to the quickassist. To be fair, the current quickassist implementations are years old, and there's a new one coming out in purley and presumably there will be a new generation of add in cards as well--and I have no idea how those will perform. Fundamentally, though, on-cpu crypto is always going to outperform off-cpu crypto when dealing with small blocks; quickassist is a huge win for servers doing SSL on streams of data, not for VPNs dealing with packets at a time.
Also, unrelated, but how do the SoC Intel i-series NICs compare to the PCIe versions? any difference?
not as far as I know. Both are PCIe as far as the software is concerned, the one on the SOC just happens to live in the same package and is connected via a special bus. The i354 in avoton/rangeley is actually 4x2.5Gbps rather than 4x1Gbps, but that doesn't matter when you're attaching it to a gigabit ethernet PHY.
-
Thank you so much! Your descriptions are very detailed and easy to follow, I really appreciate you taking your time to explain this stuff to me.
-
Thank you both so much, your discussion has helped guide me into the things that I should be looking for in my setup!
psSense have replied to say that they don't have any products that would meet my requirements so a self build seems to be my only option…I asked what people were using on my VPN providers forum and had this interesting piece of information...
'When I benchmarked my CPUs for OpenVPN with AirVPN parameters I was getting from 120Mb/s for a Intel N3105 to ~350MB/s for a i5 2500.
A modern fast i3 Xeon or otherwise might do 500Mb/s but I would check.'
-
I'm currently using an old i5-2400, it's enormous overkill for my needs but it got me started into pfsense. I'll be replacing in the future with either one of these new Atom boards or an apollo lake board if the atoms don't pan out within a reasonable amount of time.
My system draws about 40W under normal load and will bump up maybe a few watts with a lot of use and down a few at idle. But 40W is a lot to pay for a 24/7 system, depending on where you live that's somewhere between $40-100/yr to run an appliance.
With the apollo lake boards running between $55-100 for a one time purchase it's a pretty easy choice.It seems reasonable to me that by eliminating all moving parts (no fans or hdd) and switching to a much lower power board that I could get power draw down to an average of 15W or less. That comes out to $15-40/yr, and that's still a hog compared to something like the SG-1000.
So basically I would just recommend that you shop around to find the combo that gets your job done and meets the best compromise between initial buy in and cost over time.
What VAMike explained about how different architectures handle AES-NI drastically differently will have a lot of impact on parts for your use case.
If an i5-2500 can put out the VPN throughput you need then maybe an Celeron G3930T would do the same for 60W less TDP? Not saying it will but just that a modern CPU will probably get you way more bang for the buck over time.
I bought an old one not knowing a thing about pfsense or what to expect and now I'll just be buying another part. -
I've found this which I think will far exceed my requirements but good for future proofing..?
http://www.ebay.co.uk/itm/182347604580
Let me have your opinions please…
-
I've found this which I think will far exceed my requirements but good for future proofing..?
http://www.ebay.co.uk/itm/182347604580
Let me have your opinions please…
Overall it doesn't seem great for a pfsense box. It's pretty expensive, I don't see anywhere what kind of NICs or Wifi card it has, so they might not work well with pfsense, and if that's the case you're kind of screwed. The CPU is probably a great compromise between power draw and performance, but I don't know how great it would be at VPN. I don't know if OpenVPN on pfsense can utilize multiple actual cores, but I'm almost positive it can't do anything with hyperthreading. If that's the case then for VPN you effectively have a dual core CPU clocked 20% higher than a $55 J3355 Celeron, but with older AES-NI than the celeron, so it might be a draw, or maybe even worse at VPN.
Also if you're looking for wifi it might be better to get something like a ubiquiti AP, or reuse an old router as an AP than use an integrated wifi card.I'd like to be able to buy a fanless small device that can handle a 150/10Mbps VPN connection plus more to future proof myself, maybe something that can handle 300/100Mbps..?
'When I benchmarked my CPUs for OpenVPN with AirVPN parameters I was getting from 120Mb/s for a Intel N3105 to ~350MB/s for a i5 2500.
I don't know how old that benchmark is or what kind of VPN they were using or encryption level, but if the N3150 was getting 120Mb/s then based on what VAMike pointed out about the AES improvements to Goldmont the Apollo Lake lineup should be noticeably faster. Maybe not 300Mbps, but at $55 for the motherboard and CPU you could just upgrade in the future when it can't keep up anymore.
Throw some LP RAM, a little innodisk SLC sata-dom, an energy efficient PSU or pico-PSU, and an ebay i350 in with it and you have a very solid set of components.When the day comes that the CPU can't meet your needs anymore, there will probably be a new cheap/low power SoC with a little higher clock and a better set of AES-NI that you can swap in.
-
pfBasic, thank you for you help and guidance it truly is invaluable.
The box is expensive but I chose it as I'm concerned about going and buying the wrong hardware combinations, I'm not confident enough to individually purchase the CPU, RAM, hard drive, case, power supply etc as I simply don't have the knowledge… I've visited several sites and looked at various motherboards etc and then I'm completely lost.... :-[
I've read in several forums that OpenVPN cannot use multiple cores so I guess the i7 won't be used to it's full potential.
I've got a Asus RT3200 that I'll use for my wireless AP.I'm embarrassed to say that I need hand holding through buying the 'perfect' setup.
I'd like to draw upon your your past experience and ask you if you were to start again but with the knowledge that you now have what setup would you go for? (sorry to sound cheeky but can you provide links too please?) :) -
pfBasic, thank you for you help and guidance it truly is invaluable.
The box is expensive but I chose it as I'm concerned about going and buying the wrong hardware combinations, I'm not confident enough to individually purchase the CPU, RAM, hard drive, case, power supply etc as I simply don't have the knowledge… I've visited several sites and looked at various motherboards etc and then I'm completely lost.... :-[
I've read in several forums that OpenVPN cannot use multiple cores so I guess the i7 won't be used to it's full potential.
I've got a Asus RT3200 that I'll use for my wireless AP.I'm embarrassed to say that I need hand holding through buying the 'perfect' setup.
I'd like to draw upon your your past experience and ask you if you were to start again but with the knowledge that you now have what setup would you go for? (sorry to sound cheeky but can you provide links too please?) :)
[/quote]I'm happy to help, I'm no IT pro at all and am a pfsense newbie myself. Everything you read from me will be regurgitated from something I've read elsewhere.
That being said building your own box is really not difficult, and if you ask questions here about component choices you can end up with a very solid system for a great price.
Building PCs is like Legos for adults. Pick a motherboard, is recommend mini-ITX as they are very compact, then pick a case that fits mini-itx. The board you pick will have sieve for what hardware you can use with it, so if it says ddr4 get that, of it says sodimm then get a so funny. As far as actually putting the parts together it's pretty much just plug and play then install pfsense the same as you would on a pre built box.The box you picked is overall probably going to work for you, but your paying a lot of money due things you'll literally never use, and it doesn't specify the types of NICs it has, and NICs are probably the most important component in a pfsense box.
-
I'm embarrassed to say that I need hand holding through buying the 'perfect' setup.
I'd like to draw upon your your past experience and ask you if you were to start again but with the knowledge that you now have what setup would you go for? (sorry to sound cheeky but can you provide links too please?) :)For what you described I would go with something along these lines:
Buy a used non-working SFF desktop off eBay, looking for something that has a working PSU. This gets you a Case & PSU for really cheap. Some examples of what I'm generally talking about (I'm linking eBay but if you can't find what you need for free/cheap shipping then craigslist/garage sale/your local computer shop probably have something for you).:
http://www.ebay.com/itm/PARTS-REPAIR-Gateway-E-4300-SFF-Slim-Desktop-E-Series-PC-Computer-NO-HD-OS-RAM-/331568199492?hash=item4d33013744:g:MgYAAOSwKrhVZ5Sc#viTabs_0
http://www.ebay.com/itm/Dell-Optiplex-790-SFF-Desktop-for-Parts-Repair-NO-RAM-CPU-HDD-/311775026711?hash=item48973d5e17:g:XksAAOSw5cNYcB2IBuy a used intel NIC off eBay. Pretty much all of the cheap ones are manufacturers version and/or chinese knockoffs, but there are tons of threads on here testifying that these work just as well as the real deal. The PRO/1000's are cheapest but you can find i340 or i350 for very cheap, they are more advanced and lower power. Here's an example of a two port model, if you need more ports you can get up to a four port. You'll want the SFF bracket, but you can get those seperately for as cheap as a dollar if you find a great deal on a NIC with a tall bracket.
http://www.ebay.com/itm/IBM-I340-T2-Dual-Port-Ethernet-Adapter-Card-49Y4232-/332093948632?hash=item4d525782d8:g:cykAAOSwjDZYfPtFRAM that fits the board you buy:
https://smile.amazon.com/Crucial-1600MHz-PC3-12800-DDR3L-SDRAM-CT51264BF160BJ/dp/B009RBN6I6/ref=sr_1_11?s=pc&ie=UTF8&qid=1485672045&sr=1-11&keywords=ddr3l&refinements=p_n_feature_twenty_browse-bin%3A9729697011%2Cp_n_feature_four_browse-bin%3A2253866011%2Cp_n_feature_five_browse-bin%3A673263011Motherboard & CPU, again I would recommend you get the J3355B as I believe it will exceed you current needs and it gets you fanless embedded CPU for $55:
https://www.newegg.com/Product/Product.aspx?Item=N82E16813157726Assuming you have absolutely nothing laying around that you can pull parts from, then you could buy this for about $155. Put pfsense on it, push it behind a desk and it'll be the most advanced, stable router you've probably ever owned. In a few years if your needs exceed this CPU, there will proably be something with a new architecture that's cheap, low power, and fast enough for you. Just buy one of those and swap it with the J3355B and plug everything else back in (except maybe the RAM if you get a DDR4 [or 5] board).
The only moving part in this system would be the PSU fan, which you could replace with a pico PSU but it probably isn't worth it.
Another note along the lines of your described use. Unless you have a specific reason to use VPN all the time, it might not be the best choice to just route ALL of your traffic over a VPN. Not all websites play nice with VPN IPs because a lot of VPN traffic is malicious. You'll end up with a lot of captcha's that you wouldn't otherwise deal with and some websites will outright reject your traffic.
You might be better off only routing one LAN over VPN, or only sending a few of your static IPs over VPN. Not only will this increase your general usability but it could also greatly reduce your hardware needs.
I don't know what your specific needs are, but I mention it because I initially just blasted everything over a VPN but it was so annoying being blocked from websites and getting captcha's that I just put the few IPs on the VPN that actually needed it. -
I asked what people were using on my VPN providers forum and had this interesting piece of information…
'When I benchmarked my CPUs for OpenVPN with AirVPN parameters I was getting from 120Mb/s for a Intel N3105 to ~350MB/s for a i5 2500.
A modern fast i3 Xeon or otherwise might do 500Mb/s but I would check.'
I can confirm.
I'm using a N3150 and the speed is about 125Mbps when connected to a PIA server (the speed is about 280Mbs if not connected to a VPN).
As far as I know the CPU's OpenVPN performance are directly connected to AES-NI and to the single thread capabilities.
To get an idea you could take a look here:
https://forum.pfsense.org/index.php?topic=115673.0
so you should easily imagine the performance potentially reached by a Celeron J3455.
From my point of view the main reason to prefer a quad core to a dual core is that pfSense can use multiple VPN clients grouping them into a gateway group.
So activating two VPN clients (one per core) I can easily double the speed while the remaining cores are managing the other activities of the router.
And the situation can only improve with the advent of OpenVPN 2.4 that supports AES-GCM as rightly mentioned by VAMike. -
pfSense can use multiple VPN clients grouping them into a gateway group.
…
So activating two VPN clients (one per core) I can easily double the speed while the remaining cores are managing the other activities of the router.This is very interesting, I had read about doing this in other implementations but didn't know you could do it on pfSense. Do you know of a guide on here or elsewhere that would show how to do this?
Everywhere on this forum I always see the OpenVPN single core only issue mentioned as a reason to use high single core CPU's.
If it's simply a matter of configuration to effectively utilize multiple cores then that argument isn't really valid and we should all be using quad core+ CPU's as I don't know of any other CPU hogs that are single core only? -
Everywhere on this forum I always see the OpenVPN single core only issue mentioned as a reason to use high single core CPU's.
If it's simply a matter of configuration to effectively utilize multiple cores then that argument isn't really valid and we should all be using quad core+ CPU's as I don't know of any other CPU hogs that are single core only?Any single openvpn process can only use one core. You can run multiple processes on different ports (as a server) or connect to multiple vpn servers (as a client) and utilize multiple core. BUT this has limitations. No single connection will get more than one core worth of CPU, so if you're interested in downloading a file to your desktop you can be bottlenecked by the VPN even if there is unused capacity available to other openvpn processes. Similarly, as a server, no client will get more than a core even if you have idle server processes. It's somewhat more complicated to configure each client to run on a separate port or IP, or you can round-robin and potentially get an imbalanced load. At some point it's easier to just either use a different VPN solution, or spend the money and buy a faster CPU.
-
Any single openvpn process can only use one core. You can run multiple processes on different ports (as a server) or connect to multiple vpn servers (as a client) and utilize multiple core. BUT this has limitations. No single connection will get more than one core worth of CPU, so if you're interested in downloading a file to your desktop you can be bottlenecked by the VPN even if there is unused capacity available to other openvpn processes. Similarly, as a server, no client will get more than a core even if you have idle server processes. It's somewhat more complicated to configure each client to run on a separate port or IP, or you can round-robin and potentially get an imbalanced load. At some point it's easier to just either use a different VPN solution, or spend the money and buy a faster CPU.
That makes sense, still it sounds potentially worthwhile (if the configuration isn't a nightmare) until someday over the rainbow when OpenVPN supports multi-core ten pfsense adopts that version.
-
This is very interesting, I had read about doing this in other implementations but didn't know you could do it on pfSense. Do you know of a guide on here or elsewhere that would show how to do this?
Anyway it's worth to try.
I've followed the instructions from M_Devil here:
https://forum.pfsense.org/index.php?topic=115992.msg652957#msg652957 -
Thankyou!
-
This is very interesting, I had read about doing this in other implementations but didn't know you could do it on pfSense. Do you know of a guide on here or elsewhere that would show how to do this?
Anyway it's worth to try.
I've followed the instructions from M_Devil here:
https://forum.pfsense.org/index.php?topic=115992.msg652957#msg652957Thanks very much! I appreciate it, got it set up quick and it's working great, balancing load between two instances. My current CPU is already way overkill but if I move to a more sensible CPU or virtualization in the future I'm sure this will help!
-
Another worthwhile application of this is having continuous access to multiple servers. For example, I use PIA and I've found that every now and then the server I typically use will start to slow down noticeably with a significant increase in ping. In the past I've simply switched to a different server manually, then switch back a few days later.
With this I've set up the second client to connect to a different server. This way if one server starts to slow down (or goes down completely), I'm already connected to an alternative server.
For this reason it might even be worth considering a third client for some.
It really is crazy to me how effective of a tool pfSense is.