Best 2017 hardware for gigabit fiber + VPN



  • Hi all,

    I'm planning to build my first pfSense box. I live in Europe and I have a 1 gigabit fiber optical connection. I plan to use OpenVPN or IPSec or what is the fastest VPN I can get on the box. Therefore, the CPU must be at least AES-NI.

    After much reading, it seems that the best platform is a CPU released in 2013: Intel Atom C2758. It supports:

    • AES-NI
    • Intel QuickAssist
    • DPDK support (enabled software)

    Source : here

    Therefore, I'm about to go with the configuration listed here :

    • Super Micro SUPERMICRO A1SRI-2758F - Motherboard - Mini-ITX - Intel Atom C2758 - USB3.0 - 4 x Gigabit LAN - Onboard-Grafik (A1SRI-2758F-O)
    • MS-Tech CI-57 - Ultra Small Form Factor - Mini-ITX - 120 Watt (CI-57-120)
    • SanDisk SSD - SSD - 64GB - 6,4 cm (2.5") - SATA-600 (SDSSDP-064G-G25)
    • Kingston KVR16LSE11/8HB

    It seems a shame to buy a 2013 cpu in 2017 that has been reported by Intel to have a bug that will prevent him from booting :(

    So, what do you guys think about it ? Should I go for the above setup or use a i3/i7 skylake/kaby lake that supports AES-NI and forget about QAT/DPDK ? Any coments are welcome !

    EDIT: I forgot to mention that my FTTH connection is over PPoE, but I might move to another internet provider which won't be PPoE. In any case I'de rather get a future proof platform that performs well on PPoE & not PPoE. I've read that PPoE is single threaded so the base frequency for a single core must be high enough to handle gigabit speed with NAT, routing & encryption.


  • Banned

    @excalibur:

    It seems a shame to buy a 2013 cpu in 2017 that has been reported by Intel to have a bug that will prevent him from booting :(

    I've read that PPoE is single threaded so the base frequency for a single core must be high enough to handle gigabit speed with NAT, routing & encryption.

    Yeah I'd pass on the C2758, as you pointed out it's old and has issues.

    Modern CPU's can handle a lot with instruction sets than older, more powerful CPU's. AES-NI gets updated, so a modern CPU with AES-NI will perform better than an older one, although I'm not sure (I don't think) that every generation gets an upgrade. The latest Apollo / Kaby Lake CPUs did get an upgrade.
    That being said, you would need a monstrous machine to handle gigabit OpenVPN, I wouldn't plan on it. What do you actually need to see in terms of VPN performance (throughput and number of clients and/or servers)?
    If your VPN needs are reasonable and you aren't looking to utilize heavy packages, then I'm betting you can actually get by with a low end modern Pentium.
    A G4560T will give you 4 threads @ 2.9GHz and all the features you want for a good price.
    There are a lot of threads on here asking what hardware to buy for gigabit WAN but not many people report what they are getting with their hardware on gigabit WAN(to include how hard that hardware is working to achieve it, no one is surprised if an E3-1585 can do gigabit WAN, it is interesting to see how hard it's working to do so, and even more interesting how hard lower end hardware works to achieve X throughput on gigabit WAN). Because of this (and the tendency for people to want to try out a lot of packages that can be very resource intensive and expect good performance [IPS, VPN, pfBNG]) it seems to be the common tendency for people to recommend getting a small supercomputer to NAT gigabit WAN. There are SOHO routers that can NAT gigabit WAN, they have optimized hardware, so maybe that doesn't correlate, but I still find it hard to believe that one needs an intel coreX, xeon or 8 core atom to NAT gigabit WAN for a SOHO user.

    Please report back with your achieved performance on whatever hardware you end up purchasing!

    I've recently been educated on the forums that if you're looking for high performance gigabit WAN, dual channel RAM matters. So maybe take a look at a 2x4GB kit.

    From what I've read Quick Assist the repeated line for Quick Assist is that it doesn't make matter now but probably will in the future. Do what you will with that, haha.

    I'd never heard of DPDK before this thread (thanks for that!) but it looks like you don't need to worry about it when picking a CPU. The second sentence on their home page reads, "It was designed to run on any processors". http://dpdk.org/
    They do however have a list of supported NICs, it includes the standard intel PRO/1000, i340, i350 and more.
    http://dpdk.org/doc/nics

    When you mention future proofing, future proofing for what? 2GbE WAN? You'll need >1GbE NICs for that at a minimum. Just the ambiguous term "future-proof" doesn't really mean anything unless you specify what features you expect in the foreseeable future that you are planning for now.



  • Thanks a lot pfBasic for your inputs, I highly appreciate :) I didn't know that AES-NI implementation is sometimes updated when a new CPU is released, so it's good to know and I'll probably go with kaby lake for that reason.

    Regarding my VPN usage, I won't use it for privacy purposes but to overcome throttling that french ISP sometimes abuses for certain websites such as YouTube, Google Play Store, Apple App Store… With that in mind, I could even use an OpenVPN connection without encryption at all but I would be subject to man in the middle attacks, so that's not a good idea.

    Regarding my speed needs, I want to keep my gigabit speed without VPN, no matter what.

    When using the VPN, I know I probably can't get gigabit speed, and IP forwarding won't help to achieve that (OpenVPN is not very efficient when used with IP forwarding, SoftEther attempts to solve this problem but is not very well maintained). So I would say: I would like to get as much as I can get for VPN speed, but without going in a crazy budget to get more than 500 Mbit/s. I consider 500 Mbps to be more than enough to downloads apps from Play Store and watch videos. 100 Mbps would be even ok but I want to be future proof and maybe one day I'll use the VPN for all traffic of a future seedbox.

    The G4560T CPU seem like a good deal, though I don't know how it performs in VPN encryption/decryption nor in single threaded mode due to PPoE connection. I'll have to investigate on that.

    I'll probably go with 2x8GB ECC in order to get benefit from dual channel.

    QuickAssist seems like a pandora box, everyone talks about it but nobody gets it. I won't spend much time or money on that. We'll see in the future and I'll upgrade my box if it has an impact on my usage.

    When I talked about future proofing, I was referring to the type of connection: right now I have PPoE FTTH, maybe later I will have non PPoE FTTH, and I want my setup to perform well in both case: high-frequency base clock for single thread mode, and multi-core for non PPoE connections. Sorry if I wasn't clear about that.


  • Banned

    No worries, I'm glad it was of use to you. I'm just passing along what I've learned reading here and elsewhere!

    Without any privacy needs you can have some very low level encryption that won't tax your system much. You should be happy with your VPN throughput.

    With PPPoE fiber and VPN both being single threaded I would think that four threads would still be plenty, but hopefully someone who has attempted what you're trying without expensive high end hardware can chime in with their experiences!



  • Yeah I'm still searching for a user sharing his experience in using a kaby lake cpu and FreeBSD. I'm afraid of some bugs with latest intel chipset and FreeBSD.

    You said that Kaby Lake uses an upgraded version of AES-NI. Can you point me a link for that? I've googled but didn't find anything revelant.


  • Banned

    My mistake, it was Goldmont (Apollo Lake) that recently upgraded AES-NI, not Kaby Lake, they also included SHA hashing instructions.

    https://en.wikipedia.org/wiki/Goldmont

    Throughput and latency of instructions for accelerating encryption/description (AES) and carry-less multiplication (PCLMULQDQ) have been improved significantly in the Goldmont microarchitecture.

    But in terms of general improvements to AES-NI over time I found a few interesting things:

    https://software.intel.com/sites/default/files/m/d/4/1/d/8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf
    table 2 shows a 2010 130W Gulftown 6 core 12 thread 3.33GHz i7-980X (MSRP $1059) achieving ~.49 cycles per byte @ AES-256-CBC utilizing all threads for serial encryption.
    To actually achieve this performance you would need 12 instances of VPN running as it is single threaded, also this test is highly optimized.
    1 thread on the cpu works at ~5.7 cycles/byte @ the same level of encryption, this is probably the more useful reference benchmark.
    This is a good reference because it is the first year of AES-NI on a very high end CPU.

    This shows ~198% (theoretical) performance increase from Westmere (2010) to Sandy/Ivy Bridge (2011/2012) on slides 43 & 44.
    https://www.cosic.esat.kuleuven.be/ecrypt/AESday/slides/Use_of_the_AES_Instruction_Set.pdf

    This shows a 2015 65W Skylake 4 core 8 thread 3.4 GHz i7-6700 (MSRP $312) achieving ~2.57 cycles/byte but doesn't state all of the parameters unless you buy the book.
    https://books.google.com/books?id=xUG8DAAAQBAJ&pg=PA55&lpg=PA55&dq=intel+aes-ni+cycles+per+byte&source=bl&ots=YCVDZOy60N&sig=w1oh0vzlffT4bJxqyl_pCkYCiic&hl=en&sa=X&ved=0ahUKEwihsYCOwuXSAhUX5GMKHUKpAs8Q6AEIXTAJ#v=onepage&q=intel aes-ni cycles per byte&f=false

    I haven't seen more recent benchmarks of AES-NI in terms of cycles per byte. I don't know how Goldmont stacks up to Skylake or Kaby lake but it is probably still inferior even if it is more modern.
    Either way gigabit +VPN +PPPoE fiber is probably too much to ask of Apollo Lake. I think you could do gigabit on Apollo Lake if you are just doing NAT, but that's just a guess.

    I've seen lots of claims that AES-NI doesn't use HT, but obviously in this intel paper it does, other user benchmarks show this as well.
    I don't know if OpenVPN can use HT?
    https://www.reddit.com/r/hardware/comments/2ckwai/aesni_and_hyperthreading_in_i7_cpu/

    In addition to improvements to AES-NI, pfSense OpenVPN can now use GCM, which is much faster than CBC.
    https://software.intel.com/en-us/articles/aes-gcm-encryption-performance-on-intel-xeon-e5-v3-processors
    https://calomel.org/aesni_ssl_performance.html
    http://ece.gmu.edu/coursewebpages/ECE/ECE646/F15/project/F15_presentations/Session_3_Crypto_Libraries/1_Special_Instructions.pdf



  • @pfBasic:

    My mistake, it was Goldmont (Apollo Lake) that recently upgraded AES-NI, not Kaby Lake, they also included SHA hashing instructions.

    https://en.wikipedia.org/wiki/Goldmont

    Throughput and latency of instructions for accelerating encryption/description (AES) and carry-less multiplication (PCLMULQDQ) have been improved significantly in the Goldmont microarchitecture.

    Note that's in relation to silvermont, which was really slow. Goldmont is still slower than a recent performance-oriented (as opposed to power-oriented) architecture like skylake. To the OP: the thing to google for is "cycles per byte", which is how many CPU cycles it takes to process a byte of encrypted data. At 1 cycle/byte, a 1 GHz processor can perform 1GByte/s of encryption. Intel got AES128-GCM under 1 cycle/byte on broadwell, and successive architectures (skylake/kaby lake) have made minor improvements. (See https://software.intel.com/en-us/articles/improving-openssl-performance ) AES-CBC is significantly slower than AES-GCM because it's harder to pipeline. Silvermont is tremendously slower because it has reduced parallelism in order to run with less power consumption. It's somewhere around 7 cycles/byte, which means that at the same clock speed, a skylake CPU will perform crypto around 7 times as fast as a silvermont CPU. (That's a huge difference, which is why I avoid the silvermont CPUs, which includes the N3150 etc., when crypto performance is a factor.) Goldmont is still slower than skylake, but only by a factor of 1.5 or 2 or so, which isn't as huge. (For context, while we're talking about hundreds of megabytes, or gigabytes, with AES-NI, a J1900 is going to do crypto at tens of megabytes. And OpenVPN speeds won't some anywhere near the CPUs' abilities to do crypto, because other parts of the OpenVPN architecture become huge bottlenecks at high data rates.)



  • @excalibur:

    When using the VPN, I know I probably can't get gigabit speed, and IP forwarding won't help to achieve that (OpenVPN is not very efficient when used with IP forwarding, SoftEther attempts to solve this problem but is not very well maintained). So I would say: I would like to get as much as I can get for VPN speed, but without going in a crazy budget to get more than 500 Mbit/s. I consider 500 Mbps to be more than enough to downloads apps from Play Store and watch videos. 100 Mbps would be even ok but I want to be future proof and maybe one day I'll use the VPN for all traffic of a future seedbox.

    All the advice given here is spot on.  Regarding OpenVPN performance, have a read at some of the replies to this thread I started last year when first getting into using an OpenVPN provider for the majority of my traffic: https://forum.pfsense.org/index.php?topic=115992.0.

    There's a lot there, but the TL;DR is that you can use multiple OpenVPN connections simultaneously, therefore allowing each of your CPU cores to handle one OpenVPN process.  With services that can handle multiple connections (Steam downloads, for example) you theoretically could max out your connection with the right CPU and provider with a single download from one workstation.  With services limited to a single connection, you could still have multiple devices maxing out a single connection each with aggregate throughput much higher than what you otherwise might see.  This concept takes advantage of pfSense's gateway groups, the same as you might use in a situation where you had two or more ISP connections, except that in this case the multiple connections are virtual, made via OpenVPN to different endpoints.

    So, with a 1Gbps connection, and one core currently having to handle all of the PPPoE needs, perhaps a modern Kaby lake CPU with 4 physical cores (vs the 2 physical cores in the G4560T) might be of real benefit.  I say "might" because I haven't the 1Gbps connection to try it with.  Just something to consider.



  • That's a lot of reading to digest but also a lot of useful information, thank you guys.

    So to sum it up, with OpenVPN now supporting GCM, and regarding the benchmarks here, we can safely say that any 2 GHz kaby lake processor would be enough to handle 1 Gbps AES-256-GCM encryption. (the benchmarks shows 2.20 GHz broadwell encrypting @ 1.08 cycles per byte).

    The problem appears to be that OpenVPN has some bottleneck elsewhere so it's not enough to have 1 cycle per byte for GCM encryption. To get the maximum performance, we can use gateways pfsense groups as @whosmatt suggests. That implies that we need 4 cores to scale well, with OpenVPN running on 3 or 4 cores. Or maybe pfsense / FreeBSD / OpenVPN supports intel hyper threads and two cores with 4 HT could be enough.

    But I must keep in mind that I'm not really interested in privacy, I just use a VPN to get away from ISP throttling or hide my identity when sharing a torrent. Therefore, I could use IPSec / IKEv2 which seems to perform better than OpenVPN, as pointed @kroko here. Combined with gateway groups, I might even be able to reach 1 Gbps.  :-*

    In the end, I must now pick the right CPU. I'm still hesitating between an i3-7300 (2x4.00 GHz, TDP 51W) or an i5-7600 (4x3.50 GHz, TDP 65W). I don't know if it's better to have a base frequency @ 4.00 GHz with 2 cores or a base frequency @ 3.50 GHz with 4 cores. Furthermore, the i5-7600 supports Turbo Boost and can go up to 4.10 GHz, but I don't know if openssl activates TurboBoost when encrypting / decrypting (I don't even know how Turbo Boost works at all). And at last, I'd like to keep the form factor pretty small: mini-box m300 if possible or if the CPU fan requires more room I'll go with a Silverstone sst-ml05b.



  • @excalibur:

    That's a lot of reading to digest but also a lot of useful information, thank you guys.

    So to sum it up, with OpenVPN now supporting GCM, and regarding the benchmarks here, we can safely say that any 2 GHz kaby lake processor would be enough to handle 1 Gbps AES-256-GCM encryption. (the benchmarks shows 2.20 GHz broadwell encrypting @ 1.08 cycles per byte).

    The openssl crypto can perform at multiple gigabytes per second, but the openvpn overhead will make your throughput limit much, much lower. It's possible to configure openvpn to use larger blocks (setting tunnel mtu to something like 32 or 64k) which will let you hit 1gbps, but then the tunnel is extremely sensitive to packet loss and your real world performance may actually be lower. (Also, why on earth would you use AES-256 instead of AES-128?)

    The problem appears to be that OpenVPN has some bottleneck elsewhere so it's not enough to have 1 cycle per byte for GCM encryption. To get the maximum performance, we can use gateways pfsense groups as @whosmatt suggests. That implies that we need 4 cores to scale well, with OpenVPN running on 3 or 4 cores. Or maybe pfsense / FreeBSD / OpenVPN supports intel hyper threads and two cores with 4 HT could be enough.

    If this sort of scaling is sufficient, you likely don't need one core per openvpn process. (The way openvpn hits its bottleneck tends to be that it can't keep enough packets in flight to saturate the CPU, not that it maxes out the CPU.) Note that you won't ever get a full gigabit on a single connection this way, but that might be sufficient for your needs.



  • I'm planning to build my first pfSense box. I live in Europe and I have a 1 gigabit fiber optical connection. I plan to use OpenVPN or IPSec or what is the fastest VPN I can get on the box. Therefore, the CPU must be at least AES-NI.

    If the CPU is strong or powerful enough it might be not necessary to have AES-NI inside, it might be good for IPsec for
    sure but at the moment for OpenVPN it is may be not really pushing the throughput. Doing an OpenSSL test on the
    same unit is not the same as doing it between two units over the internet.

    After much reading, it seems that the best platform is a CPU released in 2013: Intel Atom C2758. It supports:

    • AES-NI
    • Intel QuickAssist
    • DPDK support (enabled software)
    • AES-NI is nice to have and I personally would even buy a CPU or SoC with it inside, for sure.
    • QuickAssist hm, not so really to answer but it can be a really gainer and for the professional and the enterprise segment
      it might be in the future a really need or nice to have it in! If I install such cards in my Linux, BSD and MS Windows Servers
      and then the firewall would be the bottleneck? Why? And if on both or more sites of a VPN connection QuickAssist is handling
      the internally compression it might be a good sounding thing to get also profit from that in the lower end areas such the home
      network area, for sure not for all peoples this must matching well or fitting their needs.
    • DPDK is a SDK from Intel for network based units and the SDPK is a SDK for storage units to code better and more nearly
      the hardware or to unleash their full potential, so if this might be getting inside of the FreeBSD and/or pfSense source code
      it can be pushing the TCP/IP transporting rate up to x3 faster then today, and please remember with the same hardware!
      So many limitations like single CPU core threaded PPPoE, the entire and common throughput, up to the ability to saturate
      and fairly use a 10 GBit/s line or link, will be then gone! So it might be nice to know that Supermicro is starting to produce
      the next generation of the boards in the mid of 2017 based on an Intel D-1500 series that is offering all of this three things
      natively! (DPDK & SPDK, AES-NI & QuickAssist)

    Source : here

    Things often changes more then we love it.

    Therefore, I'm about to go with the configuration listed here :

    • Super Micro SUPERMICRO A1SRI-2758F - Motherboard - Mini-ITX - Intel Atom C2758 - USB3.0 - 4 x Gigabit LAN - Onboard-Grafik (A1SRI-2758F-O)
    • MS-Tech CI-57 - Ultra Small Form Factor - Mini-ITX - 120 Watt (CI-57-120)
    • SanDisk SSD - SSD - 64GB - 6,4 cm (2.5") - SATA-600 (SDSSDP-064G-G25)
    • Kingston KVR16LSE11/8HB

    Might be a real pfSense bomb but not really hitting the 1 GBit/s mark! Perhaps around 850 MBit/s! 
    Without PPPoE you will be really nice sorted, perhaps to much, but this might be also pointed to the use case
    and the installed packets.

    It seems a shame to buy a 2013 cpu in 2017 that has been reported by Intel to have a bug that will prevent him from booting :(

    Who is pressing you? If the PPPoE will be surely not there a smaller unit will be also matching well to your needs!
    Jetway NF9HG-2930 is doing then the job for you for around 350 Euro and able to realize much more!

    So, what do you guys think about it ? Should I go for the above setup or use a i3/i7 skylake/kaby lake that supports AES-NI and forget about QAT/DPDK ? Any coments are welcome !

    Only for SPI/NAT & VPN it will be to much or with other words go and have a look for the Qoton J1900 box for $260
    that will not route that 1 GBit/s at the WAN interface but is nice and cheap. If you are willing to be future proof the
    C2758 will be a really bomb, or you take something between this parts, like a Intel core i3 cpu and you will be happy!


  • Banned

    @pfBasic:

    I'd never heard of DPDK before this thread (thanks for that!) but it looks like you don't need to worry about it when picking a CPU.

    I just found some more info on DPDK and pfSense and thought I'd share here since you asked:

    https://www.netgate.com/blog/pfsense-around-the-world-better-ipsec-tryforward-and-netmap-fwd.html

    Back in February, I wrote a blog post that discussed our plans for pfSense software version 2.3, which is now in alpha, and our plans for pfSense 3.0. While I promoted DPDK then, we’ve since found that netmap provides a simpler API, and substantially better safety, as the device drivers remain in the kernel, rather than running in userspace with DPDK. Still, DPDK provides a set of libraries, such as longest-prefix match, which uses a variation of the DIR-24-8 algorithm for routing lookups, which we should find useful in our pursuit of the ultimate open source software router.

    It looks like the intent is to move away from DPDK but will still implement it in some form? Hopefully someone more knowledgeable can chime in but I thought I'd share the post.



  • It looks like the intent is to move away from DPDK but will still implement it in some form?

    Perhaps not for the general or public usage? Who knows. Link

    Hopefully someone more knowledgeable can chime in but I thought I'd share the post.

    Would be nice to hear some news about that DPDK, netmap-fwrd and QuickAssist this might be the
    most interesting things for me to hear about.



  • @BlueKobold:

    I'm planning to build my first pfSense box. I live in Europe and I have a 1 gigabit fiber optical connection. I plan to use OpenVPN or IPSec or what is the fastest VPN I can get on the box. Therefore, the CPU must be at least AES-NI.

    If the CPU is strong or powerful enough it might be not necessary to have AES-NI inside, it might be good for IPsec for
    sure but at the moment for OpenVPN it is may be not really pushing the throughput. Doing an OpenSSL test on the
    same unit is not the same as doing it between two units over the internet.

    I really wish you would stop spreading FUD about AES-NI and OpenVPN. There is a significant performance benefit from using AES-NI hardware with OpenVPN, end of story.

    So, what do you guys think about it ? Should I go for the above setup or use a i3/i7 skylake/kaby lake that supports AES-NI and forget about QAT/DPDK ? Any coments are welcome !

    Only for SPI/NAT & VPN it will be to much or with other words go and have a look for the Qoton J1900 box for $260
    that will not route that 1 GBit/s at the WAN interface but is nice and cheap. If you are willing to be future proof the
    C2758 will be a really bomb, or you take something between this parts, like a Intel core i3 cpu and you will be happy!

    Neither the C2758 nor the J1900 can do gigabit OpenVPN. If that's the requirement, go with a fast current i3 (you still probably won't hit a full gigabit without some questionable tuning tradeoffs, but you'll get a lot closer). Forget QAT, it's a pipe dream. You don't need DPDK.



  • careful with the c2758 as it may have the clock generator bug depending on the revision: https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/

    you could get a ryzen 1700 with a b350 board as well


  • Banned

    @messerchmidt:

    you could get a ryzen 1700 with a b350 board as well

    I am looking forward to seeing Ryzen CPU's popping up in the pfSense world to see where they fit in for budget horsepower.



  • @messerchmidt:

    you could get a ryzen 1700 with a b350 board as well

    It would be nice to see some relevant benchmarks for that platform.



  • This maybe helps goo.gl/WWGIcT

    That Ryzen really kicks Xeon behinds.

    The past two weeks I'm like a hawk on reading steroids. Trying to find the definitive answer on the "how to obtain 1 gbps (VPN) throughput without applying for a new mortgage" question. This thread is a great help so thank you all guys!



  • @Korny:

    This maybe helps goo.gl/WWGIcT

    Nope, those are pretty irrelevant benchmarks. :) At this point I don't have any real reason to think that a $400 ryzen will outperform a $120 i3 for this application. (It's not an application that scales well with increasing core count, and it does respond well to increasing clock speed. Up to a point–it also gets bottlenecked by packet/buffer sizing issues and increasing CPU performance tends to see diminishing returns.) It's a conversation that will get more interesting when we see what AMD offers on the lower end because building a pfsense on a 1700x doesn't make a whole lot more sense than building one on an E5. If AMD offers something comparable to denverton at a much better price/performance ratio than we've seen so far from avoton/rangely (intel's denverton strategy is still mostly a mystery), then that will shake up this space.

    Trying to find the definitive answer on the "how to obtain 1 gbps (VPN) throughput without applying for a new mortgage" question.

    Either that's hyperbole or you have a really small mortgage. The answer to this is pretty straightforward–buy the highest clocked i3 or i5 you can find/afford. You are unlikely to hit 1Gbps with a single OpenVPN stream regardless of the CPU. If you use multiple OpenVPN instances or use a different implementation (ipsec) then the 1Gbps target isn't that hard or expensive. Certainly less than the ryzen 1700x.



  • This thread has been cold for a few months, but it's still seems relevant. I'm also looking for the best inexpensive hardware for a full time VPN tunnel to saturate a 1GB fibre connection.

    Looking about, it seems like these are good candidates:

    Although the J1900, while cheap doesn't offer the AES-NI instructions, which seems like a non-starter for a VPN device.

    Any thoughts?

    Most polished/appliance-like look. Atom x7-3950… less CPU power than the Haswell device

    Most power, and bad-ass looking heat fin case. Haswell processor

    Least powerful, but cheap. There are even cheaper J1900 devices from Qoton


  • Banned

    Not even close. You won't fit gigabit VPN even with the latest high clock i7.

    Gigabit openvpn is limited by openvpn at this point.

    You can get gigabit openvpn with gateway groups on an i3, but that setup has its own set of limitations and advantages.



  • @pfBasic:

    Not even close. You won't fit gigabit VPN even with the latest high clock i7.

    Gigabit openvpn is limited by openvpn at this point.

    You can get gigabit openvpn with gateway groups on an i3, but that setup has its own set of limitations and advantages.

    Hmmm… what about IPSec IKEv2? I'm less worried about industrial espionage and more worried about my ISP selling/analyzing my connection log, traffic and browser history. Perhaps a lower level of encryption would be adequate?

    -dw


  • Banned

    I've never used ipsec but I would guess you'd have no problem with that. You can use the oldest most broken / compromised encryption you want for that.

    Your ISP will not attempt to decrypt your encrypted traffic no matter how easy it might be to do so.



  • @daveweinstein:

    @pfBasic:

    Not even close. You won't fit gigabit VPN even with the latest high clock i7.

    Gigabit openvpn is limited by openvpn at this point.

    You can get gigabit openvpn with gateway groups on an i3, but that setup has its own set of limitations and advantages.

    Hmmm… what about IPSec IKEv2? I'm less worried about industrial espionage and more worried about my ISP selling/analyzing my connection log, traffic and browser history. Perhaps a lower level of encryption would be adequate?

    -dw

    IPsec isn't less secure than openvpn, it's just more of a pain to set up and much harder to reliably access from arbitrary locations on the internet. If you can use IPsec it's likely to perform better, but if you don't control both ends it might be hard to get working.



  • Did someone say an I5-7600k can't do a gigabit/s but they think the lastest I3 can?



  • @Ryu945:

    Did someone say an I5-7600k can't do a gigabit/s but they think the lastest I3 can?

    No, they said even the fastest CPU can't achieve single stream gigabit because of non-CPU bottlenecks which dominate far below 1gbps. With multiple OpenVPN instances you can achieve 1gbps in aggregate even with a relatively modest CPU.



  • Explain what this bottle neck is clearly then.  Can I just use one PFsense router and run multiple instances of the VPN.  Then tell the same Pfsense router to merge it as a multi-WAN connection?


  • Banned

    Somewhere in openvpn software it simply does not scale to gigabit.

    So you create multiple instances, which will utilize multiple cores/threads. Create a gateway group and you can bypass the restriction for some types of traffic but not all. I.e., anything that uses only one connection will be limited to the max throughout of one openvpn instance.



  • Explain what this bottle neck is clearly then.

    On OpenVPN the TUN/TAP architecture for sure.

    Can I just use one PFsense router and run multiple instances of the VPN.

    On OpenVPN you might be able to set up several tunnels and they all can be running on one cpu core each!
    Over IPsec you might be able to set up also more then one IPsec tunnel too, but with the need of more IP addresses.



  • @BlueKobold:

    Explain what this bottle neck is clearly then.

    On OpenVPN the TUN/TAP architecture for sure.

    Not really. There are more fundamental problems with the openvpn protocol that prevent it from approaching the limits of tun/tap. In my experience when it maxes out on a high speed link, it will do so before it runs out of CPU. Fundamentally, the problem is that it can't keep enough packets in flight to saturate a higher speed link. Too much of the code is synchronous: in a simplified view, the receiver will get a packet, process it, send it on, tell the sender it's ready for another one, etc. In a more asynchronous/threaded model the receiver would get a packet, tell the sender it's ready for another one, start processing the first one, get a second one, tell the sender it's ready for another one, start processing the second one, tell the sender it's ready for another one, send the first packet on, etc. At that point the tun interface becomes a bottleneck, but one you could throw hardware at (throwing hardware at openvpn now doesn't really change things much).



  • Does the attached diagram I made shed light on the subject?



  • Banned

    Yeah, you can get in the neighborhood of 300Mbps AES-128 with an SoC J3355 Celeron @ 2.0 GhZ.

    Throwing a 4.2 GhZ i3-7350k at it only gets you in the 650Mbps range. Beyond that it didn't get much faster.

    While that may seem like linear scaling, it isn't. One part is an SoC Celeron architecture, the other is an actively cooled desktop part with a very high clock meant to be overclocked.



  • Or you could just use a higher performance VPN such as IKEv2 with AES GCM to get you 600-700 Mbps with much more modest HW requirements if you're using your box as a VPN server.

    I've yet to find a public network that won't connect through IPSec IKEv2 but I do have an OpenVPN tcp 443 server running as backup just in case.

    If you're using your box as a vpn client your best bet is using gateway groups to run multiple OpenVPN client connections. That will get you again about 6-700 Mbps inbound on multi connection traffic.


  • Banned

    Yeah, gateway groups are the best answer for most people. It won't work for all types of traffic but will work for a lot of it.

    With gateway groups you can get probably 900Mbps from a low power $75 J3455-ITX.



  • Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate


  • Banned

    There isn't a simple X hardware is best answer.

    It depends on how much $ you want to spend, if gateway groups are a solution for your use case, if IPSec is a solution for you or are you set on OpenVPN, etc.

    The simple answer for the best hardware will always be whatever modern CPU with the highest clock speed. So that would be a very expensive overclocked CPU. But since that's ridiculous, and there are diminishing returns no one does that.

    IMO the i3-7350k would be the absolute upper limit for 99.9% of home use pfSense (and probably 90% of commercial) installations @$150.
    That CPU would also be massive Overkill for almost all pfSense setups. You simply aren't going to exceed the limits of 2 cores with hyperthreading @ 4.2GhZ pushing packets at home. If you do find a way to do that then you are probably doing something entirely unnecessary.

    Most home users are probably best suited by a modern (Apollo lake as of now) SoC, or an old eBay SFF workstation desktop.



  • @malabarka:

    Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate

    Get a Chinese Qotom mini PC which has 4 Intel LAN ports, Core i5 or for slightly less money an i3. Both have AES- NI.

    https://www.aliexpress.com/store/product/Latest-New-core-I5-5250U-4-LAN-Home-computer-router-server-support-pfsense-linux-firewall-Cent/108231_32798137911.html?spm=2114.12010608.0.0.XFsGIe



  • @pfBasic:

    Somewhere in openvpn software it simply does not scale to gigabit.

    So you create multiple instances, which will utilize multiple cores/threads. Create a gateway group and you can bypass the restriction for some types of traffic but not all. I.e., anything that uses only one connection will be limited to the max throughout of one openvpn instance.

    @pfBasic:

    I've never used ipsec but I would guess you'd have no problem with that. You can use the oldest most broken / compromised encryption you want for that.

    Your ISP will not attempt to decrypt your encrypted traffic no matter how easy it might be to do so.

    What aspect of the software is maxing out the hardware?  I want to see if I can find hardware that can handle the problem the software is causing.  I can't do that search if I don't know how the software is maxing out the hardware.

    For example.  Is the CPU fast enough but there is a cache limitation problem when it comes to 1 Gbps?  Is it a bus speed problem with how the software is sending the data?  The software has to hit some hardware limitation, otherwise it would be going faster.


  • Banned

    At this time you are not going to find hardware to solve the problem.

    Someday if/when OpenVPN is updated, but not now.

    If you want to try though, go buy an i7-7740X, put it on liquid helium, overclock it and let us know how close you get to gigabit with AES-128-GCM. Please post pics!  ;) My bet is 780Mbps!



  • @Waqar.UK:

    @malabarka:

    Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate

    Get a Chinese Qotom mini PC which has 4 Intel LAN ports, Core i5 or for slightly less money an i3. Both have AES- NI.

    https://www.aliexpress.com/store/product/Latest-New-core-I5-5250U-4-LAN-Home-computer-router-server-support-pfsense-linux-firewall-Cent/108231_32798137911.html?spm=2114.12010608.0.0.XFsGIe

    yeah that looks a decent package.