Hardware for Gigabit Fiber and Openvpn?
-
All,
I've been doing a lot of research regarding the power required for 1gb connections and OpenVPN. I realize no matter what I do I won't be able to get a single OpenVPN connection up to 1gb, but I have no idea how far I can push it as a client connected to a 3rd party vpn provider. The best thread I found on processors vs VPN speed is https://forum.pfsense.org/index.php?topic=115673.0
Initially I looked at the c2758 Atom, but discounted believing it would not afford sufficient VPN speed
From my research, it looks like the best options are to go with a kabylake, xeon e3, or xeon d.
Specifically, I'm looking at the following options (I don't have space for a full-size 1U - all the 1u's listed are 15" long or less. I also don't want to have to build from scratch if I can avoid it.)
Dell R210 ii xeon e3 1240v2 - This seems to be the favorite budget option
Supermicro 5019S-ML with xeon e3 1240v5 or 1270v5
Supermicro 5018D-FN4T with xeon d-1541
Mitxpc MPC-GBH270MX500 with up to i7 7700I'm pretty confident that all of the above would have no trouble with the 1gb throughput, but I'm not as confident as to which ones would yield the highest throughput through a tunnel. My goal would be to get in the neighborhood of 500-700mbps throughput with vpn
Also, the price difference from the dell r210 to the Supermicro 5018d is pretty substantial, about $1000, so I'm also wondering if I go with one of the higher end options, what would be the impact of running PFsense virtualized on esxi? Ideally, I'd love to have a virtual pfsense appliance, 2 copies of 2012r2 running on the same box (1 for downloads behind vpn, 1 for plex routed out of the vpn), and maybe zabbix or other monitoring tool.
I'd appreciate any insight the community would be willing to share.
-
Well, that e3 1240v2 actually wouldn't be all that great at OpenVPN.
OpenVPN performance is basically dependent on 3 things when you are shopping for CPUs:
-
Modern Architecture
-
Base Clock Frequency
-
AES-NI
Architecture updates improve performance throughout (sometimes not by much) but the e2 1240v2 is 5 years old now and EoL.
OpenVPN is single threaded, so clock speed winds, not core count. (You can run multiple instances in a gateway group [and this is probably the only way you'll get gigabit VPN on any hardware] but it has limitations).
AES-NI now is not the same as AES-NI 5 years ago, not even close. It gets updated over time and is dramatically more efficient now.
All of those things combine to say, buy new stuff.
Old repurposed hardware works great on pfSense, but when you are trying to push a lot of bandwidth through an OpenVPN pipe, you really will benefit from new technology.What is really important to know for this kind of application is what else will you do with pfSense?
A lot of posts on here start with "what should I buy for high VPN throughput?" and then halfway through the OP will throw in that they also want to run an IDS/IPS, which is even more CPU intensive than OpenVPN.
If you just want to NAT/Firewall/OpenVPN, then check out an i3-7350K.
Latest Architecture
@ 4.2GHz I believe it has the fastest base frequency of any stock Intel CPU (don't quote me on that, but either way, it's fast).
Modern AES-NIIf you are looking to do IDS/IPS then you might need to go to an i7-7700K.
It's pretty much the same as the i3-7350K but with 4 cores/8 threads instead of 2 cores/4 threads.Keep in mind, I seriously doubt that you will hit gigabit speeds on either of those CPUs on a single OpenVPN client. I'm guessing that you will get ~750Mbps.
In a gateway group, no problem.VPN provider will be important too for those speeds, unless you are hosting your own server or VPS something? Then you'll be pretty familiar with it's capabilities.
Shop around and do some trial/short term subscriptions until you find one that performs well for you.
PIA is probably a good place to start.
AES-GCM would be a nice to have VPN provider feature (almost none of them support it) if you plan on using 2.4 now or in the future. It will increase your throughput (although I have no idea by how much at those speeds).
-
-
Initially, I probably wouldn't use Suricata. Ultimately, I would like that capability. Can I ask why the 7700 over the e3 1270 v5? I'm actually leaning towards a 1u simply for the replaceable psu. The mini pc with the 7700 has some mini psu that I'm pretty sure I could never replace in the event of failure.
750 mbps through vpn on a gig uplink would be fine with me.
-
7700K, not 7700. The reason would be 600MHz higher clock frequency for OpenVPN throughput.
The Xeon has ECC, the 7700K has a GPU. Neither are important for pfSense. If you have to pick one then ECC is obviously more relevant, but you really don't need ECC in your firewall.
If you want to virtualize, then it depends on what else you want running on it. If you want some kind of video encode/decode, then i7, if you want something that ECC actually matters on, then Xeon.
I've never used a 1U, but I'm assuming your concern is cooling? 80W vs 91W TDP
Things like this (not for the correct socket) suggest that you can do high TDP CPUs in a 1U case, but I honestly don't know if it's a good idea, or if that product is any good. http://www.rego.com.tw/news_detail.php?id=139
Ultimately, you're going to have to choose on a compromise between VPN throughput and overall computing power and feature sets.
Honestly I'd still be most interested in the i3-7350K. One core will get you high VPN throughput, and you will probably be able to get suricata to do line speed IDS/IPS. It is a really high clock speed. It also costs less than half of the other two and only has 60W TDP, which is probably more realistic in a 1U.
-
I'm actually leaning towards a 1u simply for the replaceable psu. The mini pc with the 7700 has some mini psu that I'm pretty sure I could never replace in the event of failure.
If you are building new as 1u, personally, I would choose a 2u as it would give a little more flexibility to choose a nice cooler, a little more leg room to fit parts, etc. It won't impact your ability to replace PSU as well. I'm also in a similar situation leaning between a 2u and micro/mini ATX.
Hope it helps.
-
only has 60W TDP, which is probably more realistic in a 1U.
The Us don't really matter for power dissipation, the design of the cooling system does; 1U systems with a pair of CPUs which are each 90W aren't uncommon.
-
Regarding the mini pc http://www.mitxpc.com/proddetail.php?prod=MPC-GBH270MX500 (this is basically a brix/intel nuc equivalent)
it runs a 7700 3.6ghz-4.2ghz, not the 7700k so it would seem like there'd be only marginal benefit (I think) over a E3 1270v5 at 3.6-4.0ghz.I'm leaning towards the 1U Supermicro 5019S-L or Supermicro 5019S-ML because the power supplies are easily replaced, and PSU failure is something I've experienced a lot as systems get older.
Additionally the 1u will have more space to work with than a minipc unit like the one aboveI doubt I'd be able to replace the power supply in the mitxpc unit.
Of course I could custom build a solution, but then it becomes more of a time investment and a side project (also the itx case offerings are usually not great in my experience). I'd much rather buy something easy that I can swap in an ssd and hit the ground running. As far as the 7700k goes, that thing is a monster, but this is the only thing I've found that would support it http://www.wiredzone.com/supermicro-servers-compact-single-processor-sys-1019s-m2-10025167 and judging by the number of fans on that, I'm guessing it will sound like a fighter jet. I'd prefer to have something a bit less on the noisy side. I know it's a pain to get something in a 1u form factor that balances noise and power, but I'm still hopeful.
The reason why I'm focused on a small 1u 15" length is I don't have space for a full 25"-27" long server.
For virtualization I'd want to run 2 instances of 2012 and/or one monitoring instance of something like zabbix. One of the 2012's would be outside the VPN for plex and the other would be inside for downloads and other internal applications. It sounds like I'd need to dedicate at least 3-4 cores to pfsense and that would leave me with 1 dual core 2012 instance for plex and then 2 single cores left over.
Maybe the better way to go is to get a 1u and put in a 7350/7300 i3 and run pfsense bare metal (as pfBasic suggested) and get a separate 8 core/16 thread for virtualization. It will end up costing significantly more, but is that the best way to go long term? Or will I need the more than 4 cores for pfsense in the next few years?
-
Regarding the mini pc http://www.mitxpc.com/proddetail.php?prod=MPC-GBH270MX500 (this is basically a brix/intel nuc equivalent)
it runs a 7700 3.6ghz-4.2ghz, not the 7700k so it would seem like there'd be only marginal benefit (I think) over a E3 1270v5 at 3.6-4.0ghz.Overclocking feature in the i7 7700K, in case you want to overcllock it.
-
Maybe the better way to go is to get a 1u and put in a 7350/7300 i3 and run pfsense bare metal (as pfBasic suggested) and get a separate 8 core/16 thread for virtualization. It will end up costing significantly more, but is that the best way to go long term? Or will I need the more than 4 cores for pfsense in the next few years?
I'm thinking of the same question too – separate boxes for pfSense and virtualization server vs. one box with virtual pfSense instance -- and I'm leaning towards keeping things separate, even if it costs more. I'm a big fan of baremetal for critical infrastructure (firewalls, routers, DNS, DHCP)... at least one baremetal, the rest can be virtual, based on some harsh lessons learned in the enterprise with all DNS servers virtualized and the storage array crashing and unable to provide the entire company with DNS, DHCP, etc. for hours (basically total outage, company probably lost millions of dollars in revenue. thank goodness wasn't my job area). I realize the industry push is to software defined networking and virtual appliances, but having at least one infrastructure appliance on baremetal seems to be the safe approach.
I built a 2U overkill box in 2013. Xeon Haswell E3-1220v3 @ 3.1GHz, Supermicro Micro-ATX motherboard, Rosewill 2U case, 16 GB ECC RAM, but I'm looking to downsize my pfSense box to a Kaby Lake i3 and probably a mini-ITX mobo and case and picoPSU power supply just to save a ton of electricity. Can then re-purpose the current server to be a virtualization host or something that actually uses the power.
Pipe dreams are that OpenVPN eventually goes multi-core/multi-threaded, but it's been a long shot for years, and I don't think there's much movement. What's more realistic is that VPN providers upgrade to OpenVPN 2.4 and support AES-GCM, which should squeeze out more throughput.
Some day I'll build a beefy 1U or 2U Xeon pfSense box with 10Gigabit NICs, but I'll wait till I have solar panels on the house to the point I don't care about power consumption or heat dissipation. For now, I'll use Xeons for legit servers and keep pfSense pretty lean and quiet. Leaning towards i3-7320 (51W TDP) or i3-7350K (60W TDP) like you...just to max out OpenVPN throughput and leave the possibility open for Suricata in the future.
-
I didn't want this to be my first post here on someone else's thread, but…
I've been looking to come back to pfSense from Mikrotik as I'm getting a 1000/250 Mbps connection installed next week, and think I'm in the same boat as OP. I want to be able to utilize as much as the speed I'm paying for as possible while running VPN, having some NAT rules, and wanting to run other applications in pfsense (I haven't looked closely yet, but IP blocking, content filtering/ad blocking would be nice). I'm also looking to use OpenVPN to route only certain traffic from certain servers on my home network through to a PIA endpoint. I'm looking at the hardware below. Maybe this will be suitable for the OP as well.
ASUS RS200-E9-PS2-F 1U Rackmount Server LGA 1151 - $512
https://www.newegg.com/Product/Product.aspx?Item=N82E16816110142
Intel Core i3-7350K Kaby Lake 4.2GHz LGA 1151 - $129
http://www.microcenter.com/product/473230/Core_i3-7350K_Kaby_Lake_42GHz_LGA_1151_Boxed_ProcessorI'd also add a couple sticks of RAM and an SSD, but I'm unsure on which yet.
How would that mobo, with those Intel NICs, and that CPU fair on a 1000/250 Mbps connection running what I and OP mentioned? I was looking to spend around $700, and this, I believe, would get me the best speeds and processing power at that price. Is this accurate? Would I be better off finding something with better Intel NICs?
Thanks!
-
I'm still going back and forth on this. I'm thinking at this point of getting this initially http://www.mitxpc.com/proddetail.php?prod=5018D-FN4T
I know I won't get anywhere near full line speed with ovpn, but I'm thinking I can intially get this setup while I wait for KabyLake capable server/ mini pc options to become more prevalent.
My other thought is go first with baremetal on an i6300 which I know isn't kabylake, but I'm still thinking I'd get pretty high speeds.
http://www.mitxpc.com/proddetail.php?prod=EKGBH170M557
Also, @eptesicus how do you think the noise level will be on that Asus server?
@eptesicus, also doesn't look like the 7350 is supported for that server looks like 7320 and 7300 are though
https://www.asus.com/Commercial-Servers-Workstations/RS200-E9-PS2-F/HelpDesk_CPU/
-
I'm still going back and forth on this. I'm thinking at this point of getting this initially http://www.mitxpc.com/proddetail.php?prod=5018D-FN4T
I know I won't get anywhere near full line speed with ovpn, but I'm thinking I can intially get this setup while I wait for KabyLake capable server/ mini pc options to become more prevalent.
My other thought is go first with baremetal on an i6300 which I know isn't kabylake, but I'm still thinking I'd get pretty high speeds.
http://www.mitxpc.com/proddetail.php?prod=EKGBH170M557
Also, @eptesicus how do you think the noise level will be on that Asus server?
@eptesicus, also doesn't look like the 7350 is supported for that server looks like 7320 and 7300 are though
https://www.asus.com/Commercial-Servers-Workstations/RS200-E9-PS2-F/HelpDesk_CPU/
Good eye. I e-mailed ASUS about that CPU. Because of the crazy cheap price on it, I was hoping that it'd work on that motherboard, even without being able to utilize the unlocking capabilities of the 'K', but ASUS would only reiterate what was on the list. I ended up order the server and getting the i3-7100 for $115. I couldn't justify another $40 to get 0.2 GHz and 1 MB cache from the i3-7320, which was also a decent deal. I'm hoping I won't regret it. But if I have big regrets, then it would have been because I purchased an i3 period.
Here's what I snagged… It should all be here tomorrow to get playing.
i3-7100 - https://www.amazon.com/gp/product/B01NCESRJX/
ADATA SU800 128GB SSD - https://www.amazon.com/gp/product/B01K8A29BE/
Crucial 8GB Kit (4GBx2) RAM - https://www.amazon.com/gp/product/B00MMLUZ2I/
ASUS RS200-E9-PS2-F - https://www.newegg.com/Product/Product.aspx?Item=N82E16816110142As far as noise goes... I'll work on lessening that. I have two Supermicro 846E16's that made a lot of noise at first, but I got the fans in them down to 10% speed. They're ok in the home office, and I wouldn't want this 1U server to be any louder, so I'll work on the fans if I can without running into temperature issues.
The downside to this server already is finding rack rails... Even though I have a model number for them, I don't believe they exist. Finding the IPMI/KVM module is also going to be a pain, but I can add that later on.
-
http://www.vpnusers.com/viewtopic.php?f=7&t=7270
Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425
Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.
@eptesicus very interested to know reworld throughput once you get yourself setup.
-
http://www.vpnusers.com/viewtopic.php?f=7&t=7270
Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425
Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.
@eptesicus very interested to know reworld throughput once you get yourself setup.
I was looking into the Supermicro 5018D-FN4T as well, but was really hoping that it didn't have 10Gb ports. Similar servers without those ports can be found for $600-700. If they had a front I/O server, exactly like the FN4T, but without 10Gb, I would have bought it over the ASUS. But, I can't justify spending that much right now on the firewall. I hope it works out well for you!
I'll be getting everything today, and will setup it up this weekend. My install won't happen until Thursday, so I'll be doing my testing then. I'll update you with my findings.
By the way, how do you like Xabbix? Have you used it before? I ended up trying many monitoring tools (Xabbix, OpenNMS, OP5, etc.) and ended up settling on PRTG. However, I'm maxed out on the free number of sensors (100), and am starting to look into other monitoring tools again.
-
You're going to be disappointed.
When you say symmetrical do you mean simultaneous upload and download?
850Mbps? Nope. 425Mbps? I seriously doubt it.
OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.
You can do gateway groups but a lot of traffic can't utilize it.
Also, iperf is synthetic not real world.
-
You're going to be disappointed.
When you say symmetrical do you mean simultaneous upload and download?
850Mbps? Nope. 425Mbps? I seriously doubt it.
OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.
You can do gateway groups but a lot of traffic can't utilize it.
Also, iperf is synthetic not real world.
What about the i3-7100? The clock speed for that is 3.9GHz.
-
@eptesicus – Any updates? How's the hardware looking?
I'm thinking of going with the same box with the 7320
-
High clock i3's are probably your best bet for high OpenVPN throughput.
I don't believe you'll get gigabit speeds on OpenVPN regardless of clock speed. I've never heard of anyone achieving that.
Your best bet (within a reasonable budget) for OpenVPN performance would be an i3-7350K, it's relatively inexpensive and has I think one of the highest clock speeds of any intel CPU.
That being said, it won't blow an i3-7100 out of the water.
-
@eptesicus – Any updates? How's the hardware looking?
I'm thinking of going with the same box with the 7320
I finally got some time to look at this thread again…
I spent hours communicating with ASUS regarding the RS200-E9-PS2-F. The server I received had an earlier BIOS version that didn't support my CPU. They kept suggesting that I should buy another CPU that was supported by the BIOS it was shipped with, update the BIOS, and then return that CPU and reinstall the i3 I have... Or... Ship the server back to ASUS to have them update the BIOS. They refused to ship me an updated BIOS chip, even though Supermicro and ASRock have done it for me in the past. I was opposed to buying a supported CPU because of most shop's no-return policy on CPUs... Also, I should mention that most modern ASUS systems have the ability to update the BIOS without any CPU or RAM installed. This board does not. My RAM isn't listed on the extremely short (and filled with extremely hard to find RAM makes and models) list, so the customer service rep was saying that I needed to buy some memory on the QVL The whole process was extremely frustrating, but I ended up finding a shop that accepted returns on opened CPUs. I got the BIOS updated with no problem, and thankfully the RAM I bought worked.
Now that it's all together, and I don't have to talk to ASUS anymore, I love the system. I'm extremely thankful that the BIOS supports the ability to change the fan speeds, but I'll need to do some tweaking. On the latest BIOS version, I'm finding that the CPU temps aren't correct like it is in the OS. I did swap out the heatsink for a Thermaltake Engine 27, but I don't notice a difference in temps. Once I get gigabit internet on Thursday, I'll test the temps under some VPN load. I'm unfamiliar with FreeBSD, so I'm not quite sure how to do some load testing on it just yet. Thinking about it - if I get time tomorrow, I'll try to find a way to load test the system with the factory heatsink and the aftermarket cooler.
Also... I know I'm going to get some flack for this on here, but in looking at my open-source firewall options again, I decided to install OPNsense on it. I'm going to test that first, but will look into installing pfsense second if I'm unsatisfied.
If I didn't say it already... With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.
-
Fiber is getting installed as we speak! We'll see how this goes, and I'll let you know how the router performs.