Hardware for Gigabit Fiber and Openvpn?

Finger79

Maybe the better way to go is to get a 1u and put in a 7350/7300 i3 and run pfsense bare metal (as pfBasic suggested) and get a separate 8 core/16 thread for virtualization. It will end up costing significantly more, but is that the best way to go long term? Or will I need the more than 4 cores for pfsense in the next few years?

I'm thinking of the same question too – separate boxes for pfSense and virtualization server vs. one box with virtual pfSense instance -- and I'm leaning towards keeping things separate, even if it costs more. I'm a big fan of baremetal for critical infrastructure (firewalls, routers, DNS, DHCP)... at least one baremetal, the rest can be virtual, based on some harsh lessons learned in the enterprise with all DNS servers virtualized and the storage array crashing and unable to provide the entire company with DNS, DHCP, etc. for hours (basically total outage, company probably lost millions of dollars in revenue. thank goodness wasn't my job area). I realize the industry push is to software defined networking and virtual appliances, but having at least one infrastructure appliance on baremetal seems to be the safe approach.

I built a 2U overkill box in 2013. Xeon Haswell E3-1220v3 @ 3.1GHz, Supermicro Micro-ATX motherboard, Rosewill 2U case, 16 GB ECC RAM, but I'm looking to downsize my pfSense box to a Kaby Lake i3 and probably a mini-ITX mobo and case and picoPSU power supply just to save a ton of electricity. Can then re-purpose the current server to be a virtualization host or something that actually uses the power.

Pipe dreams are that OpenVPN eventually goes multi-core/multi-threaded, but it's been a long shot for years, and I don't think there's much movement. What's more realistic is that VPN providers upgrade to OpenVPN 2.4 and support AES-GCM, which should squeeze out more throughput.

Some day I'll build a beefy 1U or 2U Xeon pfSense box with 10Gigabit NICs, but I'll wait till I have solar panels on the house to the point I don't care about power consumption or heat dissipation. For now, I'll use Xeons for legit servers and keep pfSense pretty lean and quiet. Leaning towards i3-7320 (51W TDP) or i3-7350K (60W TDP) like you...just to max out OpenVPN throughput and leave the possibility open for Suricata in the future.

eptesicus

I didn't want this to be my first post here on someone else's thread, but…

I've been looking to come back to pfSense from Mikrotik as I'm getting a 1000/250 Mbps connection installed next week, and think I'm in the same boat as OP. I want to be able to utilize as much as the speed I'm paying for as possible while running VPN, having some NAT rules, and wanting to run other applications in pfsense (I haven't looked closely yet, but IP blocking, content filtering/ad blocking would be nice). I'm also looking to use OpenVPN to route only certain traffic from certain servers on my home network through to a PIA endpoint. I'm looking at the hardware below. Maybe this will be suitable for the OP as well.

ASUS RS200-E9-PS2-F 1U Rackmount Server LGA 1151 - $512
https://www.newegg.com/Product/Product.aspx?Item=N82E16816110142
Intel Core i3-7350K Kaby Lake 4.2GHz LGA 1151 - $129
http://www.microcenter.com/product/473230/Core_i3-7350K_Kaby_Lake_42GHz_LGA_1151_Boxed_Processor

I'd also add a couple sticks of RAM and an SSD, but I'm unsure on which yet.

How would that mobo, with those Intel NICs, and that CPU fair on a 1000/250 Mbps connection running what I and OP mentioned? I was looking to spend around $700, and this, I believe, would get me the best speeds and processing power at that price. Is this accurate? Would I be better off finding something with better Intel NICs?

Thanks!

idealanthony

I'm still going back and forth on this. I'm thinking at this point of getting this initially http://www.mitxpc.com/proddetail.php?prod=5018D-FN4T

I know I won't get anywhere near full line speed with ovpn, but I'm thinking I can intially get this setup while I wait for KabyLake capable server/ mini pc options to become more prevalent.

My other thought is go first with baremetal on an i6300 which I know isn't kabylake, but I'm still thinking I'd get pretty high speeds.

http://www.mitxpc.com/proddetail.php?prod=EKGBH170M557

Also, @eptesicus how do you think the noise level will be on that Asus server?

@eptesicus, also doesn't look like the 7350 is supported for that server looks like 7320 and 7300 are though

https://www.asus.com/Commercial-Servers-Workstations/RS200-E9-PS2-F/HelpDesk_CPU/

eptesicus

@idealanthony:

I'm still going back and forth on this. I'm thinking at this point of getting this initially http://www.mitxpc.com/proddetail.php?prod=5018D-FN4T

I know I won't get anywhere near full line speed with ovpn, but I'm thinking I can intially get this setup while I wait for KabyLake capable server/ mini pc options to become more prevalent.

My other thought is go first with baremetal on an i6300 which I know isn't kabylake, but I'm still thinking I'd get pretty high speeds.

http://www.mitxpc.com/proddetail.php?prod=EKGBH170M557

Also, @eptesicus how do you think the noise level will be on that Asus server?

@eptesicus, also doesn't look like the 7350 is supported for that server looks like 7320 and 7300 are though

https://www.asus.com/Commercial-Servers-Workstations/RS200-E9-PS2-F/HelpDesk_CPU/

Good eye. I e-mailed ASUS about that CPU. Because of the crazy cheap price on it, I was hoping that it'd work on that motherboard, even without being able to utilize the unlocking capabilities of the 'K', but ASUS would only reiterate what was on the list. I ended up order the server and getting the i3-7100 for $115. I couldn't justify another $40 to get 0.2 GHz and 1 MB cache from the i3-7320, which was also a decent deal. I'm hoping I won't regret it. But if I have big regrets, then it would have been because I purchased an i3 period.

Here's what I snagged… It should all be here tomorrow to get playing.

i3-7100 - https://www.amazon.com/gp/product/B01NCESRJX/
ADATA SU800 128GB SSD - https://www.amazon.com/gp/product/B01K8A29BE/
Crucial 8GB Kit (4GBx2) RAM - https://www.amazon.com/gp/product/B00MMLUZ2I/
ASUS RS200-E9-PS2-F - https://www.newegg.com/Product/Product.aspx?Item=N82E16816110142

As far as noise goes... I'll work on lessening that. I have two Supermicro 846E16's that made a lot of noise at first, but I got the fans in them down to 10% speed. They're ok in the home office, and I wouldn't want this 1U server to be any louder, so I'll work on the fans if I can without running into temperature issues.

The downside to this server already is finding rack rails... Even though I have a model number for them, I don't believe they exist. Finding the IPMI/KVM module is also going to be a pain, but I can add that later on.

idealanthony

http://www.vpnusers.com/viewtopic.php?f=7&t=7270

Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425

Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.

@eptesicus very interested to know reworld throughput once you get yourself setup.

eptesicus

@idealanthony:

http://www.vpnusers.com/viewtopic.php?f=7&t=7270

Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425

Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.

@eptesicus very interested to know reworld throughput once you get yourself setup.

I was looking into the Supermicro 5018D-FN4T as well, but was really hoping that it didn't have 10Gb ports. Similar servers without those ports can be found for $600-700. If they had a front I/O server, exactly like the FN4T, but without 10Gb, I would have bought it over the ASUS. But, I can't justify spending that much right now on the firewall. I hope it works out well for you!

I'll be getting everything today, and will setup it up this weekend. My install won't happen until Thursday, so I'll be doing my testing then. I'll update you with my findings.

By the way, how do you like Xabbix? Have you used it before? I ended up trying many monitoring tools (Xabbix, OpenNMS, OP5, etc.) and ended up settling on PRTG. However, I'm maxed out on the free number of sensors (100), and am starting to look into other monitoring tools again.

pfBasic

You're going to be disappointed.

When you say symmetrical do you mean simultaneous upload and download?

850Mbps? Nope. 425Mbps? I seriously doubt it.

OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.

You can do gateway groups but a lot of traffic can't utilize it.

Also, iperf is synthetic not real world.

eptesicus

@pfBasic:

You're going to be disappointed.

When you say symmetrical do you mean simultaneous upload and download?

850Mbps? Nope. 425Mbps? I seriously doubt it.

OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.

You can do gateway groups but a lot of traffic can't utilize it.

Also, iperf is synthetic not real world.

What about the i3-7100? The clock speed for that is 3.9GHz.

idealanthony

@eptesicus – Any updates? How's the hardware looking?

I'm thinking of going with the same box with the 7320

pfBasic

High clock i3's are probably your best bet for high OpenVPN throughput.

I don't believe you'll get gigabit speeds on OpenVPN regardless of clock speed. I've never heard of anyone achieving that.

Your best bet (within a reasonable budget) for OpenVPN performance would be an i3-7350K, it's relatively inexpensive and has I think one of the highest clock speeds of any intel CPU.

That being said, it won't blow an i3-7100 out of the water.

eptesicus

@idealanthony:

@eptesicus – Any updates? How's the hardware looking?

I'm thinking of going with the same box with the 7320

I finally got some time to look at this thread again…

I spent hours communicating with ASUS regarding the RS200-E9-PS2-F. The server I received had an earlier BIOS version that didn't support my CPU. They kept suggesting that I should buy another CPU that was supported by the BIOS it was shipped with, update the BIOS, and then return that CPU and reinstall the i3 I have... Or... Ship the server back to ASUS to have them update the BIOS. They refused to ship me an updated BIOS chip, even though Supermicro and ASRock have done it for me in the past. I was opposed to buying a supported CPU because of most shop's no-return policy on CPUs... Also, I should mention that most modern ASUS systems have the ability to update the BIOS without any CPU or RAM installed. This board does not. My RAM isn't listed on the extremely short (and filled with extremely hard to find RAM makes and models) list, so the customer service rep was saying that I needed to buy some memory on the QVL The whole process was extremely frustrating, but I ended up finding a shop that accepted returns on opened CPUs. I got the BIOS updated with no problem, and thankfully the RAM I bought worked.

Now that it's all together, and I don't have to talk to ASUS anymore, I love the system. I'm extremely thankful that the BIOS supports the ability to change the fan speeds, but I'll need to do some tweaking. On the latest BIOS version, I'm finding that the CPU temps aren't correct like it is in the OS. I did swap out the heatsink for a Thermaltake Engine 27, but I don't notice a difference in temps. Once I get gigabit internet on Thursday, I'll test the temps under some VPN load. I'm unfamiliar with FreeBSD, so I'm not quite sure how to do some load testing on it just yet. Thinking about it - if I get time tomorrow, I'll try to find a way to load test the system with the factory heatsink and the aftermarket cooler.

Also... I know I'm going to get some flack for this on here, but in looking at my open-source firewall options again, I decided to install OPNsense on it. I'm going to test that first, but will look into installing pfsense second if I'm unsatisfied.

If I didn't say it already... With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

eptesicus

Fiber is getting installed as we speak! We'll see how this goes, and I'll let you know how the router performs.

pfBasic

@idealanthony:

If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

A better recommendation would be to avoid that board at all costs, instead of buying two cpus.

VAMike

@eptesicus:

If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

I'd send back the ASUS before making some third party eat the cost of discounting a CPU as open box.

A Former User

If it helps anyone on this thread, I built this system a few months ago:

CPU: Intel Core i3-7320 4.1GHz Dual-Core Processor
CPU Cooler: Noctua NH-L9i 33.8 CFM CPU Cooler
Motherboard: ASRock H270M-ITX/ac Mini ITX LGA1151 Motherboard
Memory: Corsair Vengeance LPX 8GB (2 x 4GB) DDR4-2400 Memory
Storage: Samsung 830 Series 128GB 2.5" Solid State Drive
Case: Mini-Box M350 Case w/ 150W PicoPSU
pfSense 2.4 beta

OpenVPN throughput test:

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

This test finishes in 6.39s, giving about 500Mbps theoretical limit. During the test, htop shows 50-70% CPU usage on one core although it does vary quite a bit. I can max out my 70Mbps cable connection and it doesn't break a sweat, but plan to move to 1Gb fiber in the near future.

Snort:

I haven't finished setting Snort up yet, but @datum posted some benchmarks here https://forum.pfsense.org/index.php?topic=123801.msg689437#msg689437. Snort CPU usage is highly dependent on the rule sets enabled though. It's really hard to get an idea of performance impact since people usually don't post their rule/inspection settings.

Power Usage
13W Idle/low load, 45-55W heavy load

patrick0525