Hardware for Gigabit Fiber and Openvpn?
-
I'm still going back and forth on this. I'm thinking at this point of getting this initially http://www.mitxpc.com/proddetail.php?prod=5018D-FN4T
I know I won't get anywhere near full line speed with ovpn, but I'm thinking I can intially get this setup while I wait for KabyLake capable server/ mini pc options to become more prevalent.
My other thought is go first with baremetal on an i6300 which I know isn't kabylake, but I'm still thinking I'd get pretty high speeds.
http://www.mitxpc.com/proddetail.php?prod=EKGBH170M557
Also, @eptesicus how do you think the noise level will be on that Asus server?
@eptesicus, also doesn't look like the 7350 is supported for that server looks like 7320 and 7300 are though
https://www.asus.com/Commercial-Servers-Workstations/RS200-E9-PS2-F/HelpDesk_CPU/
Good eye. I e-mailed ASUS about that CPU. Because of the crazy cheap price on it, I was hoping that it'd work on that motherboard, even without being able to utilize the unlocking capabilities of the 'K', but ASUS would only reiterate what was on the list. I ended up order the server and getting the i3-7100 for $115. I couldn't justify another $40 to get 0.2 GHz and 1 MB cache from the i3-7320, which was also a decent deal. I'm hoping I won't regret it. But if I have big regrets, then it would have been because I purchased an i3 period.
Here's what I snagged… It should all be here tomorrow to get playing.
i3-7100 - https://www.amazon.com/gp/product/B01NCESRJX/
ADATA SU800 128GB SSD - https://www.amazon.com/gp/product/B01K8A29BE/
Crucial 8GB Kit (4GBx2) RAM - https://www.amazon.com/gp/product/B00MMLUZ2I/
ASUS RS200-E9-PS2-F - https://www.newegg.com/Product/Product.aspx?Item=N82E16816110142As far as noise goes... I'll work on lessening that. I have two Supermicro 846E16's that made a lot of noise at first, but I got the fans in them down to 10% speed. They're ok in the home office, and I wouldn't want this 1U server to be any louder, so I'll work on the fans if I can without running into temperature issues.
The downside to this server already is finding rack rails... Even though I have a model number for them, I don't believe they exist. Finding the IPMI/KVM module is also going to be a pain, but I can add that later on.
-
http://www.vpnusers.com/viewtopic.php?f=7&t=7270
Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425
Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.
@eptesicus very interested to know reworld throughput once you get yourself setup.
-
http://www.vpnusers.com/viewtopic.php?f=7&t=7270
Looks like I might be able to expect ~425 symmetrical through vpn with the d1541 server. Although this is using a softether and not on pfsense. I'm assuming I'll be somewhere close to ~425
Also considering http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/2/ where he's using the same server for routing 2gb over fiber, and using a secondary 1gbps for vpn.
@eptesicus very interested to know reworld throughput once you get yourself setup.
I was looking into the Supermicro 5018D-FN4T as well, but was really hoping that it didn't have 10Gb ports. Similar servers without those ports can be found for $600-700. If they had a front I/O server, exactly like the FN4T, but without 10Gb, I would have bought it over the ASUS. But, I can't justify spending that much right now on the firewall. I hope it works out well for you!
I'll be getting everything today, and will setup it up this weekend. My install won't happen until Thursday, so I'll be doing my testing then. I'll update you with my findings.
By the way, how do you like Xabbix? Have you used it before? I ended up trying many monitoring tools (Xabbix, OpenNMS, OP5, etc.) and ended up settling on PRTG. However, I'm maxed out on the free number of sensors (100), and am starting to look into other monitoring tools again.
-
You're going to be disappointed.
When you say symmetrical do you mean simultaneous upload and download?
850Mbps? Nope. 425Mbps? I seriously doubt it.
OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.
You can do gateway groups but a lot of traffic can't utilize it.
Also, iperf is synthetic not real world.
-
You're going to be disappointed.
When you say symmetrical do you mean simultaneous upload and download?
850Mbps? Nope. 425Mbps? I seriously doubt it.
OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.
You can do gateway groups but a lot of traffic can't utilize it.
Also, iperf is synthetic not real world.
What about the i3-7100? The clock speed for that is 3.9GHz.
-
@eptesicus – Any updates? How's the hardware looking?
I'm thinking of going with the same box with the 7320
-
High clock i3's are probably your best bet for high OpenVPN throughput.
I don't believe you'll get gigabit speeds on OpenVPN regardless of clock speed. I've never heard of anyone achieving that.
Your best bet (within a reasonable budget) for OpenVPN performance would be an i3-7350K, it's relatively inexpensive and has I think one of the highest clock speeds of any intel CPU.
That being said, it won't blow an i3-7100 out of the water.
-
@eptesicus – Any updates? How's the hardware looking?
I'm thinking of going with the same box with the 7320
I finally got some time to look at this thread again…
I spent hours communicating with ASUS regarding the RS200-E9-PS2-F. The server I received had an earlier BIOS version that didn't support my CPU. They kept suggesting that I should buy another CPU that was supported by the BIOS it was shipped with, update the BIOS, and then return that CPU and reinstall the i3 I have... Or... Ship the server back to ASUS to have them update the BIOS. They refused to ship me an updated BIOS chip, even though Supermicro and ASRock have done it for me in the past. I was opposed to buying a supported CPU because of most shop's no-return policy on CPUs... Also, I should mention that most modern ASUS systems have the ability to update the BIOS without any CPU or RAM installed. This board does not. My RAM isn't listed on the extremely short (and filled with extremely hard to find RAM makes and models) list, so the customer service rep was saying that I needed to buy some memory on the QVL The whole process was extremely frustrating, but I ended up finding a shop that accepted returns on opened CPUs. I got the BIOS updated with no problem, and thankfully the RAM I bought worked.
Now that it's all together, and I don't have to talk to ASUS anymore, I love the system. I'm extremely thankful that the BIOS supports the ability to change the fan speeds, but I'll need to do some tweaking. On the latest BIOS version, I'm finding that the CPU temps aren't correct like it is in the OS. I did swap out the heatsink for a Thermaltake Engine 27, but I don't notice a difference in temps. Once I get gigabit internet on Thursday, I'll test the temps under some VPN load. I'm unfamiliar with FreeBSD, so I'm not quite sure how to do some load testing on it just yet. Thinking about it - if I get time tomorrow, I'll try to find a way to load test the system with the factory heatsink and the aftermarket cooler.
Also... I know I'm going to get some flack for this on here, but in looking at my open-source firewall options again, I decided to install OPNsense on it. I'm going to test that first, but will look into installing pfsense second if I'm unsatisfied.
If I didn't say it already... With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.
-
Fiber is getting installed as we speak! We'll see how this goes, and I'll let you know how the router performs.
-
If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.
A better recommendation would be to avoid that board at all costs, instead of buying two cpus.
-
If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.
I'd send back the ASUS before making some third party eat the cost of discounting a CPU as open box.
-
If it helps anyone on this thread, I built this system a few months ago:
CPU: Intel Core i3-7320 4.1GHz Dual-Core Processor
CPU Cooler: Noctua NH-L9i 33.8 CFM CPU Cooler
Motherboard: ASRock H270M-ITX/ac Mini ITX LGA1151 Motherboard
Memory: Corsair Vengeance LPX 8GB (2 x 4GB) DDR4-2400 Memory
Storage: Samsung 830 Series 128GB 2.5" Solid State Drive
Case: Mini-Box M350 Case w/ 150W PicoPSU
pfSense 2.4 betaOpenVPN throughput test:
openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbcThis test finishes in 6.39s, giving about 500Mbps theoretical limit. During the test, htop shows 50-70% CPU usage on one core although it does vary quite a bit. I can max out my 70Mbps cable connection and it doesn't break a sweat, but plan to move to 1Gb fiber in the near future.
Snort:
I haven't finished setting Snort up yet, but @datum posted some benchmarks here https://forum.pfsense.org/index.php?topic=123801.msg689437#msg689437. Snort CPU usage is highly dependent on the rule sets enabled though. It's really hard to get an idea of performance impact since people usually don't post their rule/inspection settings.
Power Usage
13W Idle/low load, 45-55W heavy load -
Similar Results on 2.4 beta:
CPU: i3-6100 3.7GHz
openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbcThis test finishes in 6.99s,
Running: i340-t4 nic, in-line Surricate, Squid, PfBlockerNG, OpenVPN
Speed test runs at 70/70 down/up and cpu runs at 10-13% -
Recently got gigabit fiber and run Ethernet straight form the ONT. I currently have pfsense virtualized on a poweredge with a decent 6x12 running on top of ESXI 6.5. I've tried e1000 and vmxnet3 drivers and messed around with hardware offloading. (better results with e1000) Long story short after much testing and tweaking.. my throughput was kind of meh. I saw around half the throughput with openvpn client. I have an edgerouter x sfp for lab purposes and I was getting around 800mbps.
That led me to getting this:
IN WIN CE685.FH300TB3 Black MicroATX Slim Case
ASRock H270M-ITX/ac LGA 1151 Intel H270 Mini ITX Motherboards
Intel Core i3-7350K Kaby Lake Dual-Core 4.2 GHz LGA 1151 61W
CORSAIR Vengeance LPX 8GB 288-Pin DDR4 SDRAM DDR4 2400
Noctua NH-L9i 95mm SSO2 CPU Cooler
Spare 120GB SSD
I will report back with my results. For science!
-
For those on pfSense 2.4, curious what –cipher aes-256-gcm and aes-128-gcm will give compared to just aes-256-cbc and aes-128-cbc. For fun and profit!
-
For those on pfSense 2.4, curious what –cipher aes-256-gcm and aes-128-gcm will give compared to just aes-256-cbc and aes-128-cbc. For fun and profit!
https://forum.pfsense.org/index.php?topic=105238.msg709164#msg709164
A synthetic test was all I could really do. My server is GCM but upload speed is only 15Mbps. Very few VPN providers use GCM at this time.
TL;DR, It doesn't make a big difference in synthetic tests but I'm not convinced that test is valuable for testing this.
https://calomel.org/aesni_ssl_performance.html
This study shows a big difference in AES-NI for GCM over CBC.
-
@Finger79, some additional data points for you. With the i3-7320:
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc Sun May 7 08:18:22 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode 6.41 real 6.40 user 0.00 sys time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm Sun May 7 08:18:38 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode 5.89 real 5.88 user 0.00 sys time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc Sun May 7 08:18:54 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode 6.28 real 6.28 user 0.00 sys time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm Sun May 7 08:19:09 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode 5.73 real 5.72 user 0.01 sysWhich gives us:
AES-256-CBC, 3200/6.41 => 499.22 Mbps
AES-256-GCM, 3200/5.89 => 543.29 Mbps
AES-128-CBC, 3200/6.28 => 509.55 Mbps
AES-128-GCM, 3200/5.73 => 558.46 Mbps -
@eptesicus - Were you able to get the asus to install pfsense?
I can't for the life of me get it to boot/ install off a thumbdrive. It just gives me an error code AE and sits there.
The computer is working fine, I've tested it with a windows thumbdrive and it has no problem booting off of that.
I created the thumb drive with the AMD64 img file using win32 disk imager
Edit- So 2.3.4 would not install for the life of me. Switched to 2.4 beta from 5/9, worked like a champ… Now I just need to go through the exercise of setup
-
Could be a UEFI setting in the BIOS… I don't think the 2.3.x branch supports UEFI like 2.4 does.
If you change the boot mode to Legacy, you'll probably be able to install 2.3.4.
-
@eptesicus - Were you able to get the asus to install pfsense?
I ended up installing OPNSense instead.
