Hardware for Gigabit Fiber and Openvpn?

eptesicus · Apr 28, 2017, 5:08 PM

You're going to be disappointed.

When you say symmetrical do you mean simultaneous upload and download?

850Mbps? Nope. 425Mbps? I seriously doubt it.

OpenVPN is single threaded. Cores don't matter, clock speed does. 2.1GHz isn't fast enough.

You can do gateway groups but a lot of traffic can't utilize it.

Also, iperf is synthetic not real world.

What about the i3-7100? The clock speed for that is 3.9GHz.

idealanthony · May 2, 2017, 5:35 AM

@eptesicus – Any updates? How's the hardware looking?

I'm thinking of going with the same box with the 7320

pfBasic · May 2, 2017, 5:41 AM

High clock i3's are probably your best bet for high OpenVPN throughput.

I don't believe you'll get gigabit speeds on OpenVPN regardless of clock speed. I've never heard of anyone achieving that.

Your best bet (within a reasonable budget) for OpenVPN performance would be an i3-7350K, it's relatively inexpensive and has I think one of the highest clock speeds of any intel CPU.

That being said, it won't blow an i3-7100 out of the water.

eptesicus · May 3, 2017, 2:57 AM

@idealanthony:

@eptesicus – Any updates? How's the hardware looking?

I'm thinking of going with the same box with the 7320

I finally got some time to look at this thread again…

I spent hours communicating with ASUS regarding the RS200-E9-PS2-F. The server I received had an earlier BIOS version that didn't support my CPU. They kept suggesting that I should buy another CPU that was supported by the BIOS it was shipped with, update the BIOS, and then return that CPU and reinstall the i3 I have... Or... Ship the server back to ASUS to have them update the BIOS. They refused to ship me an updated BIOS chip, even though Supermicro and ASRock have done it for me in the past. I was opposed to buying a supported CPU because of most shop's no-return policy on CPUs... Also, I should mention that most modern ASUS systems have the ability to update the BIOS without any CPU or RAM installed. This board does not. My RAM isn't listed on the extremely short (and filled with extremely hard to find RAM makes and models) list, so the customer service rep was saying that I needed to buy some memory on the QVL The whole process was extremely frustrating, but I ended up finding a shop that accepted returns on opened CPUs. I got the BIOS updated with no problem, and thankfully the RAM I bought worked.

Now that it's all together, and I don't have to talk to ASUS anymore, I love the system. I'm extremely thankful that the BIOS supports the ability to change the fan speeds, but I'll need to do some tweaking. On the latest BIOS version, I'm finding that the CPU temps aren't correct like it is in the OS. I did swap out the heatsink for a Thermaltake Engine 27, but I don't notice a difference in temps. Once I get gigabit internet on Thursday, I'll test the temps under some VPN load. I'm unfamiliar with FreeBSD, so I'm not quite sure how to do some load testing on it just yet. Thinking about it - if I get time tomorrow, I'll try to find a way to load test the system with the factory heatsink and the aftermarket cooler.

Also... I know I'm going to get some flack for this on here, but in looking at my open-source firewall options again, I decided to install OPNsense on it. I'm going to test that first, but will look into installing pfsense second if I'm unsatisfied.

If I didn't say it already... With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

eptesicus · May 4, 2017, 12:44 PM

Fiber is getting installed as we speak! We'll see how this goes, and I'll let you know how the router performs.

pfBasic · May 4, 2017, 7:53 PM

@idealanthony:

If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

A better recommendation would be to avoid that board at all costs, instead of buying two cpus.

VAMike · May 4, 2017, 7:24 PM

@eptesicus:

If I didn't say it already… With all of the frustration I've had with ASUS and getting the BIOS updated... I'd stay away from the 7350K. Go with something on the approved CPU list, but just be prepared to buy a cheap CPU (I went with a $29 G3900) to update the BIOS.

I'd send back the ASUS before making some third party eat the cost of discounting a CPU as open box.

A Former User · May 4, 2017, 9:05 PM

If it helps anyone on this thread, I built this system a few months ago:

CPU: Intel Core i3-7320 4.1GHz Dual-Core Processor
CPU Cooler: Noctua NH-L9i 33.8 CFM CPU Cooler
Motherboard: ASRock H270M-ITX/ac Mini ITX LGA1151 Motherboard
Memory: Corsair Vengeance LPX 8GB (2 x 4GB) DDR4-2400 Memory
Storage: Samsung 830 Series 128GB 2.5" Solid State Drive
Case: Mini-Box M350 Case w/ 150W PicoPSU
pfSense 2.4 beta

OpenVPN throughput test:

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

This test finishes in 6.39s, giving about 500Mbps theoretical limit. During the test, htop shows 50-70% CPU usage on one core although it does vary quite a bit. I can max out my 70Mbps cable connection and it doesn't break a sweat, but plan to move to 1Gb fiber in the near future.

Snort:

I haven't finished setting Snort up yet, but @datum posted some benchmarks here https://forum.pfsense.org/index.php?topic=123801.msg689437#msg689437. Snort CPU usage is highly dependent on the rule sets enabled though. It's really hard to get an idea of performance impact since people usually don't post their rule/inspection settings.

Power Usage
13W Idle/low load, 45-55W heavy load

patrick0525 · May 5, 2017, 10:49 AM

Similar Results on 2.4 beta:

CPU: i3-6100 3.7GHz

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

This test finishes in 6.99s,

Running: i340-t4 nic, in-line Surricate, Squid, PfBlockerNG, OpenVPN
Speed test runs at 70/70 down/up and cpu runs at 10-13%

itchris · May 6, 2017, 10:33 PM

Recently got gigabit fiber and run Ethernet straight form the ONT. I currently have pfsense virtualized on a poweredge with a decent 6x12 running on top of ESXI 6.5. I've tried e1000 and vmxnet3 drivers and messed around with hardware offloading. (better results with e1000) Long story short after much testing and tweaking.. my throughput was kind of meh. I saw around half the throughput with openvpn client. I have an edgerouter x sfp for lab purposes and I was getting around 800mbps.

That led me to getting this:

IN WIN CE685.FH300TB3 Black MicroATX Slim Case

ASRock H270M-ITX/ac LGA 1151 Intel H270 Mini ITX Motherboards

Intel Core i3-7350K Kaby Lake Dual-Core 4.2 GHz LGA 1151 61W

CORSAIR Vengeance LPX 8GB 288-Pin DDR4 SDRAM DDR4 2400

Noctua NH-L9i 95mm SSO2 CPU Cooler

Spare 120GB SSD

I will report back with my results. For science!

Finger79 · May 6, 2017, 11:31 PM

For those on pfSense 2.4, curious what –cipher aes-256-gcm and aes-128-gcm will give compared to just aes-256-cbc and aes-128-cbc. For fun and profit!

pfBasic · May 7, 2017, 1:26 AM

@Finger79:

For those on pfSense 2.4, curious what –cipher aes-256-gcm and aes-128-gcm will give compared to just aes-256-cbc and aes-128-cbc. For fun and profit!

https://forum.pfsense.org/index.php?topic=105238.msg709164#msg709164

A synthetic test was all I could really do. My server is GCM but upload speed is only 15Mbps. Very few VPN providers use GCM at this time.

TL;DR, It doesn't make a big difference in synthetic tests but I'm not convinced that test is valuable for testing this.

https://calomel.org/aesni_ssl_performance.html

This study shows a big difference in AES-NI for GCM over CBC.

A Former User · May 7, 2017, 1:37 PM

@Finger79, some additional data points for you. With the i3-7320:

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Sun May  7 08:18:22 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
        6.41 real         6.40 user         0.00 sys

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Sun May  7 08:18:38 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
        5.89 real         5.88 user         0.00 sys

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
Sun May  7 08:18:54 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
        6.28 real         6.28 user         0.00 sys

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
Sun May  7 08:19:09 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
        5.73 real         5.72 user         0.01 sys

Which gives us:

AES-256-CBC, 3200/6.41 => 499.22 Mbps
AES-256-GCM, 3200/5.89 => 543.29 Mbps
AES-128-CBC, 3200/6.28 => 509.55 Mbps
AES-128-GCM, 3200/5.73 => 558.46 Mbps

idealanthony · May 10, 2017, 9:03 AM

@eptesicus - Were you able to get the asus to install pfsense?

I can't for the life of me get it to boot/ install off a thumbdrive. It just gives me an error code AE and sits there.

The computer is working fine, I've tested it with a windows thumbdrive and it has no problem booting off of that.

I created the thumb drive with the AMD64 img file using win32 disk imager

Edit- So 2.3.4 would not install for the life of me. Switched to 2.4 beta from 5/9, worked like a champ… Now I just need to go through the exercise of setup

MikeV7896 · May 10, 2017, 3:48 PM

Could be a UEFI setting in the BIOS… I don't think the 2.3.x branch supports UEFI like 2.4 does.

If you change the boot mode to Legacy, you'll probably be able to install 2.3.4.

eptesicus · May 12, 2017, 7:06 PM

@idealanthony:

@eptesicus - Were you able to get the asus to install pfsense?

I ended up installing OPNSense instead.

pfBasic · May 22, 2017, 6:29 PM

@itchris:

That led me to getting this:

Intel Core i3-7350K Kaby Lake Dual-Core 4.2 GHz LGA 1151 61W

I will report back with my results. For science!

How did this end up working out for you?

What kind of OpenVPN speeds are you getting, at what encryption levels and at what CPU utilization? Have you used IDS/IPS on this at all?

kapara · May 22, 2017, 8:38 PM

Be aware that if you are using SMB you will see horrible performance yet if you do an iperf test you will saturate the maximum. I thought IPSec was having problems on my pfsense. I could only get 4mbit on a 10mbit link yet iperf could push 9.86 mbit.

EricE · May 25, 2017, 12:05 AM

@pfBasic:

Your best bet (within a reasonable budget) for OpenVPN performance would be an i3-7350K, it's relatively inexpensive and has I think one of the highest clock speeds of any intel CPU.

It's an awesome value gaming CPU too: https://www.cpubenchmark.net/singleThread.html

Number three on the list - amazing for an i3.