New system questions

mattyd

I'm currently running an EdgeRouter Lite, and am getting a little frustrated with the L2TP IPSec VPN not working. I got no answer to my forum post, and none of the similar issues I found was ever resolved. Because of that, and because I like to try new things, I was considering swapping over to a pfSense setup instead. I came up with a few questions that I haven't been able to find answers for when Googling around and would appreciate some help.

The Atom processors that the both the SG-2440 and SG-4860 are built from are getting a bit long in the tooth, it seems. Any feel for if there are Skylake or Kaby Lake replacements for those parts coming? I'm ready to buy, but if the landscape is changing in the next 6-8 months or so, I'm willing to wait.
How does the ERL compare to the SG-2440? The SG-4860?

As far as I can tell Xeon D is where Intel is going for embedded stuff going forward? I think that is overkill for my router but will be doing one of those builds for my NAS.

Requirements:
3) I currently have 50/50 FIOS internet but I am considering jumping up to 100/100, 300/300 FIOS. Or to drop my phone and go with Comcast 250 (for $60/month!, but much slower upstream) in the next year or so.
4) I travel a fair amount for work and occasional personal trips, and I like to VPN to my network so I can access my files and Plex server for movies. Looking to do IPSEC or OpenVPN here.
5) If the hotel has a good enough network and I'm traveling for fun, I'll also dump backups of my pictures back to the home network as well. So I'd like to be able to pump as much data through the VPN as my bandwidth allows.
5A) I believe I read that the SG-4860 was currently good for about 100mbit currently, with more coming when AES NI support comes online? Where does the SG-2440 sit?
6) No bittorrent, other than what Battle.Net does on my behalf.
7) Crashplan backups come in from four or five friends and family, but no other "services" in the way of inbound traffic.
8) I'd prefer to run the management server for my UniFi APs on the router if possible. If not, it can stay on my PC if it has to.

From reading, I am pretty sure the SG-2440 will keep up with what I currently have, but I'd rather not be replacing hardware in a year if I change internet plans. Or if something on a more current architecture and smaller process comes along with double the perf at the same power consumption. ;)

I don't think my requirements are heavy, except maybe the higher bitrate VPN. At a hotel with wired internet, I DID once manage to push 30mbit through the ERL with PPTP in the middle of the night, but I don't know if I was limited by the ERL or by the hotel's internet.

Assume I lose my mind and pony up for 300/300 internet. Or if FIOS decides they need to compete with the 2gbps Comcast offering by coming out with an attractive gigabit plan. If I go with the SG-2440, will I regret it in a year? Am I going to regret the SG-4860 in a year? Are either/both beefier than the ERL? I read the ERL falls off pretty fast if you try to do QOS or want IPSec.

I didn't really want to go all the way to $700 if I can avoid it, because that will delay my NAS update. Plus, at the $700 point, some kind of PC starts to look a lot more attractive. The downsides being not supporting the project at that point, more hassle, and power consumption. But on the plus side, there would be no question of being underpowered for the effective lifetime of the hardware either.

I'm aware of the cheaper Netgate versions, but if the hardware won't be something I regret, I'm okay with the premium to support the project. I'm just waffling over whether or not that hardware will be good to go and worth the money for 3-5 years, or if I have to spend more time and effort putting something else together.

If the recommendation is to go with something else, what is the best way to support the project? pfSense Gold? Direct donation to a 501 C charity?

Thanks in advance!

mattlach

I don't have any experience with those atom boards you mention.

I have built a few pfSense systems with low power (35W) Haswell chips (the Celerons even have ECC support) and they have always saturated my 150/150 FiOS connection.

My most recent build (which I am still in the process of setting up) is using a PC Engines APU2C4 embedded type board. I'm very excited about this build.

It's a tiny board you can buy from the manufacturer with a matched tiny aluminum case. It runs a quad core AMD Jaguar based chip at 1Ghz, has three on board Intel 210AT Ethernet adapters, runs completely passively (with cooling coming from CPU contact to the case) and only consumes 6W as measured at the wall, and has 4GB of onboard ECC RAM.

I only just got it running and pfSense installed on it last night so I haven't done extensive testing yet, but with iPerf I was able to push 595Mbit/s through it, which is way beyond what I need for my 150/150 connection. I'm planning on testing OpenVPN speeds with AES-NI too, but I haven't gotten to it yet.

It doesn't have video output though, so you need to do the install via serial console (which is kind of a fun blast from the past)

Another great part is the price. I ordered mine direct shipped from the manufacturer in Switzerland. Got here to the East Coast of the US in 3 days.

APU2C4 board: $114
CASE1D2BLKU Black Enclosure: $9.40
AC12VUS2 US AC adapter: $4.10
MSATA16D 16GB msata SSD: $16.00
Shipping & handling: $29.40
Total: $172.90

I was skeptical at first regarding this low power part, but it really does a great job.

After using pfSense since 2010, I - too - recently tested a Ubiquiti EdgeRouter for a little bit. Mine was a PoE model. I wound up eating the restocking fee at Newegg due to it's miserable performance with QoS enabled. Thank god pfSense supports codel!

mattyd

I'll be curious to hear how your QOS and VPN testing go!

That is a pretty nice price point and perfectly fine for the next couple of years at least. The Comcast 2gbps plan is $300/month, but it looks like the 300/300 plans have disappeared off the useless Frontier website. There is hopefully a decent chance of getting reasonably priced gigabit in the next couple of years though. At that price point, that little box seems like it might be a reasonable stop gap at minimum.

Do you know if AMD has anything similar to the Intel QuickAssist? Support for that is also supposedly coming to pfSense soon as well and should provide a VPN perf bump.

I don't mind putting hardware together, but there are so many options that it is a little overwhelming, and I couldn't find much in the way of comparative benchmarks to what I have now, which makes it hard to understand what HW spec I actually need.

Thanks for the suggestion!

mattlach

@mattyd:

I'll be curious to hear how your QOS and VPN testing go!

Traditional QoS quickly bogs down even the most powerful system, not because the CPU is inadequate, but because of buffer bloat. So - essentially - you could keep increasing CPU power, RAM speed, get expensive NIC's with QoS offload, and in the end you may find that it doesn't actually help at all.

This is why there has been some amount of excitement in the Traffic Shaping subforum regarding codel queues, and even more excitement regarding a possible upcoming fq_codel.

(No one has promised fq_codel is coming to pfSense yet, but we are hopeful)

It accomplishes most of what a cumbersome buffer-bloat creating QoS queue does, with very little in the way of slow downs and buffer bloat. This is what I am planning on using, and I don't expect it to have much of an impact on the load of my little APU system.

VPN testing likely will - however - even though it does have AES-NI, I plan on testing this this evening.

mattyd

I really appreciate the info you've provided, and sharing your results.

I tried doing traffic monitoring on my ERL, but the resulting graphs are not always correct and the UI gets very slow when that is turned on. So I didn't try to pile QOS on top. Ditching my phone line means I get 100/100 internet for the same price as my 50/50 now, but I'll want QOS so things don't interfere with phone calls. Nothing more intense than that, probably. But adding features with more capable hardware is something I'll probably do, just because I can't leave well enough alone. :)

I think I'm down to looking at the SG-2440 and the APU2C4 system you have. If anyone else looking at the thread has experience with both, I'd be interested in how they stack up.

Straight up perf-wise, I'd guess that they are mostly equivalent with a definite single-thread advantage for the Intel. Reading around, it seems most of the important things are all multi-threaded now, so single-thread advantage probably isn't a huge deal. QuickAssist support would give the Intel system a bigger edge, though no counting eggs before they hatch.

The Atom processors that the both the SG-2440 and SG-4860 are built from are getting a bit long in the tooth, it seems. Any feel for if there are Skylake or Kaby Lake replacements for those parts coming? I'm ready to buy, but if the landscape is changing in the next 6-8 months or so, I'm willing to wait.

This can only be answered by the pfSense staff team and not really from us users or customers.

How does the ERL compare to the SG-2440? The SG-4860?

pfSense is a x86 based software firewall that can be turned pending on the hardware given abilities
into a fully featured UTM device and inside of the EdgeRouter is a MIPS CPU and a Linux based OS
with the style of Vyatta CLI. So to compare the incomparable is not really wise.

As far as I can tell Xeon D is where Intel is going for embedded stuff going forward? I think that is overkill for my router but will be doing one of those builds for my NAS.

This is mostly pending on more then one or two points;

number of users and devices
the entire traffic amount
used and offered services
Internet connection speed
VLAN, QoS and routing work

I currently have 50/50 FIOS internet but I am considering jumping up to 100/100, 300/300 FIOS. Or to drop my phone and go with Comcast 250 (for $60/month!, but much slower upstream) in the next year or so.

For 50/50 the APU1D4 or the newer one APU2C4 is all you need. The SG-2220 will do the job also with ease
but for more I would go with the SG-2440 unit it comes with Intel QAT and AES-NI.

I travel a fair amount for work and occasional personal trips, and I like to VPN to my network so I can access my files and Plex server for movies. Looking to do IPSEC or OpenVPN here.

Apples iOS and MacOS devices are using by default IPSec and all windows machines can be using the
ShrewSoftVPN client.

If the hotel has a good enough network and I'm traveling for fun, I'll also dump backups of my pictures back to the home network as well. So I'd like to be able to pump as much data through the VPN as my bandwidth allows.
5A) I believe I read that the SG-4860 was currently good for about 100mbit currently, with more coming when AES NI support comes online? Where does the SG-2440 sit?

speeding up the entire throughput will be nice with AES-NI and IPSec (AES-GCM)
the SG4860 will be able to deliver several 100 MBit/s over IPSec and also some
100 MBit/s at the WAN port.

No bittorrent, other than what Battle.Net does on my behalf.

Crashplan backups come in from four or five friends and family, but no other "services" in the way of inbound traffic.

This will be mostly pending on the used protocols.

I'd prefer to run the management server for my UniFi APs on the router if possible. If not, it can stay on my PC if it has to.

Put it on a RaspBerry PI 2.0
Put it on the PlexServer in a VM
Or place it on a small box such an Alix or APU1D2/APU1D4

I would go with the SG-4860 and then counting for 5 till 6 years in usage.
$700 / 6 years = ~$10 for each month of usage for 5 users its then ~$2 for each month and "nose"!
Not to small and not to big.

Other options are:

APU1D4 ~50 - 150 MBit/s @WAN
Firewall, OpenVPN, snort and pfBlockerNG
APU2C4 ~150 - 250 MBit/s @WAN
Intel NICs, AES-NI, 4 core CPU, ECC RAM, for firewall, IPSec VPN, snort & pfBlockerNG
Jetway NF9HG-2930 ~500 MBit/s - 1000 MBit/s @WAN (pending on PPoE usage or not)
Firewall, snort, squid, pfBlockerNG, OpenVPN or IPSec VPN
ASUSQ87T ~500 MBit/s - multiple 1 GBit/s @WAN (pending on the CPU and installed NIC)
fully UTM with many installed packets, AES-NI (pending on the installed CPU), 10/100/1000/10000
MBit/s pending on the installed NIC(s). But if needed able to upgrade with, NIC, CPU, RAM and
Intel QAT adapter from pfSense or Netgate store.

mattlach

Oh, and as far as Unifi client goes, I'm not 100% sure as I run a virtualization server separate from my pfSense box for everything not router related, but I feel like the best way to accomplish this would be to run bhyve and do a virtual Ubuntu LTS server install in it, and then install the unifi client in that.

If you search the forums there are probably plenty of examples.

mattyd

Thank you both for the responses!

After too much time poking around this weekend, somehow a bunch of stuff to build a setup around the A1SRI-2758F wound up in carts across three different vendors. I hate when that happens. For about $550 after shipping and a leftover Newegg gift card I had, I'll have a 2758 based system with 16gb and 120gb Intel SSD.

There is no question that it is an unholy level of overkill for all my future anticipated needs (note: the very best sort of overkill), but for about the same out of pocket as the Netgate version of the 4860 . . . it was hard to resist. I wasn't quite sold on the dual core 2440 even though it probably would have been perfectly fine.

I still almost just gave in and ordered the 4860 in the process of trying to sort the case (Supermicro SC101i) and power supply out (wound up with a Seasonic 60W brick).

If I had to spend most of my free time from the weekend over again, I would probably just buy the SG-4860. Even at $700, the integration and single point of warranty are worth it even before the support incidents.

All up, I was looking at $530 (no gift card) for a 2558 based system with all the other components. That is $20 less than the Netgate version (though it does have more ram and storage), but I surely spent more than $20 worth of time figuring things out.

Definitely worth thinking about if you don't like the M350 case that all the builds I found seemed to prefer.

But this way I have a platform that is much beefier to play with ESXi on (and can still put up to another 16GB in if I need it for VMs), so that is a win. Short of > 1GBps, I'm set until the hardware dies. And even with faster than gigabit internet, I can probably upgrade myself into it with the PCIe slot for a 10GBps net card and a case to expose it.

mattlach

@mattyd:

Thank you both for the responses!

After too much time poking around this weekend, somehow a bunch of stuff to build a setup around the A1SRI-2758F wound up in carts across three different vendors. I hate when that happens. For about $550 after shipping and a leftover Newegg gift card I had, I'll have a 2758 based system with 16gb and 120gb Intel SSD.

There is no question that it is an unholy level of overkill for all my future anticipated needs (note: the very best sort of overkill), but for about the same out of pocket as the Netgate version of the 4860 . . . it was hard to resist. I wasn't quite sold on the dual core 2440 even though it probably would have been perfectly fine.

I still almost just gave in and ordered the 4860 in the process of trying to sort the case (Supermicro SC101i) and power supply out (wound up with a Seasonic 60W brick).

If I had to spend most of my free time from the weekend over again, I would probably just buy the SG-4860. Even at $700, the integration and single point of warranty are worth it even before the support incidents.

All up, I was looking at $530 (no gift card) for a 2558 based system with all the other components. That is $20 less than the Netgate version (though it does have more ram and storage), but I surely spent more than $20 worth of time figuring things out.

Definitely worth thinking about if you don't like the M350 case that all the builds I found seemed to prefer.

But this way I have a platform that is much beefier to play with ESXi on (and can still put up to another 16GB in if I need it for VMs), so that is a win. Short of > 1GBps, I'm set until the hardware dies. And even with faster than gigabit internet, I can probably upgrade myself into it with the PCIe slot for a 10GBps net card and a case to expose it.

Ahh, nice. ESXi. I was wondering what you were going to do with all that RAM. :p

If you want to virtualize things, give Proxmox a gander too. I used to use ESXi, but got fed up with their poor support (not patching known bugs, etc) on the free version, and the fact that the paid version was out of reach cost wise for me.

One of the great things about Proxmox is that it combines VM's and Linux Containers, making it able for you to do a lot more with less.

I have a server with dual 6 core Xeons (24 logical cores in total) and 192GB of RAM, which - when I as running ESXi - left me some comfort space to grow. Since migrating to Proxmox my server is totally overkill due to me making lots of use of containers instead of VM's. I've been toying with he idea of scaling back to save power. I rarely see CPU use over 2-3%, and have never seen it over 25%.

I used to run pfSense in ESXi, but I built a dedicated box for pfSense because I got tired of my network going down every time I wanted to tinker with the server :p

That, and security wise, having your main server exposed to your WAN just makes me a little uncomfortable, even though it is PROBABLY fine.

mattyd

I'll definitely consider that! My understanding was ESXi would provide sufficient isolation so that I could run pfSense and another VM that had maybe a radius server for WiFi and the UniFi server. Even down to keeping the physical NICs separated from each VM. I am guessing that level of isolation is not included in Promox?

Even so, I agree with you and would not have the confidence to lump NAS or internal servers AND the router onto the same hardware. It would be just stuff related to the router and networking probably. And yes, I know it is still horrible overkill even if I get into more and fancier packages. To be honest, if NewEgg or Amazon had directly been a vendor for the 2558 board, it would have been a much tougher call. At least some of spending the extra on the 2758 was being able to get a seller I felt better about. The $20 shipping one vendor was asking on the 2558 was 1/3 of the price difference right there!

Plus, once I had mentally committed to spending SG-4860 level funds, it was a tiny step to go on to getting bigger hardware for the same cost. I personally have never in my life said "Wow, I really regret buying more hardware than I thought I'd need." ;) 13W more idle isn't free for a 24/7 appliance, but ~ $12.5/yr isn't back breaking either. That it gives me more room to play with stuff I'd like to learn, like ESXi, and now Proxmox is just a bonus.

And I still probably should have just bought the SG-4860. ;)

That is a pretty beefy server. I'm interested that it wasn't totally overkill before moving to Proxmox! Did you just have a lot of VMs that chewed up resources through reservation? Xeon-D 1540 gets you 16 virtual cores and up to 128gb of ram in a 45W TDP. But pretty spendy. Built in 10GBps is pretty sweet though. http://ark.intel.com/products/87039/Intel-Xeon-Processor-D-1540-12M-Cache-2_00-GHz I'm probably doing the 1520 for my NAS build, but I'm only replacing a Synology 1513+ with 3GB of ram and not 24 logical Xeon cores with 192GB of RAM.

mattlach

@mattyd:

I'll definitely consider that! My understanding was ESXi would provide sufficient isolation so that I could run pfSense and another VM that had maybe a radius server for WiFi and the UniFi server. Even down to keeping the physical NICs separated from each VM. I am guessing that level of isolation is not included in Promox?

You can do this on both proxmox and ESXi, but it requires a VT-D capable system (CPU, Motherboard and BIOS must all support it) in order to pass the NIC's (or at least the WAN port) through to your pfSense guest. This is MUCH better than the alternative of WAN Port -> Virtual switch -> pfSense guest -> virtual switch -> LAN port, but I'm still a lot more comfortable having them separate, and it has the added benefit of being able to download something or read web pages while I'm tinkering with my server, as the internet hasn't gone down :p

@mattyd:

That is a pretty beefy server. I'm interested that it wasn't totally overkill before moving to Proxmox! Did you just have a lot of VMs that chewed up resources through reservation?

Well, honestly, the CPU's were always overkill. The reason I got them was because it was the only way to get all the PCIe lanes and RAM support I wanted at the time.

You'll find that as soon as you start virtualizing things, you'll wind up with an insatiable appetite for RAM.

My first ESXi server was an AMD E350 Zacate board with 8GB of RAM. I only ran pfSense and an Ubuntu guest for a home built NAS on that, but I quickly felt limited.

Then I got an 8 core AMD FX-8120 and maxed it out with 32GB of RAM. The CPU was always sufficient but I always found myself chafing under the 32GB max RAM limit.

That's when I got my two used Xeon L5640's cheap on eBay and stuck them in a new old stock Supermicro X8DTE board I also found on eBay. I started out with 64GB of RAM, but quickly filled all 12 RAM slots with 8GB sticks for 96GB. Used that for the longest time. Right before I migrated away from ESXi, I ordered 12 16GB sticks and maxed out the board with 192GB of RAM. It was great, but when I switched from ESXi to proxmox I went from "modest room to grow" to "absolutely ridiculous overkill" due to all the RAM using LXC containers saved me.

It's still not terrible, as I ahve used my extra free RAM to increase the size of my ARC (ZFS read cache) to speed up my NAS significantly, but had I known what I know now, I would never have spent the money on that RAM.

@mattyd:

Xeon-D 1540 gets you 16 virtual cores and up to 128gb of ram in a 45W TDP. But pretty spendy. Built in 10GBps is pretty sweet though. http://ark.intel.com/products/87039/Intel-Xeon-Processor-D-1540-12M-Cache-2_00-GHz I'm probably doing the 1520 for my NAS build, but I'm only replacing a Synology 1513+ with 3GB of ram and not 24 logical Xeon cores with 192GB of RAM.

Yeah, I've been eye-balling those Xeon D's on and off.

I like my current server, but it is a little long in the tooth, with only 3GB/s SATA, and USB2, and it uses more power than I'd like.

Supermicro's X10SDV-7TP8F seems amazing with it's 6x gigabit Ethernet ports and onboard LSI SAS controller, but unfortunately it is the 16 core (32 logical) version which is total overkill for me, and has a 65W TDP.

The X10SDV-7TP4F has a more appropriate (for me) 8 core (16 logical) CPU, and 45W TDP, but unfortunately sacrifices the 6 gigabit ports, and instead only has two. I could work with this though. It has two 8x PCIe slots, I could stick my quad port Intel server nic in one of them.

I've found this board for ~$900, which is a fair price I think. What has stopped me from going all in is the fact that I'd need to buy all new DDR4 RAM, which adds another $600 to the price tag. The payoff time in electricity costs for this investment, even with Massachusetts relatively high electricity costs (at least for the U.S.) is something like 12-15 years, so I have not been able to make that justification yet.

Another concern is that I'd be stuck on 128GB of RAM, as that is where the Xeon D maxes out. It would certainly be enough for my needs today, but it doesn't offer much in the way of room to grow.

For reference, here are the full specs of my current server:

Norco RPC-4216 case
Dual Xeon L5640 (2x 6core 12 logical) 2.2Ghz base, max turbo 2.8Ghz
192GB registered ECC DDR3 1333Mhz RAM
Supermicro X8DTE motherboard 2 with on board Intel NIC's
2x LSI 9211-8i SAS conntrollers in JBOD (IT) mode
Intel Pro/1000 Quad port NIC
Boot Drives: Two 512GB Samsung 850 EVO SSD's in ZFS mirror
Swap Drive: 128GB Samsung 850 Pro SSD
MythTV Live Recording/Buffer Drive: 128GB Samsung 850 Pro SSD
MythTV Scheduled Recording Drive: 1TB Samsung 850 EVO SSD
Main NAS drives 12x WD Red 4TB ZFS configured as two 6 drive RAIDz2 pools
ZFS ZIL/SLOG drives: Two mirrored 100GB Intel S3700 SSD's
ZFS L2ARC (read Cache): Two striped 512GB Samsung 850 Pro SSD's

So, one of the guests on my server is also my TV/DVR system. It uses one SSD as a Live TV buffer, and a large 1TB one gets scheduled recordings written to it. Every night a cron job dumps the oldest recordings off of it onto my main 48TB NAS storage.

The boot drives also host all my VM/Container guest images.

Boot drives, swap drive and the two MythTV drives are hooked up to on board Sata (limited to 3Gb/s)

The 12 4TB WD Red's, ZIL/SLOG SSD's and L2ARC SSD's are all connected to my two LSI SAS controllers.

The two on board NIC's are bridged and directly connected to a private network that is accessed only by my MythTV guest and my two networked TV tuners.

The four ports on my Intel Server NIC are all Link Aggregated using 802.3ad LACP and connected to my main house switch (My ProCurve 1810G-24)

The only reason I have a private network for my tuners is because SiliconDusts HD Homerun Prime have a nasty habit of announcing themselves to everything on the network (Windows machines, smartphones, Tablets, Smart TV's etc.) saying essentially "hey look at me, I'm here, you xcan use me to watch TV", but MythTV expects to have exclusive use of the tuners. If anyone snatches a tuner away, it can result in interrupted recordings, etc. In their wisdom, Silicondust offered no way to disable this announcing they do, so I was forced to put them on private subnet to isolate them from the rest of the network.

Anyway, I digress.

mattyd

Digression or not, I appreciate the extra info and feedback! Definitely relevant to my interests.

I probably won't have a server anywhere close to yours, I can justify a lot of things (see: previous thread entries where I went somehow from APU2C4 to C2778), but that is a truly massive amount of storage. 12 spinning discs and 9 SSDs, wow! I'll have only 5 spinning discs and a boot SSD. I'm just going to go from 5x2TB to 5x4TB and pick up transcoding ability and general purpose server functionality in the process. I already have link aggregation, so I'll be keeping that, too.

Thanks for taking the time to help educate me. I'll definitely be referring back to my notes from this thread when it is time to build my new NAS.