Recommandations on hardware for gigabit WAN

LeetDonkey

Hello

I'm looking to build a new pfsense box.
The box should be able to handle a 1000/1000 internet connection.
The motherboard should be mini-itx size
Also, on the LAN side I have a game server, so stable low latency is vital

I would also like the box to push a decent amount of bandwidth using OpenVPN or something similar, actually the closer to gigabit throughput the better.
I'm not using any of the packages as it is right now, but I would like a large overhead on performance if the need rises in the future.

Going through the forums I've been looking at Supermicro A1SRi-2758F
Will this be good enough for my use? Should I consider anything else? Any recommendations on amount of RAM?

Thanks

Guest

I'm looking to build a new pfsense box.
The box should be able to handle a 1000/1000 internet connection.

If you will be able to get a static public IP address from your ISP that you are able to put it in the WAN setup
the better for you and also faster it will be, related to the circumstance that over PPPoE "only"one CPU core
will be in usage!!! But with a static IP address you could be using all CPU cores in the WAN area and so it
will be also faster then. The SG-4860 or SG8860 will be also fine for your needs as I see it right and a real
pre-tuned pfSense image is then also pre-installed and available for this boxes!

The motherboard should be mini-itx size
Also, on the LAN side I have a game server, so stable low latency is vital

Then set up a DMZ and LAN zone and buy two switches that are really fast and managed.
Layer2 for the DMZ and Layer2 or Layer3 for the LAN area as you will need it. I would preffer
to solve this with the following switches art this time.

Cisco SG200-series Layer2
Cisco SG300-series Layer3
D-Link DGS1510-series Layer3
Zyxel GS1910-series Layer2

I would also like the box to push a decent amount of bandwidth using OpenVPN or something similar, actually the closer to gigabit throughput the better.

The SG-4860 is able to realize 500 MBit/s over IPSec (AES-GCM) and the SG-8860 much more over this
throughput! But the SG-8860 is not silent as you wish this to be for a more private or home

I'm not using any of the packages as it is right now, but I would like a large overhead on performance if the need rises in the future.

There are some interesting units for you, that should perhaps also named here in this thread;

Supermicro C2758 (as you found it) + RAM + case + SSD + PSU (the self made box ever)
SG-8860 from pfSense shop + mSATA (enough for all + Support and future proof)
mini ITX Board + E3-12xx v3/v4 CPU + RAM + case + SSD + PSU (really powerful)
Supermicro X10SDV-6C+-TLN4F + RAM + case + M.2 SSD + PSU (pfSense bomb)

Going through the forums I've been looking at Supermicro A1SRi-2758F

It is the pfSense basis within at this time, but in some month (Q1/2016) we will see more Supermicro D-15x8
boards coming out and being more and more sold cheaper and cheaper, so this could be changing in 2016.

Also a Gigabyte GA-6ILSL with an Intel Xeon E3-12xx v3 is very powerful and fine for any setup you will be
needing and you can pay.

Will this be good enough for my use?

Yes, for sure it will be enough for that action you where told about here.

Should I consider anything else? Any recommendations on amount of RAM?

2 GB - normal pfSense firewall only
2 GB - 4 GB pfSense firewall, VPN and Snort
4 GB - 8 GB pfSense firewall, VPN, Snort and Squid & SquidGuard
8 GB - 16 GB pfSense firewall, VPN, Snort, Squid & SquidGuard, HAVP & ClamAV,
plus NIC tuning (mbuf size), Squid tuning (Squid RAM size) and/or high
throughput to servers or DMZ clients and LAN clients.

Possible spare parts for the Supermicro A1SRi-2758F are the following;

Supermicro SCS101i mini ITX case
Supermicro 80 Watt PSU
Supermicro SC721 TQ-250B - Mini Tower - Mini-ITX 250 Watt
if the PCIe slot should be used to add another PCIe card and CD/DVD player and more HDDs
M350 mini ITX case & 150 Watt Pico PSU & 197 Watt external PSU matching your country regulations
Kingston ValueRAM DDR3L 1600 MHz / PC3-12800 CL11 ECC RAM
Supermicro 9655V or 9655H TPM module (if needed)
Samsung840 Pro SSD

….push a decent amount of bandwidth using OpenVPN or something similar,

I would be like using the IPSec VPN together with the AES-GCM algorithm over OpenVPN now!
AES-NI is used by AES-GCM and this is used by IPSec, so you will be able to get 4x or x5 more throughput
then using other things for your VPN. Only my opinion over that. If Intel QuickAssist or AES-GCM will be ready
for usage in pfSense you might be also able to push the OpenVPN part too.

LeetDonkey

Hello

Thank you for your detailed answer, I just want to clarify a few things:

This is my current setup:

This is not working well, the game server has some pretty high cpu and RAM demands and to begin with I would like to split up pfSense and the gameserver and stop using esxi.

This is what I had imagined my network would look like after splitting it up:

However the way you explain it is this I think:

Is this correct?
I am unsure how I would benefit from your suggestion as I have never used managed switches or vlans, but if there's an advantage to it in my setup I will try to read up on it

Guest

Picture 3 is in my opinion the best choice to realize it.

rajl

Realistically speaking, picture 2 and picture 3 will both work equally well for you. You can get almost the exact same results either way with some careful planning.

Picture 2 will require you to use vLANs and vLAN tagging to segregate your game server onto a DMZ. Picture 3 can be done with or without vLANs, and no vLAN tagging is required.
Picture 3 will allow you to beter prioritize traffic to the game server. As a simple example, if the LAN and the DMZ are attached to two separate NICs on your router, you could prioritize all traffic on the DMZ port over traffic on the LAN port. In contrast, picture 2 has all game and LAN traffic moving from the single switch to the router along a single fiber. You can still prioritize game traffic over LAN traffic, but it won't be as effective because the switch could send the game and LAN traffic to your router in any sequence or order it wants. A managed switch with vLANS can alleviate this by allowing you to implement QoS on the switch, prioritizing which traffic gets sent to the router first. With FIG. 3, you don't have to worry about QoS on the switch, so a cheaper unmanaged switch would work just fine.

LeetDonkey

@rajl:

Realistically speaking, picture 2 and picture 3 will both work equally well for you. You can get almost the exact same results either way with some careful planning.

Picture 2 will require you to use vLANs and vLAN tagging to segregate your game server onto a DMZ. Picture 3 can be done with or without vLANs, and no vLAN tagging is required.

Picture 3 will allow you to beter prioritize traffic to the game server. As a simple example, if the LAN and the DMZ are attached to two separate NICs on your router, you could prioritize all traffic on the DMZ port over traffic on the LAN port. In contrast, picture 2 has all game and LAN traffic moving from the single switch to the router along a single fiber. You can still prioritize game traffic over LAN traffic, but it won't be as effective because the switch could send the game and LAN traffic to your router in any sequence or order it wants. A managed switch with vLANS can alleviate this by allowing you to implement QoS on the switch, prioritizing which traffic gets sent to the router first. With FIG. 3, you don't have to worry about QoS on the switch, so a cheaper unmanaged switch would work just fine.

Thank you for clearing it up for me, while reading it, I suddenly realized that security-wise my current approach is not the best idea. (Game server on same LAN as the rest of my equipment and just basic port forwards.)
I think I'll do your version of fig. 3 - two unmanged switches since I already have a couple of unmanaged gigabit switches lying around it would not give me any additional costs.

Edit:
On second thought I might go with a managed switch, I just realized that I could actually prioritize all the other PC's as well - The possibilities!

LeetDonkey

I've been putting some extra thought into the actual pfSense hardware:

SuperMicro A1SRI-2758F-O(4 x i354) + 2x8GB ECC RAM ~ 495€
ASRock H110M-ITX(1 x I219v) + 2x8GB Non-ECC RAM +I5-6600 + Intel 350-T2 ~ 529€
Asus P10S-I(2 x I210at) + 2x8GB ECC RAM + E3-1220 V5 ~ 538€
Gigabyte GA-6LISL(2 x I210at) + 2x8GB ECC RAM + E3-1220 v3 ~ 567€

A bit over my price point:
–---------------------------------------------------------------------------
Supermicro X10SDV-6C+-TLN4F + 2x8GB ECC RAM ~ 848€

All of them will be using the following:

Enclosure: Streacom F7C Alpha without ODD (This will allow me to use the PCI-E slot on the motherboard if I should need it)
PSU: PicoPSU 12V
SSD: Samsung 850 EVO

I know that the Asrock+I5 setup isn't exactly servergrade but the cost is on par with the other options with the exception of a better network card than the two xeon options.

I must say I'm leaning towards either the Asus P10S setup or the Supermicro A1SRI-2758F setup.
If anyone wants to share their thoughts on any of the listed setups I would appreciate it, especially the I354 vs I350 vs I210 options.

I realize with only two network interfaces I would need to go for a managed switch or an additional PCI-E network card.

edmund

Definitely #3 - with separate ports you can segregate your security and use the traffic shaper to give the gaming server priority over the users - the users will hardly notice this but the game server will probably appear to be a lot more responsive.

Guest

SuperMicro A1SRI-2758F-O(4 x i354) + 2x8GB ECC RAM ~ 495€

Would be a cool set up and reaching for a longer time as you might be imagine.

Realistically speaking, picture 2 and picture 3 will both work equally well for you. You can get almost the exact same results either way with some careful planning.

With the network schematic shown in picture 3, mostly the schematic shown in picture 2 will be prevent!
Or in shorter words picture three is preventing you from doing the "failure" of picture 2.

Picture 2 will require you to use vLANs and vLAN tagging to segregate your game server onto a DMZ. Picture 3 can be done with or without vLANs, and no vLAN tagging is required.

With inter-VLAN hopping you would be not really cut each from another one. And the
traffic from the LAN would also not spread away from the DMZ homed game server.

For the LAN you should go with a managed switch and for the DMZ you only need a dump Layer2 Switch.

LeetDonkey

Thank you all for your answers

So this is my final shopping list:

Supermicro A1SRI-2758F-O ~ 388€
2 x Kingstom ValueRAM Server Premier 8GB (KVR16LSE11/8KF) ~ 116€
120GB Intel 320 (Got one I don't use) ~ 0€
PicoPUSU-80 (Got an AC adapter I don't use) ~ 28€
M300 Enclosure ~ 70€ (Changed it from the streacom F7C since I'll get a smaller footprint, and the PCI above the board should be a non-issue with this particular motherboard)
Shipping for PicoPSU and M300 ~ 13.50€

Total cost ~ 615,5€

It will be a couple of days before I order, so if anyone wants to suggest some last minute changes, please let me know.

I'm still contemplating the managed switch, but I'm considering the Cisco sg200-08 based on a search on the pfSense forums

Guest

I'm still contemplating the managed switch, but I'm considering the Cisco sg200-08 based on a search on the pfSense forums

The SG200-08 is for ~115 € here and the Cisco SG300-08 is for ~139 € here so it could be nice to get a Layer3
switch for some coin more! Think about, what you need or want also think on upcoming things in your network.

The Layer3 switch is able to route the entire traffic and also between the VLANs if they are some in usage
and the pfSense box will be freed form this load.

Alternatively switches:

ZYXEL GS1920-24 24-port GbE L2 Smart Switch ~99 €
ZYXEL GS1920-24 28-port GbE L2 Smart Switch ~135 € (24 RJ45 + 4 SFP ports)
Netgear GS108Tv2 ~65 €
Netgear JGS524Ev2 ~155 €
Netgear JGS516PE ~199 €
TP-Link TL-SG3424 ~199 €
Cisco SG300-10 ~ 190 €

All in all this are nice switches I am self or friends have in usage without any problems.

Keljian

Personally, I would stick to one box/esxi.

How much load (in terms of MHz as esxi talks in MHz) is the game server putting on the box and how much memory?

Remember also that kvm/proxmox/ovirt etc have less overhead than esxi and could run Centos in openVZ

I would opt for a secondhand haswell based i7/asrock b85 box with 32 gig of ram, a 10 gbit Nic (Chelsio t420-so-cr) hooked up to a mikrotik crs226. Have separate vswitches and do it all virtually.

LeetDonkey

@Keljian:

Personally, I would stick to one box/esxi.

How much load (in terms of MHz as esxi talks in MHz) is the game server putting on the box and how much memory?

Remember also that kvm/proxmox/ovirt etc have less overhead than esxi and could run Centos in openVZ

I would opt for a secondhand haswell based i7/asrock b85 box with 32 gig of ram, a 10 gbit Nic (Chelsio t420-so-cr) hooked up to a mikrotik crs226. Have separate vswitches and do it all virtually.

If memory serves it seems to hover at around 12000MHz on a decent night with players populating most of the servers.
Symtpoms when the servers are loaded are:

Server framerate fluctuates
Latency rises for all players
Players get dropped for no apparent reason

I'm not sure the build you are suggesting is an upgrade to my current ESXI system? or a standalone pfSense box?
Anyways unless I get one of them extended ITX boards from Asrock it will be almost impossible to get 32GB DDR3

My current ESXI build is a I7-4770 with 16GB of RAM, this would be repurposed to running CentOS for my game server, no hypervisor.

Keljian

My suggestion would be to beef up your esxi machine- is itx a requirement? There are some 8 and 10 core xeons showing up on eBay secondhand cheaply…

LeetDonkey

@Keljian:

My suggestion would be to beef up your esxi machine- is itx a requirement? There are some 8 and 10 core xeons showing up on eBay secondhand cheaply…

The footprint is somewhat important, however it is quite possible even with mini-itx:
http://www.asrockrack.com/general/productdetail.asp?Model=EPC612D4I#Specifications

It was my first intention to upgrade the ESXI, however the upgrade path is quite expensive.

Asrock EPC612D4I - 260€
CPU: E5-2670V3 from Hong Kong(Ebay - 388€
RAM: 2x16GB DDR4 SO-DIMM ECC ~ 351€
Enclosure: Cooltek Coolcube ~45€
PSU: Silverstone SX500-LG ~ 99€
SSD: Reused ~ 0€

Total: 1143€
I might be able to get 300€ for my old ESXI server so the total cost would be
843€

if I decided to go micro-atx the cost would be:
ASRock EPC612D4U ~ 280€
CPU: E5-2670V3 from Hong Kong(Ebay - 388€
RAM: 2x16GB DDR4 DIMM ECC ~ 233€
Enclosure: Cooltek Coolcube Maxi ~58€
PSU: Sea Sonic G Series 360 ~ 73€
SSD: Reused ~ 0€
Total: 1032€

Total cost after selling current ESXI: 732€

The cost would still be higher than simply splitting it up and getting a dedicated pfSense box.
On top of that everything is passively cooled as it is right now, I could make a Xeon almost as quiet, but it would cost me at least 100€ more to do so.
The idle power consumption would most likely be on par with my 4770+Supermicro pfSense box.

So, if we take the ESXI box one step further and include my current NAS option:

HFX PowerNAS
I7-4770T
Running Windows 10
Sabnzbd
Mediabrowser server
Flexraid

ASRock EPC612D4U ~ 280€
CPU: E5-2670V3 from Hong Kong(Ebay - 388€
RAM: 2x16GB DDR4 DIMM ECC ~ 233€
Enclosure: Fractal Design Node 804 ~ 112€
PSU: Sea Sonic Platinum Series 400 Fanless ~ 132€
SSD: Reused ~ 0€
CPU Cooler: Noctua NH-D14 ~74 €
I added some options to make it more quiet

Total: 1219€
Suppose I can get 300€ for my current ESXI server and 300€ for my HFX Powernas NAS:
Total after they're sold: 619€

An interesting idea, I'll need to think about it.

Keljian

My home server is described in a bit of detail here: https://forums.servethehome.com/index.php?threads/constellation-my-new-home-home-office-server-build.6801/

It has plenty of cycles free, but I don't run game servers.

I highly recommend the 10gbit nic (chelsio or mellanox - Both are cheap on ebay - $70USD for a dual port chelsio, or $20 for a single port mellanox) into a mikrotik CRS226 or the smaller version CRS210, to feed your home network. That said, you can't have that and a HBA in the same box if you go ITX.

One particularly nice thing about the Chelsio is it shows up as multiple devices in ESXi (al-la SR-IOV) so you can do passthrough to more than one VM easily if you want to.

Other options for you to consider:
Xeon-D (8 core/16 thread low power, 2.6ghz turbo) - Broadwell based low power. High threads, 2x10gbit integrated 10gb-T nics

Xeon E5-2670 - AKA sandy bridge generation 8 core - lots of these going on ebay cheaply these days. http://www.ebay.com.au/itm/151964337392 for instance - 2.6 GHz, 3.3 GHz Turbo, 8 Cores, 16 Threads, 20 MB cache.

LeetDonkey

@Keljian:

My home server is described in a bit of detail here: https://forums.servethehome.com/index.php?threads/constellation-my-new-home-home-office-server-build.6801/

It has plenty of cycles free, but I don't run game servers.

I highly recommend the 10gbit nic (chelsio or mellanox - Both are cheap on ebay - $70USD for a dual port chelsio, or $20 for a single port mellanox) into a mikrotik CRS226 or the smaller version CRS210, to feed your home network. That said, you can't have that and a HBA in the same box if you go ITX.

Other options for you to consider:
Xeon-D (8 core/16 thread low power, 2.6ghz turbo) - Broadwell based low power. High threads, 2x10gbit integrated 10gb-T nics

Xeon E5-2670 - AKA sandy bridge generation 8 core - lots of these going on ebay cheaply these days. http://www.ebay.com.au/itm/151964337392 for instance - 2.6 GHz, 3.3 GHz Turbo, 8 Cores, 16 Threads, 20 MB cache.

To be precise, I'm running CS:GO servers, unfortunately they're not multithreaded on Linux meaning each instance likes to hog a full core on my 4770, this is somewhat problematic if I get a low-frequency high-core count CPU.
Secondly, the Sandy Bridge xeons are cheap, but their idle consumption seem to be substantial compared to their Haswell-E counterparts.
The cost of a Sandy Bridge vs Haswell-E will of course be able to pay for the increased consumption of the Sandy Bridge for ~ 4 years, but it sorta bothers me.

Still, it is something to consider.

There's one thing I don't fully understand about your recommendations - The 10gbit NIC - Why is it a good idea when my LAN is 1 gigabit and WAN is also 1 gigabit? Wouldn't it just sit at 10% capacity since the bottleneck is well.. everywhere else?

Also, The Mikrotik switch, what's the advantage of getting this switch instead of say - the Cisco SG300-10?

I've always used unmanaged switches so this is new territory for me

Keljian

Ok so you run a CS:GO server, nas, pfsense(presumably dhcp), and likely other stuff on your esxi box.

The nas and some cs:go traffic is local, as is dhcp and a few other things. So you don't limit your local bandwidth or latency passing data through to the net, it makes sense to feed your switch with 10gbit.

You may only use 4-5 gbit at peak, but 10gbit gear is relatively cheap anyhow and you get a bit of future proofing in the mix

My reason for suggesting the Mikrotik switches is they are effectively "dumb" switches with some management functions, but more specifically they easily and cheaply link 10gbit sfp+ as a trunk to multiple port 1 gbit twisted pair. So as a core switch they are good value. They can also handle vlans

LeetDonkey

@Keljian:

Ok so you run a CS:GO server, nas, pfsense(presumably dhcp), and likely other stuff on your esxi box.

The nas and some cs:go traffic is local, as is dhcp and a few other things. So you don't limit your local bandwidth or latency passing data through to the net, it makes sense to feed your switch with 10gbit.

My reason for suggesting the Mikrotik switches is they are effectively "dumb" switches with some management functions, but more specifically they easily and cheaply link 10gbit sfp+ as a trunk to multiple port 1 gbit twisted pair. So as a core switch they are good value. They can also handle vlans

At the moment they are split up NAS on its own hardware by itself and pfSense + CentOS(For game server) On an ESXI box, however if I was going to beef up the ESXI machine it seemed like a good idea to consolidate all machines onto 1 ESXI host, it would be easier to justify the increased cost and footprint this way.
with ECC memory and servergrade hardware I could also consider Freenas instead of FlexRAID

Good point on the mikrotik switch + 10gbit nic, I'll have to take that into account.
On a sidenote I began to wonder if I could do away with the media converter and attach the fiber from my ISP directly to pfSense if I got a dual port chelsio, but I have to research that before making any assumptions.

I'll have consider my options, thank you for providing some new ideas on how to set up my network

Keljian

To further clarify:

The way I see it you have two streams, external (as in up and down from the internet) and local

External traffic you will limit to Game serving/downloading/uploading etc.

Local traffic could be:

Traffic To/From your NAS
DHCP
Traffic between clients on your network
Upstream (external/internet) traffic to the internet from your clients
Downstream traffic (external/internet) traffic to your clients
Game traffic to the server
Update traffic if you share updates across the clients on your network
Wireless traffic including negotiations/wireless overhead
Bonjour server/client traffic
Mythtv traffic, if you run something like a Silicondust HDhomerun
Remote Desktop traffic to your VMs
IP Camera feeds

While you can have all of this on 1gbit, if you have multiple VMs accessing the local network clients and visa versa there could be latency induced due to lack of bandwidth, or traffic during peaks. If you did it on 10gbit, basically you end up with a topology that looks like two or more switches (the VSwitches, and the physical switches) and a 10gbit trunk between them.

Therefore even if your local traffic takes up 5gbit or so with the NAS VM streaming 4K to multiple clients @ 1gbit, it's not going to impact upon your game performance as you still have plenty of spare network bandwidth to share around. Obviously if you had 10 clients grabbing files from your file server at 1gbit and your fileserver VM could manage 10gbit worth of data being read at the same time, that would flood the network, but it's a highly unlikely scenario in home use.

As for running fibre to pfsense - I can't say whether this will work - you'll have to speak to your fibre provider and work out the setup you have. Typically speaking fibre providers don't like this as they run VoIP and other services over the fibre also, thus have redundancies built in for emergencies.