PfSense box for 100/40 w/ traffic shaping + some room

mathiasringhof

Hi,

I just got my feet wet with pfSense and a VM / router-on-a-stick configuration - love it! I'm exploring options to move the VM to physical box for various reasons.

I saw the official hardware requirements and read quite a few threads, most of which were concentrating on OpenVPN. While I will have people connecting to me my main requirement is traffic shaping. I initially went with a Ubiquiti USG which was NOT able to do so while saturating the connection and I want to avoid returning another device.

My connection is 100 down / 40 up and I want to ensure that my video conferences / work VPN get bandwidth even though torrents run, my wife watches Netflix and someone is trying to watch from my Plex server (for example work gets prio, wifey next, Plex and external dude get the leftovers).
I understand that Intel NICs are preferred and read quite a few times CPU is not THAT important, but again, I'm still digging into the posts here and on places like ServeTheHome and reddit.

I already put a bunch of configurations together (all have 4gb RAM and up to 128gb HDD)
https://docs.google.com/spreadsheets/d/1HF0IIQZs2sYIeKY-nER_JhpiZaqKibplbTedNr-SeFI/edit#gid=0

Two I like most so far:
Intel NUC i3-7100U - 3600+ CPU mark which I hope is plenty room for growth, but single NIC
Shuttle DS68U - 1700+ CPU mark, but two Intel NICs.. I'm just not sure if the CPU will fit the bill tbh

I saw that folks seem to like the PC-Engine APU2C4 (more but slower cores compared to the Shuttle) so I'm trying to take those experiences as a reference point. Amazon has a few China PCs, but I'm honestly super scared of counterfeit Intel chips.. just had a dual nic card that died after a few hours. Also, not that much cheaper than the shuttle, just more Ethernet ports that I don't (think I) need.

I haven't found a good answer yet, though, and might just stick to the more cumbersome but scalable VM environment until I better understand the load the VM produces. But maybe I'm missing a few key pieces here or some of you can share their experience / link to threads I might have missed.

EricE

OpenVPN is single threaded, so base clock speed is pretty important.

As others have pointed out in other threads, not all AES-NI support is the same, so newer CPUs can be vastly better than older ones with AES-NI support.

That i3 has a decent clock and it's pretty new so it should be a good choice for OpenVPN performance. Your speeds seem somewhat modest too which also helps.

VAMike

I took the vpn comment to mean that they're was a work laptop with a vpn or such. If there's no vpn requirement on the router, an apu2 is more than enough power for a hundred megabit link with plenty of room to grow. I don't know of a better priced option. (Unless you're in the EU, then the apu2 may not be competitive.) Otherwise, this is a low enough requirement that pretty much any hardware should work (except for the usual caveats about freebsd's nic driver support).

belt9

Yeah you can achieve your goals with tons of hardware combos.

I like the J3355B in an M300 enclosure using a PCIe riser for a dual (or quad if you want it) port NIC (I recommend an eBay server pull i340t2 or 4). Combine that with 2x2GB SO-DIMM DDR3(L), a little 16GB SSD and a picoPSU 80 (non-WI). In the US that will all cost you something like $200, that's for a totally silent (fanless) and compact system with no moving parts. Less if you have some of your own parts you can reuse.
But as stated you can certainly use even less hardware than that.

As far as traffic shaping goes, I would recommend you install pfSense 2.4 (RELEASE goes live in like a week I think?), and use Limiters with fq_codel.
You have to implement fq_codel with CLI & the shellcmd package, but it's really easy.
For what you described, fq_codel will impress you and it takes about 2 minutes to setup.

It will keep your latency very low under load, and won't let torrent type traffic or any other bandwidth hog tank the network.
It would probably solve your problems even without weighting, but you can very easily set three categories of weights as you decsribed.
Something like:
Work: 60
Wifey: 30
Plex: 10

Work get's 60% of the pipe, Wifey 30%, Plex 10%. The beauty is that if the pipe isn't full it won't limit anyone. So even the low guy on the totem pole (plex) gets to use all of the bandwidth if nothing else is using it, but as soon as Work or Wifey start using the network they will limit Plex as necessary.
The end result is everyone's happy, all the time.

Read more on that in the following thread if you're interested.
https://forum.pfsense.org/index.php?topic=126637.0

mathiasringhof

Thanks everyone for taking the time to respond! It sounds like traffic shaping is NOT the CPU hog I feared it was. Whatever the hell they put into that USG most be really subpar hardware then…

OpenVPN is single threaded, so base clock speed is pretty important.

Oh, didn't know that, that puts some things into perspective. That makes the higher clocked dual core Celerons sound more appealing..

I took the vpn comment to mean that they're was a work laptop with a vpn or such

Sorry, I apparently rambled a bit on in the middle of the night.

What I meant regarding VPN:

there is an OpenVPN Server in my home network with 0-2 concurrent connections, mostly people using my Plex. Right now it's hosted on my home server, but I see some benefits of moving it to the router
I personally use Cisco AnyConnect on my work laptop to connect to.. work. Not that the router cares, except my traffic shaping requirement.

I don't see myself using the router as an OpenVPN client to tunnel traffic through PIA or something similar.

Unless you're in the EU, then the apu2 may not be competitive.

Yep, located in Germany, so it's about 210€. The shuttle is about 290€. In the states I'd probably just get a used Dell R210 ii for 200 bucks and be done with it. Alas…

I like the J3355B in an M300 enclosure using a PCIe riser for a dual (or quad if you want it) port NIC (I recommend an eBay server pull i340t2 or 4). Combine that with 2x2GB SO-DIMM DDR3(L), a little 16GB SSD and a picoPSU 80 (non-WI). In the US that will all cost you something like $200, that's for a totally silent (fanless) and compact system with no moving parts. Less if you have some of your own parts you can reuse.

Thanks, I'll look into building a small whitebox this weekend. I never built something small so I struggled a bit with the external power supply and the small cases, but that's a great starting point.

As far as traffic shaping goes, I would recommend you install pfSense 2.4 (RELEASE goes live in like a week I think?), and use Limiters with fq_codel.

That sounds wonderful, thanks for the info + link I'll dig into that.

/edit
I just noticed that the Celeron 3855U and the J3355B have pretty similar single threaded performance, but the 3855U is about 50% faster in multicore. Most likely due to TurboBoost, also the 3855 is about 50% more TDP.
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Celeron+J3355+%40+2.00GHz&id=2960&full
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Celeron+3855U+%40+1.60GHz

Sounds like both might just work for what I want to do…

johnpoz

What Hardware are you running on VM?

I have pfsense running as vm on OLD HP microserver N40L on esxi 6.5, It could handle my 75/10 connection without any problem – would get 80+ and 12 up.. I was doing some fq_codel to remove buffer bloat but not any other sort of shaping.. When I put in my new 500/50 line the other day no the VM could not keep up but it was doing 120ish down and the up it handled..

I had to go with USG to handle the speed of the line, which it does do but I am not using any shaping on it - from what I was reading yeah it falls down pretty hard then..

I have only had the usg online for a few days - and while it can route the packets at speed.. Other than that its very limited.. They are getting there I think.. But I want my pfsense back!! I hope to have some pfsense hardware in Nov.. But I am going with actual pfsense/netgate hardware.. The new sg-3100 is shipping in a few days, I would think that should handle your needs without even breaking a sweat, etc.

VAMike

@mathiasringhof:

I just noticed that the Celeron 3855U and the J3355B have pretty similar single threaded performance, but the 3855U is about 50% faster in multicore. Most likely due to

The low end traditional celeron parts (skylake and later) are all pretty good for this sort of application. (Insert note here about how annoying it is that intel is now calling everything a celeron. The 3855U and the J3355 are completely different architectures. The U series will be much faster for some tasks, but firewalling isn't one of them.) The main reason the J3355 comes up (note there's no "b", that's a motherboard product name) so much is that it's got a bit lower thermal requirement, it's a bit cheaper, there are a few decent low-cost boards, and it's enough power to cover a pretty big range of requirements. The 3855U isn't a bad choice, but I'm not aware of as many low-cost reasonably available/tested boards using it. Stepping up a bit into even higher performance G series celerons is useful for people trying to do full VPN, but unnecessary for the performance you're talking about. (That said, it may be sensible depending on availability in your local market if the costs end up pretty similar. Around here the G series final price would probably be cheaper than the U series, because they're much higher volume, but would still be twice as much as the J series.)

mathiasringhof

What Hardware are you running on VM?

My VM host is Xeon E3 1230v6 so powerful enough. if I'm going the VM route I just need to add a proper network card (otherwise all other VMs / containers share the one remaining and my pfSense uses a single NIC for everything). If my newest purchase from eBay isn't a counterfeit piece of junk like the card I got from Amazon it's were I'll start.

I had to go with USG to handle the speed of the line, which it does do but I am not using any shaping on it - from what I was reading yeah it falls down pretty hard then..

Just activating Smart Queues reduced bandwidth like crazy…

I have only had the usg online for a few days - and while it can route the packets at speed.. Other than that its very limited.. They are getting there I think.. But I want my pfsense back!! I hope to have some pfsense hardware in Nov.. But I am going with actual pfsense/netgate hardware..

The UI is very pretty but I was surprised how little stuff was there. pfSense on the other hand got me surprised just how much I can do with it. :)

The new sg-3100 is shipping in a few days, I would think that should handle your needs without even breaking a sweat, etc.

The prices in Germany are complete bonkers. 665€ for the Atom based SG-2440. 420€ for the atom-based SG-3100.

Insert note here about how annoying it is that intel is now calling everything a celeron. The 3855U and the J3355 are completely different architectures. The U series will be much faster for some tasks, but firewalling isn't one of them

Yes, it has been very interesting to learn about the different chip series Intel puts out there. Atom C, D, E, Apollo Lake, Skylake, jeez. What makes you say the SkylakeU are not faster at the pfSense stuff than the Apollo Lake Celerons? I'm tying my assessments to Passmark scores right now, but that might not be optimal.

The 3855U isn't a bad choice, but I'm not aware of as many low-cost reasonably available/tested boards using it.

I've picked the Shuttle DS68U, which seems to be well received from what I could find: http://www.shuttle.eu/products/slim/ds68u/overview/

And while the mainboards are pretty chip I couldn't find any that use Intel NICs, so I have to get a case with space for a network card. Adding all up I ended up at 277€, which is very close to the Shuttle with 293€:
https://docs.google.com/spreadsheets/d/1HF0IIQZs2sYIeKY-nER_JhpiZaqKibplbTedNr-SeFI/edit#gid=0

I might be doing it wrong and I'll continue to look into it, but as of right now I see the main choices between:

Just VM the thing, safe the money and bite the bullet when you have to do maintenance
You'll never notice not having two NICs, buy the damn NUC i3
You'll never notice only having a Celeron CPU, buy the damn Shuttle

johnpoz

The prices in Germany are complete bonkers. 665€ for the Atom based SG-2440. 420€ for the atom-based SG-3100.

You sure those prices are not bundled with support? They have started offering enterprise level support so yeah the price jumps up if you pick support vs community support which is 0$ ;)

pricing.png_thumb

belt9

$350 is pretty steep for an ARM CPU.

johnpoz

There is a bit more too it than just the CPU ;) Don't forget it comes with year of gold as well.

Why don't you add up the price of building that box with the specs.. Then take into account the development cost of pfsense that buying hardware direct from them supports, etc. etc. Now compare that price to say what you get with buying say comparable product vs some box made in china that your going to put pfsense on ;)

I too would love them to be cheaper ;) But not like they are all that crazy.. And I for sure understand budget committees (spouses) for your home purchases.. Which forced me to get the "cheap" usg until such time as budget can allow for pfsense hardware.. And I still got an eye roll when it showed - WTF did you order now ;)

Maybe it is just me, but I would much rather wait a month or two to get pfsense hardware vs some china box.. Which isn't all that much cheaper when you add it all up.. What you going to save 100-150$ My buddy got one of those cheap boxes off amazon.. Ran into the bios issue, they sure an the hell not fixing it, etc.

belt9

Haha yeah, the budget committee sure wouldn't stand for that (and I agree with her).

It is a good value with gold.

For me I use a SFF used i5-2400 workstation with 8gb ram. It's power hungry but was very cheap and it's powerful.

I also like the $2-250 j3355b builds.

Basically it would be nice to have an option to buy official without gold for those that don't want it.
But that might not be realistic for netgate with their profit margins.

mathiasringhof

You sure those prices are not bundled with support? They have started offering enterprise level support so yeah the price jumps up if you pick support vs community support which is 0$ ;)

Yep, I can add support from that local partner on top though. It's similar to say Apple, were a $699 device costs 799€.

There is a bit more too it than just the CPU ;) Don't forget it comes with year of gold as well.

Yep and I would love to have that / support the company. I'm not complaining about the price they offer and as a company would love to get that premium support. But since it's just me playing around with my home network those appliances are not in price range, and that's OK!

Maybe it is just me, but I would much rather wait a month or two to get pfsense hardware vs some china box.. Which isn't all that much cheaper when you add it all up.. What you going to save 100-150$ My buddy got one of those cheap boxes off amazon.. Ran into the bios issue, they sure an the hell not fixing it, etc.

The main thing that scares me about the China boxes is the knock off thing. They all claim Intel chipsets & NICs, but how can you be sure? Especially the NICs are being copied like crazy apparently.

Anyhow, thanks everyone for the support and responses, I really appreciate it. If there are more suggestions or links to threads with mini ITX builds, keep them coming I haven't written that route off!

VAMike

@mathiasringhof:

Insert note here about how annoying it is that intel is now calling everything a celeron. The 3855U and the J3355 are completely different architectures. The U series will be much faster for some tasks, but firewalling isn't one of them

Yes, it has been very interesting to learn about the different chip series Intel puts out there. Atom C, D, E, Apollo Lake, Skylake, jeez. What makes you say the SkylakeU are not faster at the pfSense stuff than the Apollo Lake Celerons? I'm tying my assessments to Passmark scores right now, but that might not be optimal.

passmark is useless. To be clear, a skylake outperform an apollo lake at the same clock speed or at a slight clock speed disadvantage (which is the case between the 3855U and the J3355). What I meant is that for some tasks the skylake would stomp all over the apollo lake, but firewalling isn't one of those tasks–the performance will be a lot closer. So if the U series ends up being price competitive just get it.

Haha yeah, the budget committee sure wouldn't stand for that (and I agree with her).

if all customers and/or users would be submitting 5 € - 10 € a year that would be not so hard to finance that
project. And as second, if you spend 20 years 5 € it is not to much but with Gold support you will get something back!

For me I use a SFF used i5-2400 workstation with 8gb ram. It's power hungry but was very cheap and it's powerful.
I also like the $2-250 j3355b builds.

I love more the APU2C4 bundles from the varia store here in Germany, they offers mostly good parts and are also
not so high in price.

Basically it would be nice to have an option to buy official without gold for those that don't want it.
But that might not be realistic for netgate with their profit margins.

I don´t know what you think what a pfSense version change will be producing in costs!? From 2.1.5 to 2.2x it
was something around ~$92.000,00 what I was reading once a time here in that forum from one of the developers.

The prices in Germany are complete bonkers. 665€ for the Atom based SG-2440.

You will get three miniPCIe slots + 1 SIM slot on top of this!

420€ for the atom-based SG-3100.

Please compare this unit to the SolidRun ClearFog pro unit with case and a qualified SoC or SoM!
It comes with more ports, crypto offloading engine inside of the CPU and it is ARM based as many many users
were asking for something like this in the past or formers days. My personally mind on this, is that many peoples
at first are calling and asking for somethings or more, and then if this will be available they all run away or have no
money to pay a small fee such 5 € for home usage and perhaps 10 € for professional usage inside of company networks.

belt9

the general idea is that most of the official solutions are priced well out of the budget of many home users and are also not competitive with what a home user could put together on their own or buy from a third party.

This is all totally understandable and fine - netgate is clearly not marketing home users as their primary buyer for most of their products.

mathiasringhof

Apologies for not stating clearly that I'm comparing US vs German prices, not complaining about the pricing for Netgate hardware in general. I do see the value they bring to the table, but I question the addition of 100+€ from that partner. But as I said, this is not so uncommon, not sure why though.

I'm also interested in the Gold subscription as I've heard very good things about the book. So I'd be paying roughtly $250. I'd definitely consider buying that, especially since then I can actually get confirmation from Netgate themselves before the purchase that it would (probably) fit my needs.

I love more the APU2C4 bundles from the varia store here in Germany, they offers mostly good parts and are also
not so high in price.

Hey thanks, I saw their offer on Amazon but good to hear they use good components. One line of thinking was to start with that and if for whatever reason I don't have enough power on this one, use it as a slave in a HA setup. Haven't looked into that too much, but it would enable me to use a VM with plenty of power and a backup unit in case the server gets rebooted / dies / explodes / flies away.

belt9

Gold is a great purchase if you're trying to learn pfSense, whether you purchase an official product or not.

Hey thanks, I saw their offer on Amazon but good to hear they use good components.

For the lower Internet connection speeds here in Germany it will be one of the best and often sold hardware
in combination with pfSense as I am right informed. It is running here for 100 MBit/s down and 50 MBit/s up
for ~ 70 employees together with IPSec VPN, Squid & SquidGuard, snort and pfblockerNG, all is fine.

One line of thinking was to start with that and if for whatever reason I don't have enough power on this one, use it as a slave in a HA setup.

You will be able to run it in one big 1U" case as well available from the Varia-Store, here is a link to that dual 1U" case;
APU2C4 - 1 U" - rack mount case

Haven't looked into that too much, but it would enable me to use a VM with plenty of power and a backup unit in case the server gets rebooted / dies / explodes / flies away.

That could be also very interesting, but I love more the real hardware HA setup, if one server is "gone" mostly also
both VMs are also "gone" please don´t forget this too!

For more power you could also have a look on the new Supermicro Atom C3000 line
But the network drivers will be not really matching to all NICs that are SoC integrated!!!

Stronger and faster then the Intel Atom C2000 series, but slower and less powerful then the Intel Xeon D-15xx series.
it is not only interesting what kind of Internet connection speed you are running, also the amount of installed packets,
running applications, offered services or used protocols will be also important likes the amount of users and their
produced traffic such mailing, surfing, gaming or audio/video streaming!