Hardware advice, gigabit Internet
-
I have a Protectli FW4A (e3845 CPU with AES, and 8GB of RAM) there, I said it. I understand I was bad, I don't need any more berating. I already got some from Gonzo over on Reddit
However, I still would like to give pfSense a shot, despite the god of gods smacking me for being a troll.
I bought this thing expecting to run a different router OS on it, and also not expecting Cox to install fiber in my area for another lifetime.
Since then I had an outage, and the tech doing the repair mentioned that Cox was going live that same day with DOCSIS 3.1 in my neighborhood, and started offering 1Gbps down X 35 Mbps up network speeds for just a few dollars more per month. So, I upgraded.
I also had just move to pfSense the week prior.
From the modem with a pretty decent laptop with a gigabit Ethernet NIC, I was getting about ~830 Mbps down and nearly 50Mpbs up.
The device has Intel 82583V NICs so I turned on all of the offloading features, and removed all packages from the device to get a baseline. I tested from the same laptop cabled to the router and from speedtest-cli on the router. I also tried different times of the day and to different servers. I was getting only around 500Mbps at best.
I contacted the owner of the Protectli earlier and he said that they have done synthetic lab tests and reached 930-940Mbps. That is with no features enabled like IPS. He did say that pfSense gives the best performance on the device as compared to some of the other router OSes that I have played with before.
At this point, my entire picture has changed. I don't think that the E3845 will do it for me even if 833Mbps is my ceiling.
I used to want to run pfBlackerNG, Snort, and Squid on it. I am backing off of squid as most people feel it doesn't make any sense on a gig connection for home users, with limited users downloading the same things repeatedly. But I would also like to run a few other things like a syslog server and ntopng/softflowd.
I also would really like to eventually get to dialing in https MITM scanning eventually. I would really like to see what that 40-60% of my traffic is going to, and have it scanned for malware, etc. Although, I know that many people see this as a bad thing and possibly pointless. If I get to that point I will do some more reading on the subject, and ask advice later, but I want to size for potentially using that feature also.
So, should am I misconfiguring the e3845 box, or is being able to hit upwards of 800Mbps with some of these features on just not in the cards for this CPU?
If so, I am now all over the place. I don't really want to go for the home run with a Xeon. I would really like passive cooling, but will settle for quiet fans. I don't really want to virtualize as my server is a C2758, so it might not be strong enough for all this and my two VMs and two containers. A separate device allows me to work on it without bringing down my Internet and annoying the other dwellers. I am open to a lot of other options, from putting a box together myself with a miniITX motherboard to a Netgate appliance, or something in between. I would prefer to keep the price down around $500. RAM and mSATA can be extra, as I probably already have them depending on specs.
-
I also would really like to eventually get to dialing in https MITM scanning eventually. I would really like to see what that 40-60% of my traffic is going to, and have it scanned for malware, etc. Although, I know that many people see this as a bad thing and possibly pointless. If I get to that point I will do some more reading on the subject, and ask advice later, but I want to size for potentially using that feature also.
So, this here is the difference between a $200 box and a $1000+ box. I'd say forget about it, because you're more likely to weaken your security than improve it. But if you want to do this at gigabit rates you need multiple heavy cores. Otherwise, most lower mid range hardware should be able to hit your numbers just firewalling. That said, I'd also expect better from the e3845. This isn't pppoe is it?
-
no PPPoE, I could test with by loading up a standard desktop OS on it and seeing what type of throughput I get with the E3845 directly to the modem.
-
From the modem with a pretty decent laptop with a gigabit Ethernet NIC, I was getting about ~830 Mbps down and nearly 50Mpbs up.
Ok the might be then the raw throughput without any kind of SPI/NAT or firewall rules working in the front of that
laptop. If this is a real and pur modem it was a good test, in any kind of art and wise, but it also counts the complete
wiring too, and if there is something bad or broken or old or what ever, this will also count into that game play!
Please don´t forget this.The device has Intel 82583V NICs so I turned on all of the offloading features, and removed all packages from the device to get a baseline. I tested from the same laptop cabled to the router and from speedtest-cli on the router. I also tried different times of the day and to different servers. I was getting only around 500Mbps at best.
If you want to know it really and owns also enough time please do a fresh and full install on the mSATA and then please
try out in front of the WAN port to install a small switch without a configuration. At best will be here something likes the
Netgear GS105E, GS108E or GS108Tv2, without any config on them they will be acting as a dump switch, but very fast
too!!! So if you do then also a "lab" test with iPerf 3.0 or NetIO through the WAN interface you will be going to get the
best result for , what your appliance will be able to realize for you, and this mostly protocol Independent on top of this.I contacted the owner of the Protectli earlier and he said that they have done synthetic lab tests and reached 930-940Mbps. That is with no features enabled like IPS. He did say that pfSense gives the best performance on the
device as compared to some of the other router OSes that I have played with before.930 - 940 MBIt/s + the TCP/IP overhead and + the SPI/NAT and firewall rules passing through it is nearly
a real 1 GBit/s please don´t forget this.At this point, my entire picture has changed. I don't think that the E3845 will do it for me even if 833Mbps is my ceiling.
This sounds perhaps hard to you now, but if you are reaching with the "pure modem" in front of the laptop
833 MBit/s and then also with the pfSense firewall in front of the home network too, I mean 833 MBit/s,
all must be fine then or am I wrong now with that? What do you imagine here, the pfSense is throwing buckets
of bits and put it into the line, to fill the space up to 1 GBit/s?no PPPoE,
That means you will be able to get all out of your pfSense box, so could it be that the Internet line is not able to
realize more then that?I could test with by loading up a standard desktop OS on it and seeing what type of throughput I get with the E3845 directly to the modem.
Is this "modem" perhaps a router and you are doing double NAT now and this will normally loose between 3% - 5%
of your entire throughput!I used to want to run pfBlackerNG, Snort, and Squid on it.
And if you count now only ClamAV on top of that action, you get a fully UTM device, please open Google
and do a search on UTM devices to see and get closer to what you have to pay for a UTM with 1 GBit/s
WAN throughput and then please let us talk about your small Protectli FW4A (e3845 CPU with AES,
and 8GB of RAM) that is not competing that UTM device! -
The modem is a Arris TM3402a. http://www.arris.com/products/touchstone-tm3402-emta/
I have used this brand of modem before, and while they do have models with NAT, those models hand out private IPs, and have a WebGUI for configuration changes. This is what they were trying to push when I got into the store, until I told them I wanted DOCSIS 3.1 and no NATing: http://www.arris.com/products/touchstone-tg2472-cable-voice-gateway/
Mine has no webgui, and the pfSense WAN port receives a public IP add.
So if you do then also a "lab" test with iPerf 3.0 or NetIO through the WAN interface you will be going to get the
best result for , what your appliance will be able to realize for you, and this mostly protocol Independent on top of this.I will try this over the weekend and report back. Also with a fresh install. Also with a different box entirely.
That means you will be able to get all out of your pfSense box, so could it be that the Internet line is not able to
realize more then that?So, my thinking is if I can pull 833Mbps, I would like to see the box able to handle at least that before I start turning services on. I would be really happy if I could keep my speed above 500 with those services turned on.
This sounds perhaps hard to you now, but if you are reaching with the "pure modem" in front of the laptop
833 MBit/s and then also with the pfSense firewall in front of the home network too, I mean 833 MBit/s,
all must be fine then or am I wrong now with that? What do you imagine here, the pfSense is throwing buckets
of bits and put it into the line, to fill the space up to 1 GBit/s?My thinking here is that if the lab can produce 930Mbps with gig hardware, then my 833Mbps line should be reduced to ~750Mbps after on the LAN side without services turned on. But in fact I am getting am at best getting 500Mbps… meaning I am loosing 39% to this box, and it isn't consistent. Sometimes it's much lower then that.
My thinking is the equipment or the config. The config is already fairly fresh. Yes I did install a couple of packages, but I later removed them to set a baseline first. Once the baseline was decent, then I would start turning things on, one at a time to measure their impact.
By the way I also plugged the same laptop which hit 833 into a switch port on a consumer ASUS NAT and tested. With other people on the network streaming the typical stuff I hit 749, 755, and 819 on three subsequent tests.
And if you count now only ClamAV on top of that action, you get a fully UTM device, please open Google
and do a search on UTM devices to see and get closer to what you have to pay for a UTM with 1 GBit/s
WAN throughput and then please let us talk about your small Protectli FW4A (e3845 CPU with AES,
and 8GB of RAM) that is not competing that UTM device!Yes I left out ClamAV. I have conditioned myself not to say UTM because of so many devices that call themselves UTMs that don't do what others do, but that is the goal.
I will do the testing and get back. Thanks for all of the great feedback und mit freundlichen Grüßen!
Quote from: xionoix on October 17, 2017, 09:33:38 pm
I also would really like to eventually get to dialing in https MITM scanning eventually. I would really like to see what that 40-60% of my traffic is going to, and have it scanned for malware, etc. Although, I know that many people see this as a bad thing and possibly pointless. If I get to that point I will do some more reading on the subject, and ask advice later, but I want to size for potentially using that feature also.
So, this here is the difference between a $200 box and a $1000+ box. I'd say forget about it, because you're more likely to weaken your security than improve it. But if you want to do this at gigabit rates you need multiple heavy cores. Otherwise, most lower mid range hardware should be able to hit your numbers just firewalling. That said, I'd also expect better from the e3845. This isn't pppoe is it?
What kind of hardware could hit this with the these meaty packages turned on? would a C2758, a 6th gen i5, or do I need to go to xeons to do this?
-
What kind of hardware could hit this with the these meaty packages turned on? would a C2758, a 6th gen i5, or do I need to go to xeons to do this?
There are many way you could walk through that theme, and as often shown their are more then one or two camps
that prefer the one or other way to realize it, as mostly too it comes to the budget on top of this and then it is not
even so clear or easy to council something right matching to all points and needs or wishes.The first camp is preferring a small Intel Core i3, i5 or i7 CPU that is powerful enough and cheaper to get the
hands on. But together with installed packets such Squid & SquidGuard, SARG, Snort and pfBlockerNG and then
getting out one real (1 GBit/s) after passing through all named above packets it should be then a high scaling
CPU.The second camp is preferring to use server grade CPU´s either from Intel or AMD, because they are produced to
save electric power and run 24/7/365, and on top of this they are mostly able to get cheap if it is a refurbished
CPU from a server pull! So the smallest CPU from then is the Intel Xeon E3 in eiter version v2/v3/v4/v5 they
looks like to be attractive to many users related to some edge numbers, RAM support form 16 GB to 32 GB,
4 core with HT (4C/4T) and TurboBoost v1/v2, from low scaling to high up scaling, AES-NI support and
if refurbished you can get them cheap and mostly older boards too!The newer CPUs such as Intel C3000 and Intel Xeon D-15x8 or the newest (N) platform will be the top end
what I would be looking for, but they are also higher in price range or not really on the market now, or fully
supported now by pfSense.Some thoughts on this by my own, if you want to get pfBlockerNG & DNSBL & TLD running fine with many lists
you were choosing the amount of RAM can be to small on many systems, and it the RAM is also to slow too, it
can be that the entire memory system will be saturated, your CPU might be fine but and powerful enough but the
memory system is to slow or the amount of RAM is to low or small!Please don´t forget this too if you are willing
to use that packet, it is awesome but needs also RAM that is matching right to the choose amount of lists you
have subscribed to. -
So I decided to take a different approach to make sure that I didn't have another issue with the line or modem and to at least baseline that.
I loaded up a vanilla install of Ubuntu Desktop 17 on the E3845 box and connected directly to the modem and was able to pull a few speed tests at over 900Mbps.
Since I have seen so much emphasis on the NIC chipsets, I assumed the drivers can be a major factor here. So I tried a copy of IPFire on this E3845. I was also getting ~500Mbps throughput on the LAN side of the E3845 box.
Next, I grabbed a workstation from work to test, and I am getting 900+Mbps with Surricata VRT registered rules turned on, and ClamAV pfBlockerNG enabled. It is a Dell T5610 Xeon E5-2630 v2 with 32GB of RAM. I then threw in a couple of Broadcom bcm5722kfb1g that I had laying around.
As expected it doesn't break a sweat. RAM utilization hasn't broken 8% and I can't see that the CPU has broken 10% utilization. To top it off this box only uses between 5 & 10 watts more then the E3845 box did, and it's noise level is less then my server which has a C2758 and a 400W power supply.
So, other then the size of this beast I am super happy with the performance of this box.
I think I will look for something to replace my C2758 server in the two year old used market for Xeon processors. Then I will virtualize on top of that. Lots of good write-ups on virtualizing on Proxmox and I built a VM for testing on the C2758 platform as well. It was easy to setup, but performance was about the same as the E3845.
What is interesting in all of this is that I didn't see any indicators in the reporting that the E3845 wasn't able to handle the gigabit speed, and I don't see anyone's comments that it shouldn't be able to handle gigabit running as just a firewall with no services.
However, it was very inconsistent in it's performance. Maybe I wasn't tweaking it just right, but it's been a PIA to me.
-
So I decided to take a different approach to make sure that I didn't have another issue with the line or modem and to at least baseline that.
Very good, this will be many times the unseen and unknown part in that game play.
I loaded up a vanilla install of Ubuntu Desktop 17 on the E3845 box and connected directly to the modem and was able to pull a few speed tests at over 900Mbps.
Ok, fairly that must be said here at this point, Linux is more near hardware coded and is also running more agile
with more speed as FreeBSD is doing, so you should be counting some more horse power for FreeBSD to get the
same result as with Linux. And by the way the main computer or IT industry (not the whole, but many) was
cutting of the the dirver support for BSD based systems in former days, but not really for Linux! And on top
of this Ubuntu is not doing SPI and NAT!Since I have seen so much emphasis on the NIC chipsets, I assumed the drivers can be a major factor here. So I tried a copy of IPFire on this E3845. I was also getting ~500Mbps throughput on the LAN side of the E3845 box.
Also here with some tunings right matching to the underlying OS (Linux) you will be getting more out as now, but
not so many!Next, I grabbed a workstation from work to test, and I am getting 900+Mbps with Surricata VRT registered rules turned on, and ClamAV pfBlockerNG enabled. It is a Dell T5610 Xeon E5-2630 v2 with 32GB of RAM. I then threw in a couple of Broadcom bcm5722kfb1g that I had laying around.
Yee-haw, that is nothing you should be comparing against the named hardware from above! Other CPU, other cores
and much RAM! It is a really pfSense bomb!You can try to turn on the pfBlockerNG together with a DNSBL & TLD to
watch out the RAM usage will be enough or not if you want.I think I will look for something to replace my C2758 server in the two year old used market for Xeon processors. Then I will virtualize on top of that. Lots of good write-ups on virtualizing on Proxmox and I built a VM for testing on the C2758 platform as well. It was easy to setup, but performance was about the same as the E3845.
If version 3.0 will be out I expect perhaps a really multi cpu core usage and if this will be done right then
you could expect much more from your actual box as you perhaps do now!However, it was very inconsistent in it's performance. Maybe I wasn't tweaking it just right, but it's been a PIA to me.
The C2758 is not really much needing to tune and since the version 2.4.x the hassle of turning the TRIM on
is gone forever. So what exactly do you mean with a PIA for you? Perhaps I was not really understanding it
in the right way.-
enable PowerD (high adaptive)
-
enable TRIM if SSD or mSATA are in usage
-
shorten the num.queues to 1, 2 or 4 for each NIC
-
high up the mbuf size to 125000, 250000, 500000 or 1000000
-
if the IPMI port is shared to the WAN port as fall back you might be running
-
increasing or high up the squid default memory size
This are things you could do, and not you must do!
-
-
@BlueKobold
Thanks for your continued responses, lots of help trying to figure things out…This T5610 is just on loan. I need to get it back soon, so I want to switch back to the E3845 or to virtualize on the C2758. The real-world performance seems to be pretty similar between the two. CPU comparison sites also show single core performance seems similar, and the differences really only show up when using multicore apps or AES-NI. I don't have any tunnels open right now. I occasionally VPN back in. Not that I want to close that door, it just isn't a priority at the moment. My other VMs don't have high CPU utilization they are relatively low peaking around 60-85% a couple of times a day for only a couple of minutes.
The C2758 is not really much needing to tune and since the version 2.4.x the hassle of turning the TRIM on
is gone forever. So what exactly do you mean with a PIA for you? Perhaps I was not really understanding it
in the right way.- enable PowerD (high adaptive)
- enable TRIM if SSD or mSATA are in usage
- shorten the num.queues to 1, 2 or 4 for each NIC
- high up the mbuf size to 125000, 250000, 500000 or 1000000
- if the IPMI port is shared to the WAN port as fall back you might be running
- increasing or high up the squid default memory size
This are things you could do, and not you must do!
it would be ideal to get the C2758 running the router in a VM on Proxmox and not spend any more money.
I am only using the on-board ports for management. I have an i350 QP in it for VMs and WAN. I used these docs for my setup: https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox and https://pve.proxmox.com/wiki/PfSense_Guest_Notes
Do you know if virtualizing on Proxmox balances the load on other cores? I am on the community repo. I am using the kvm64 CPU. I thought I read ESXi users see those benefits.
However, I just have a feeling that neither of these CPUs have enough power to run these apps. Assuming for a minute that I go with a dedicated piece of hardware for pfSense, should I be looking at an i5 at a minimum to be running all of these pfSense packages?