SOLVED: Troubleshooting throughput
-
I recently built a pfsense system with the goal of being able to push 1Gb/s up and down. I have 1 Gb/s symmetrical service from my ISP, using PPPoE (unfortunate, but it is what it is). With my provider's firewall device, I am able to push 950+ up and down, so I know the bandwidth is there if I can push it.
My key hardware consists of an Asrock J3455B-ITX board with an embedded Intel J3455 quad-core Celeron and a Broadcom 5709-based dual port NIC. I went with a Broadcom card to hopefully avoid the issue seen with Intel NICs and PPPoE. I also have 4 gigs of RAM and a small SSD for storage.
When I test this hardware, I am only able to get 500Mb/s download, similar to my previous SG-2440. However, my new system is able to push well over 900 Mb/s upload, where my SG-2440 had a upload rate of 500Mb/s, similar to its download rate.
Since my new system is able to hit target on the receive side, I'm hoping there is a possibility for increasing my download rate to a similar rate. Running top -P, I can see that during download testing, the core that is being hit the heaviest of the four cores is at about 53% utilization. During the upload test, the heaviest hit core goes up to about 65%.
Following the guide at https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards, I tried add the following lines to my /boot/loader.conf.local file:
kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0But that only seemed to slow my upload speed to the 600 Mb/s range, so I ended up removing it.
Any thoughts as to what can I can do to increase my download rate?
-
Certainly no expert, but I think the problem could lie in your NIC.
From what I've read on here, there isn't an Intel problem with PPPoE, it's an issue with receive side scaling (rss) queuing. Apparently the igb driver has multiple rss queues whereas the em driver is single queue? Again, I'm just regurgitating what I've read on here.
The user in the linked thread achieved nearly 90% of gigabit with a C2758.
As I understand it PPPoE is single threaded.
The 2758 is a full GHz faster than the J3455 but the J3455 is a Celeron as Opposed to Atom and it is an old atom, those things count for a lot.
Passmark (FWIW - not a very relevant bench + sample size is low) shows the single core performance of the J3455 being ~53% faster than the C2758. That's probably overzealous, but the point is that the J3455 should be powerful enough for routing PPPoE at Gigabit speeds - your top outputs confirm this.So that points back to your NIC, I'd give an Intel PRO/1000 a shot - those use em drivers, and aside from running pretty hot they are excellent.
Check out the following thread:
https://forum.pfsense.org/index.php?topic=107610.msg733717#msg733717 -
The bad news: you should have gone with the J3355 instead of the J3455, for ~25% better single thread performance. Single thread performance is your bottleneck for this application. The issue with RSS & PPPoE isn't intel-specific, all of the freebsd drivers (AFAIK) will drop incoming PPPoE packets into the same receive queue.
-
Thanks belt9 and VAMike. I'll try to dig up an em Intel card and see if that improves my performance. As far as the J3355 being better, I guess that ship has sailed ::), but I'll keep it in mind if it looks like the CPU is becoming a bottleneck.
-
Really often, you will be seeing only an positive effect to any point, if your are combining some tunings.
Also a fresh and full install without any packets will be best option to start those tests. After finding out
what tunings or settings will be fine matching you could install the rest and restore from a backup all
your things fast.-
Go to your BIOS and have a look if their is something like hyper threading and if this is enabled or disabled,
try out to enable it! -
Also have a look if the PowerD (high adaptive) is enabled or not, try enabling it.
(Not needed or necessary to this things but) try out to enable TRIM support for your mSATA SSD
kern.ipc.nmbclusters="131072"
Can be different from each NIC to other NICs!Try out 125000, 250000, 500000 or 1000000 but please be careful
if you have not enough RAM inside of your pfSense box you will ending up in a booting loop perhaps.It could also be helpful to try out to regulate the amount of num.queues to 1, 2, or 4 if needed.
I recently built a pfsense system with the goal of being able to push 1Gb/s up and down. I have 1 Gb/s symmetrical service from my ISP, using PPPoE (unfortunate, but it is what it is). With my provider's firewall device, I am able to push 950+ up and down, so I know the bandwidth is there if I can push it.
Your ISPs router or firewall is mostly ASIC/FPGA supported and speeds up the throughput, pfSense is not
doing so and is not doing anything like this.My key hardware consists of an Asrock J3455B-ITX board with an embedded Intel J3455 quad-core Celeron and a Broadcom 5709-based dual port NIC. I went with a Broadcom card to hopefully avoid the issue seen with Intel NICs and PPPoE. I also have 4 gigs of RAM and a small SSD for storage.
The igb(4) driver is also running on multiple cpu core now, as I am right informed, but the entire PPPoE part
is single threaded and so you will be not reaching more by using a NIC from here or there.Any thoughts as to what can I can do to increase my download rate?
- An Intel NIC that is using the em driver Intel Pro 1000/PT dual or quad NIC will be fine
- More RAM if the mbuf size increasing will be fine working for you if you will be installing more packets such
snort, Squid and pfBlockerNG that will need or use also more RAM together.
-
-
It's been a long time, but I thought I'd close this out. I located an Intel NIC to replace the Broadcom card, but I saw no change in throughput after switching to it. At around the same time, my wife decided she was tired of her old laptop (which only had 100 Mb/s NIC), so I pulled some old parts together to build her a desktop PC, located in the same room as the pfsense firewall in question. Using her new PC with a gig NIC, I was able to push ~950 Mb/s or so through the firewall.
Previously, all of my gig-capable systems were on the other side of a MoCa (ethernet-coax) bridge, which obviously weren't useful as realistic tests, so I was using my Macbook air with a gig-e dongle for testing. I guess the gig-e dongle is not capable of actually pushing a full gig, so that was where my bottleneck was, and the fairly obvious question of whether the dongle could be the actual bottleneck didn't ever occur to me. I guess since it was a thunderbolt adapter, as opposed to USB, I just assumed it would be able to push a full gig.
Anyway, thanks for all the suggestions, it turns out it was one of those dumb issues and nothing to do with pfsense at all.
-
It's been a long time, but I thought I'd close this out. I located an Intel NIC to replace the Broadcom card, but I saw no change in throughput after switching to it. At around the same time, my wife decided she was tired of her old laptop (which only had 100 Mb/s NIC), so I pulled some old parts together to build her a desktop PC, located in the same room as the pfsense firewall in question. Using her new PC with a gig NIC, I was able to push ~950 Mb/s or so through the firewall.
Previously, all of my gig-capable systems were on the other side of a MoCa (ethernet-coax) bridge, which obviously weren't useful as realistic tests, so I was using my Macbook air with a gig-e dongle for testing. I guess the gig-e dongle is not capable of actually pushing a full gig, so that was where my bottleneck was, and the fairly obvious question of whether the dongle could be the actual bottleneck didn't ever occur to me. I guess since it was a thunderbolt adapter, as opposed to USB, I just assumed it would be able to push a full gig.
Anyway, thanks for all the suggestions, it turns out it was one of those dumb issues and nothing to do with pfsense at all.
Thunderbolt is considerable faster than usb so that’ wasn’t the problem. It’s essentially pcie x1. I’ve measure 991mbs (once but usually around 980) with my 2015 MBP i7 quad core with the Apple dongle on Comcast gigabit.
-
I don't think it was the Thunderbolt adapter either. USB might have made more sense tho as they often cannot go beyond 200Mbit.