Help me tune my Netgate SG-2220

[pwest]

I've been happily running pfSense on my  SG-2220 on my home LAN for over a year, but recently upgraded from 24Mbps Uverse to Gigabit Xfinity.

With the Xfinity modem connected directly to my PC, my download speeds are about 940Mbps, but when the SG-2220 is in between the PC and the modem (I believe I've got pretty much everything turned off, and no other devices are connected) I'm seeing sustained throuputs of about 520Mbps with occasional bursts as high as 750Mps.  My WAN connection to the cable modem is DHCP (no PPPoE), I'm using new Cat 6 cables,and running the latest version of pfSense (2.3.2-RELEASE-p1 (amd64)).  Looking at the Dashboard, mbuf, cpu, and memory usage are very low.

So, maybe 500Mbps is 'all' it's good for, or maybe some of the dozens of tuning parameters available could speed things up.  Any suggestions/recommendations would be appreciated.

Thanks,
Phl

[whosmatt]

@pwest:

Looking at the Dashboard, mbuf, cpu, and memory usage are very low.

The dashboard isn't a very good place to look at CPU usage IMO, unless it's a long sustained period, as in not a speed test.  Best bet will be to get an ssh session going and look at top.  Look at the %idle metric during a speedtest, or better, a sustained download from a source you know can support that kind of throughput, something like downloading a game from Steam, if you're into that sort of thing.

Oh, and congrats on getting a 1Gbps connection!  Your issue is one many of us only dream of having to address :)

[pwest]

@whosmatt:

The dashboard isn't a very good place to look at CPU usage IMO, unless it's a long sustained period, as in not a speed test.  Best bet will be to get an ssh session going and look at top.  Look at the %idle metric during a speedtest, or better, a sustained download from a source you know can support that kind of throughput, something like downloading a game from Steam, if you're into that sort of thing.

OK: I connected w/ssh, ran top, downloaded a 5GB image and never saw CPU: system go much above 50%.

Oh, and congrats on getting a 1Gbps connection!  Your issue is one many of us only dream of having to address :)

Thanks, it's one of those first-world 'problems' – my hardware can't keep up w/my internet connection!

-Phil

[whosmatt]

@pwest:

OK: I connected w/ssh, ran top, downloaded a 5GB image and never saw CPU: system go much above 50%.

That's only one part of the equation.  % idle tells the story. Closing any GUI (web) sessions will free up some CPU cycles.  Check % interrupt as well.  But mostly watch % idle.  Subtract that number from 100 and that's the total CPU usage (from pf, the web interface, NIC drivers, etc).  If % idle hits 0, your CPU is 100% consumed.

[kolpinkb]

Try adding the following lines to your loader.conf.local file under /boot

hw.igb.rxd=4096
hw.igb.txd=4096
kern.ipc.nmbclusters=1000000

Uncheck the following options under Advanced>Networking

Then reboot and run your speestests again.

[pwest]

I've made the parameter changes suggested above.  The processor usage may have decreased a bit, but the bottom line is about the same with max download speed being about 500mbps and minimum % Idle typically between 25%-30%.

Concerning what download/speedtest source to use, I've got a bit of a problem there.  Speedtest.com consistently provides the fastest results (a bit above 500), but the test is a bit short making the %Idle reading inconsistent.  Speedtest.fronteir.com produces througputs almost as high, but the test lasts longer.  Other tools that I tried:
wget -O /dev/null http://speedtest.tele2.net/1GB.zip  //nice big transfer, but overall speed was only 160bmps and %Idle 80%
speedtest-cli //short test time with hroughput of 500mbps, %idles seen as low as 19%

I'd like to find a good source for a file on a server with fat pipes that's at least 10 GB.  Can I grab something from steam somehow?  I'd rather not install the whole gaming infrastructure, create an account, etc.

Thanks again for all the help on this!  (You'd think the people from Netgate would supply a tuning guide)
-Phil

[kolpinkb]

Interesting that those settings didn't help.  I have an Intel i350-T4v2 that without those settings enabled I couldn't get squid to push the full 1Gps the card supports.

Try setting up a demo HTTP server box on the wan port.  Then try pulling a file on the LAN side.  An ssd on the test box will really help.  Even a Core i3 laptop can act as the server.

I've personally used one as a test pfSense box on such hardware and I was for sure able to pull 1GB with squid using an ssd.  You can install pfSense on it and then place a 1-5 GB file in the /var/www/… directory and try downloading it.

[VAMike]

@thehammer86:

I've personally used one as a test pfSense box on such hardware and I was for sure able to pull 1GB with squid using an ssd.  You can install pfSense on it and then place a 1-5 GB file in the /var/www/… directory and try downloading it.

Pro tip: from your www directory run something like "truncate -s 2G empty.file" to create a sparse (no disk blocks involved) file for testing purposes. That way, you take disk speed out of the equation.

[pwest]

I sent a note to Netgate support asking if they had any tuning recommendations, and if they thought their system should be able to do better than 500-600mbps, and I got this note back:
–-------------------------------------------------
6-700Mbps is about what I would expect though from the 2220. It won't push 940Mbps.
That is very dependent on the type of traffic in use though as always when trying to measure throughput.

Pulling a large file while observing 'top -aSH' has always proved the most reliable method for testing high throughput for me. I found that many online tests, like speedtest.net, were inconsistent at high values.

If you imported a config check you have powerd enabled. Without that the SG series boxes will run at their lowest speed. I doubt yours is though as you wouldn't get 500Mbps in that case.

Thanks,
Steve

Turns out, I did not have PowerD checked.  I did so, and selected the HiAdaptive option and am now getting download speeds consistently over 800mbps and an sometimes over 900mbps!  %Idle drops to 5%-10%.  Although I'm still interested in continuing to tune and refine system settings to improve processing efficiency, I'm pretty happy with the overall throughput now.

-Phil