Poor performance on igb driver

stephenw10

Hmm, interesting. I wouldn't have expected msi to any better than msix.
What sort of figures did you see?

Steve

bblacey

@stephenw10 said in Poor performance on igb driver:

Hmm, interesting. I wouldn't have expected msi to any better than msix.
What sort of figures did you see?

Steve

Hmmm, I'm back to msix interrupts so that was a red herring. I'm able to fully saturate my 400/20 link (achieve 470/24) with both inbound and outbound firewall rules enabled. Here is my current config that seems to achieve this:

[2.4.4-RELEASE][root@firewall.home]/root: cat /boot/loader.conf 
kern.cam.boot_delay=10000
# Tune the igb driver
hw.igb.rx_process_limit=800  #100
hw.igb.rxd=4096  #default 1024
hw.igb.txd=4096  #default 1024
# Disable msix interrupts on igb driver either via hw.pci or the narrower hw.igb
#hw.pci.enable_msix=0   #default 1 (enabled, disable to nudge to msi interrupts)
#hw.igb.enable_msix=0
#net.inet.tcp.tso=0  #confirmed redundant with disable in GUI
#hw.igb.fc_setting=0
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1
boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"
comconsole_speed="115200"
autoboot_delay="3"
hw.usb.no_pf="1"

Basically, I'm using the defaults other than increasing the igb driver rx_process_limit, rxd and txd. I have disabled tso, lro and checksum offloading via the gui under System->Advanced->Networking (checked means disabled) and set kern.ipc.nmbclusters to 262144 under System->Advanced->Tunables.

Hardware:

CPU: Intel(R) Atom(TM) CPU  C2358  @ 1.74GHz (1750.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

You might want to go back to the pfSense defaults and enable all networking offloading options are disabled (checked in the GUI), then tweak the igb driver elements as I did above test and then adjust key tunables such as kern.ipc.nmbclusters but more isn't necessarily better.

MarcoP

I've just noticed you are on PPPoE, would increasing MSS clamping on the interface or setting MTU at 1492 help.

stephenw10

You should put custom settings in /boot/loader.conf.local to avoid them being overwritten at upgrade. Create that file if it's not there.

Steve

Nonconformist

Hi @bdaniel7, any luck on achieving gigabit speeds after your tweaks? I’ve been running into the same issues as you with the same Qotom box.

Posted about it [here] (https://forum.netgate.com/topic/137196/slow-gigabit-download-on-a-quadcore-intel-celeron-j1900-2-41ghz), and then used the tweaks in this thread.

Still getting only about 730mbps on wired.

bdaniel7

@nonconformist
Hi, nope, I couldn't get any speed higher than 550 Mbps when I tried the tweaks.
Then I abandoned the subject, due to lack of time.

I will try the tweaks from the article you posted.

MarcoP

any dropped packets?

netstat -ihw 1

MarcoP

Since you have cores waiting, you could try to avoid locks when switching between them with:
net.isr.bindthreads="1"

Nonconformist

@marcop Couldn't check through the week so doing this over the weekend. Long story short, no dropped packets.

net.isr.bindthreads="1"

actually brought the download/upload speeds down to 680/800 from 740/934.

Reading more about this, it's beginning to look like achieving 1G download isn't possible with the igb0 driver with a PPPoE WAN connection.

stephenw10

Are you guys actually running at 2.4GHz? I asked about cpu freq values earlier and there were none and looking back it appears to be running at 2GHz.

If that's true then you need to enable speedstep in the BIOS and make sure it's loading.
You should see some values reported by:
sysctl dev.cpu.0.freq_levels

Steve

stephenw10

The dashboard should show:

Intel(R) Celeron(R) CPU J1900 @ 1.99GHz
Current: 1992 MHz, Max: 1993 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: No

Sysctl something like:

dev.cpu.0.freq_levels: 1993/2000 1992/2000 1909/1825 1826/1650 1743/1475 1660/1300 1577/1125 1494/950 1411/775 1328/600

Otherwise you're seeing a 20% performance reduction.

Steve

Nonconformist

@stephenw10 Didn't find a setting on the BIOS to enable Speedstep.

Nevertheless, it seems like the Celeron J1900 QOTOM box cannot achieve gigabit down/up from what I've gathered from a fair bit of research online. Read numerous threads about some limits with the PCIe lanes on this hardware which mean achieving gigabit is a long shot, and definitely not practical with PfblockerNG and Snort/Suricata and VPN running. I'm giving up on trying to get this box to do it. This leaves me with a tricky decision as a home user with about 35 devices behind PFSense:

Should I upgrade to better hardware, considering the lack of gigabit speeds and the fact that this processor doesn't support AES-NI anyway?
Does it make sense to upgrade at all, given that the practical applications of gigabit speeds are few, if any at this point in time?

Just out of curiosity - if I were to upgrade and get one of the official Netgate boxes, should it be looking at the SG-1000, SG-3100 or SG-5100?

stephenw10

You would need either the SG-3100 or SG-5100. The 5100 has significantly more processing power though, you could run VPN packages etc and still push Gigabit.

I'm curious though, do you not see any CPU frequency levels reported even after enabling powerd?

Steve

bdaniel7

I ended up asking from my ISP an ONT that can run as router and can initiate the pppoe connection.
And I changed the pfSense' WAN interface to dhcp.

I tested the speed from a Dell Vostro 3500 laptop running Windows 7 with an Intel card, connected to the ONT, and I got a bit over 900 Mbps down.
While testing through pfSense, I got only 540 Mbps down.

So the problem still lies with either pfSense or the igb driver.

stephenw10

Hmm, the igb driver on other hardware has no problem passing Gigabit. So I'd have to say it's whatever is specific to your box. I'm surprised the CPU cannot pass more, 540Mbps is Atom D525 levels. So perhaps there is some PCIe bus issue on that board or something similar.

Steve

MarcoP

If we aren't talking about forwarding, but a client-server speed test, I can confirm igb does Gigabit fairly easy.

I just setup a spare port to test without forwarding, VLANs, firewall rules or netmap. Test done with iperf3, pfSense as server and I get an average of 936Mbits/sec both ways (-c -R).

loader.conf.local:

hw.igb.max_interrupt_rate="16000"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.intr_storm_threshold="10000"
net.link.ifqmaxlen="2048"

tunables:

dev.igb.0.eee_disabled 1
net.inet.tcp.hostcache.enable 0 - use this for tests only

Nonconformist

@stephenw10 Did quite a bit of research with settings, tweaking them over and over again. Also installed other firewall/router OS distributions to isolate whether the problem was with PFSense itself, or is a hardware limitation on these boxes.

Long story short - I can safely say the Qotom J1900 cannot pass gigabit down/up in a practical, real world situation. The bottleneck isn't the CPU, but the PCIe lanes.

Having said that, I've no real practical application of a Gigabit up/down connection today - so putting this on the backburner and living with whatever >500mbps I'm getting, until I decide to upgrade this box.

chrcoluk

Just to add in case people still have pppoe performance caps due to cpu power, make sure you enable powerd on units that support intel turbo.

Without powerd turbo clocks wont kick in. powerd isnt just for power saving.

Also it seems "some" igb chips do support rss properly with pppoe, I switched to a pppoe isp last month and see my rss is working properly on the igb driver. I have 2 queues for both isr and igb, the rx is almost a 50/50 split, and its tx thats lopsided at 90/10.

captain_finnegan

net.isr.dispatch=deferred

I just wanted to register and confirm that this worked really well for me after trawling for solutions online for days. I upgraded from 350/50 to 900/100 a few weeks back and it was bugging me that I couldn't access all the speed.

My setup is as follows:

BT Broadband Fibre 900 FTTP 900/100 (speed tests at 930-ish/120ish via BT Smart Hub 2)
PPPoE ONT
J1900 CPU - Asrock Q1900DC-ITX - 4gb ram
Intel i340-T4 Quad NIC (running in PCIe 2.0 x1 slot) (igb driver)

Prior to using net.isr.dispatch=deferred, I was averaging 550-790/120 on Fast.com and Speedtest.net

I'm now getting full line speed. I'll never use it all but it was just bugging me that I couldn't access 100% of what I was paying for.

ztolley

@captain_finnegan

Finally, I've been looking for someone that has UK FTTP and runs a PFSense box that can keep up with the 900/100 service.

I have Zen and Fritzbox and whilst that keeps up I seem to have to reboot it now and again and I ideally want something a bit more secure and lets me monitor how my connection performs. Now I need to track down how I can re-create that setup.