Poor performance on igb driver

[bdaniel7]

Hello,

I'm running pfSense 2.4.3-p1, on a Qotom board with the following specs:

Intel(R) Celeron(R) CPU J1900 @ 1.99GHz
4 CPUs: 1 package(s) x 4 core(s)
8 GB RAM
120 GB SATA SSD
4 x Intel I211AT network ports.

The network ports are served by the igb driver.
I have a 1Gbps (best effort) fibre optics from my provider, but the router WAN is performing quite poorly.
When connected directly to the mediaconverter, I'm getting 750 Mbps, which is fair considering the terms provided by the ISP (850 Mbps average, 940 Mbps maximum).

When connected to the LAN port of pfSense, I'm getting only 500 Mbps, give or take 50 Mbps, with CPU going to 30% during tests.
Both the WAN and LAN are using SFTP, Cat 6a cables.
Now, are there any settings that I can use to improve the speed?

FreeBSD always boasted on high speed with Intel cards, so.. what gives?

Thanks.

[stephenw10]

How are you testing the throughput?

Check the CPU loading across the cores by running at the CLI top -aSH during the test.

Check the Status > Interfaces page for any errors on the either interface.

Steve

[MarcoP]

The quad core CPU might have one core at 100% and that is what's being used on your test, if you run parallel connections your total throughput might be higher.

[bdaniel7]

I'm testing using a Speedtest client running on Windows 10 Pro, that connects to a server in the ISP network.

There are no In/Out Errors, nor Collisions, recorded in the Interfaces page.

I tried running the client from 2 different computers, at the same time.
Both clients got about half of the total speed, 340 and 317 Mbps.

Re: CPU loading across the cores, I saw that not all cores were used. Maybe only 2 were used, with 20% and 80% load, while the others were on zero.

[MarcoP]

@bdaniel7 said in Poor performance on igb driver:

Intel(R) Celeron(R) CPU J1900 @ 1.99GHz

If the test was done via HTTPS it seems that this is the most your non AES-NI capable CPU can do, try using HTTP to confirm this please, otherwise I would have no other ideas.

Cheers.

[bdaniel7]

@marcop
I ran the test using an exe, not from browser, so I don't know if it was on HTTPS.

[Animosity022]

Are you running any traffic shaping or anything else?

[Birke]

have you tried any of the tipps mentioned here: https://www.netgate.com/docs/pfsense/hardware/tuning-and-troubleshooting-network-cards.html?

[stephenw10]

I would expect a J1900 to pass that fairly easily in normal test conditions.

Can we see the output from top when the test is running?

Steve

[bdaniel7]

@Birke

I have set most of the settings from that tunning page.
The only thing I didn't set was hw.igb.num_queues=1, I have it set to 0.

I tried with Hardware Checksum Offloading, Hardware TCP Segmentation Offloading, Hardware Large Receive Offloading disabled and enabled, I didn't notice any difference.

@Animosity022
No traffic shaping.

I have the following services enabled:

dhcpd
dpinger
ntpd
openvpn OpenVPN server: Home LAN
openvpn_2 OpenVPN client:
sshd
syslogd
unbound

@stephenw10

How do I record/export the live output from top?
Should I record a video?

[stephenw10]

You shouldn't need to set the igb queues to 1 any longer. That was a bug in much older versions.

Just hit q in top when it's showing something useful and it will quit out and leave whatever was there available to copy and paste out.

Are you routing traffic over OpenVPN?

Steve

[tman222]

Hi @bdaniel7

I also agree that the CPU should be able to handle 1Gbit speeds fairly easily, especially if you are not trying run any IDS/IPS on top regular kernel packet processing.

FreeBSD's network defaults aren't tuned too well for very high speed connections by default (although this is getting better in newer versions). Here is a link to a thread with some more parameters you can tune on your Intel NIC's:

https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-upload

Of those parameters, I"d probably adjust the RX/TX descriptors and processing limits first and see if that yields any improvements.

Hope this helps.

[bdaniel7]

I'm only using OpenVPN to access the internal network from outside.
Which is happening when I'm at the office.

[stephenw10]

How are you testing when that is shown? What is connected to igb0 and igb1?

Is the CPU actually running at 1.9GHz? Do you have powerd enabled?

Try running sysctl dev.cpu.0.freq when the test is running.

Steve

[bdaniel7]

igb0 is WAN, igb1 is LAN.

I'm starting top -aSH as you suggested, then during the peak transfer, I exit from top with q.

I had powerD enabled, with all (AC power, Battery power, Unknown power) set to Maximum.
I disabled powerD but there is no difference.

And I get this sysctl: unknown oid 'dev.cpu.0.freq'

[tman222]

@tman222 said in Poor performance on igb driver:

Hi @bdaniel7

I also agree that the CPU should be able to handle 1Gbit speeds fairly easily, especially if you are not trying run any IDS/IPS on top regular kernel packet processing.

FreeBSD's network defaults aren't tuned too well for very high speed connections by default (although this is getting better in newer versions). Here is a link to a thread with some more parameters you can tune on your Intel NIC's:

https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-upload

Of those parameters, I"d probably adjust the RX/TX descriptors and processing limits first and see if that yields any improvements.

Hope this helps.

Hi @bdaniel7 - have you also tried tuning some of the additional parameters that I suggested? If yes, what were the results?

[stephenw10]

Sorry I meant where are you testing between? Speedtest client on igb1 connecting to a server via igb0?

Steve

[bdaniel7]

@stephenw10
Yes, the mediaconverter is connected to igb0, my Windows 10 client is connected to the igb1 port.

[stephenw10]

I don't see it having been asked so, are you connecting using PPPoE?

Steve

[bdaniel7]

@stephenw10
Yes, I'm using PPPoE.