Intel ET2 Quad Very High CPU usage - IGB driver

Guest · Feb 12, 2018, 12:57 AM

@johnkeates:

A software bridge will eat up resources pretty fast. I suspect it shouldn't matter if you are not using it, but then again maybe it even copies packets to the bridge regardless. Can you test without the bridge?

Just tested without the Bridge and its the same result:
CPU: 1.7% user, 0.0% nice, 82.1% system, 8.9% interrupt, 7.4% idle

[SUM] 0.0-60.2 sec 1.85 GBytes 264 Mbits/sec

Are you sure this CPU can do 1000Mbits over TCP ?

What process is using all that CPU? It shouldn't be doing anything special for basic port-to-port traffic.

ralms · Feb 12, 2018, 1:27 AM

@johnkeates:

What process is using all that CPU? It shouldn't be doing anything special for basic port-to-port traffic.

iPerf

I just tested again and this time it managed to do 320Mbits/sec and the interrupt got really high (40%), but from what I check the Intel ET2 should be compatible.

ralms · Feb 12, 2018, 2:50 AM

@johnkeates:

What process is using all that CPU? It shouldn't be doing anything special for basic port-to-port traffic.

So after searching a bit, it seems that LAGG in LACP can cause this.
My LACP is using ports igb2 and igb3.

Print in attachment.

From this view it seems that the NIC is causing most of the CPU load, but I wonder why.

EDIT:
I will try to install the driver from Intel tomorow.

Guest · Feb 12, 2018, 9:24 AM

Looks like you hit the Intel NIC queue problem, this happens to some quad port users. Check the forum to fix this :)

ralms · Feb 12, 2018, 3:16 PM

@johnkeates:

Looks like you hit the Intel NIC queue problem, this happens to some quad port users. Check the forum to fix this :)

I've been searching everywhere and I can't find what you've mentioned.

From some pages I've found I done the following:
Set kern.ipc.nmbclusters to 1000000

Disabled TSO, LRO and Hardware Checksum Offload

So far no effect (so I will re-enable the Hardware options).

Do you have a link for a post with a possible fix please?

I will try to update the driver now since most people seem to complain of PfSense using very old drivers.

Guest · Feb 12, 2018, 5:41 PM

Most of the problems have to do with the NIC dumping all traffic on a single queue and then causing an interrupt storm.
This page seems to have both the tunables and sysctls to fix this https://wiki.freebsd.org/NetworkPerformanceTuning

Also read: https://forum.pfsense.org/index.php?topic=86732.0 & https://forum.pfsense.org/index.php?topic=123462.15

dreamslacker · Feb 14, 2018, 1:09 PM

@ralms:

So after searching a bit, it seems that LAGG in LACP can cause this.
My LACP is using ports igb2 and igb3.

Print in attachment.

From this view it seems that the NIC is causing most of the CPU load, but I wonder why.

EDIT:
I will try to install the driver from Intel tomorow.

Because the LAGG is a software driver. You are not offloading anything to the NIC as when you use it on its own. I've seen the same thing happen before on a Rangeley 8-core. Just had to break the LAGG and use the physical interfaces directly to resolve.

Harvy66 · Feb 14, 2018, 6:13 PM

@ralms:

@johnkeates:

What process is using all that CPU? It shouldn't be doing anything special for basic port-to-port traffic.

So after searching a bit, it seems that LAGG in LACP can cause this.
My LACP is using ports igb2 and igb3.

Print in attachment.

From this view it seems that the NIC is causing most of the CPU load, but I wonder why.

EDIT:
I will try to install the driver from Intel tomorow.

Yeah, don't run iperf from pfSense. It's vastly different than running iperf through pfSense.

ralms · Feb 14, 2018, 8:21 PM

@dreamslacker:

Because the LAGG is a software driver. You are not offloading anything to the NIC as when you use it on its own. I've seen the same thing happen before on a Rangeley 8-core. Just had to break the LAGG and use the physical interfaces directly to resolve.

I've tried outside the LAG, it has a bit more performance but not anything significant, the main issue is still there.

@Harvy66:

Yeah, don't run iperf from pfSense. It's vastly different than running iperf through pfSense.

Since a SpeedTest is giving the same results, I dont think its a main problem right now to troubleshoot it.

stephenw10 · Feb 16, 2018, 12:49 AM

Yeah I would still expect you to see Gigabit easily but it's a much better test to use other devices for the iperf client and server.

Just as an example I can see line rate Gigabit (~940Mbps) with pfSense as one end of the iperf test, as you're doing, on an old E4500. That's using em NICs.

Steve