Intel NIC problem with MSIX

mastahfr

I've 3 intel NIC on this PC.
One is "Intel 82579V" and the two others are "Intel 82571GB - Intel PRO/1000 PT Dual Port".
All the three NIC only have MSI enable but not MSI-X.

I've tried under pfsense 2.0.3 amd64 and 2.1.RC0 amd64

dmesg:
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xe020-0xe03f mem 0xf7d80000-0xf7d9ffff,0xf7d60000-0xf7d7ffff irq 16 at device 0.0 on pci1
em0: Using an MSI interrupt

em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xe000-0xe01f mem 0xf7d20000-0xf7d3ffff,0xf7d00000-0xf7d1ffff irq 17 at device 0.1 on pci1
em1: Using an MSI interrupt

em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xf080-0xf09f mem 0xf7e00000-0xf7e1ffff,0xf7e39000-0xf7e39fff irq 20 at device 25.0 on pci0
em2: Using an MSI interrupt

pciconf -lvc:
em0@pci0:1:0:0: class=0x020000 card=0x125e8086 chip=0x105e8086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4)
em1@pci0:1:0:1: class=0x020000 card=0x125e8086 chip=0x105e8086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4)
em2@pci0:0:25:0: class=0x020000 card=0x20348086 chip=0x15038086 rev=0x04 hdr=0x00
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP

sysctl hw.em:
hw.em.eee_setting: 0
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024
hw.em.rxd: 1024
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66

So obviously, MSI-X is enabled but not working. Anyone have any particular clue on why it doesn't want to activate ?
I've search countless of time around, without finding any evidence why it's failing at activating MSI-X.

Thanks.</intel(r)></intel(r)></intel(r)>

mastahfr

So far, I've tried the Intel PRO/1000 PT Dual Port into an older computer.
The MSI-X is working properly.

I've also tried this computer (the one not able to activate MSI-X, but just MSI) on an ubuntu distro (13.x). All the PCI NIC can't activate MSI-X neither.
So I'm assuming that it is a motherboard issue (or bios).

Motherboard is Intel DH77DF (1155 socket) and CPU is Intel G630T (1155 socket).

I'm really disappointed by Intel if the motherboard is at fault. This MSI-X spec is like 8 years old, how come they can't manage to implement it properly now days…

If any of you have any idea what could cause this behaviour, I would be glad to hear it ...

As for now I recommend that you avoid at least DH77DF, if not Intel motherboard... until the problem is solved.

Btw, I need MSI-X because the CPU is dying under shit tons of interrupts at 300mb/s (9000-10000+/s and per NIC, so a total of 19000-20000+/s for WAN+LAN) which bring my CPU at 40-50%. The problem will be even more present as I'll get high speed from WAN.

wallabybob

@mastahfr:

Btw, I need MSI-X because the CPU is dying under shit tons of interrupts at 300mb/s (9000-10000+/s and per NIC, so a total of 19000-20000+/s for WAN+LAN) which bring my CPU at 40-50%. The problem will be even more present as I'll get high speed from WAN.

It is not clear to me how MSI-X would reduce the interrupt rate compared with MSI. Please explain.

Have you tried modifying the interrupt moderation parameters?

mastahfr

@wallabybob:

It is not clear to me how MSI-X would reduce the interrupt rate compared with MSI. Please explain.

Judging by http://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MSI-X, MSI-X should do a lot less interrupts.

@wallabybob:

Have you tried modifying the interrupt moderation parameters?

Not at all, what are those params ?

wallabybob

@mastahfr:

Judging by http://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MSI-X, MSI-X should do a lot less interrupts.

Sorry, but I don't see how MSI-X would reduce interrupt rate. That section says MSI-X can have more interrupts (ways of interrupting the system) and can have an interrrupt cause generate an interrupt to a particular CPU (for example, receive interrupts to cpu0 and transmit interrupts to cpu1) but it is not clear to me how either capability could, by itself, reduce interrupt rate.

Interrupt moderation parameters are described in the FreeBSD man page for the em driver: see http://www.freebsd.org/cgi/man.cgi?query=em&apropos=0&sektion=0&manpath=FreeBSD+8.1-RELEASE&arch=default&format=html and note the rx_int_delay and tx_int_delay parameters. By delaying interrupts somewhat the interrupt rate can be reduced by the processor doing more work per interrupt, for example, processing 3 received frames in one interrupt rather than 3 separate interrupts for the 3 frames.

mastahfr

Maybe I did understand MSIX wrong. Though it's probably usefull with a quadcore or more.

I'll have a look at the FreeBSD page about em and try to tune the settings.

Thanks.