Performance issues and interrupt storm with em driver [RC2 AMD64]

ftass

Hello!

I have built a router/firewall with the following hardware:
*Supermicro X7SPA-HF (dual on board intel nics (82574L), atom d510)
*Addon dual port Intel PRO/1000 PT

The on board nics are used for local traffic, and are bonded as lagg0. The addon dual port card are used for two wan connections.

I'm getting around 300Mbit/s throughput, with one cpu core handling all the interrupts. The performance is ok but considering only one cpu core is in use I got a feeling it can be tuned a bit. One concern I have is that the machine died yesterday, displaying only "interrupt storm on irq 258" on the console. Seems like something ain't quite right.

vmstat -i:
interrupt total rate
irq1: atkbd0 5 0
irq18: ehci0 uhci5 2 0
irq19: uhci2 uhci4+ 29328 2
cpu0: timer 25348462 1978
irq256: em0 5645078 440
irq257: em1 951927 74
irq258: em2:rx 0 1150847 89
irq259: em2:tx 0 2969097 231
irq261: em3:rx 0 1464717 114
irq262: em3:tx 0 2274026 177
cpu1: timer 25348338 1978
cpu3: timer 25348338 1978
cpu2: timer 25348337 1978
Total 115878502 9043

dmesg:

em0: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xcc00-0xcc1f mem 0xfe8e0000-0xfe8fffff,0xfe8c0000-0xfe8dffff irq 16 at device 0.0 on pci1
em0: Using an MSI interrupt
em0: [FILTER]
em1: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xc880-0xc89f mem 0xfe8a0000-0xfe8bffff,0xfe880000-0xfe89ffff irq 17 at device 0.1 on pci1
em1: Using an MSI interrupt
em1: [FILTER]
em2: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xdc00-0xdc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2
em2: Using MSIX interrupts with 3 vectors
em2: [ITHREAD]
em2: [ITHREAD]
em2: [ITHREAD]
em3: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xec00-0xec1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3
em3: Using MSIX interrupts with 3 vectors
em3: [ITHREAD]
em3: [ITHREAD]
em3: [ITHREAD]

Em2 and em3 are the integrated nics, which apparantly need both tx and rx interrupts?

I have tried messing around with kernel parameters but not really getting any noticable gains. A lot of what I have found searching the interwebs seems to be outdated information.

sysctl net.isr
net.isr.numthreads: 1
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.direct: 1
net.isr.direct_force: 1

I have tried making changes to these tunables by adding them to the system->advanced->system tunables list but without any luck. They did not even get set, not sure what I did wrong.

Also tried sysctl dev.em.X.rx_int_delay/dev.em.X.tx_int_delay without any noticable difference.

Not sure where I should go from here. Maybe disabling acpi/apic?</intel(r)></intel(r)></intel(r)></intel(r)>