Performance issues and interrupt storm with em driver [RC2 AMD64]
-
Hello!
I have built a router/firewall with the following hardware:
*Supermicro X7SPA-HF (dual on board intel nics (82574L), atom d510)
*Addon dual port Intel PRO/1000 PTThe on board nics are used for local traffic, and are bonded as lagg0. The addon dual port card are used for two wan connections.
I'm getting around 300Mbit/s throughput, with one cpu core handling all the interrupts. The performance is ok but considering only one cpu core is in use I got a feeling it can be tuned a bit. One concern I have is that the machine died yesterday, displaying only "interrupt storm on irq 258" on the console. Seems like something ain't quite right.
vmstat -i:
interrupt total rate
irq1: atkbd0 5 0
irq18: ehci0 uhci5 2 0
irq19: uhci2 uhci4+ 29328 2
cpu0: timer 25348462 1978
irq256: em0 5645078 440
irq257: em1 951927 74
irq258: em2:rx 0 1150847 89
irq259: em2:tx 0 2969097 231
irq261: em3:rx 0 1464717 114
irq262: em3:tx 0 2274026 177
cpu1: timer 25348338 1978
cpu3: timer 25348338 1978
cpu2: timer 25348337 1978
Total 115878502 9043dmesg:
em0: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xcc00-0xcc1f mem 0xfe8e0000-0xfe8fffff,0xfe8c0000-0xfe8dffff irq 16 at device 0.0 on pci1
em0: Using an MSI interrupt
em0: [FILTER]
em1: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xc880-0xc89f mem 0xfe8a0000-0xfe8bffff,0xfe880000-0xfe89ffff irq 17 at device 0.1 on pci1
em1: Using an MSI interrupt
em1: [FILTER]
em2: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xdc00-0xdc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2
em2: Using MSIX interrupts with 3 vectors
em2: [ITHREAD]
em2: [ITHREAD]
em2: [ITHREAD]
em3: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xec00-0xec1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3
em3: Using MSIX interrupts with 3 vectors
em3: [ITHREAD]
em3: [ITHREAD]
em3: [ITHREAD]Em2 and em3 are the integrated nics, which apparantly need both tx and rx interrupts?
I have tried messing around with kernel parameters but not really getting any noticable gains. A lot of what I have found searching the interwebs seems to be outdated information.
sysctl net.isr
net.isr.numthreads: 1
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.direct: 1
net.isr.direct_force: 1I have tried making changes to these tunables by adding them to the system->advanced->system tunables list but without any luck. They did not even get set, not sure what I did wrong.
Also tried sysctl dev.em.X.rx_int_delay/dev.em.X.tx_int_delay without any noticable difference.
Not sure where I should go from here. Maybe disabling acpi/apic?</intel(r)></intel(r)></intel(r)></intel(r)>