Performance issues and interrupt storm with em driver [RC2 AMD64]

  • Hello!

    I have built a router/firewall with the following hardware:
    *Supermicro X7SPA-HF (dual on board intel nics (82574L), atom d510)
    *Addon dual port Intel® PRO/1000 PT

    The on board nics are used for local traffic, and are bonded as lagg0. The addon dual port card are used for two wan connections.

    I'm getting around 300Mbit/s throughput, with one cpu core handling all the interrupts. The performance is ok but considering only one cpu core is in use I got a feeling it can be tuned a bit. One concern I have is that the machine died yesterday, displaying only "interrupt storm on irq 258" on the console. Seems like something ain't quite right.

    vmstat -i:
    interrupt                          total       rate
    irq1: atkbd0                           5          0
    irq18: ehci0 uhci5                     2          0
    irq19: uhci2 uhci4+                29328          2
    cpu0: timer                     25348462       1978
    irq256: em0                      5645078        440
    irq257: em1                       951927         74
    irq258: em2:rx 0                 1150847         89
    irq259: em2:tx 0                 2969097        231
    irq261: em3:rx 0                 1464717        114
    irq262: em3:tx 0                 2274026        177
    cpu1: timer                     25348338       1978
    cpu3: timer                     25348338       1978
    cpu2: timer                     25348337       1978
    Total                          115878502       9043


    em0: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xcc00-0xcc1f mem 0xfe8e0000-0xfe8fffff,0xfe8c0000-0xfe8dffff irq 16 at device 0.0 on pci1
    em0: Using an MSI interrupt
    em0: [FILTER]
    em1: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xc880-0xc89f mem 0xfe8a0000-0xfe8bffff,0xfe880000-0xfe89ffff irq 17 at device 0.1 on pci1
    em1: Using an MSI interrupt
    em1: [FILTER]
    em2: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xdc00-0xdc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2
    em2: Using MSIX interrupts with 3 vectors
    em2: [ITHREAD]
    em2: [ITHREAD]
    em2: [ITHREAD]
    em3: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xec00-0xec1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3
    em3: Using MSIX interrupts with 3 vectors
    em3: [ITHREAD]
    em3: [ITHREAD]
    em3: [ITHREAD]

    Em2 and em3 are the integrated nics, which apparantly need both tx and rx interrupts?

    I have tried messing around with kernel parameters but not really getting any noticable gains. A lot of what I have found searching the interwebs seems to be outdated information.

    sysctl net.isr
    net.isr.numthreads: 1
    net.isr.defaultqlimit: 256
    net.isr.maxqlimit: 10240
    net.isr.bindthreads: 0
    net.isr.maxthreads: 1 1
    net.isr.direct_force: 1

    I have tried making changes to these tunables by adding them to the system->advanced->system tunables list but without any luck. They did not even get set, not sure what I did wrong.

    Also tried sysctl dev.em.X.rx_int_delay/dev.em.X.tx_int_delay without any noticable difference.

    Not sure where I should go from here. Maybe disabling acpi/apic?</intel(r)></intel(r)></intel(r)></intel(r)>