Interrupt storm detected on "irq259:"; throttling interrupt source



  • I am running PFsense 2.2.6 on FreeBSD 10.1 Release p25 on a Supermicro box with 2 onboard NICs and 1 additional Intel NIC.
    The firewall seems to freeze once in every 15 or 20 days, I've to then reboot the firewall to get it going, but when it is frozen I see the following message on the screen
    "interrupt storm detected on "irq259:"; throttling interrupt source"

    IRQ 259 is asigned to one of my onboard NIC, below are my vmstat and dmesg output. Any help in resolving this would be greatly appreciated.

    $ vmstat -i
    interrupt                          total      rate
    irq18: ehci0 uhci5                    2          0
    irq19: uhci2 uhci4*              2972666        21
    irq20: hpet0                  155048624      1123
    irq23: uhci3 ehci1                    23          0
    irq256: em0:rx 0              710649926      5149
    irq257: em0:tx 0              683697025      4953
    irq258: em0:link                      59          0
    irq259: em1:rx 0              582254163      4218
    irq260: em1:tx 0              803087670      5819
    irq261: em1:link                    3423          0
    Total                        2937713581      21286

    $ dmesg
    Copyright © 1992-2014 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.1-RELEASE-p25 #0 c39b63e(releng/10.1)-dirty: Mon Dec 21 15:20:13 CST 2015
        root@pfs22-amd64-builder:/usr/obj.RELENG_2_2.amd64/usr/pfSensesrc/src.RELENG_2_2/sys/pfSense_SMP.10 amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Atom(TM) CPU D510  @ 1.66GHz (1666.70-MHz K8-class CPU)
      Origin = "GenuineIntel"  Id = 0x106ca  Family = 0x6  Model = 0x1c  Stepping = 10
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant, performance statistics
    real memory  = 4294967296 (4096 MB)
    avail memory = 4097843200 (3908 MB)
    Event timer "LAPIC" quality 400
    ACPI APIC Table: <121710 APIC1048>
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
    cpu0 (BSP): APIC ID:  0
    cpu1 (AP/HT): APIC ID:  1
    cpu2 (AP): APIC ID:  2
    cpu3 (AP/HT): APIC ID:  3
    ioapic0: Changing APIC ID to 4
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806345e0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80634690, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80634740, 0) error 1
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8060ce10, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8060cec0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8060cf70, 0) error 1
    random: <software, yarrow="">initialized
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <smci>on motherboard
    acpi0: Power Button (fixed)
    acpi0: reservation of fee00000, 1000 (3) failed
    acpi0: reservation of 0, a0000 (3) failed
    acpi0: reservation of 100000, bff00000 (3) failed
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 450
    Event timer "HPET1" frequency 14318180 Hz quality 440
    Event timer "HPET2" frequency 14318180 Hz quality 440
    Event timer "HPET3" frequency 14318180 Hz quality 440
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    uhci0: <intel 82801i="" (ich9)="" usb="" controller="">port 0xbc00-0xbc1f irq 16 at device 26.0 on pci0
    usbus0 on uhci0
    uhci1: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb880-0xb89f irq 21 at device 26.1 on pci0
    usbus1 on uhci1
    uhci2: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb800-0xb81f irq 19 at device 26.2 on pci0
    usbus2 on uhci2
    ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfebfbc00-0xfebfbfff irq 18 at device 26.7 on pci0
    usbus3: EHCI version 1.0
    usbus3 on ehci0
    pcib1: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xcc00-0xcc1f mem 0xfe820000-0xfe83ffff,0xfe880000-0xfe8fffff,0xfe81c000-0xfe81ffff irq 16 at device 0.0 on pci1
    em0: Using MSIX interrupts with 3 vectors
    em0: Ethernet address: 68:05:ca:3a:ed:17
    pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xdc00-0xdc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2
    em1: Using MSIX interrupts with 3 vectors
    em1: Ethernet address: 00:25:90:32:2b:2c
    pcib3: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xec00-0xec1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3
    em2: Using MSIX interrupts with 3 vectors
    em2: Ethernet address: 00:25:90:32:2b:2d
    uhci3: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb480-0xb49f irq 23 at device 29.0 on pci0
    usbus4 on uhci3
    uhci4: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb400-0xb41f irq 19 at device 29.1 on pci0
    usbus5 on uhci4
    uhci5: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb080-0xb09f irq 18 at device 29.2 on pci0
    usbus6 on uhci5
    ehci1: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfebfb800-0xfebfbbff irq 23 at device 29.7 on pci0
    usbus7: EHCI version 1.0
    usbus7 on ehci1
    pcib4: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    vgapci0: <vga-compatible display="">mem 0xfc000000-0xfcffffff,0xfdffc000-0xfdffffff,0xfe000000-0xfe7fffff irq 17 at device 4.0 on pci4
    vgapci0: Boot video device
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel ich9="" sata300="" controller="">port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f,0xa400-0xa40f irq 19 at device 31.2 on pci0
    ata2: <ata channel="">at channel 0 on atapci0
    ata3: <ata channel="">at channel 1 on atapci0
    atapci1: <intel ich9="" sata300="" controller="">port 0xa000-0xa007,0x9c00-0x9c03,0x9880-0x9887,0x9800-0x9803,0x9480-0x948f,0x9400-0x940f irq 19 at device 31.5 on pci0
    ata4: <ata channel="">at channel 0 on atapci1
    ata5: <ata channel="">at channel 1 on atapci1
    acpi_button0: <power button="">on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0
    orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    ppc0: cannot reserve I/O port range
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    random: unblocking device.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 12Mbps Full Speed USB v1.0
    usbus3: 480Mbps High Speed USB v2.0
    usbus4: 12Mbps Full Speed USB v1.0
    usbus5: 12Mbps Full Speed USB v1.0
    usbus6: 12Mbps Full Speed USB v1.0
    usbus7: 480Mbps High Speed USB v2.0
    ugen1.1: <intel>at usbus1
    uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
    ugen0.1: <intel>at usbus0
    uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
    ugen3.1: <intel>at usbus3
    uhub2: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
    ugen2.1: <intel>at usbus2
    uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
    ugen5.1: <intel>at usbus5
    uhub4: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus5
    ugen4.1: <intel>at usbus4
    uhub5: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus4
    ugen6.1: <intel>at usbus6
    uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus6
    ugen7.1: <intel>at usbus7
    uhub7: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus7
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    uhub3: 2 ports with 2 removable, self powered
    uhub4: 2 ports with 2 removable, self powered
    uhub5: 2 ports with 2 removable, self powered
    uhub6: 2 ports with 2 removable, self powered
    uhub7: 6 ports with 6 removable, self powered
    uhub2: 6 ports with 6 removable, self powered
    ugen2.2: <winbond electronics="" corp="">at usbus2
    ukbd0: <winbond 0="" 2="" electronics="" corp="" hermon="" usb="" hidmouse="" device,="" class="" 0,="" rev="" 1.10="" 0.01,="" addr="">on usbus2
    kbd2 at ukbd0
    ugen4.2: <aten>at usbus4
    ukbd1: <hid keyboard="">on usbus4
    kbd3 at ukbd1
    ada0 at ata2 bus 0 scbus0 target 0 lun 0
    ada0: <adata sp800="" 5.0.2a="">ATA-8 SATA 2.x device
    ada0: Serial Number 03314125500100000031
    ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
    ada0: 30533MB (62533296 512 byte sectors: 16H 63S/T 16383C)
    ada0: Previously was known as ad4
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    Timecounter "TSC" frequency 1666701130 Hz quality 1000
    Trying to mount root from ufs:/dev/ufsid/569c75c6c0ba5910 [rw]…
    WARNING: / was not properly dismounted
    vlan0: changing name to 'em1_vlan900'
    vlan1: changing name to 'em1_vlan903'
    tun1: changing name to 'ovpns1'
    pflog0: promiscuous mode enabled
    ovpns1: link state changed to UP
    em0: link state changed to UP
    em1: link state changed to UP
    em1_vlan903: link state changed to UP
    em1_vlan900: link state changed to UP
    em1: promiscuous mode enabled
    em1_vlan900: promiscuous mode enabled
    em0: promiscuous mode enabled
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source
    interrupt storm detected on "irq259:"; throttling interrupt source</adata></hid></aten></winbond></winbond></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></cpu></cpu></cpu></cpu></at></keyboard></generic></system></isa></power></ata></ata></intel></ata></ata></intel></isa></pci-isa></vga-compatible></acpi></acpi></intel></intel></intel></intel></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel></intel></intel></intel></acpi></acpi></high></at></at></acpi></acpi></acpi></acpi></smci></software></software,></version></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>



  • @Pawandev:

    interrupt storm detected on "irq259:"; throttling interrupt source

    I experienced a teardrop attack in my internal network which caused my machine to act this way. Found it with softflowd and PRTG. Fixed the offending server ability to tko the firewall by removing the server's  default gateway.