Interrupt storm detected on "irq259:"; throttling interrupt source
-
I am running PFsense 2.2.6 on FreeBSD 10.1 Release p25 on a Supermicro box with 2 onboard NICs and 1 additional Intel NIC.
The firewall seems to freeze once in every 15 or 20 days, I've to then reboot the firewall to get it going, but when it is frozen I see the following message on the screen
"interrupt storm detected on "irq259:"; throttling interrupt source"IRQ 259 is asigned to one of my onboard NIC, below are my vmstat and dmesg output. Any help in resolving this would be greatly appreciated.
$ vmstat -i
interrupt total rate
irq18: ehci0 uhci5 2 0
irq19: uhci2 uhci4* 2972666 21
irq20: hpet0 155048624 1123
irq23: uhci3 ehci1 23 0
irq256: em0:rx 0 710649926 5149
irq257: em0:tx 0 683697025 4953
irq258: em0:link 59 0
irq259: em1:rx 0 582254163 4218
irq260: em1:tx 0 803087670 5819
irq261: em1:link 3423 0
Total 2937713581 21286$ dmesg
Copyright
1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.1-RELEASE-p25 #0 c39b63e(releng/10.1)-dirty: Mon Dec 21 15:20:13 CST 2015
root@pfs22-amd64-builder:/usr/obj.RELENG_2_2.amd64/usr/pfSensesrc/src.RELENG_2_2/sys/pfSense_SMP.10 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Atom(TM) CPU D510 @ 1.66GHz (1666.70-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106ca Family = 0x6 Model = 0x1c Stepping = 10
Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 4097843200 (3908 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <121710 APIC1048>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP/HT): APIC ID: 3
ioapic0: Changing APIC ID to 4
ioapic0 <version 2.0="">irqs 0-23 on motherboard
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806345e0, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80634690, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80634740, 0) error 1
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8060ce10, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8060cec0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8060cf70, 0) error 1
random: <software, yarrow="">initialized
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <smci>on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
uhci0: <intel 82801i="" (ich9)="" usb="" controller="">port 0xbc00-0xbc1f irq 16 at device 26.0 on pci0
usbus0 on uhci0
uhci1: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb880-0xb89f irq 21 at device 26.1 on pci0
usbus1 on uhci1
uhci2: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb800-0xb81f irq 19 at device 26.2 on pci0
usbus2 on uhci2
ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfebfbc00-0xfebfbfff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib1: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xcc00-0xcc1f mem 0xfe820000-0xfe83ffff,0xfe880000-0xfe8fffff,0xfe81c000-0xfe81ffff irq 16 at device 0.0 on pci1
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 68:05:ca:3a:ed:17
pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
pci2: <acpi pci="" bus="">on pcib2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xdc00-0xdc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 16 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:32:2b:2c
pcib3: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
pci3: <acpi pci="" bus="">on pcib3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xec00-0xec1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 17 at device 0.0 on pci3
em2: Using MSIX interrupts with 3 vectors
em2: Ethernet address: 00:25:90:32:2b:2d
uhci3: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb480-0xb49f irq 23 at device 29.0 on pci0
usbus4 on uhci3
uhci4: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb400-0xb41f irq 19 at device 29.1 on pci0
usbus5 on uhci4
uhci5: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb080-0xb09f irq 18 at device 29.2 on pci0
usbus6 on uhci5
ehci1: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfebfb800-0xfebfbbff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib4: <acpi pci-pci="" bridge="">at device 30.0 on pci0
pci4: <acpi pci="" bus="">on pcib4
vgapci0: <vga-compatible display="">mem 0xfc000000-0xfcffffff,0xfdffc000-0xfdffffff,0xfe000000-0xfe7fffff irq 17 at device 4.0 on pci4
vgapci0: Boot video device
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel ich9="" sata300="" controller="">port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f,0xa400-0xa40f irq 19 at device 31.2 on pci0
ata2: <ata channel="">at channel 0 on atapci0
ata3: <ata channel="">at channel 1 on atapci0
atapci1: <intel ich9="" sata300="" controller="">port 0xa000-0xa007,0x9c00-0x9c03,0x9880-0x9887,0x9800-0x9803,0x9480-0x948f,0x9400-0x940f irq 19 at device 31.5 on pci0
ata4: <ata channel="">at channel 0 on atapci1
ata5: <ata channel="">at channel 1 on atapci1
acpi_button0: <power button="">on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0
orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen1.1: <intel>at usbus1
uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
ugen0.1: <intel>at usbus0
uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
ugen3.1: <intel>at usbus3
uhub2: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
ugen2.1: <intel>at usbus2
uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
ugen5.1: <intel>at usbus5
uhub4: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus5
ugen4.1: <intel>at usbus4
uhub5: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus4
ugen6.1: <intel>at usbus6
uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus6
ugen7.1: <intel>at usbus7
uhub7: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub7: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
ugen2.2: <winbond electronics="" corp="">at usbus2
ukbd0: <winbond 0="" 2="" electronics="" corp="" hermon="" usb="" hidmouse="" device,="" class="" 0,="" rev="" 1.10="" 0.01,="" addr="">on usbus2
kbd2 at ukbd0
ugen4.2: <aten>at usbus4
ukbd1: <hid keyboard="">on usbus4
kbd3 at ukbd1
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <adata sp800="" 5.0.2a="">ATA-8 SATA 2.x device
ada0: Serial Number 03314125500100000031
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 30533MB (62533296 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC" frequency 1666701130 Hz quality 1000
Trying to mount root from ufs:/dev/ufsid/569c75c6c0ba5910 [rw]…
WARNING: / was not properly dismounted
vlan0: changing name to 'em1_vlan900'
vlan1: changing name to 'em1_vlan903'
tun1: changing name to 'ovpns1'
pflog0: promiscuous mode enabled
ovpns1: link state changed to UP
em0: link state changed to UP
em1: link state changed to UP
em1_vlan903: link state changed to UP
em1_vlan900: link state changed to UP
em1: promiscuous mode enabled
em1_vlan900: promiscuous mode enabled
em0: promiscuous mode enabled
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source
interrupt storm detected on "irq259:"; throttling interrupt source</adata></hid></aten></winbond></winbond></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></cpu></cpu></cpu></cpu></at></keyboard></generic></system></isa></power></ata></ata></intel></ata></ata></intel></isa></pci-isa></vga-compatible></acpi></acpi></intel></intel></intel></intel></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel></intel></intel></intel></acpi></acpi></high></at></at></acpi></acpi></acpi></acpi></smci></software></software,></version></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
interrupt storm detected on "irq259:"; throttling interrupt source
I experienced a teardrop attack in my internal network which caused my machine to act this way. Found it with softflowd and PRTG. Fixed the offending server ability to tko the firewall by removing the server's default gateway.