High interrupt load when VGA cable (un)plugged
-
The KVM switch was here when I arrived this morning. Here are the results:
- Using just the VGA portion of the smart switch DID NOT solve the problem.
- Using just the KEYBOARD portion of the switch DID solve the problem.
Conclusion: Whatever FreeBSD does when it detects a monitor attach/detach caused an interrupt storm on IRQ16 UNLESS a PS/2 keyboard was attached… probably an unhandled interrupt due to a faulty assumption that a PS/2 keyboard driver is always present.
Solutions:
- Do nothing....Just reboot after doing any console work (if acceptable)
- or Leave a PS/2 keyboard always attached
- or Leave a VGA monitor always attached
- or Use a KVM switch that emulates PS/2 keyboards on all ports - even when not selected.
For reference: This system is:
pfSense: 2.1 Nano-4G-VGA-i386
motherboard: GIGABYTE C1007UN-D.
processor: Dual Core embedded Celeron (1007u) at 1.5GHz.
graphics: Intel integrated
chipset: NM70
storage: 64G Sandisk SSD
total power usage: 18 watts
KVM: StatTech SV411K ($46 USD with all cables).It appears that the graphics and ehci0 share interrupt 16 (the one experiencing the interrupt storm.)
-
I have a very similar setup. Same hardware. The difference is I'm using the 64bit nano image instead.
I experienced this same issue but I wasn't sure if it was the keyboard (USB) or the VGA being disconnected that would trigger it.
My workaround is to never use the VGA console or attach a keyboard. If I power up the system in this manner, there is no interrupt storm. If you still need the console, there is the serial port which should have the same availability as VGA or SSH. These may be suitable alternatives.
-
My workaround is to never use the VGA console or attach a keyboard. If I power up the system in this manner, there is no interrupt storm. If you still need the console, there is the serial port which should have the same availability as VGA or SSH. These may be suitable alternatives.
Since I have the VGA monitor and keyboard within reach for the other 3 servers, the KVM makes sense for me. Otherwise, I would probably just do a reboot (or schedule one for the middle of the night) after doing any console work. After all, I don't really expect to be at the console often (if at all.) The GUI and SSH access works well.
So-far-so-good using the KVM; still no interrupt storms despite all of my initial set-up playing.
-
Thanks for reporting. I do also experience interrupt storms and "random" reboots on pfSense-2.1-RELEASE-4g-amd64-nanobsd_vga when no (USB) keyboard is attached.
The interrupt storm in progress:
systat -iostat 1 /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average |||| /0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100 cpu user| nice| system| interrupt|XXXXXXXX idle|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXtop -SH last pid: 73849; load averages: 0.83, 0.79, 0.76 up 0+05:57:14 23:17:11 147 processes: 6 running, 114 sleeping, 27 waiting CPU: 0.0% user, 0.0% nice, 0.0% system, 19.0% interrupt, 81.0% idle Mem: 109M Active, 38M Inact, 116M Wired, 1980K Cache, 88M Buf, 1576M Free Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 64K CPU3 3 356:24 100.00% idle{idle: cpu3} 10 root 171 ki31 0K 64K CPU1 1 355:13 100.00% idle{idle: cpu1} 10 root 171 ki31 0K 64K RUN 2 354:34 100.00% idle{idle: cpu2} 11 root -64 - 0K 448K CPU0 0 53:05 77.98% intr{irq16: ehci0} 10 root 171 ki31 0K 64K CPU0 0 302:51 26.95% idle{idle: cpu0} 294 root 76 20 6908K 1380K kqread 0 2:49 0.00% check_reload_status 11 root -32 - 0K 448K WAIT 2 0:46 0.00% intr{swi4: clock} 0 root -16 0 0K 240K sched 1 0:44 0.00% kernel{swapper} 36564 root 76 0 144M 47440K accept 2 0:09 0.00% php{php} 14 root -40 - 0K 160K - 2 0:04 0.00% usb{usbus1} 11 root -64 - 0K 448K WAIT 3 0:03 0.00% intr{irq23: ehci1} 14 root -40 - 0K 160K - 1 0:02 0.00% usb{axe0} 14 root -44 - 0K 160K - 1 0:01 0.00% usb{usbus1} 13 root -16 - 0K 16K - 0 0:00 0.00% yarrow 29789 root 44 0 24220K 4160K kqread 1 0:00 0.00% lighttpd 14 root -40 - 0K 160K - 3 0:00 0.00% usb{usbus1} 11 root -32 - 0K 448K WAIT 1 0:00 0.00% intr{swi4: clock} 14 root -40 - 0K 160K - 2 0:00 0.00% usb{usbus0}systat -vmstat 1 2 users Load 0.67 0.77 0.75 Feb 3 23:16 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156300 18300 1020328 23724 1615532 count All 189512 23004 1074838k 43000 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 158k total 63 313k 4 407 156k 119 zfod 156k ehci0 16 ozfod 33 ehci1 23 0.0%Sys 19.0%Intr 0.0%User 0.0%Nice 81.0%Idle %ozfod 400 cpu0: time | | | | | | | | | | | daefr igb1:que 0 +++++++++ prcfr igb1:que 1 47 dtbuf totfr 5 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 400 cpu1: time intrn 400 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 400 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111696 act tps 0 0 0 0 0 0 39408 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613552 free 89872 bufI am now going to see what happens when the HDMI converter is unplugged. Maybe disable the Intel IGD in BIOS.
-
That's pretty special. :P
Does the bios have a way to remap interrupts? I remember seeing a setting like that…
-
The Intel Desktop Board DPPTCK - BIOS Version 0046 - 09/27/2013 configuration has these settings:
| » USB | |
| | |
| USB Legacy | <enabled></enabled> |
| | |
| USB Port 01: | <enable></enable> |
| USB Port 02: | <enable></enable> |
| USB Port 03: | <disable></disable> |
| USB Port 04: | <disable></disable> |
| USB Port 05: | <enable></enable> |
| | |
| » Thunderbolt Controller | <disabled></disabled> |
| Thunderbolt PCIe Cache-line Size | <32> |
| Security Level | <normal mode="" w="" o="" nhi=""></normal> |
| SMI/Notify Support | <enabled></enabled> |
| SwSMI Support | <enabled></enabled> |
| Notify Support | <enabled></enabled> |
| Thunderbolt Surprise-Removal | <disabled></disabled> |
| Thunderbolt Wake Delay | 2500 |
| Thunderbolt SwSMI Delay | 10 |
| | |
| Num Lock | <enabled></enabled> |
| | |
| » SATA Drives | |
| | |
| Chipset SATA Controller Configuration | |
| | |
| Chipset SATA Mode | <ahci></ahci> |
| mSATA Port | [Not Installed] |
| | |
| Hard Disk Pre-Delay | 0 |
| | |
| » Video | |
| | |
| Integrated Graphics Device | <always enable=""></always> |
| | |
| IGD DVMT Memory | <32 MB> |
| IGD Primary Video Port | |
| IGD Secondary Video Port | |
| Intel Graphics Performance Analyzers | <disabled></disabled> |
| | |
| » PCI/PCIe Add-In Slots | |
| | |
| » Slot FMC mini PCIE | Using: x1 Speed: 5.0…. |
| Mini PCIe Config Map Slot FMC | || 00 | 02 | 04 | 06 | 08 | 0A | 0C | 0E |
–-+------+------+------+------+------+------+------+------+
00 | 8086 | 1521 | 0000 | 0010 | 0001 | 0200 | 0010 | 0080 |
10 | 0000 | F7D8 | 0000 | 0000 | E021 | 0000 | 4000 | F7E0 |
20 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | FFFF | 0000 |
30 | 0000 | F7D0 | 0040 | 0000 | 0000 | 0000 | 0104 | 0000 |
40 | 5001 | C823 | 2008 | 0000 | 0000 | 0000 | 0000 | 0000 |
50 | 7005 | 0180 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
60 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
70 | A011 | 0009 | 0003 | 0000 | 2003 | 0000 | 0000 | 0000 |
80 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
90 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | FFFF | FFFF |
A0 | 0010 | 0002 | 8CC2 | 1000 | 2000 | 0019 | EC42 | EC42 |
B0 | 0040 | 1012 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
C0 | 0000 | 0000 | 081F | 0000 | 0000 | 0000 | 0000 | 0000 |
D0 | 0002 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
E0 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
F0 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 | 0000 |
---+------+------+------+------+------+------+------+------+| PCI Vendor ID:Device ID | <8086:1521> |
| | |
| » Slot HMC mini PCIE | Not Populated |
| | |
| PCI Latency Timer | <248> |
| | |
| » Power | |
| | |
| Intel Dynamic Power Technology | |
| Enhanced Intel SpeedStep Technology | <enabled></enabled> |
| Processor Power Efficiency Policy | |
| OS ACPI C2 Report | <enabled></enabled> |
| OS ACPI C3 Report | <enabled></enabled> |
| | |
| System Power Options | |
| | |
| Intel Rapid Start Technology | <disabled></disabled> |
| Hibernation Timer | <immediate></immediate> |
| Intel Smart Connect Technology | <disabled></disabled> |
| After Power Failure | |
| Deep S4/S5 | <disabled></disabled> |
| Wake on LAN from S4/S5 | |
| S3 State Indicator | |
| Wake System from S5 | <enabled></enabled> |
| Wakeup Date | 0 |
| Wakeup Minute | 0 |
| Wakeup Second | 0 |
| PCIe ASPM Support | <enable></enable> |
| Native ACPI OS PCIe Support | <enabled></enabled> |Suggestions what (and why) to change in BIOS are welcome…
-
Some more details:
$ dmesg | grep -i "irq 16" vgapci0: <vga-compatible display=""> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 ehci0: <ehci (generic)="" usb="" 2.0="" controller=""> mem 0xf7f08000-0xf7f083ff irq 16 at device 26.0 on pci0 pcib1: <acpi pci-pci="" bridge=""> irq 16 at device 28.0 on pci0 pcib3: <acpi pci-pci="" bridge=""> irq 16 at device 28.4 on pci0</acpi></acpi></ehci></vga-compatible>$ pciconf -l hostb0@pci0:0:0:0: class=0x060000 card=0x20448086 chip=0x01548086 rev=0x09 hdr=0x00 vgapci0@pci0:0:2:0: class=0x030000 card=0x20448086 chip=0x01668086 rev=0x09 hdr=0x00 none0@pci0:0:22:0: class=0x078000 card=0x20448086 chip=0x1e3a8086 rev=0x04 hdr=0x00 ehci0@pci0:0:26:0: class=0x0c0320 card=0x20448086 chip=0x1e2d8086 rev=0x04 hdr=0x00 none1@pci0:0:27:0: class=0x040300 card=0x20448086 chip=0x1e208086 rev=0x04 hdr=0x00 pcib1@pci0:0:28:0: class=0x060400 card=0x20448086 chip=0x1e108086 rev=0xc4 hdr=0x01 pcib2@pci0:0:28:1: class=0x060400 card=0x20448086 chip=0x1e128086 rev=0xc4 hdr=0x01 pcib3@pci0:0:28:4: class=0x060400 card=0x20448086 chip=0x1e188086 rev=0xc4 hdr=0x01 ehci1@pci0:0:29:0: class=0x0c0320 card=0x20448086 chip=0x1e268086 rev=0x04 hdr=0x00 isab0@pci0:0:31:0: class=0x060100 card=0x20448086 chip=0x1e568086 rev=0x04 hdr=0x00 atapci0@pci0:0:31:2: class=0x010601 card=0x20448086 chip=0x1e038086 rev=0x04 hdr=0x00 none2@pci0:0:31:3: class=0x0c0500 card=0x20448086 chip=0x1e228086 rev=0x04 hdr=0x00 igb0@pci0:2:0:0: class=0x020000 card=0x0000ffff chip=0x15218086 rev=0x01 hdr=0x00 igb1@pci0:2:0:1: class=0x020000 card=0x0000ffff chip=0x15218086 rev=0x01 hdr=0x00$ dmesg | grep -i "warning" ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, truncating length 268 to 244 (20101013/tbfadt-392) atrtc0: Warning: Couldn't map I/O.$ vmstat -i interrupt total rate irq16: ehci0 2775 1 irq23: ehci1 71066 38 cpu0: timer 730136 399 irq261: igb1:que 0 652 0 irq262: igb1:que 1 134 0 irq263: igb1:que 2 91 0 irq264: igb1:que 3 205 0 irq265: igb1:link 2 0 cpu1: timer 730130 399 cpu2: timer 730130 399 cpu3: timer 730130 399 Total 2995451 1640A few minutes after unplugging the HDMI connector:
$ vmstat -i interrupt total rate irq16: ehci0 53844356 24352 irq23: ehci1 82074 37 cpu0: timer 884004 399 irq261: igb1:que 0 737 0 irq262: igb1:que 1 172 0 irq263: igb1:que 2 122 0 irq264: igb1:que 3 235 0 irq265: igb1:link 2 0 cpu1: timer 883998 399 cpu2: timer 883998 399 cpu3: timer 883998 399 Total 57463696 25989And a few minutes later the interrupt rate on ehci0 / irq16 has even doubled:
$ vmstat -i interrupt total rate irq16: ehci0 163276384 56050 irq23: ehci1 102179 35 cpu0: timer 1165015 399 irq261: igb1:que 0 869 0 irq262: igb1:que 1 235 0 irq263: igb1:que 2 221 0 irq264: igb1:que 3 238 0 irq265: igb1:link 2 0 cpu1: timer 1165009 399 cpu2: timer 1165008 399 cpu3: timer 1165008 399 Total 168040168 57686$ pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x20448086 chip=0x01548086 rev=0x09 hdr=0x00 class = bridge subclass = HOST-PCI vgapci0@pci0:0:2:0: class=0x030000 card=0x20448086 chip=0x01668086 rev=0x09 hdr=0x00 class = display subclass = VGA none0@pci0:0:22:0: class=0x078000 card=0x20448086 chip=0x1e3a8086 rev=0x04 hdr=0x00 class = simple comms ehci0@pci0:0:26:0: class=0x0c0320 card=0x20448086 chip=0x1e2d8086 rev=0x04 hdr=0x00 class = serial bus subclass = USB none1@pci0:0:27:0: class=0x040300 card=0x20448086 chip=0x1e208086 rev=0x04 hdr=0x00 class = multimedia subclass = HDA pcib1@pci0:0:28:0: class=0x060400 card=0x20448086 chip=0x1e108086 rev=0xc4 hdr=0x01 class = bridge subclass = PCI-PCI pcib2@pci0:0:28:1: class=0x060400 card=0x20448086 chip=0x1e128086 rev=0xc4 hdr=0x01 class = bridge subclass = PCI-PCI pcib3@pci0:0:28:4: class=0x060400 card=0x20448086 chip=0x1e188086 rev=0xc4 hdr=0x01 class = bridge subclass = PCI-PCI ehci1@pci0:0:29:0: class=0x0c0320 card=0x20448086 chip=0x1e268086 rev=0x04 hdr=0x00 class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x20448086 chip=0x1e568086 rev=0x04 hdr=0x00 class = bridge subclass = PCI-ISA atapci0@pci0:0:31:2: class=0x010601 card=0x20448086 chip=0x1e038086 rev=0x04 hdr=0x00 class = mass storage subclass = SATA none2@pci0:0:31:3: class=0x0c0500 card=0x20448086 chip=0x1e228086 rev=0x04 hdr=0x00 class = serial bus subclass = SMBus igb0@pci0:2:0:0: class=0x020000 card=0x0000ffff chip=0x15218086 rev=0x01 hdr=0x00 class = network subclass = ethernet igb1@pci0:2:0:1: class=0x020000 card=0x0000ffff chip=0x15218086 rev=0x01 hdr=0x00 class = network subclass = ethernet -
I'm quickly making a response because I have somewhere I have to be, but I also just unplugged my monitor and had this happen
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 64K CPU2 2 571.4H 100.00% [idle{idle: cpu2}]
11 root 171 ki31 0K 64K CPU0 0 570.7H 100.00% [idle{idle: cpu0}]
11 root 171 ki31 0K 64K RUN 1 568.2H 100.00% [idle{idle: cpu1}]
12 root -64 - 0K 448K CPU3 3 5:02 53.86% [intr{irq16: ehci0}]
11 root 171 ki31 0K 64K CPU3 3 574.3H 48.68% [idle{idle: cpu3}]Luckily I have a quad core and I had plenty CPU to spare. I just left my monitor unplugged and rebooted for now.
-
Sounds like standard broken BIOS (ACPI). Is this the latest one? Also, you might try your luck with a 2.2 snapshot, except that they've already been wiped. ::) >:(
Other than that
- Doctor, it hurts when I do this.
– So don't do that.
- Doctor, it hurts when I do this.
-
Sounds like standard broken BIOS (ACPI). Is this the latest one? Also, you might try your luck with a 2.2 snapshot, except that they've already been wiped. ::) >:(
Other than that
- Doctor, it hurts when I do this.
– So don't do that.
Of the many places a bug could occur, unplugging a monitor from a "headless" unit is a great bug to have. I'm just going to wait for 2.2 stable, then try it out again.
Thanks for the reply.
BTW, if anyone else reads this and is curious, I'm using a MSI B85I LGA 1150 Intel B85 HDMI with the built in Haswell GPU. I flashed the most recent BIOS version from around 1.5 months ago.
- Doctor, it hurts when I do this.
-
I'm just going to wait for 2.2 stable, then try it out again.
Might take quite some time. Meanwhile, snaps seem to be back.