{irq16: em1 ehci0} taking up 75% of cpu
-
@Bai:
I'm running 2.0 RC3 on an i3. Shortly after I boot it, one of the cpu cores gets a heavy load on it. Looking at the System Activity, it shows {irq16: em1 ehci0}
em1 is my WAN card. I'm not running any packages, and other than the high cpu activity, everything seems to be working.
I've seen this sort of thing happen when the BIOS provides incorrect interrupt data. For example, if the BIOS says a device interrupts on irq18 when it really interrupts on irq16 the driver's interrupt handler will get attached to irq18 but the real interrupt goes to irq16 where it doesn't get handled (because the correct handler doesn't get called) and the interrupt request doesn't get cleared until there is an irq18 interrupt and the correct handler is called. Do you have the latest BIOS?
@Bai:
I wonder if it made a difference that I installed pfSense using a usb cd-rom.
I think it is unlikely.
Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a
-
I've seen this sort of thing happen when the BIOS provides incorrect interrupt data. For example, if the BIOS says a device interrupts on irq18 when it really interrupts on irq16 the driver's interrupt handler will get attached to irq18 but the real interrupt goes to irq16 where it doesn't get handled (because the correct handler doesn't get called) and the interrupt request doesn't get cleared until there is an irq18 interrupt and the correct handler is called. Do you have the latest BIOS?
Yep.
I think it is unlikely.
I figured, but I thought I'd mention it just in case.
Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a
The names of a what?
-
@Bai:
Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a
The names of a what?
Sorry, just the output of the shell command pciconf -l -v; devinfo -r; vmstat -i will do.
-
Well, I turned off Legacy USB and USB 3 in the BIOS, and the problem seems to have gone away. It's now idling at a nice 39W. :)
Will the output of the commands still be worthwhile?
-
@Bai:
Will the output of the commands still be worthwhile?
No - seems like a USB related problem.
-
@Bai:
Will the output of the commands still be worthwhile?
No - seems like a USB related problem.
Actually, it looks like it's not. :( It just took a while before it came back. :(
Here's the output.
$ pciconf -l -v; devinfo -r; vmstat -i hostb0@pci0:0:0:0: class=0x060000 card=0x20038086 chip=0x01008086 rev=0x09 hdr=0x00 class = bridge subclass = HOST-PCI vgapci0@pci0:0:2:0: class=0x030000 card=0x20038086 chip=0x01028086 rev=0x09 hdr=0x00 class = display subclass = VGA none0@pci0:0:22:0: class=0x078000 card=0x20038086 chip=0x1c3a8086 rev=0x04 hdr=0x00 class = simple comms em0@pci0:0:25:0: class=0x020000 card=0x20038086 chip=0x15038086 rev=0x05 hdr=0x00 class = network subclass = ethernet ehci0@pci0:0:26:0: class=0x0c0320 card=0x20038086 chip=0x1c2d8086 rev=0x05 hdr=0x00 class = serial bus subclass = USB none1@pci0:0:27:0: class=0x040300 card=0x20038086 chip=0x1c208086 rev=0x05 hdr=0x00 class = multimedia subclass = HDA pcib1@pci0:0:28:0: class=0x060400 card=0x20038086 chip=0x1c108086 rev=0xb5 hdr=0x01 class = bridge subclass = PCI-PCI ehci1@pci0:0:29:0: class=0x0c0320 card=0x20038086 chip=0x1c268086 rev=0x05 hdr=0x00 class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 card=0x20038086 chip=0x1c4a8086 rev=0x05 hdr=0x00 class = bridge subclass = PCI-ISA atapci0@pci0:0:31:2: class=0x010601 card=0x20038086 chip=0x1c028086 rev=0x05 hdr=0x00 class = mass storage subclass = SATA none2@pci0:0:31:3: class=0x0c0500 card=0x20038086 chip=0x1c228086 rev=0x05 hdr=0x00 class = serial bus subclass = SMBus pcib2@pci0:1:0:0: class=0x060400 card=0x20038086 chip=0x88921283 rev=0x10 hdr=0x01 class = bridge subclass = PCI-PCI em1@pci0:2:0:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00 class = network subclass = ethernet em2@pci0:2:1:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00 class = network subclass = ethernet em3@pci0:2:2:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00 class = network subclass = ethernet nexus0 cryptosoft0 apic0 ram0 I/O memory addresses: 0x0-0x9cfff 0x100000-0x1fffffff 0x20200000-0x3fffffff 0x40200000-0xda969fff 0xdabc9000-0xdabc9fff 0xdae8e000-0xdaffffff npx0 acpi0 Interrupt request lines: 9 I/O ports: 0x10-0x1f 0x22-0x3f 0x44-0x5f 0x62-0x63 0x65-0x6f 0x72-0x7f 0x80 0x84-0x86 0x88 0x8c-0x8e 0x90-0x9f 0xa2-0xbf 0xe0-0xef 0x400-0x453 0x454-0x457 0x458-0x47f 0x4d0-0x4d1 0x500-0x57f 0x1180-0x119f I/O memory addresses: 0xf8000000-0xfbffffff 0xfec00000-0xfecfffff 0xfed08000-0xfed08fff 0xfed10000-0xfed19fff 0xfed1c000-0xfed1ffff 0xfed20000-0xfed3ffff 0xfed90000-0xfed93fff 0xfee00000-0xfee0ffff 0xff000000-0xffffffff cpu0 ACPI I/O ports: 0x414 0x415 acpi_perf0 est0 p4tcc0 cpufreq0 cpu1 ACPI I/O ports: 0x414 0x415 acpi_perf1 est1 p4tcc1 cpufreq1 cpu2 ACPI I/O ports: 0x414 0x415 acpi_perf2 est2 p4tcc2 cpufreq2 cpu3 ACPI I/O ports: 0x414 0x415 acpi_perf3 est3 p4tcc3 cpufreq3 pcib0 pci0 I/O ports: 0xf000-0xf03f 0xf040-0xf05f 0xf080-0xf09f 0xf0a0-0xf0a3 0xf0b0-0xf0b7 0xf0c0-0xf0c3 0xf0d0-0xf0d7 I/O memory addresses: 0xe0000000-0xefffffff 0xfe000000-0xfe3fffff 0xfe720000-0xfe723fff 0xfe724000-0xfe7240ff 0xfe729000-0xfe72900f hostb0 vgapci0 em0 Interrupt request lines: 256 I/O memory addresses: 0xfe700000-0xfe71ffff 0xfe728000-0xfe728fff ehci0 Interrupt request lines: 16 I/O memory addresses: 0xfe727000-0xfe7273ff usbus0 uhub0 uhub2 pcib1 pci1 pcib2 pci2 I/O memory addresses: 0xfe420000-0xfe43ffff 0xfe480000-0xfe49ffff 0xfe4e0000-0xfe4fffff em1 Interrupt request lines: 16 I/O ports: 0xe080-0xe0bf I/O memory addresses: 0xfe500000-0xfe51ffff em2 Interrupt request lines: 17 I/O ports: 0xe040-0xe07f I/O memory addresses: 0xfe4a0000-0xfe4bffff em3 Interrupt request lines: 18 I/O ports: 0xe000-0xe03f I/O memory addresses: 0xfe440000-0xfe45ffff ehci1 Interrupt request lines: 23 I/O memory addresses: 0xfe726000-0xfe7263ff usbus1 uhub1 uhub3 isab0 isa0 pmtimer0 sc0 vga0 I/O ports: 0x3c0-0x3df I/O memory addresses: 0xa0000-0xbffff ata0 Interrupt request lines: 14 I/O ports: 0x1f0-0x1f7 0x3f6 ata1 Interrupt request lines: 15 I/O ports: 0x170-0x177 0x376 atkbdc0 I/O ports: 0x60 0x64 atkbd0 Interrupt request lines: 1 atapci0 Interrupt request lines: 19 I/O ports: 0xf060-0xf07f I/O memory addresses: 0xfe725000-0xfe7257ff ata2 ad4 subdisk4 ata3 ata4 ata5 ata6 ata7 acpi_sysresource0 acpi_sysresource1 atpic0 atdma0 attimer0 atrtc0 Interrupt request lines: 8 I/O ports: 0x70-0x71 acpi_sysresource2 npxisa0 acpi_sysresource3 acpi_sysresource4 acpi_button0 pci_link0 pci_link1 pci_link2 pci_link3 pci_link4 pci_link5 pci_link6 pci_link7 acpi_sysresource5 acpi_sysresource6 acpi_timer0 ACPI I/O ports: 0x408-0x40b acpi_hpet0 I/O memory addresses: 0xfed00000-0xfed003ff interrupt total rate irq16: em1 ehci0 779815612 16681 irq17: em2 18172 0 irq18: em3 4435 0 irq19: atapci0 81729 1 irq23: ehci1 93211 1 cpu0: timer 93493755 2000 irq256: em0 604657 12 cpu1: timer 93493955 2000 cpu2: timer 93493973 2000 cpu3: timer 93494072 2000 Total 1154593571 24699 -
I think Steve's suggestion was that you disable USB2 and USB3. From what you said you disabled USB1 and USB3. Please try Steve's suggestion. USB keyboard and mouse should work fine with USB1.
-
I would try disabling 'high speed' with that loader tunable. You clearly don't need high speed, ehci, for a keyboard.
Steve
-
I think Steve's suggestion was that you disable USB2 and USB3. From what you said you disabled USB1 and USB3. Please try Steve's suggestion. USB keyboard and mouse should work fine with USB1.
Those were the only options in the BIOS for USB.
I would try disabling 'high speed' with that loader tunable. You clearly don't need high speed, ehci, for a keyboard.
Steve
I'll give that a shot. Which ones do you suggest? All three or just the last one?
-
I would probably try no_hs first then lost interupt.
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB. :-\Steve
-
I would probably try no_hs first then lost interupt.
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB. :-\Steve
No luck with no_hs. It ran fine for a little bit, but then quickly returned to it's previous state. Is there a way to make sure that it's recognizing the loader.conf.local?
And I don't really have a choice on using USB. I don't have any PS2 ports. :)
-
I would expect to be able to see the sysctl OIDs but all I see is:
[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb hw.usb.no_boot_wait: 0 hw.usb.debug: 0 hw.usb.usb_lang_mask: 255 hw.usb.usb_lang_id: 9 hw.usb.template: 0 hw.usb.power_timeout: 30 hw.usb.uath.regdomain: 0 hw.usb.uath.countrycode: 0 hw.usb.urtw.preamble_mode: 2 hw.usb.urtw.debug: 0 hw.usb.ucom.cons_baud: 9600 hw.usb.ucom.cons_unit: -1No EHCI at all.
Steve
-
I would expect to be able to see the sysctl OIDs but all I see is:
[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb hw.usb.no_boot_wait: 0 hw.usb.debug: 0 hw.usb.usb_lang_mask: 255 hw.usb.usb_lang_id: 9 hw.usb.template: 0 hw.usb.power_timeout: 30 hw.usb.uath.regdomain: 0 hw.usb.uath.countrycode: 0 hw.usb.urtw.preamble_mode: 2 hw.usb.urtw.debug: 0 hw.usb.ucom.cons_baud: 9600 hw.usb.ucom.cons_unit: -1No EHCI at all.
Steve
I get hw.usb.ehci.no_hs 1 when I do it. So it's picking it up. I guess it's time to try the others.
-
None of them worked.
Also, my keyboard no longer works.
Any other suggestions?
-
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB. :-\
Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.
@Bai:
None of them worked.
Also, my keyboard no longer works.
Any other suggestions?
Suggestions:
-
Use a motherboard with a chipset that has been available for at least six months at the time of release of FreeBSD 8.1. (I recall that I saw an older pfSense release lock up on startup on a motherboard with AMD chipset if USB was enabled. The next version of pfSense which had a more recent FreeBSD worked fine on the same motherboard when USB was enabled.)
-
Keep searching - maybe a FreeBSD user has found a solution for FreeBSD 8.1
-
Ignore it, you still have three working cores which is probably much more than you need.
-
Disable motherboard USB entirely and use a PCI USB 2.0 card (which almost certainly will have a USB chipset that has been around for a while and consequently has well debugged drivers). I don't know if the BIOS will support this.
-
Disable motherboard USB and set BIOS to ignore "no keyboard". What do you need the keyboard for once the system is configured.
-
-
Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.
That's what I thought, and yet:
[2.0-RC3][root@pfsense.fire.box]/root(2): sysctl -a | grep ehci dev.usbus.2.%parent: ehci0 dev.ehci.0.%desc: Intel 6300ESB USB 2.0 controller dev.ehci.0.%driver: ehci dev.ehci.0.%location: slot=29 function=7 dev.ehci.0.%pnpinfo: vendor=0x8086 device=0x25ad subvendor=0x8086 subdevice=0x25ad class=0x0c0320 dev.ehci.0.%parent: pci0Hmmm. :-\
@Bai Shen. If it has disabled your keyboard it is clearly doing something. Do you still have Legacy USB disabled in the bios? Perhaps you have ended up disabling ehci when that is all that was left still functioning.
Steve
-
- Keep searching - maybe a FreeBSD user has found a solution for FreeBSD 8.1
Not as far as I can tell. There's a bug filed for it, but no resolution so far.
- Ignore it, you still have three working cores which is probably much more than you need.
That's what I'm doing. The biggest annoyance is that it prevents the processor from idling and therefore uses more power than it should.
- Disable motherboard USB entirely and use a PCI USB 2.0 card (which almost certainly will have a USB chipset that has been around for a while and consequently has well debugged drivers). I don't know if the BIOS will support this.
Maybe. But right now I'm using all of the PCI slots for NICs. I'll be picking up PCIe NICs later, but for now I'm using the ones I have.
- Disable motherboard USB and set BIOS to ignore "no keyboard". What do you need the keyboard for once the system is configured.
I've thought about doing that. But I've had instances before where I had to use the console on the actual box to reset/change configurations. So I'm hesitant to do that atm. Plus I'm not sure how it'll work to turn it back on as there's no other way to connect a keyboard.
-
@Bai Shen. If it has disabled your keyboard it is clearly doing something. Do you still have Legacy USB disabled in the bios? Perhaps you have ended up disabling ehci when that is all that was left still functioning.
Steve
No, I made sure to turn legacy and usb3 back on before messing with the loader.conf.local.
-
Serial console instead of keyboard?
I would definitely leave USB3 disabled.
Perhaps you can force one of the two devices onto a different IRQ.Steve
Edit: Assuming you are still using the DH67CL, are you running the lastest bios?
Edit: It seems (though I can't find detailed instruction) that you should be able to set IRQ 16 as unavailable to PCI auto configuration. That should force your LAN card onto a different IRQ.
-
Serial console instead of keyboard?
I don't think it has a serial port, but I could be wrong. I don't have any infrastructure to support that either.
I would definitely leave USB3 disabled.
How come?
Perhaps you can force one of the two devices onto a different IRQ.
Steve
Edit: Assuming you are still using the DH67CL, are you running the lastest bios?
Edit: It seems (though I can't find detailed instruction) that you should be able to set IRQ 16 as unavailable to PCI auto configuration. That should force your LAN card onto a different IRQ.
Yep, I'm running the latest bios.
I'll have to look through the bios at the PCI config. I don't recall seeing anything like that before, but I wasn't looking for it. How does setting the NIC to a different IRQ fix the problem? Wouldn't I still get the interrupts from the ehci?
