{irq16: em1 ehci0} taking up 75% of cpu
-
I would expect to be able to see the sysctl OIDs but all I see is:
[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb hw.usb.no_boot_wait: 0 hw.usb.debug: 0 hw.usb.usb_lang_mask: 255 hw.usb.usb_lang_id: 9 hw.usb.template: 0 hw.usb.power_timeout: 30 hw.usb.uath.regdomain: 0 hw.usb.uath.countrycode: 0 hw.usb.urtw.preamble_mode: 2 hw.usb.urtw.debug: 0 hw.usb.ucom.cons_baud: 9600 hw.usb.ucom.cons_unit: -1
No EHCI at all.
Steve
-
I would expect to be able to see the sysctl OIDs but all I see is:
[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb hw.usb.no_boot_wait: 0 hw.usb.debug: 0 hw.usb.usb_lang_mask: 255 hw.usb.usb_lang_id: 9 hw.usb.template: 0 hw.usb.power_timeout: 30 hw.usb.uath.regdomain: 0 hw.usb.uath.countrycode: 0 hw.usb.urtw.preamble_mode: 2 hw.usb.urtw.debug: 0 hw.usb.ucom.cons_baud: 9600 hw.usb.ucom.cons_unit: -1
No EHCI at all.
Steve
I get hw.usb.ehci.no_hs 1 when I do it. So it's picking it up. I guess it's time to try the others.
-
None of them worked.
Also, my keyboard no longer works.
Any other suggestions?
-
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB. :-\
Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.
@Bai:
None of them worked.
Also, my keyboard no longer works.
Any other suggestions?
Suggestions:
-
Use a motherboard with a chipset that has been available for at least six months at the time of release of FreeBSD 8.1. (I recall that I saw an older pfSense release lock up on startup on a motherboard with AMD chipset if USB was enabled. The next version of pfSense which had a more recent FreeBSD worked fine on the same motherboard when USB was enabled.)
-
Keep searching - maybe a FreeBSD user has found a solution for FreeBSD 8.1
-
Ignore it, you still have three working cores which is probably much more than you need.
-
Disable motherboard USB entirely and use a PCI USB 2.0 card (which almost certainly will have a USB chipset that has been around for a while and consequently has well debugged drivers). I don't know if the BIOS will support this.
-
Disable motherboard USB and set BIOS to ignore "no keyboard". What do you need the keyboard for once the system is configured.
-
-
Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.
That's what I thought, and yet:
[2.0-RC3][root@pfsense.fire.box]/root(2): sysctl -a | grep ehci dev.usbus.2.%parent: ehci0 dev.ehci.0.%desc: Intel 6300ESB USB 2.0 controller dev.ehci.0.%driver: ehci dev.ehci.0.%location: slot=29 function=7 dev.ehci.0.%pnpinfo: vendor=0x8086 device=0x25ad subvendor=0x8086 subdevice=0x25ad class=0x0c0320 dev.ehci.0.%parent: pci0
Hmmm. :-\
@Bai Shen. If it has disabled your keyboard it is clearly doing something. Do you still have Legacy USB disabled in the bios? Perhaps you have ended up disabling ehci when that is all that was left still functioning.
Steve
-
- Keep searching - maybe a FreeBSD user has found a solution for FreeBSD 8.1
Not as far as I can tell. There's a bug filed for it, but no resolution so far.
- Ignore it, you still have three working cores which is probably much more than you need.
That's what I'm doing. The biggest annoyance is that it prevents the processor from idling and therefore uses more power than it should.
- Disable motherboard USB entirely and use a PCI USB 2.0 card (which almost certainly will have a USB chipset that has been around for a while and consequently has well debugged drivers). I don't know if the BIOS will support this.
Maybe. But right now I'm using all of the PCI slots for NICs. I'll be picking up PCIe NICs later, but for now I'm using the ones I have.
- Disable motherboard USB and set BIOS to ignore "no keyboard". What do you need the keyboard for once the system is configured.
I've thought about doing that. But I've had instances before where I had to use the console on the actual box to reset/change configurations. So I'm hesitant to do that atm. Plus I'm not sure how it'll work to turn it back on as there's no other way to connect a keyboard.
-
@Bai Shen. If it has disabled your keyboard it is clearly doing something. Do you still have Legacy USB disabled in the bios? Perhaps you have ended up disabling ehci when that is all that was left still functioning.
Steve
No, I made sure to turn legacy and usb3 back on before messing with the loader.conf.local.
-
Serial console instead of keyboard?
I would definitely leave USB3 disabled.
Perhaps you can force one of the two devices onto a different IRQ.Steve
Edit: Assuming you are still using the DH67CL, are you running the lastest bios?
Edit: It seems (though I can't find detailed instruction) that you should be able to set IRQ 16 as unavailable to PCI auto configuration. That should force your LAN card onto a different IRQ.
-
Serial console instead of keyboard?
I don't think it has a serial port, but I could be wrong. I don't have any infrastructure to support that either.
I would definitely leave USB3 disabled.
How come?
Perhaps you can force one of the two devices onto a different IRQ.
Steve
Edit: Assuming you are still using the DH67CL, are you running the lastest bios?
Edit: It seems (though I can't find detailed instruction) that you should be able to set IRQ 16 as unavailable to PCI auto configuration. That should force your LAN card onto a different IRQ.
Yep, I'm running the latest bios.
I'll have to look through the bios at the PCI config. I don't recall seeing anything like that before, but I wasn't looking for it. How does setting the NIC to a different IRQ fix the problem? Wouldn't I still get the interrupts from the ehci?
-
@Bai:
I would definitely leave USB3 disabled.
How come?
Because it's highly probable that FreeBSD didn't support it when 8.1 was released. Does it even support it now?
@Bai:
I'll have to look through the bios at the PCI config. I don't recall seeing anything like that before, but I wasn't looking for it. How does setting the NIC to a different IRQ fix the problem? Wouldn't I still get the interrupts from the ehci?
It may be an IRQ conflict causing the interrupt storm.
Any interrupts set to Available in Setup are considered to be available for
use by the add-in card.Implies that you can set to unavailable.
Steve
-
Because it's highly probable that FreeBSD didn't support it when 8.1 was released. Does it even support it now?
No idea. I'm not using them, but figured I'd leave them on so I don't plug something in down the road and wonder why it's not working. :)
It may be an IRQ conflict causing the interrupt storm.
Ah, gotcha.
Any interrupts set to Available in Setup are considered to be available for
use by the add-in card.Implies that you can set to unavailable.
Steve
-nods- I'll take a look when I get home.
-
I ended up being sidetracked by other things and never messed with the box any more. This weekend, I shut it down to rearrange some cables. Since I brought it back up, I haven't seen the problem again. No idea what the difference is as I don't recall changing anything.
Just wanted to give y'all an update.
-
Strange.
Perhaps something to do with the cable routing as you suggest.Steve
-
Strange.
Perhaps something to do with the cable routing as you suggest.Steve
All I did was unplug the Kill-A-Watt from the power cable. Dunno.
-
This looks like a problem I had on an i3 550 that went out the door. I believe the fix was to turn off USB legacy support in the BIOS.
-
Well, I rebooted due to updating to 2.0-RELEASE and it's back. Only now it's irq11
-
You are now seeing {irq11: em1 ehci0} taking a lot of CPU?
Please post output of pfSense shell command vmstat -i
-
You are now seeing {irq11: em1 ehci0} taking a lot of CPU?
Yes.
Please post output of pfSense shell command vmstat -i
Okay. I'll do that when I get back to the machine.
-
That the irq has changed from 16 to 11 suggests to me you have (possibly inadvertently) disabled multiprocessing in the BIOS or disabled the IOAPIC or disabled acpi.
-
EHCI provides access for 'high speed' devices as opposed to low and full speed.
The ehci driver has some tunables you could add to /boot/loader.conf.local to try.
LOADER TUNABLES
Tunables can be set at the loader(8) prompt before booting the kernel or
stored in loader.conf(5).hw.usb.ehci.lostintrbug
This tunable enables the lost interrupt quirk. The default value
is 0 (off).hw.usb.ehci.iaadbug
This tunable enables the EHCI doorbell quirk. The default value
is 0 (off).hw.usb.ehci.no_hs
This tunable disables USB devices to attach like HIGH-speed ones
and will force all attached devices to attach to the FULL- or
LOW-speed companion controller. The default value is 0 (off).Steve
Steve I had a similar problem to the OP on my X9SCV build. {irq16: ehci0} was using ~50% of one cpu. I did as you suggested and added hw.usb.ehci.no_hs="1" to loader.conf and it seems to have fixed it. I'll keep an eye on it. Thanks
-
Glad that worked for you and thanks for reporting back. :)
Steve
-
Dear All, I've read this topic carefully and tried many ways to solve this problem. But there is no result till now.
I use pfsense 2.0.1 release (amd64) with ZOTACLGA 1155 Z68-ITX, Intel Celeron G530, Box, 2x2.4 GHz, 2 GB DDR3, HDD WD2500AAKX, 2 x onboard Realtek RTL8111E and AzureWave AR5B95. My ISP gives me 1 Gbps full duplex uplink and static IP. So everything good and I have realy 800-900 Mbps throughput, but only one bad. Last time I noticed interrupts on CPU 30-40%, and I saw this on irq16: ath0 ehci0. I disabled USB3, disabled Audio-controller and other unused things in BIOS. No result. Then I removed the WiFi card AR5B95 and CPU load slow down to 20-30%. I tried to change loader.conf with hw.usb.ehci.no_hs="1" and rebooted the system. But no result again. This is a link to RRD graphs.
Here is the output from $ vmstat -i and $ devinfo -v, to illustrate the problem:
interrupt total rate irq16: ehci0 6679556627 168445 ------> + approx. 450 000 every 1 second irq19: atapci0+ 57770 1 irq23: ehci1 79325 2 cpu0: timer 79305108 1999 irq256: re0 98275744 2478 irq257: re1 96862546 2442 cpu1: timer 79304926 1999 Total 7033442046 177370
$ devinfo -v nexus0 ....... acpi0 ....... pcib0 pnpinfo _HID=PNP0A08 _UID=0 at handle=\_SB_.PCI0 pci0 ....... ehci0 pnpinfo vendor=0x8086 device=0x1c2d subvendor=0x19da subdevice=0xa198 class=0x0c0320 at slot=26 function=0 handle=\_SB_.PCI0.USBE usbus0 uhub0 uhub2 pnpinfo vendor=0x8087 product=0x0024 devclass=0x09 devsubclass=0x00 sernum="" release=0x0000 intclass=0x09 intsubclass=0x00 at bus=1 hubaddr=1 port=0 devaddr=2 interface=0 ....... ehci1 pnpinfo vendor=0x8086 device=0x1c26 subvendor=0x19da subdevice=0xa198 class=0x0c0320 at slot=29 function=0 handle=\_SB_.PCI0.EUSB usbus1 uhub1 uhub3 pnpinfo vendor=0x8087 product=0x0024 devclass=0x09 devsubclass=0x00 sernum="" release=0x0000 intclass=0x09 intsubclass=0x00 at bus=1 hubaddr=1 port=1 devaddr=2 interface=0 .......
So may be anybody know what cause this problem, or what I have to do again to track down this? I'm not very experienced with FreeBSD so I don't know what to try next.
-
Update. Problem solved!
Point is that motherboard Z68-ITX has video only with HDMI 1.4a or with DisplayPort (mini-DP). So I connected LCD TV in to HDMI, installed pfsense, assigned interfaces, entered commands for loader.conf, etc. But I did not disconnected the HDMI after it.Then on the russian pfSense forum guys asked me about HDMI/mini-DP and suggested to disconnect LCD. I did it without shutting down the router and looked into vmstat -i. The interrupts appeared on irq16: ehci0 immediately after I disconnected the cable.
After I disconnected the display then I rebooted the router. After it I haven't seen the problem again.
-
Thanks for reporting back.
Interesting solution, I wonder if the HDMI implementation on that board supports an ethernet connection?Steve
-
So my machine is still acting up. What's weird is that it works fine for about an hour and then the problems start.
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
I haven't looked too much into it as everything seems to be working correctly. The only problem is the extra power it's using.
-
You are now seeing {irq11: em1 ehci0} taking a lot of CPU?
Please post output of pfSense shell command vmstat -i
Just realized I never did the vmstat check. So here it is.
interrupt total rate
irq16: em1 ehci0 267777779 57797
irq17: em2 11528 2
irq18: em3 11468 2
irq19: atapci0 37902 8
irq23: ehci1 9358 2
cpu0: timer 9309748 2009
irq256: em0 53580 11
cpu1: timer 9309520 2009
Total 286520883 61843 -
The vmstat output shows a very high interrupt rate on irq16 again.
If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?
You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.
-
@Bai:
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
You could try setting:
hw.pci.enable_msix=0
or
hw.pci.enable_msi=0
in /boot/loader.conf.local.At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.
Steve
-
The vmstat output shows a very high interrupt rate on irq16 again.
If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?
You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.
I tried disabling the USB devices in the BIOS previously with no luck. Also some of the other usb settings. I haven't tried changing the USB ports, though. I'll give that a shot.
@Bai:
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
You could try setting:
hw.pci.enable_msix=0
or
hw.pci.enable_msi=0
in /boot/loader.conf.local.At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.
Steve
What do those settings do?
I'll give 2.1 a shot when I rebuild my box. I have some PCIe nics that I want to move my higher traffic segments to.
-
They control how the pci bus handles interupts:
http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:it doesn't get past the em1: MSI Interrupt message
What is this message?
Steve
-
They control how the pci bus handles interupts:
http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:it doesn't get past the em1: MSI Interrupt message
What is this message?
Steve
Unfortunately, I don't remember more than that. And I'd have to reboot the box to find out. It's only been doing this since I updated it. Not sure what changed.
I'll try changing the settings and see if that makes a difference.
-
Does anybody have an update on this? I recently commissioned an Intel DH57JG board and I'm seeing the same issue. Disabling legacy USB in the BIOS, and setting hw.usb.ehci.no_hs="1" in /boot/loader.conf.local both make the problem go away for a few hours, and then it comes back. This is a production system, so I'd like to know if anybody has it licked before I spend the next week getting up at 4 am to try one option at a time.
-
I recently commissioned an Intel DH57JG board and I'm seeing the same issue.
Which build of pfSense?
If you haven't tried a snapshot build of 2.1 It might be worth doing so to take advantage of more up to date device drivers.
-
2.0.1
I have considered moving to 2.1, I'm just not really excited about testing software on this particular machine. I'll have to try my options though, one early morning at a time ;)
-
I tried 2.1 but the foray was brief, as some show-stopping bugs had me rebooting within minutes. The good news is that by rebooting into 2.0.1 without a single USB device connected, this problem does not occur for me, even after 20 days uptime.
Still looking forward to a snapshot that lets me use a keyboard without the accompanying IRQ storm.
-
I've just been ignoring it for the time being as it hasn't affected the functionality AFAIK. I need to do an upgrade and add some more nics. I might give 2.1 a shot. Not sure.