{irq16: em1 ehci0} taking up 75% of cpu
-
Update. Problem solved!
Point is that motherboard Z68-ITX has video only with HDMI 1.4a or with DisplayPort (mini-DP). So I connected LCD TV in to HDMI, installed pfsense, assigned interfaces, entered commands for loader.conf, etc. But I did not disconnected the HDMI after it.Then on the russian pfSense forum guys asked me about HDMI/mini-DP and suggested to disconnect LCD. I did it without shutting down the router and looked into vmstat -i. The interrupts appeared on irq16: ehci0 immediately after I disconnected the cable.
After I disconnected the display then I rebooted the router. After it I haven't seen the problem again.
-
Thanks for reporting back.
Interesting solution, I wonder if the HDMI implementation on that board supports an ethernet connection?Steve
-
So my machine is still acting up. What's weird is that it works fine for about an hour and then the problems start.
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
I haven't looked too much into it as everything seems to be working correctly. The only problem is the extra power it's using.
-
You are now seeing {irq11: em1 ehci0} taking a lot of CPU?
Please post output of pfSense shell command vmstat -i
Just realized I never did the vmstat check. So here it is.
interrupt total rate
irq16: em1 ehci0 267777779 57797
irq17: em2 11528 2
irq18: em3 11468 2
irq19: atapci0 37902 8
irq23: ehci1 9358 2
cpu0: timer 9309748 2009
irq256: em0 53580 11
cpu1: timer 9309520 2009
Total 286520883 61843 -
The vmstat output shows a very high interrupt rate on irq16 again.
If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?
You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.
-
@Bai:
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
You could try setting:
hw.pci.enable_msix=0
or
hw.pci.enable_msi=0
in /boot/loader.conf.local.At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.
Steve
-
The vmstat output shows a very high interrupt rate on irq16 again.
If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?
You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.
I tried disabling the USB devices in the BIOS previously with no luck. Also some of the other usb settings. I haven't tried changing the USB ports, though. I'll give that a shot.
@Bai:
Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled. Not sure if that has any bearing on things or not.
You could try setting:
hw.pci.enable_msix=0
or
hw.pci.enable_msi=0
in /boot/loader.conf.local.At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.
Steve
What do those settings do?
I'll give 2.1 a shot when I rebuild my box. I have some PCIe nics that I want to move my higher traffic segments to.
-
They control how the pci bus handles interupts:
http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:it doesn't get past the em1: MSI Interrupt message
What is this message?
Steve
-
They control how the pci bus handles interupts:
http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:it doesn't get past the em1: MSI Interrupt message
What is this message?
Steve
Unfortunately, I don't remember more than that. And I'd have to reboot the box to find out. It's only been doing this since I updated it. Not sure what changed.
I'll try changing the settings and see if that makes a difference.
-
Does anybody have an update on this? I recently commissioned an Intel DH57JG board and I'm seeing the same issue. Disabling legacy USB in the BIOS, and setting hw.usb.ehci.no_hs="1" in /boot/loader.conf.local both make the problem go away for a few hours, and then it comes back. This is a production system, so I'd like to know if anybody has it licked before I spend the next week getting up at 4 am to try one option at a time.
-
I recently commissioned an Intel DH57JG board and I'm seeing the same issue.
Which build of pfSense?
If you haven't tried a snapshot build of 2.1 It might be worth doing so to take advantage of more up to date device drivers.
-
2.0.1
I have considered moving to 2.1, I'm just not really excited about testing software on this particular machine. I'll have to try my options though, one early morning at a time ;)
-
I tried 2.1 but the foray was brief, as some show-stopping bugs had me rebooting within minutes. The good news is that by rebooting into 2.0.1 without a single USB device connected, this problem does not occur for me, even after 20 days uptime.
Still looking forward to a snapshot that lets me use a keyboard without the accompanying IRQ storm.
-
I've just been ignoring it for the time being as it hasn't affected the functionality AFAIK. I need to do an upgrade and add some more nics. I might give 2.1 a shot. Not sure.
