Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    {irq16: em1 ehci0} taking up 75% of cpu

    Scheduled Pinned Locked Moved Hardware
    54 Posts 6 Posters 23.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      vlad-i-mir
      last edited by

      Dear All, I've read this topic carefully and tried many ways to solve this problem. But there is no result till now.

      I use pfsense 2.0.1 release (amd64) with ZOTAC®LGA 1155 Z68-ITX, Intel Celeron G530, Box, 2x2.4 GHz, 2 GB DDR3, HDD WD2500AAKX, 2 x onboard Realtek RTL8111E and AzureWave AR5B95. My ISP gives me 1 Gbps full duplex uplink and static IP. So everything good and I have realy 800-900 Mbps throughput, but only one bad. Last time I noticed interrupts on CPU 30-40%, and I saw this on irq16: ath0 ehci0. I disabled USB3, disabled Audio-controller and other unused things in BIOS. No result. Then I removed the WiFi card AR5B95 and CPU load slow down to 20-30%. I tried to change loader.conf with hw.usb.ehci.no_hs="1" and rebooted the system. But no result again. This is a link to RRD graphs.

      Here is the output from $ vmstat -i and $ devinfo -v, to illustrate the problem:

      interrupt total rate
      irq16: ehci0 6679556627 168445     ------> + approx. 450 000 every 1 second
      irq19: atapci0+ 57770 1
      irq23: ehci1 79325 2
      cpu0: timer 79305108 1999
      irq256: re0 98275744 2478
      irq257: re1 96862546 2442
      cpu1: timer 79304926 1999
      Total 7033442046 177370
      
      $ devinfo -v
      nexus0
      .......
      acpi0
      .......
      pcib0 pnpinfo _HID=PNP0A08 _UID=0 at handle=\_SB_.PCI0
      pci0
      .......
      ehci0 pnpinfo vendor=0x8086 device=0x1c2d subvendor=0x19da subdevice=0xa198 class=0x0c0320 at slot=26 function=0 handle=\_SB_.PCI0.USBE
      usbus0
      uhub0
      uhub2 pnpinfo vendor=0x8087 product=0x0024 devclass=0x09 devsubclass=0x00 sernum="" release=0x0000 intclass=0x09 intsubclass=0x00 at bus=1 hubaddr=1 port=0 devaddr=2 interface=0
      .......
      ehci1 pnpinfo vendor=0x8086 device=0x1c26 subvendor=0x19da subdevice=0xa198 class=0x0c0320 at slot=29 function=0 handle=\_SB_.PCI0.EUSB
      usbus1
      uhub1
      uhub3 pnpinfo vendor=0x8087 product=0x0024 devclass=0x09 devsubclass=0x00 sernum="" release=0x0000 intclass=0x09 intsubclass=0x00 at bus=1 hubaddr=1 port=1 devaddr=2 interface=0
      .......
      

      So may be anybody know what cause this problem, or what I have to do again to track down this? I'm not very experienced with FreeBSD so I don't know what to try next.

      1 Reply Last reply Reply Quote 0
      • V
        vlad-i-mir
        last edited by

        Update. Problem solved!
        Point is that motherboard Z68-ITX has video only with HDMI 1.4a or with DisplayPort (mini-DP). So I connected LCD TV in to HDMI, installed pfsense, assigned interfaces, entered commands for loader.conf, etc. But I did not disconnected the HDMI after it.

        Then on the russian pfSense forum guys asked me about HDMI/mini-DP and suggested to disconnect LCD. I did it without shutting down the router and looked into vmstat -i. The interrupts appeared on irq16: ehci0 immediately after I disconnected the cable.

        After I disconnected the display then I rebooted the router. After it I haven't seen the problem again.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Thanks for reporting back.
          Interesting solution, I wonder if the HDMI implementation on that board supports an ethernet connection?

          Steve

          1 Reply Last reply Reply Quote 0
          • B
            Bai Shen
            last edited by

            So my machine is still acting up.  What's weird is that it works fine for about an hour and then the problems start.

            Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled.  Not sure if that has any bearing on things or not.

            I haven't looked too much into it as everything seems to be working correctly.  The only problem is the extra power it's using.

            1 Reply Last reply Reply Quote 0
            • B
              Bai Shen
              last edited by

              @wallabybob:

              You are now seeing {irq11: em1 ehci0} taking a lot of CPU?

              Please post output of pfSense shell command vmstat -i

              Just realized I never did the vmstat check.  So here it is.

              interrupt                          total       rate
              irq16: em1 ehci0               267777779      57797
              irq17: em2                         11528          2
              irq18: em3                         11468          2
              irq19: atapci0                     37902          8
              irq23: ehci1                        9358          2
              cpu0: timer                      9309748       2009
              irq256: em0                        53580         11
              cpu1: timer                      9309520       2009
              Total                          286520883      61843

              1 Reply Last reply Reply Quote 0
              • W
                wallabybob
                last edited by

                The vmstat output shows a very high interrupt rate on irq16 again.

                If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?

                You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  @Bai:

                  Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled.  Not sure if that has any bearing on things or not.

                  You could try setting:
                  hw.pci.enable_msix=0
                  or
                  hw.pci.enable_msi=0
                  in /boot/loader.conf.local.

                  At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • B
                    Bai Shen
                    last edited by

                    @wallabybob:

                    The vmstat output shows a very high interrupt rate on irq16 again.

                    If you don't have a high packet rate on em1 and are bothered by the high interrupt rates I would be inclined to see if I could stop use of of irq16 and see how things change. For example, move USB devices to different USB sockets in an attempt to move them to ehci1, the other USB controller. The BIOS might allow disabling of USB. Does that change the interrupt rate on irq16? Disable em1. Does that change interrupt rate on irq16?

                    You could also try installing a snapshot build of pfSense 2.1. It is based on a much more recent release of FreeBSD which might work better with your hardware.

                    I tried disabling the USB devices in the BIOS previously with no luck.  Also some of the other usb settings.  I haven't tried changing the USB ports, though.  I'll give that a shot.

                    @stephenw10:

                    @Bai:

                    Also, I've since upgraded to 2.0.1 and now it doesn't get past the em1: MSI Interrupt message unless I select to boot with ACPI(I think) disabled.  Not sure if that has any bearing on things or not.

                    You could try setting:
                    hw.pci.enable_msix=0
                    or
                    hw.pci.enable_msi=0
                    in /boot/loader.conf.local.

                    At this point I think I would try one of the 2.1 snapshots since it's likely to have better support for your motherboard.

                    Steve

                    What do those settings do?

                    I'll give 2.1 a shot when I rebuild my box.  I have some PCIe nics that I want to move my higher traffic segments to.

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      They control how the pci bus handles interupts:
                      http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
                      Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:

                      it doesn't get past the em1: MSI Interrupt message

                      What is this message?

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • B
                        Bai Shen
                        last edited by

                        @stephenw10:

                        They control how the pci bus handles interupts:
                        http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
                        Disabling both of them forces legacy interupts. If you card or bus controller is not fully compliant or just has a buggy driver it could be causing this. I only suggested it because you said:

                        it doesn't get past the em1: MSI Interrupt message

                        What is this message?

                        Steve

                        Unfortunately, I don't remember more than that.  And I'd have to reboot the box to find out.  It's only been doing this since I updated it.  Not sure what changed.

                        I'll try changing the settings and see if that makes a difference.

                        1 Reply Last reply Reply Quote 0
                        • C
                          clarknova
                          last edited by

                          Does anybody have an update on this? I recently commissioned an Intel DH57JG board and I'm seeing the same issue. Disabling legacy USB in the BIOS, and setting hw.usb.ehci.no_hs="1" in /boot/loader.conf.local both make the problem go away for a few hours, and then it comes back. This is a production system, so I'd like to know if anybody has it licked before I spend the next week getting up at 4 am to try one option at a time.

                          db

                          1 Reply Last reply Reply Quote 0
                          • W
                            wallabybob
                            last edited by

                            @clarknova:

                            I recently commissioned an Intel DH57JG board and I'm seeing the same issue.

                            Which build of pfSense?

                            If you haven't tried a snapshot build of 2.1 It might be worth doing so to take advantage of more up to date device drivers.

                            1 Reply Last reply Reply Quote 0
                            • C
                              clarknova
                              last edited by

                              2.0.1

                              I have considered moving to 2.1, I'm just not really excited about testing software on this particular machine. I'll have to try my options though, one early morning at a time ;)

                              db

                              1 Reply Last reply Reply Quote 0
                              • C
                                clarknova
                                last edited by

                                I tried 2.1 but the foray was brief, as some show-stopping bugs had me rebooting within minutes. The good news is that by rebooting into 2.0.1 without a single USB device connected, this problem does not occur for me, even after 20 days uptime.

                                Still looking forward to a snapshot that lets me use a keyboard without the accompanying IRQ storm.

                                db

                                1 Reply Last reply Reply Quote 0
                                • B
                                  Bai Shen
                                  last edited by

                                  I've just been ignoring it for the time being as it hasn't affected the functionality AFAIK.  I need to do an upgrade and add some more nics.  I might give 2.1 a shot.  Not sure.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.