{irq16: em1 ehci0} taking up 75% of cpu

[stephenw10]

Yes that seems likely the same issue. Or indeed this one:
http://forums.freebsd.org/showthread.php?t=24952

I was going to suggest you disable all usb ports in the bios but it seems you are using them.  ::)
You may be able to disable ehci though. Have a look through your bios and see what you can disable.

Steve

[Bai Shen]

@stephenw10:

Yes that seems likely the same issue. Or indeed this one:
http://forums.freebsd.org/showthread.php?t=24952

I was going to suggest you disable and usb ports in the bios but it seems you are using them.  ::)
You may be able to disable ehci though. Have a look through your bios and see what you can disable.

Steve

And if I disabled the usb ports, I'd have no way of reenabling them. :)

What's the difference between ehci and usb?

[stephenw10]

EHCI provides access for 'high speed' devices as opposed to low and full speed.

The ehci driver has some tunables you could add to /boot/loader.conf.local to try.

LOADER TUNABLES

Tunables can be set at the loader(8) prompt before booting the kernel or
    stored in loader.conf(5).

hw.usb.ehci.lostintrbug
            This tunable enables the lost interrupt quirk.  The default value
            is 0 (off).

hw.usb.ehci.iaadbug
            This tunable enables the EHCI doorbell quirk.  The default value
            is 0 (off).

hw.usb.ehci.no_hs
            This tunable disables USB devices to attach like HIGH-speed ones
            and will force all attached devices to attach to the FULL- or
            LOW-speed companion controller.  The default value is 0 (off).

Steve

[Bai Shen]

@stephenw10:

EHCI provides access for 'high speed' devices as opposed to low and full speed.

The ehci driver has some tunables you could add to /boot/loader.conf.local to try.

LOADER TUNABLES

Tunables can be set at the loader(8) prompt before booting the kernel or
     stored in loader.conf(5).

hw.usb.ehci.lostintrbug
             This tunable enables the lost interrupt quirk.  The default value
             is 0 (off).

hw.usb.ehci.iaadbug
             This tunable enables the EHCI doorbell quirk.  The default value
             is 0 (off).

hw.usb.ehci.no_hs
             This tunable disables USB devices to attach like HIGH-speed ones
             and will force all attached devices to attach to the FULL- or
             LOW-speed companion controller.  The default value is 0 (off).

Steve

Ah, okay.  I'll take a look when I get home.

I wonder if it made a difference that I installed pfSense using a usb cd-rom.

[wallabybob]

@Bai:

I'm running 2.0 RC3 on an i3.  Shortly after I boot it, one of the cpu cores gets a heavy load on it.  Looking at the System Activity, it shows {irq16: em1 ehci0}

em1 is my WAN card.  I'm not running any packages, and other than the high cpu activity, everything seems to be working.

I've seen this sort of thing happen when the BIOS provides incorrect interrupt data. For example, if the BIOS says a device interrupts on irq18 when it really interrupts on irq16 the driver's interrupt handler will get attached to irq18 but the real interrupt goes to irq16 where it doesn't get handled (because the correct handler doesn't get called) and the interrupt request doesn't get cleared until there is an irq18 interrupt and the correct handler is called. Do you have the latest BIOS?

@Bai:

I wonder if it made a difference that I installed pfSense using a usb cd-rom.

I think it is unlikely.

Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a

[Bai Shen]

@wallabybob:

I've seen this sort of thing happen when the BIOS provides incorrect interrupt data. For example, if the BIOS says a device interrupts on irq18 when it really interrupts on irq16 the driver's interrupt handler will get attached to irq18 but the real interrupt goes to irq16 where it doesn't get handled (because the correct handler doesn't get called) and the interrupt request doesn't get cleared until there is an irq18 interrupt and the correct handler is called. Do you have the latest BIOS?

Yep.

@wallabybob:

I think it is unlikely.

I figured, but I thought I'd mention it just in case.

@wallabybob:

Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a

The names of a what?

[wallabybob]

@Bai:

@wallabybob:

Please post the output of pfSense shell command: pciconf -l -v; vmstat -i and the names of a

The names of a what?

Sorry, just the output of the shell command pciconf -l -v; devinfo -r; vmstat -i will do.

[Bai Shen]

Well, I turned off Legacy USB and USB 3 in the BIOS, and the problem seems to have gone away.  It's now idling at a nice 39W. :)

Will the output of the commands still be worthwhile?

[wallabybob]

@Bai:

Will the output of the commands still be worthwhile?

No - seems like a USB related problem.

[Bai Shen]

@wallabybob:

@Bai:

Will the output of the commands still be worthwhile?

No - seems like a USB related problem.

Actually, it looks like it's not. :(  It just took a while before it came back. :(

Here's the output.

$ pciconf -l -v; devinfo -r; vmstat -i
hostb0@pci0:0:0:0:	class=0x060000 card=0x20038086 chip=0x01008086 rev=0x09 hdr=0x00
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:	class=0x030000 card=0x20038086 chip=0x01028086 rev=0x09 hdr=0x00
    class      = display
    subclass   = VGA
none0@pci0:0:22:0:	class=0x078000 card=0x20038086 chip=0x1c3a8086 rev=0x04 hdr=0x00
    class      = simple comms
em0@pci0:0:25:0:	class=0x020000 card=0x20038086 chip=0x15038086 rev=0x05 hdr=0x00
    class      = network
    subclass   = ethernet
ehci0@pci0:0:26:0:	class=0x0c0320 card=0x20038086 chip=0x1c2d8086 rev=0x05 hdr=0x00
    class      = serial bus
    subclass   = USB
none1@pci0:0:27:0:	class=0x040300 card=0x20038086 chip=0x1c208086 rev=0x05 hdr=0x00
    class      = multimedia
    subclass   = HDA
pcib1@pci0:0:28:0:	class=0x060400 card=0x20038086 chip=0x1c108086 rev=0xb5 hdr=0x01
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:	class=0x0c0320 card=0x20038086 chip=0x1c268086 rev=0x05 hdr=0x00
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 card=0x20038086 chip=0x1c4a8086 rev=0x05 hdr=0x00
    class      = bridge
    subclass   = PCI-ISA
atapci0@pci0:0:31:2:	class=0x010601 card=0x20038086 chip=0x1c028086 rev=0x05 hdr=0x00
    class      = mass storage
    subclass   = SATA
none2@pci0:0:31:3:	class=0x0c0500 card=0x20038086 chip=0x1c228086 rev=0x05 hdr=0x00
    class      = serial bus
    subclass   = SMBus
pcib2@pci0:1:0:0:	class=0x060400 card=0x20038086 chip=0x88921283 rev=0x10 hdr=0x01
    class      = bridge
    subclass   = PCI-PCI
em1@pci0:2:0:0:	class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
    class      = network
    subclass   = ethernet
em2@pci0:2:1:0:	class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
    class      = network
    subclass   = ethernet
em3@pci0:2:2:0:	class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
    class      = network
    subclass   = ethernet
nexus0
  cryptosoft0
  apic0
  ram0
      I/O memory addresses:
          0x0-0x9cfff
          0x100000-0x1fffffff
          0x20200000-0x3fffffff
          0x40200000-0xda969fff
          0xdabc9000-0xdabc9fff
          0xdae8e000-0xdaffffff
  npx0
  acpi0
      Interrupt request lines:
          9
      I/O ports:
          0x10-0x1f
          0x22-0x3f
          0x44-0x5f
          0x62-0x63
          0x65-0x6f
          0x72-0x7f
          0x80
          0x84-0x86
          0x88
          0x8c-0x8e
          0x90-0x9f
          0xa2-0xbf
          0xe0-0xef
          0x400-0x453
          0x454-0x457
          0x458-0x47f
          0x4d0-0x4d1
          0x500-0x57f
          0x1180-0x119f
      I/O memory addresses:
          0xf8000000-0xfbffffff
          0xfec00000-0xfecfffff
          0xfed08000-0xfed08fff
          0xfed10000-0xfed19fff
          0xfed1c000-0xfed1ffff
          0xfed20000-0xfed3ffff
          0xfed90000-0xfed93fff
          0xfee00000-0xfee0ffff
          0xff000000-0xffffffff
    cpu0
        ACPI I/O ports:
            0x414
            0x415
      acpi_perf0
      est0
      p4tcc0
      cpufreq0
    cpu1
        ACPI I/O ports:
            0x414
            0x415
      acpi_perf1
      est1
      p4tcc1
      cpufreq1
    cpu2
        ACPI I/O ports:
            0x414
            0x415
      acpi_perf2
      est2
      p4tcc2
      cpufreq2
    cpu3
        ACPI I/O ports:
            0x414
            0x415
      acpi_perf3
      est3
      p4tcc3
      cpufreq3
    pcib0
      pci0
          I/O ports:
              0xf000-0xf03f
              0xf040-0xf05f
              0xf080-0xf09f
              0xf0a0-0xf0a3
              0xf0b0-0xf0b7
              0xf0c0-0xf0c3
              0xf0d0-0xf0d7
          I/O memory addresses:
              0xe0000000-0xefffffff
              0xfe000000-0xfe3fffff
              0xfe720000-0xfe723fff
              0xfe724000-0xfe7240ff
              0xfe729000-0xfe72900f
        hostb0
        vgapci0
        em0
            Interrupt request lines:
                256
            I/O memory addresses:
                0xfe700000-0xfe71ffff
                0xfe728000-0xfe728fff
        ehci0
            Interrupt request lines:
                16
            I/O memory addresses:
                0xfe727000-0xfe7273ff
          usbus0
            uhub0
              uhub2
        pcib1
          pci1
            pcib2
              pci2
                  I/O memory addresses:
                      0xfe420000-0xfe43ffff
                      0xfe480000-0xfe49ffff
                      0xfe4e0000-0xfe4fffff
                em1
                    Interrupt request lines:
                        16
                    I/O ports:
                        0xe080-0xe0bf
                    I/O memory addresses:
                        0xfe500000-0xfe51ffff
                em2
                    Interrupt request lines:
                        17
                    I/O ports:
                        0xe040-0xe07f
                    I/O memory addresses:
                        0xfe4a0000-0xfe4bffff
                em3
                    Interrupt request lines:
                        18
                    I/O ports:
                        0xe000-0xe03f
                    I/O memory addresses:
                        0xfe440000-0xfe45ffff
        ehci1
            Interrupt request lines:
                23
            I/O memory addresses:
                0xfe726000-0xfe7263ff
          usbus1
            uhub1
              uhub3
        isab0
          isa0
            pmtimer0
            sc0
            vga0
                I/O ports:
                    0x3c0-0x3df
                I/O memory addresses:
                    0xa0000-0xbffff
            ata0
                Interrupt request lines:
                    14
                I/O ports:
                    0x1f0-0x1f7
                    0x3f6
            ata1
                Interrupt request lines:
                    15
                I/O ports:
                    0x170-0x177
                    0x376
            atkbdc0
                I/O ports:
                    0x60
                    0x64
              atkbd0
                  Interrupt request lines:
                      1
        atapci0
            Interrupt request lines:
                19
            I/O ports:
                0xf060-0xf07f
            I/O memory addresses:
                0xfe725000-0xfe7257ff
          ata2
            ad4
              subdisk4
          ata3
          ata4
          ata5
          ata6
          ata7
    acpi_sysresource0
    acpi_sysresource1
    atpic0
    atdma0
    attimer0
    atrtc0
        Interrupt request lines:
            8
        I/O ports:
            0x70-0x71
    acpi_sysresource2
    npxisa0
    acpi_sysresource3
    acpi_sysresource4
    acpi_button0
    pci_link0
    pci_link1
    pci_link2
    pci_link3
    pci_link4
    pci_link5
    pci_link6
    pci_link7
    acpi_sysresource5
    acpi_sysresource6
    acpi_timer0
        ACPI I/O ports:
            0x408-0x40b
    acpi_hpet0
        I/O memory addresses:
            0xfed00000-0xfed003ff
interrupt                          total       rate
irq16: em1 ehci0               779815612      16681
irq17: em2                         18172          0
irq18: em3                          4435          0
irq19: atapci0                     81729          1
irq23: ehci1                       93211          1
cpu0: timer                     93493755       2000
irq256: em0                       604657         12
cpu1: timer                     93493955       2000
cpu2: timer                     93493973       2000
cpu3: timer                     93494072       2000
Total                         1154593571      24699

[wallabybob]

I think Steve's suggestion was that you disable USB2 and USB3. From what you said you disabled USB1 and USB3. Please try Steve's suggestion. USB keyboard and mouse should work fine with USB1.

[stephenw10]

I would try disabling 'high speed' with that loader tunable. You clearly don't need high speed, ehci, for a keyboard.

Steve

[Bai Shen]

@wallabybob:

I think Steve's suggestion was that you disable USB2 and USB3. From what you said you disabled USB1 and USB3. Please try Steve's suggestion. USB keyboard and mouse should work fine with USB1.

Those were the only options in the BIOS for USB.

@stephenw10:

I would try disabling 'high speed' with that loader tunable. You clearly don't need high speed, ehci, for a keyboard.

Steve

I'll give that a shot.  Which ones do you suggest?  All three or just the last one?

[stephenw10]

I would probably try no_hs first then lost interupt.
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB.  :-\

Steve

[Bai Shen]

@stephenw10:

I would probably try no_hs first then lost interupt.
The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB.  :-\

Steve

No luck with no_hs.  It ran fine for a little bit, but then quickly returned to it's previous state.  Is there a way to make sure that it's recognizing the loader.conf.local?

And I don't really have a choice on using USB.  I don't have any PS2 ports. :)

[stephenw10]

I would expect to be able to see the sysctl OIDs but all I see is:

[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb
hw.usb.no_boot_wait: 0
hw.usb.debug: 0
hw.usb.usb_lang_mask: 255
hw.usb.usb_lang_id: 9
hw.usb.template: 0
hw.usb.power_timeout: 30
hw.usb.uath.regdomain: 0
hw.usb.uath.countrycode: 0
hw.usb.urtw.preamble_mode: 2
hw.usb.urtw.debug: 0
hw.usb.ucom.cons_baud: 9600
hw.usb.ucom.cons_unit: -1

No EHCI at all.

Steve

[Bai Shen]

@stephenw10:

I would expect to be able to see the sysctl OIDs but all I see is:

[2.0-RC3][root@pfsense.fire.box]/root(1): sysctl hw.usb
hw.usb.no_boot_wait: 0
hw.usb.debug: 0
hw.usb.usb_lang_mask: 255
hw.usb.usb_lang_id: 9
hw.usb.template: 0
hw.usb.power_timeout: 30
hw.usb.uath.regdomain: 0
hw.usb.uath.countrycode: 0
hw.usb.urtw.preamble_mode: 2
hw.usb.urtw.debug: 0
hw.usb.ucom.cons_baud: 9600
hw.usb.ucom.cons_unit: -1

No EHCI at all.

Steve

I get hw.usb.ehci.no_hs 1 when I do it.  So it's picking it up.  I guess it's time to try the others.

[Bai Shen]

None of them worked.

Also, my keyboard no longer works.

Any other suggestions?

[wallabybob]

@stephenw10:

The sysctl OIDs are listed in the 8.1 source code, here, but I don't have them on my system. Perhaps because I'm not using USB.  :-\

Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.

@Bai:

None of them worked.

Also, my keyboard no longer works.

Any other suggestions?

Suggestions:

• Use  a motherboard with a chipset that has been available for at least six months at the time of release of FreeBSD 8.1. (I recall that I saw an older pfSense release lock up on startup on a motherboard with AMD chipset if USB was enabled. The next version of pfSense which had a more recent FreeBSD worked fine on the same motherboard when USB was enabled.)

• Keep searching - maybe a FreeBSD user has found a solution for FreeBSD 8.1

• Ignore it, you still have three working cores which is probably much more than you need.

• Disable motherboard USB entirely and use a PCI USB 2.0 card (which almost certainly will have a  USB chipset that has been around for a while and consequently has well debugged drivers). I don't know if the BIOS will support this.

• Disable motherboard USB and set BIOS to ignore "no keyboard". What do you need the keyboard for once the system is configured.

[stephenw10]

@wallabybob:

Some device drivers don't register sysctls until they have successfully attached at least one device. I don't know the specifics of the USB sysctls.

That's what I thought, and yet:

[2.0-RC3][root@pfsense.fire.box]/root(2): sysctl -a | grep ehci
dev.usbus.2.%parent: ehci0
dev.ehci.0.%desc: Intel 6300ESB USB 2.0 controller
dev.ehci.0.%driver: ehci
dev.ehci.0.%location: slot=29 function=7
dev.ehci.0.%pnpinfo: vendor=0x8086 device=0x25ad subvendor=0x8086 subdevice=0x25ad class=0x0c0320
dev.ehci.0.%parent: pci0

Hmmm.  :-\

@Bai Shen. If it has disabled your keyboard it is clearly doing something. Do you still have Legacy USB disabled in the bios? Perhaps you have ended up disabling ehci when that is all that was left still functioning.

Steve