High CPU usage on interrupt processing



  • Hello,

    I’ve installed pfSense 2.4.3 on Fujitsu PRIMERGY RX1330 M3.
    The full configuration is:
    MB: Fujitsu PRIMERGY RX1330 M3
    CPU: Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz
    RAM: 8G
    RAID: Fujitsu PRAID CP400i SAS (2 disks in RAID1, 1TB)
    LAN: PLAN-CP4x1Gbit-Cu-Intel-I350-T4 x2 (8 eth port on PCI + 2 integrated, also Intel)

    And everything works fine as far as I tested — all 10 LANs, RAID, no errors in dmesg.
    There’s only one thing that concerns me — high CPU load on interrupt processing.

    Server is currently idle, but interrupt processing is always at around 4% up to 18% (rare peaks).
    Is it OK or something doesn’t work as expected?

    Here’s some system output:

    top:
    last pid: 98015;  load averages:  0.21,  0.25,  0.22                                                                                                                           up 0+00:28:40  15:05:59
    42 processes:  1 running, 41 sleeping
    CPU:  0.1% user,  0.0% nice,  0.1% system,  4.2% interrupt, 95.6% idle
    Mem: 51M Active, 55M Inact, 272M Wired, 29M Buf, 7252M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
      315 root          1  22    0   263M 38960K accept  0   0:02   0.38% php-fpm
    97182 root          1  20    0 20068K  3380K CPU2    2   0:00   0.02% top
    22538 root          1  20    0 37704K  7992K kqread  2   0:00   0.01% nginx
    16689 root          5  52    0 13036K  2356K uwait   1   0:00   0.01% dpinger
    
    vmstat –i:
    interrupt                          total       rate
    irq4: uart0                          223          0
    cpu0:timer                        355780        203
    cpu3:timer                        842198        480
    cpu2:timer                        283307        162
    cpu1:timer                        274212        156
    irq284: mfi0                      126370         72
    irq285: xhci0                         55          0
    irq286: ahci0                         39          0
    irq287: igb4:que 0                 18894         11
    irq288: igb4:que 1                  2932          2
    irq289: igb4:que 2                  5402          3
    irq290: igb4:que 3                  2363          1
    irq291: igb4:link                      2          0
    irq292: igb5:que 0                  1694          1
    irq293: igb5:que 1                  1694          1
    irq294: igb5:que 2                  1694          1
    irq295: igb5:que 3                  1694          1
    irq296: igb5:link                      1          0
    irq302: igb7:que 0                  1693          1
    irq303: igb7:que 1                  1693          1
    irq304: igb7:que 2                  1693          1
    irq305: igb7:que 3                  1693          1
    irq306: igb7:link                      1          0
    irq307: igb8:que 0                  1693          1
    irq308: igb8:que 1                  1693          1
    irq309: igb8:que 2                  1693          1
    irq310: igb8:que 3                  1693          1
    irq311: igb8:link                      1          0
    irq312: igb9:que 0                  1694          1
    irq313: igb9:que 1                  1694          1
    irq314: igb9:que 2                  1694          1
    irq315: igb9:que 3                  1694          1
    irq316: igb9:link                      3          0
    Total                            1938879       1106
    

  • Netgate Administrator

    Do you have powerd enabled (speedstep)?

    Steve



  • No, powerd is not enabled.

    I attached dmesg output below, maybe it will give some additional clues to what's going on.

    Copyright (c) 1992-2017 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.1-RELEASE-p7 #10 r313908+986837ba7e9(RELENG_2_4): Mon Mar 26 18:08:25 CDT 2018
        root@buildbot2.netgate.com:/builder/ce-243/tmp/obj/builder/ce-243/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1)
    VT(efifb): resolution 1024x768
    CPU: Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz (3000.10-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
      Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
      Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
      AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
      AMD Features2=0x121<LAHF,ABM,Prefetch>
      Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
      XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
      VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
      TSC: P-state invariant, performance statistics
    real memory  = 8589934592 (8192 MB)
    avail memory = 7944486912 (7576 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <FUJ    D3375-B1>
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s)
    random: unblocking device.
    ioapic0 <Version 2.0> irqs 0-119 on motherboard
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #1 Launched!
    Timecounter "TSC-low" frequency 1500049087 Hz quality 1000
    random: entropy device external interface
    wlan: mac acl policy registered
    kbd0 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff81162bc0, 0) error 19
    random: registering fast source Intel Secure Key RNG
    random: fast provider: "Intel Secure Key RNG"
    nexus0
    cryptosoft0: <software crypto> on motherboard
    padlock0: No ACE support.
    acpi0: <FUJ D3375-B1> on motherboard
    acpi0: Power Button (fixed)
    cpu0: <ACPI CPU> on acpi0
    cpu1: <ACPI CPU> on acpi0
    cpu2: <ACPI CPU> on acpi0
    cpu3: <ACPI CPU> on acpi0
    hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 24000000 Hz quality 950
    Event timer "HPET" frequency 24000000 Hz quality 550
    Event timer "HPET1" frequency 24000000 Hz quality 440
    Event timer "HPET2" frequency 24000000 Hz quality 440
    Event timer "HPET3" frequency 24000000 Hz quality 440
    Event timer "HPET4" frequency 24000000 Hz quality 440
    atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
    atrtc0: Warning: Couldn't map I/O.
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
    pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
    pci0: <ACPI PCI bus> on pcib0
    pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
    pci1: <ACPI PCI bus> on pcib1
    igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdf100000-0xdf1fffff,0xdf28c000-0xdf28ffff at device 0.0 on pci1
    igb0: Using MSIX interrupts with 5 vectors
    igb0: Ethernet address: b4:96:91:13:32:40
    igb0: Bound queue 0 to cpu 0
    igb0: Bound queue 1 to cpu 1
    igb0: Bound queue 2 to cpu 2
    igb0: Bound queue 3 to cpu 3
    igb0: netmap queues/slots: TX 4/1024, RX 4/1024
    igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdf000000-0xdf0fffff,0xdf288000-0xdf28bfff at device 0.1 on pci1
    igb1: Using MSIX interrupts with 5 vectors
    igb1: Ethernet address: b4:96:91:13:32:41
    igb1: Bound queue 0 to cpu 0
    igb1: Bound queue 1 to cpu 1
    igb1: Bound queue 2 to cpu 2
    igb1: Bound queue 3 to cpu 3
    igb1: netmap queues/slots: TX 4/1024, RX 4/1024
    igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdef00000-0xdeffffff,0xdf284000-0xdf287fff at device 0.2 on pci1
    igb2: Using MSIX interrupts with 5 vectors
    igb2: Ethernet address: b4:96:91:13:32:42
    igb2: Bound queue 0 to cpu 0
    igb2: Bound queue 1 to cpu 1
    igb2: Bound queue 2 to cpu 2
    igb2: Bound queue 3 to cpu 3
    igb2: netmap queues/slots: TX 4/1024, RX 4/1024
    igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdee00000-0xdeefffff,0xdf280000-0xdf283fff at device 0.3 on pci1
    igb3: Using MSIX interrupts with 5 vectors
    igb3: Ethernet address: b4:96:91:13:32:43
    igb3: Bound queue 0 to cpu 0
    igb3: Bound queue 1 to cpu 1
    igb3: Bound queue 2 to cpu 2
    igb3: Bound queue 3 to cpu 3
    igb3: netmap queues/slots: TX 4/1024, RX 4/1024
    pcib2: <ACPI PCI-PCI bridge> at device 1.1 on pci0
    pci2: <ACPI PCI bus> on pcib2
    AVAGO MegaRAID SAS FreeBSD mrsas driver version: 06.709.07.00-fbsd
    mfi0: <Fury> port 0xe000-0xe0ff mem 0xdf500000-0xdf50ffff,0xdf400000-0xdf4fffff at device 0.0 on pci2
    mfi0: Using MSI
    mfi0: Megaraid SAS driver Ver 4.23
    mfi0: FW MaxCmds = 240, limiting to 128
    mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0
    mfi0: 465 (581773451s/0x0020/info) - Shutdown command received from host
    mfi0: 466 (boot + 9s/0x0020/info) - Firmware initialization started (PCI ID 005f/1000/1211/1734)
    mfi0: 467 (boot + 9s/0x0020/info) - Firmware version 4.270.01-4870
    mfi0: 468 (boot + 11s/0x0020/info) - Package version 24.7.0-0061
    mfi0: 469 (boot + 11s/0x0020/info) - Board Revision
    mfi0: 470 (boot + 49s/0x0002/info) - Inserted: PD 02(e0x3e/s0)
    xhci0: mfi0: <Intel Sunrise Point USB 3.0 controller> mem 0xdf800000-0xdf80ffff at device 20.0 on pci0
    xhci0: 32 bytes context size, 64-bit DMA
    471 (boot + 49s/0x0002/info) - Inserted: PD 02(e0x3e/s0) Info: enclPd=3e, scsiType=0, portMap=00, sasAddr=4433221102000000,000usbus00000000000000
    mfi0: 472 (boot + 49s/0x0002/info) - Inserted: PD 03(e0x3e/s1)
     on xhci0
    mfi0: usbus0: 5.0Gbps Super Speed USB v3.0
    473 (boot + 49s/0x0002/info) - Inserted: PD 03(e0x3e/s1) Info: enclPd=3e, scsiType=0, portMap=01, sasAddr=4433221103000000,0000000000000000
    mfi0: 474 (boot + 49s/0x0020/info) - Controller operating temperature within normal range, full operation restored
    mfi0: 475 (581772985s/0x0020/info) - Time established as 06/08/18 11:36:25; (50 seconds since power on)
    pci0: <simple comms> at device 22.0 (no driver attached)
    pci0: <simple comms> at device 22.1 (no driver attached)
    ahci0: <Intel Sunrise Point (RAID) AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xdf814000-0xdf815fff,0xdf818000-0xdf8180ff,0xdf817000-0xdf8177ff at device 23.0 on pci0
    ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
    ahcich0: <AHCI channel> at channel 0 on ahci0
    ahcich1: <AHCI channel> at channel 1 on ahci0
    ahcich2: <AHCI channel> at channel 2 on ahci0
    ahcich3: <AHCI channel> at channel 3 on ahci0
    ahcich4: <AHCI channel> at channel 4 on ahci0
    ahcich5: <AHCI channel> at channel 5 on ahci0
    ahciem0: <AHCI enclosure management bridge> on ahci0
    pcib3: <ACPI PCI-PCI bridge> at device 28.0 on pci0
    pci3: <ACPI PCI bus> on pcib3
    vgapci0: <VGA-compatible display> mem 0xdd000000-0xddffffff,0xde820000-0xde823fff,0xde000000-0xde7fffff at device 0.0 on pci3
    pci3: <processor> at device 0.1 (no driver attached)
    pcib4: <ACPI PCI-PCI bridge> at device 28.5 on pci0
    pci4: <ACPI PCI bus> on pcib4
    igb4: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc000-0xc01f mem 0xdf600000-0xdf67ffff,0xdf680000-0xdf683fff at device 0.0 on pci4
    igb4: Using MSIX interrupts with 5 vectors
    igb4: Ethernet address: 90:1b:0e:d3:53:23
    igb4: Bound queue 0 to cpu 0
    igb4: Bound queue 1 to cpu 1
    igb4: Bound queue 2 to cpu 2
    igb4: Bound queue 3 to cpu 3
    igb4: netmap queues/slots: TX 4/1024, RX 4/1024
    pcib5: <ACPI PCI-PCI bridge> at device 28.6 on pci0
    pci5: <ACPI PCI bus> on pcib5
    igb5: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xd000-0xd01f mem 0xdf700000-0xdf77ffff,0xdf780000-0xdf783fff at device 0.0 on pci5
    igb5: Using MSIX interrupts with 5 vectors
    igb5: Ethernet address: 90:1b:0e:d3:57:24
    igb5: Bound queue 0 to cpu 0
    igb5: Bound queue 1 to cpu 1
    igb5: Bound queue 2 to cpu 2
    igb5: Bound queue 3 to cpu 3
    igb5: netmap queues/slots: TX 4/1024, RX 4/1024
    pcib6: <ACPI PCI-PCI bridge> at device 29.0 on pci0
    pci6: <ACPI PCI bus> on pcib6
    igb6: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdec00000-0xdecfffff,0xded8c000-0xded8ffff at device 0.0 on pci6
    igb6: Using MSIX interrupts with 5 vectors
    igb6: Ethernet address: b4:96:91:13:29:70
    igb6: Bound queue 0 to cpu 0
    igb6: Bound queue 1 to cpu 1
    igb6: Bound queue 2 to cpu 2
    igb6: Bound queue 3 to cpu 3
    igb6: netmap queues/slots: TX 4/1024, RX 4/1024
    igb7: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdeb00000-0xdebfffff,0xded88000-0xded8bfff at device 0.1 on pci6
    igb7: Using MSIX interrupts with 5 vectors
    igb7: Ethernet address: b4:96:91:13:29:71
    igb7: Bound queue 0 to cpu 0
    igb7: Bound queue 1 to cpu 1
    igb7: Bound queue 2 to cpu 2
    igb7: Bound queue 3 to cpu 3
    igb7: netmap queues/slots: TX 4/1024, RX 4/1024
    igb8: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xdea00000-0xdeafffff,0xded84000-0xded87fff at device 0.2 on pci6
    igb8: Using MSIX interrupts with 5 vectors
    igb8: Ethernet address: b4:96:91:13:29:72
    igb8: Bound queue 0 to cpu 0
    igb8: Bound queue 1 to cpu 1
    igb8: Bound queue 2 to cpu 2
    igb8: Bound queue 3 to cpu 3
    igb8: netmap queues/slots: TX 4/1024, RX 4/1024
    igb9: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xde900000-0xde9fffff,0xded80000-0xded83fff at device 0.3 on pci6
    igb9: Using MSIX interrupts with 5 vectors
    igb9: Ethernet address: b4:96:91:13:29:73
    igb9: Bound queue 0 to cpu 0
    igb9: Bound queue 1 to cpu 1
    igb9: Bound queue 2 to cpu 2
    igb9: Bound queue 3 to cpu 3
    igb9: netmap queues/slots: TX 4/1024, RX 4/1024
    isab0: <PCI-ISA bridge> at device 31.0 on pci0
    isa0: <ISA bus> on isab0
    pci0: <memory> at device 31.2 (no driver attached)
    acpi_button0: <Power Button> on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart0: console (115200,n,8,1)
    ppc0: cannot reserve I/O port range
    est0: <Enhanced SpeedStep Frequency Control> on cpu0
    est1: <Enhanced SpeedStep Frequency Control> on cpu1
    est2: <Enhanced SpeedStep Frequency Control> on cpu2
    est3: <Enhanced SpeedStep Frequency Control> on cpu3
    Timecounters tick every 1.000 msec
    nvme cam probe device init
    ugen0.1: <0x8086 XHCI root HUB> at usbus0
    mfid0 on mfi0
    mfid0: 953344MB (1952448512 sectors) RAID volume (no label) is optimal
    uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    uhub0: 26 ports with 26 removable, self powered
    ugen0.2: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0
    ukbd0 on uhub0
    ukbd0: <Keyboard Interface> on usbus0
    kbd1 at ukbd0
    ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
    ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
    cd0 at ahcich5 bus 0 scbus5 target 0 lun 0
    ses0: SEMB SES Device
    cd0: <HL-DT-ST DVDRAM GUD0N PF02> Removable CD-ROM SCSI device
    cd0: Serial Number B7ARVC1211252
    cd0: 150.000MB/s transfers (SATA 1.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
    cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
    Trying to mount root from ufs:/dev/gptid/f21e550d-4955-11e8-b400-b49691133240 [rw]...
    CPU: Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz (3000.10-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
      Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
      Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
      AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
      AMD Features2=0x121<LAHF,ABM,Prefetch>
      Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
      Structured Extended Features3=0xc000000<IBPB,STIBP>
      XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
      VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
      TSC: P-state invariant, performance statistics
    tun1: changing name to 'ovpns1'
    tun2: changing name to 'ovpns2'
    tun3: changing name to 'ovpns3'
    ovpns2: link state changed to UP
    ovpns1: link state changed to UP
    ovpns3: link state changed to UP
    pflog0: promiscuous mode enabled
    igb4: link state changed to UP
    igb9: promiscuous mode enabled
    igb9: promiscuous mode disabled
    

  • Netgate Administrator

    Hmm, is the firewall actually idle? No traffic at all?

    I'd have to guess this is related to your raid setup since I don't have anything similar to test against.
    The mfi device is the highest interrupt load other than the cpu timers. I could be tickling the cpus sufficiently to cause that.

    If you;re not actually seeing an issue there I doubt it's a problem.

    You could move /tmp and /var to RAM drives as a test which prevent drive writes. That may reduce both interrupt loads.

    Steve



  • @stephenw10 said in High CPU usage on interrupt processing:

    Hmm, is the firewall actually idle? No traffic at all?

    Yeah, firewall is idle. The server is located in a small subnet and no one is currently connected to it.
    There’s some occasional broadcast traffic (e.g. DHCP requests) form other devices/servers on the network, but I doubt it can cause any trouble.

    I think you’re right about RAID.
    I moved /tmp and /var to RAM but still there’s a lot of interrupts on mfi0 device, yet gstat shows no disk IO except some rare writes.

    I’ll try remotely tinker with RAID-related settings in BIOS, maybe I find something I missed.
    Thank you for suggestions!


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy