High CPU load [ 63.87% {irq16: bge0 bge1} ]
-
Hi!
I found high CPU load. System activiy shows something like this:
63.87% {irq16: bge0 bge1}bge0 and bge1 are WAN NICs Broadcom BCM5721. I'm using GW monitoring/failower/loadbalancing.
-
What version of pfSense?
What system or motherboard? Are the NICs on motherboard? separate cards? multiport card? What WAN speed?
Is anything else on IRQ 16?
Please post the output of pfSense shell command```
vmstat -i -
$ vmstat -i
interrupt total rate
irq1: atkbd0 181 0
irq16: bge0 bge1+ 7310876674 8939
irq20: atapci1 1102708 1
irq23: ehci0 ehci1 1635786 2
cpu0: timer 1635695497 2000
cpu1: timer 1635695487 2000
Total 10585006333 12942pfSense 2.0.3
MB: Asus P8Q67
Broadcom NICs are in PCIe, Intel NIC is in PCI. All cards are separate, single-port. WAN speed is 100Mbps each but load is very low - less then 10 Mbps in averageAlso the problem it that pfSense often detect gateway as down:
May 26 11:36:50 check_reload_status: Reloading filter
May 26 11:36:40 apinger: alarm canceled: WANGW(173.194.39.160) *** loss ***
May 26 11:35:39 check_reload_status: Reloading filter
May 26 11:35:29 apinger: ALARM: WANGW(173.194.39.160) *** loss ***
May 26 11:32:56 check_reload_status: Reloading filter
May 26 11:32:46 apinger: alarm canceled: WANGW(173.194.39.160) *** loss ***
May 26 11:30:08 check_reload_status: Reloading filter -
Any ideas?
-
You are getting around 9000 interrupts per sec on irq16. bge0, bge1 and one or more other devices are on irq16. It is impossible to determine the split of interrupts between devices.
The pfSense shell command
dmesg | grep irq16
should identify all devices on irq16. Perhaps there are devices on irq16 that could be disabled in the BIOS because they aren't used. Audio? USB?
If you are worried about the CPU usage (perhaps because the system seems sluggish) I would be inclined to try pfSense 2.1 RC0 which has much more recent device drivers than the pfSense 2.0.x series. It has been my experience that FreeBSD can be a bit "troubled" on "too new" equipment, for example, hanging during boot unless a USB controller is disabled but the next version boots fine, with or without the same USB controller disabled.
-
Thanks!
I'll try to install 2.1 RC when I'll get second the same one server because I can't stop this one for an update -
Same problem with 2.1
$ dmesg | grep "irq 16"
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
bge0: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe610000-0xfe61ffff irq 16 at device 0.0 on pci1
vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
bge1: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe510000-0xfe51ffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0xe000-0xe03f mem 0xfe440000-0xfe45ffff,0xfe420000-0xfe43ffff irq 16 at device 1.0 on pci4</intel(r)></broadcom></vga-compatible></broadcom></acpi> -
Same problem with 2.1
I presume you mean you are still seeing high CPU loads for the irq16 task.
$ dmesg | grep "irq 16"
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
bge0: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe610000-0xfe61ffff irq 16 at device 0.0 on pci1
vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
bge1: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe510000-0xfe51ffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0xe000-0xe03f mem 0xfe440000-0xfe45ffff,0xfe420000-0xfe43ffff irq 16 at device 1.0 on pci4</intel(r)></broadcom></vga-compatible></broadcom></acpi>This is not an optimal configuration. On every interrupt from one of bge0, bge1, em1 the system has to call the bge interrupt service routine twice and the em interrupt service routine once. It would be better if each of the NICs was on a separate irq.
Maybe you could move the em1 to another slot and get a different irq that way. Maybe some of the NICs are MSI capable but are not using MSI because MSI has been disabled through a sysctl. (Some NICs say they are MSI capable but they have a broken MSI implementation.) Maybe the BIOS has some capability for tweaking irq assignments.
Please post the output of pfSense shell commands:```
dmesg
/etc/rc.banner -
$ dmesg
Waiting (max 60 seconds) for system processvnlru' to stop…done Waiting (max 60 seconds) for system processbufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 0 0 done
All buffers synced.
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE-p8 #0: Wed May 29 17:27:49 EDT 2013
root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) CPU G860 @ 3.00GHz (3000.05-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7
Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x59ae3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,popcnt,tscdlt,xsave>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 1868570624 (1782 MB)
ACPI APIC Table: <alaska a="" m="" i="">FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
ioapic0 <version 2.0="">irqs 0-23 on motherboard
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804a5250, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804a52f0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804a5390, 0) error 1
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <alaska a="" m="" i="">on motherboard
acpi0: [ITHREAD]
ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20101013/psargs-464)
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20101013/nsinit-452)
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
bge0: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe610000-0xfe61ffff irq 16 at device 0.0 on pci1
bge0: CHIP ID 0x00004101; ASIC REV 0x04; CHIP REV 0x41; PCI-E
miibus0: <mii bus="">on bge0
brgphy0: <bcm5750 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: [ITHREAD]
vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
pci0: <simple comms="">at device 22.0 (no driver attached)
atapci0: <intel ata="" controller="">port 0xf130-0xf137,0xf120-0xf123,0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0ff irq 18 at device 22.2 on pci0
atapci0: [ITHREAD]
ata2: <ata channel="">at channel 0 on atapci0
ata2: [ITHREAD]
ata3: <ata channel="">at channel 1 on atapci0
ata3: [ITHREAD]
uart2: <non-standard ns8250="" class="" uart="" with="" fifos="">port 0xf0e0-0xf0e7 mem 0xfe725000-0xfe725fff irq 17 at device 22.3 on pci0
uart2: [FILTER]
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xf080-0xf09f mem 0xfe700000-0xfe71ffff,0xfe724000-0xfe724fff irq 18 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: [FILTER]
ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe723000-0xfe7233ff irq 23 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
pci2: <acpi pci="" bus="">on pcib2
bge1: <broadcom netxtreme="" pcie="" ethernet="" adapter,="" asic="" rev.="" 0x004101="">mem 0xfe510000-0xfe51ffff irq 16 at device 0.0 on pci2
bge1: CHIP ID 0x00004101; ASIC REV 0x04; CHIP REV 0x41; PCI-E
miibus1: <mii bus="">on bge1
brgphy1: <bcm5750 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: [ITHREAD]
pcib3: <acpi pci-pci="" bridge="">irq 19 at device 28.7 on pci0
pci3: <acpi pci="" bus="">on pcib3
ehci1: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe722000-0xfe7223ff irq 23 at device 29.0 on pci0
ehci1: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci1
pcib4: <acpi pci-pci="" bridge="">at device 30.0 on pci0
pci4: <acpi pci="" bus="">on pcib4
em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0xe000-0xe03f mem 0xfe440000-0xfe45ffff,0xfe420000-0xfe43ffff irq 16 at device 1.0 on pci4
em1: [FILTER]
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci1: <intel cougar="" point="" sata300="" controller="">port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe721000-0xfe7217ff irq 20 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI v1.30 controller with 6 6Gbps ports, PM not supported
ata4: <ata channel="">at channel 1 on atapci1
ata4: [ITHREAD]
pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
acpi_button0: <power button="">on acpi0
acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <isa option="" roms="">at iomem 0xce000-0xcffff,0xd0000-0xd1fff,0xd2000-0xd2fff on isa0
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
est0: <enhanced speedstep="" frequency="" control="">on cpu0
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
est1: <enhanced speedstep="" frequency="" control="">on cpu1
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad8: 30533MB <spcc 120703="" solid="" state="" disk="" b29="">at ata4-master UDMA100 SATA 3Gb/s
SMP: AP CPU #1 Launched!
ugen1.1: <intel>at usbus1ugen0.1: <intel>at usbus0
uhub0:
<intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
Root mount waiting for: usbus1 usbus0
uhub1: 3 ports with 3 removable, self powered
uhub0: 3 ports with 3 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087="">at usbus0
uhub2: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
ugen1.2: <vendor 0x8087="">at usbus1
uhub3: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
uhub2: 6 ports with 6 removable, self powered
Root mount waiting for: usbus1
uhub3: 8 ports with 8 removable, self powered
ugen1.3: <a4tech>at usbus1
ums0: <a4tech 0="" 3="" usb="" optical="" mouse,="" class="" 0,="" rev="" 1.10="" 0.01,="" addr="">on usbus1
ums0: 8 buttons and [XYZ] coordinates ID=0
Root mount waiting for: usbus1
Root mount waiting for: usbus1
ugen1.4: <www.i-odd.com>at usbus1
umass0: <iodd2511>on usbus1
umass0: SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0
Attached to scbus0
Trying to mount root from ufs:/dev/ad8s1a
cd0 at umass-sim0 bus 0 scbus0 target 0 lun 0
cd0: <iodd virtual="" cdrom="" 1.42="">Removable CD-ROM SCSI-0 device
cd0: 40.000MB/s transfers
cd0: cd present [58345 x 2048 byte records]
da0 at umass-sim0 bus 0 scbus0 target 0 lun 1
da0: <iodd ext="" hard="" disk="">Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 305245MB (625142448 512 byte sectors: 255H 63S/T 38913C)
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
Consider tuning vm.kmem_size and vm.kmem_size_max
in /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
arpresolve: can't allocate llinfo for 31.42.48.1
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
ugen1.3: <a4tech>at usbus1 (disconnected)
ums0: at uhub3, port 2, addr 3 (disconnected)
ugen1.4: <www.i-odd.com>at usbus1 (disconnected)
umass0: at uhub3, port 3, addr 4 (disconnected)
(cd0:umass-sim0:0:0:0): lost device
(cd0:umass-sim0:0:0:0): removing device entry
(da0:umass-sim0:0:0:1): lost device - 0 outstanding, 0 refs
(da0:umass-sim0:0:0:1): removing device entry
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled</www.i-odd.com></a4tech></cpu></cpu></iodd></iodd></iodd2511></www.i-odd.com></a4tech></a4tech></vendor></vendor></vendor></vendor></intel></intel></intel></intel></spcc></cpu></enhanced></cpu></enhanced></generic></system></isa></at></keyboard></at></high></power></serial></ata></intel></isa></pci-isa></intel(r)></acpi></acpi></ehci></ehci></acpi></acpi></bcm5750></mii></broadcom></acpi></acpi></ehci></ehci></intel(r)></non-standard></ata></ata></intel></simple></vga-compatible></bcm5750></mii></broadcom></acpi></acpi></acpi></acpi></acpi></acpi></alaska></software></version></alaska></lahf></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,popcnt,tscdlt,xsave></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
WAN speed is 100Mbps each but load is very low - less then 10 Mbps in average
You have 2 WAN links downloading about 10Mbps. Suppose an average frame size of 5000 bits (about 600 bytes) which would yield 2000 frames/sec. Suppose 1 interrupt per received frame. Then there would be 2000x2 = 4000 interrupts/sec for the download traffic. Presumably that download traffic goes somewhere outside pfSense so that would probably give a total of 8000 interrupts/sec. That download traffic would generally require some ACKs so the observed interrupt rate of about 9k/sec is probably reasonable.
Since em0 didn't appear in the vmstat output I presume the vast bulk of the download traffic is forwarded over em1. As already discussed, the two WAN links and em1 are all on irq16 which can add to the system overhead.
It would appear the easiest way to see if reducing the sharing of interrupt load on the system would help reduce time spent in irq16 would be to remove em1 from the system and use em0 (apparently the onboard NIC) instead. (Then irq16 is shared only by bge0 and bge1). em0 uses MSI. Message Signalled Interrupts are not shared.
-
The problem was in device polling. It was good idea (for now) to disable it :)
-
The problem was in device polling. It was good idea (for now) to disable it :)
This doesn't make sense. The vmstat output showed active interrupts from the NICs so polling should have been already disabled.
-
The problem was in device polling. It was good idea (for now) to disable it :)
This doesn't make sense. The vmstat output showed active interrupts from the NICs so polling should have been already disabled.
But when I disable poling everything became fine and now CPU load is less than 10% and everything works perfect!
-
Polling made my box eat 100% CPU with vr driver. The thing is really buggy, better not touched.
-
When you use device polling the CPU load is always going to increase. Depending on your hardware you may end up using all the spare CPU cycles. The box should remain responsive though. The pay off is that the maximum throughput may increase. However: http://blog.pfsense.org/?p=115#comment-21378
Steve
-
The box should remain responsive though.
Responsive - not really. Was barely recoverable by reverting to previous config via serial console.
