High CPU/Interrupt usage with little traffic

sporkme · Feb 22, 2016, 10:40 PM

More info on the NICs:


re1:

rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus2
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re1: <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">port 0xcc00-0xccff mem 0xfe2ff000-0xfe2ff0ff irq 16 at device 0.0 on pci3
re1: Chip rev. 0x10000000
re1: MAC rev. 0x00000000
miibus2: <mii bus="">on re1

re0: 

rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xdc00-0xdcff mem 0xfe5ff000-0xfe5fffff,0xd0000000-0xd000ffff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x3c000000
re0: MAC rev. 0x00400000
miibus0: <mii bus="">on re0

bge0:

brgphy0: <bcm57780 1000base-t="" media="" interface="">PHY 1 on miibus1
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: <broadcom bcm57780="" a1,="" asic="" rev.="" 0x57780001="">mem 0xfe4f0000-0xfe4fffff irq 16 at device 0.0 on pci2
bge0: CHIP ID 0x57780001; ASIC REV 0x57780; CHIP REV 0x577800; PCI-E
miibus1: <mii bus="">on bge0</mii></broadcom></bcm57780></mii></realtek></rtl8169s></mii></realtek></rtl8169s>

geostar · Feb 24, 2016, 2:56 AM

Strange I'm running version 2.2.6 and every sunday night at round midnight the cpu usage goes high on low traffic and stays that way.
If I reboot the server the problem goes away and cpu usage goes back to 3% from 45%.

Do I have a sunday only bug ?

sporkme · Feb 24, 2016, 3:32 AM

I think I might troll around the freebsd-net list to see if anyone can spot something obvious. Not terribly strange hardware here, old enough to be well supported. There's probably some tweaking to some weird boot loader variable for the bge driver that will do something. I suspect the pfsense devs are probably prioritizing paid subs and don't much frequent the forums these days.

Harvy66 · Feb 24, 2016, 12:56 PM

@sporkme:

Not terribly strange hardware here, old enough to be well supported.

Realtek is rarely ever well supported. It's a crappy brand with crappy drivers.

sporkme · Feb 24, 2016, 9:00 PM

@Harvy66:

@sporkme:

Not terribly strange hardware here, old enough to be well supported.

Realtek is rarely ever well supported. It's a crappy brand with crappy drivers.

Regardless, it's pretty much THE brand you're going to end up with when you buy an ethernet card. I'm sure I'm not the only one running pfsense with Realtek cards. :)

Guest · Feb 24, 2016, 11:49 PM

@sporkme:

@Harvy66:

@sporkme:

Not terribly strange hardware here, old enough to be well supported.

Realtek is rarely ever well supported. It's a crappy brand with crappy drivers.

Regardless, it's pretty much THE brand you're going to end up with when you buy an ethernet card. I'm sure I'm not the only one running pfsense with Realtek cards. :)

What kind of services are running, what kind of packets are installed and in usage?
That Realtek NICs are not so well performing, based on the driver support is not really sad or making me angry,
but if I am going to use this Realtek NICs then and the CPU will be not so well offloaded likes the Intel NICs are
doing it, I should also not running wild and searching for other things then the RealTek NICs that are not so well
performing like the Intel ones.

Go with Intel or live with the odd circumstances based on the more bad RealTek driver support and cheaper
hardware parts of them. For sure not all will be so pointed in that direction, but many of them.

darkcrucible · Feb 25, 2016, 12:07 AM

There's an old thread that's similar to this. Can't tell if the interrupt source is the same. Are you plugging/unplugging the VGA cable?

https://forum.pfsense.org/index.php?topic=71589.0

sporkme · Feb 25, 2016, 12:17 AM

@darkcrucible:

There's an old thread that's similar to this. Can't tell if the interrupt source is the same. Are you plugging/unplugging the VGA cable?

https://forum.pfsense.org/index.php?topic=71589.0

That's bizarre.

Everything seems to be on IRQ 16:

[2.2.6-RELEASE][admin@gw.com]/root: grep "irq 16" /var/log/dmesg.boot
pcib1: <acpi pci-pci="" bridge=""> irq 16 at device 1.0 on pci0
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet=""> port 0xdc00-0xdcff mem 0xfe5ff000-0xfe5fffff,0xd0000000-0xd000ffff irq 16 at device 0.0 on pci1
vgapci0: <vga-compatible display=""> port 0xecd8-0xecdf mem 0xfe800000-0xfebfffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
pcib2: <acpi pci-pci="" bridge=""> irq 16 at device 28.0 on pci0
bge0: <broadcom bcm57780="" a1,="" asic="" rev.="" 0x57780001=""> mem 0xfe4f0000-0xfe4fffff irq 16 at device 0.0 on pci2
re1: <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet=""> port 0xcc00-0xccff mem 0xfe2ff000-0xfe2ff0ff irq 16 at device 0.0 on pci3
atapci0: <intel ich7="" udma100="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf irq 16 at device 31.1 on pci0
[2.2.6-RELEASE][admin@gw.sporklab.com]/root:</intel></realtek></broadcom></acpi></vga-compatible></realtek></acpi>

Harvy66 · Feb 25, 2016, 1:04 PM

@sporkme:

@Harvy66:

@sporkme:

Not terribly strange hardware here, old enough to be well supported.

Realtek is rarely ever well supported. It's a crappy brand with crappy drivers.

Regardless, it's pretty much THE brand you're going to end up with when you buy an ethernet card. I'm sure I'm not the only one running pfsense with Realtek cards. :)

Correct, you're also not the only one complaining about their Realtek NIC having issues. I only use Intel on all of my computers, even if that means I have to purchase a $70 NIC because my motherboard doesn't have one.

Guest · Feb 25, 2016, 5:44 PM

bge0: <broadcom bcm57780="" a1,="" asic<="" pre="">
Could it perhaps be, that the ASIC on the NIC is causing this higher interrupt usage? 
So it would be perhaps also a chance to get rid of the Broadcom and Realtek NICs and
you might be testing it once more again out only with an Intel Quad LAN Port NIC alone.

Perhaps you will see then better results like the actual one.</broadcom>

sporkme · Feb 28, 2016, 11:33 PM

Well, updated BIOS from A05 to A07 and after the reboot for that, CPU usage is back to normal and has remained so for a few days. So either that BIOS update corrected something or the reboot temporarily masked the problem. I suspect the BIOS was the fix since my RRD graphs show that there was no dip in CPU usage after previous reboots.

As for Realtek, I still think it's best to work with them if possible. Plenty of home users of pfSense that are not going to spend $60/each on NICs. The Realtek's may suck if you really need full line rate 24/7, but as long as I can get 100Mb/s in each direction, I'm happy (as I would think would be the case with most home users).

One of my FreeBSD buddies does state that they do officially support Realtek, and rather than telling people to go run Linux (where the Realteks are not as flaky) or switch to something else, users should open bug reports if there seems to be a real driver issue.