100%+ CPU Spikes causing DNS outages - 2.4.5-RELEASE-p1 (amd64) - unbound/unbound-controller
-
Hi,
Ive been running 2.4.5-RELEASE-p1 (amd64) since release and all ok however since a recent reboot i now have unbound causing 100%+ CPU spikes. This was present in 2.4.5-RELEASE (amd64) but not in 2.4.4-RELEASE-p3 (amd64). this causes DNS outages frequently for 10-15 seconds at a time. culprit is either: unbound/unbound-controller.
There are not errors recorded in the system or DNS resolver logs.
-
one example of unbound-control from top, albeit it not completely hitting 100%
https://imgur.com/a/7ivbJnO
-
and another with unbound. From the looks of my monitoring, its unbound that gets stuck at 100% cpu most often for a period between 10-20 seconds.
https://imgur.com/a/lMRm3Yv
-
I can confirm that I have also suffered what appears to be temporary network issues. Unfortunately I have never actually caught the culprit.
Taking a sniff at top just now does in fact show after about 16 days uptime that unbound has eaten 79:13 cpu time. ...which is way higher than snort at around 7 minutes. It appears that unbound has become a cpu hog; but I can't really find anything interesting in the logs either.
For the record hardware info:
pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0xb30119da chip=0x22808086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series SoC Transaction Register' class = bridge subclass = HOST-PCI vgapci0@pci0:0:2:0: class=0x030000 card=0xb30119da chip=0x22b18086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Integrated Graphics Controller' class = display subclass = VGA ahci0@pci0:0:19:0: class=0x010601 card=0xb30119da chip=0x22a38086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series SATA Controller' class = mass storage subclass = SATA xhci0@pci0:0:20:0: class=0x0c0330 card=0xb30119da chip=0x22b58086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series USB xHCI Controller' class = serial bus subclass = USB none0@pci0:0:26:0: class=0x108000 card=0xb30119da chip=0x22988086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series Trusted Execution Engine' class = encrypt/decrypt hdac0@pci0:0:27:0: class=0x040300 card=0xb30119da chip=0x22848086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series High Definition Audio Controller' class = multimedia subclass = HDA pcib1@pci0:0:28:0: class=0x060400 card=0xb30119da chip=0x22c88086 rev=0x35 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCI Express Port' class = bridge subclass = PCI-PCI pcib2@pci0:0:28:1: class=0x060400 card=0xb30119da chip=0x22ca8086 rev=0x35 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCI Express Port' class = bridge subclass = PCI-PCI pcib3@pci0:0:28:2: class=0x060400 card=0xb30119da chip=0x22cc8086 rev=0x35 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCI Express Port' class = bridge subclass = PCI-PCI pcib4@pci0:0:28:3: class=0x060400 card=0xb30119da chip=0x22ce8086 rev=0x35 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCI Express Port' class = bridge subclass = PCI-PCI isab0@pci0:0:31:0: class=0x060100 card=0xb30119da chip=0x229c8086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCU' class = bridge subclass = PCI-ISA none1@pci0:0:31:3: class=0x0c0500 card=0xb30119da chip=0x22928086 rev=0x35 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx SMBus Controller' class = serial bus subclass = SMBus re0@pci0:2:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x0c hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller' class = network subclass = ethernet re1@pci0:3:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x0c hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller' class = network subclass = ethernet iwm0@pci0:4:0:0: class=0x028000 card=0x40108086 chip=0x31658086 rev=0x81 hdr=0x00 vendor = 'Intel Corporation' device = 'Wireless 3165' class = networkYes I realized realek sucks; but I still love this little fanless zotac box and the USB items don't look to be much better.
It might be related to this lovenote on reddit:
https://www.reddit.com/r/PFSENSE/comments/fr70my/unbound_high_cpu_when_wan_gateway_down/Last time I looked at dpinger there wasn't a solid correlation but maybe I'm just silly.
-
@skogs Glad to know im not alone!
Im running on a little microPC also which was fully stable on all the 2.4.4 iterations, only with 2.4.5 has unbound been an issue.
pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x22128086 chip=0x0f008086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx Series SoC Transaction Register' class = bridge subclass = HOST-PCI vgapci0@pci0:0:2:0: class=0x030000 card=0x22128086 chip=0x0f318086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx Series Graphics & Display' class = display subclass = VGA ahci0@pci0:0:19:0: class=0x010601 card=0x0f238086 chip=0x0f238086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series SATA AHCI Controller' class = mass storage subclass = SATA xhci0@pci0:0:20:0: class=0x0c0330 card=0x0f358086 chip=0x0f358086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx, Celeron N2000 Series USB xHCI' class = serial bus subclass = USB sdhci_pci0@pci0:0:23:0: class=0x080501 card=0x00000000 chip=0x0f508086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series eMMC 4.5 Controller' class = base peripheral subclass = SD host controller none0@pci0:0:26:0: class=0x108000 card=0x0f188086 chip=0x0f188086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine' class = encrypt/decrypt hdac0@pci0:0:27:0: class=0x040300 card=0x72708086 chip=0x0f048086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx Series High Definition Audio Cont roller' class = multimedia subclass = HDA pcib1@pci0:0:28:0: class=0x060400 card=0x0f488086 chip=0x0f488086 rev=0x11 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series PCI Express Root Port 1' class = bridge subclass = PCI-PCI pcib2@pci0:0:28:1: class=0x060400 card=0x0f4a8086 chip=0x0f4a8086 rev=0x11 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series PCI Express Root Port 2' class = bridge subclass = PCI-PCI pcib3@pci0:0:28:2: class=0x060400 card=0x0f4c8086 chip=0x0f4c8086 rev=0x11 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series PCI Express Root Port 3' class = bridge subclass = PCI-PCI pcib4@pci0:0:28:3: class=0x060400 card=0x0f4e8086 chip=0x0f4e8086 rev=0x11 hdr=0x01 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series PCI Express Root Port 4' class = bridge subclass = PCI-PCI isab0@pci0:0:31:0: class=0x060100 card=0x0f1c8086 chip=0x0f1c8086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor Z36xxx/Z37xxx Series Power Control Unit' class = bridge subclass = PCI-ISA none1@pci0:0:31:3: class=0x0c0500 card=0x0f128086 chip=0x0f128086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Atom Processor E3800 Series SMBus Controller' class = serial bus subclass = SMBus igb0@pci0:1:0:0: class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I211 Gigabit Network Connection' class = network subclass = ethernet igb1@pci0:2:0:0: class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I211 Gigabit Network Connection' class = network subclass = ethernet igb2@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I211 Gigabit Network Connection' class = network subclass = ethernet igb3@pci0:4:0:0: class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I211 Gigabit Network Connection' class = network subclass = ethernetI have no WAN issues/drops, its all up when this hap[ns as programs and other applications continue to work during this period, however anything that is newly requesting DNS resolution fails.
-
I'm running 2.4.5 on a Zotac mini PC and this problem is driving me crazy. It lasts for between 1-3 minutes. I can ping everything by IP, but name resolution doesn't work AND the PFSense GUI is unresponsive, even when accessing it by IP address (it's probably trying to do some recursive lookup).
-
@uh2 as i say, ive only had this issue since 2.4.5...
Its getting worse for me on a daily basis, at first it was every hour or so, it now seems to be every 15 minutes.
-
Hello!
Maybe related to
https://forum.netgate.com/topic/115482/frequent-unbound-restarts/26
John
Lex parsimoniae
-
@serbus thanks i'll take a look
-
@serbus thanks for this, i think "DHCP Registration" has provided a workaround for me!
-
Had a pair of outages tonight that unbound seems to have made much longer than necessary.
Lots of the following in the log.unbound: [23707:1] error: outgoing tcp: bind: Can't assign requested address unbound: [23707:1] error: outgoing tcp: bind: Can't assign requested address unbound: [23707:3] error: no TCP outgoing interfaces of family unbound: [23707:3] notice: for addr 2001:4860:4860::8844 port 853 unbound: [23707:3] error: no TCP outgoing interfaces of family unbound: [23707:3] notice: for addr 2001:4860:4860::8844 port 853 -
Unbound-control is what pfBlockerNG-devel uses if you enable the option to do live updates isn’t it?
-
@motific Not sure; not using pfblocker-ng.
