pfBlockerNG - unbound-control process spikes CPU to 100% every few seconds [SOLVED]

nimrod

Im running pfSense v2.6.0 on Protectli FW6D with 16GB of RAM. I have latest pfBlockerNG installed, and im getting 100% CPU usage spikes every 7 seconds.

The output of

top -a

Is returning this:

last pid: 67809;  load averages:  0.60,  0.71,  0.83                                                                             up 0+12:26:03  16:10:47
60 processes:  1 running, 59 sleeping
CPU:  2.7% user,  0.0% nice,  0.1% system,  0.0% interrupt, 97.1% idle
Mem: 1447M Active, 500M Inact, 2795M Wired, 11G Free
ARC: 762M Total, 640M MFU, 113M MRU, 32K Anon, 2848K Header, 6733K Other
     311M Compressed, 1038M Uncompressed, 3.33:1 Ratio
Swap: 1024M Total, 1024M Free

Message from syslogd@pfSense at Jul  9 15:57:31 ...   C   TIME    WCPU COMMAND

17570 unbound]: /ind1  78    0   210M   197M CPU1     1   0:03  99.92% /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf stats_noreset

Or if its easier to see:

I found that disabling pfBlockerNG is stopping these high CPU usage spikes. Is this normal behavior, or im having some sort of miss configuration? These spikes are keeping unit very hot to the touch.

Thank you.

johnpoz

@nimrod this is a known issue with pfblocker, there is a simple fix

https://redmine.pfsense.org/issues/13154

see at the bottom of the redmine for simple patch.

nimrod

@johnpoz said in Strange issue with CPU usage.:

@nimrod this is a known issue with pfblocker, there is a simple fix

https://redmine.pfsense.org/issues/13154

see at the bottom of the redmine for simple patch.

Hi @johnpoz.

I applied the patch via system patches, rebooted appliance, but issue is still there. I also tried

killall /usr/local/bin/php_pfb

But all i get is this:

No matching processes were found

Should i try and update to 2.7.0 devel, or there is something else i can try.

Thanks for your help.

johnpoz

@nimrod I don't know if this has been fixed in 2.7 - since it seems to be a package issue.. Is there a newer package for 2.7 pfsense?

There have been multiple posts about this issue, and this clearly should clear it up.. For example as you can see in my graph of cpu usage.

nimrod

@johnpoz said in Strange issue with CPU usage.:

@nimrod I don't know if this has been fixed in 2.7 - since it seems to be a package issue.. Is there a newer package for 2.7 pfsense?

There is no new package. However, the process im having issues with is

/usr/local/sbin/unbound-control

unlike everyone else that had issues with:

/usr/local/bin/php_pfb

I think the patch that i applied fixes the issue with php_pfb.

There have been multiple posts about this issue, and this clearly should clear it up.. For example as you can see in my graph of cpu usage.

I see. But did you had issues with php_pfb or unbound-control process ?

johnpoz

@nimrod no have not seen any issues with pfblocker or unbound at all.. Only issue I was having was elevated cpu usage, and with that came a couple of degrees higher temps of pfsense itself..

I am not a typical pfblock user though - I only use it to create native alias via geoip, and other lists that I then use in my own rules.

And I use unbound in a pretty vanilla way, I just let it resolve, I don't do any tls forwarding, etc.

stephenw10

Are you running DNS-BL in Python mode?

Does it go away if you use Unbound mode instead?

Steve

nimrod

@stephenw10 said in Strange issue with CPU usage.:

Are you running DNS-BL in Python mode?

Does it go away if you use Unbound mode instead?

Steve

Hi Steve.

No. Its running in default Unbound mode. So far i tried this:

Disabling TLD and reloading all.
Upgrading to pfSense v2.7.x DEVEL.
Clean install of pfSense 2.6.0, and then full config restore.
Disabling DNS forwarder mode.
Applying patch that @johnpoz suggested.

Nothing helped. Only disabling pfBlocker completely eliminates the issue.

I found two posts on pfSense Reddit page here and here, and both of them are describing exactly the same issue as i am having. First one was resolved by upgrading to pfSense v2.5.1, and the other one was miss configured resolver.

johnpoz

@nimrod said in Strange issue with CPU usage.:

Applying patch that @johnpoz suggested.

That patch cleared it up for me - but when first did it I messed up and forgot the space between the 's and didn't work - so make sure if you hand edited that line that there is a space where the ( was, so it was like '(' and should be ' ' with space between.

I didn't catch my mistake for quite some time, week or so at least. I had done it and didn't really bother to watch cpu usage.. because mine was never spiking even that high, it was just higher than before. But after I put in the space it made a drastic difference as you can see from the graph I posted.

nimrod

@johnpoz said in Strange issue with CPU usage.:

@nimrod said in Strange issue with CPU usage.:

Applying patch that @johnpoz suggested.

That patch cleared it up for me - but when first did it I messed up and forgot the space between the 's and didn't work - so make sure if you hand edited that line that there is a space where the ( was, so it was like '(' and should be ' ' with space between.

I didn't catch my mistake for quite some time, week or so at least. I had done it and didn't really bother to watch cpu usage.. because mine was never spiking even that high, it was just higher than before. But after I put in the space it made a drastic difference as you can see from the graph I posted.

I used System Patches package to apply the patch. The code i pasted looks like this:

diff --git a/net/usr/local/pkg/pfblockerng/pfblockerng.inc b/net/usr/local/pkg/pfblockerng/pfblockerng.inc
index 7fa8c1d2f8bf..2abbef30578b 100644
--- a/net/usr/local/pkg/pfblockerng/pfblockerng.inc
+++ b/net/usr/local/pkg/pfblockerng/pfblockerng.inc
@@ -4136,7 +4136,7 @@ function pfb_filterrules() {
         foreach ($results as $result) {
             if (substr($result, 0, 1) == '@') {

-                $r = explode(')', $result, 2);
+                $r = explode(' ', $result, 2);

                 // pfSense > v2.6 uses an 'ridentifier' string
                 if (strpos($result, 'ridentifier') != FALSE) {

johnpoz

@nimrod I would directly look at the file and make sure it actually applied, if your not seeing any difference in your cpu usage after applying..

you can find the inc file in this dir

/usr/local/pkg/pfblockerng

nimrod

@johnpoz said in Strange issue with CPU usage.:

@nimrod I would directly look at the file and make sure it actually applied, if your not seeing any difference in your cpu usage after applying..

you can find the inc file in this dir

/usr/local/pkg/pfblockerng

It seems ok. I also see pfblockerng.inc.orig file, which i think is the backup created by system patches package.

This file is 300 megs, and it has 10000 lines of code in it. I guess, the file size depends on how many feeds/lists i have loaded.

johnpoz

@nimrod said in Strange issue with CPU usage.:

300 megs

no - 300k, not 300M hehehe

nimrod

@johnpoz said in Strange issue with CPU usage.:

@nimrod said in Strange issue with CPU usage.:

300 megs

no - 300k, not 300M hehehe

Yeah. 300k, sry. My mind is going out...

Anyway, @stephenw10 gave me idea with his question. So i actually switched pfBlocker from Unbound mode to Python mode. And guess what. Issue is gone.

Now im puzzled.

Is Python mode less effective, or inferior in any way to unbound mode ?

johnpoz

@nimrod I don't use DNSBL, so not sure - but if you click the little i next to it says it uses less memory. And can do more advanced DNSBL

But again I don't use that..

nimrod

Had to switch back to unbound mode, because python mode is just not working correctly when TLD is enabled.

Of course, CPU is spiking to 100% again, and i dont know what else to try and resolve this.

stephenw10

I'd probably open a thread in the pfBlocker forum section with all the details you have found.

nimrod

Update.

Setting pfBlocker into Unbound python mode.
Performing DNSBL reload.
Setting pfBlocker into Unbound mode.
Performing DNSBL reload.

These 4 steps are resolving the issue. However, when i reboot, issue comes back again. Then i repeat those 4 steps from above, and issue is gone again.

Is there any way to locate what changes are made when switching modes ? What logs should i analyze ?

Edit: @johnpoz, @stephenw10, or @jimp can you please move this thread to pfBlocker forum section ?

Thank you.

nimrod

Here is another update.

When i perform 4 steps that i described in previous post, pfBlocker is blocking hosts, but its not showing what feed was used, and there are no whitelisting buttons available.

It looks like this:

When i reboot, blocking is still working as it should, but feed is also displayed properly along with whitelisting buttons. It looks like this:

I hope ill narrow this down some more.

stephenw10

Moved. You may want to re-title it so others can find it more easily.