PfBlockerNG high CPU
-
I couldn't get the floating rule to work, but I was able to use the port forward rules to essentially change the sinkhole address just for the Shield like this:
This is just an improved bandaid, though. I just get DNSBL logging back for the rest of my network. I still think the disparty between blocked packets and total queries is part of the core issue.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
I've checked the state tables again to confirm the destination port and it's always 443.
This means the pfBlockerNG-devel's web server throws out the 'blocked page' on every request.
Because the Shield device is stupid enough to re try again, and again there is actually just one real solution : have it repaired/updated/moved.
A web browser wouldn't do that when it loads a page with publicity content.
Your idea to sink hole it seems good ! -
@Gertjan said in PfBlockerNG high CPU:
@lunaticfringe80 said in PfBlockerNG high CPU:
I've checked the state tables again to confirm the destination port and it's always 443.This means the pfBlockerNG-devel's web server throws out the 'blocked page' on every request.
Because the Shield device is stupid enough to re try again, and again there is actually just one real solution : have it repaired/updated/moved.
A web browser wouldn't do that when it loads a page with publicity content.
Your idea to sink hole it seems good !LOL! Have it repaired? Should I call up Nvidia and tell them I'm trying to block their device from phoning in metrics, but my chosen software to do that is getting overwhelmed, so I want them to fix it so it won't try so hard to phone in metrics?
Why the assumption it's the Shield that is broken and not pfblockerng? From where I'm sitting, it looks like pfblockerng is vulnerable to (unintentional?) DoS attacks from streaming set-top-boxes.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
@Gertjan said in PfBlockerNG high CPU:
@lunaticfringe80 said in PfBlockerNG high CPU:
I've checked the state tables again to confirm the destination port and it's always 443.This means the pfBlockerNG-devel's web server throws out the 'blocked page' on every request.
Because the Shield device is stupid enough to re try again, and again there is actually just one real solution : have it repaired/updated/moved.
A web browser wouldn't do that when it loads a page with publicity content.
Your idea to sink hole it seems good !LOL! Have it repaired? Should I call up Nvidia and tell them I'm trying to block their device from phoning in metrics, but my chosen software to do that is getting overwhelmed, so I want them to fix it so it won't try so hard to phone in metrics?
Why the assumption it's the Shield that is broken and not pfblockerng? From where I'm sitting, it looks like pfblockerng is vulnerable to (unintentional?) DoS attacks from streaming set-top-boxes.
If thats the case, then a fair sized network with IoT would bring the pfsense box to its knees fairly easy.
We wouldnt want that would we??
-
@Cool_Corona said in PfBlockerNG high CPU:
@lunaticfringe80 said in PfBlockerNG high CPU:
@Gertjan said in PfBlockerNG high CPU:
@lunaticfringe80 said in PfBlockerNG high CPU:
I've checked the state tables again to confirm the destination port and it's always 443.This means the pfBlockerNG-devel's web server throws out the 'blocked page' on every request.
Because the Shield device is stupid enough to re try again, and again there is actually just one real solution : have it repaired/updated/moved.
A web browser wouldn't do that when it loads a page with publicity content.
Your idea to sink hole it seems good !LOL! Have it repaired? Should I call up Nvidia and tell them I'm trying to block their device from phoning in metrics, but my chosen software to do that is getting overwhelmed, so I want them to fix it so it won't try so hard to phone in metrics?
Why the assumption it's the Shield that is broken and not pfblockerng? From where I'm sitting, it looks like pfblockerng is vulnerable to (unintentional?) DoS attacks from streaming set-top-boxes.
If thats the case, then a fair sized network with IoT would bring the pfsense box to its knees fairly easy.
We wouldnt want that would we??
What do you mean "if" that's the case? Have I not adequately demonstrated it to be? You don't need a fair sized IoT network, just a single STB will do. If denial is all you've got left to offer, I'll quit wasting my time providing all this testing data and go back to just using what works.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
LOL! Have it repaired? Should I call up Nvidia and tell them I'm trying to block their device from phoning in metrics, but my chosen software to do that is getting overwhelmed, so I want them to fix it so it won't try so hard to phone in metrics?
Euuh .... well, yes ! That's the origin of the problem.
It's like you trying to login to this forum, and not using the right password. You'll get an error page. A couple of milliseconds later, you try again, and again. How many states do you think you are going to use on the Netgate (pfSense) front end router / how many sessions files will get opened on their Web servers ?
The thing is, this is a script, no a human, so it chain-guns away ....
I've a fail2ban running on my mail/web/dns servers just for this, as it is a known ever going on problem. fal2ban was invented decades ago just for this. It boils down to use scripts on the server side to protect against scripts on the user's side.The thing is, the device (an app ?) uses a 443 port, which is normally a port used by a web browser to connect to a web server. The traffic will get redirected to the pfBlocker-NG's web server. And it keeps hammering, as it doesn't accept "no" for an answer.
For software that hammers, what about some a traffic shaping rules ? That's the way to protect a web server (mail server, whatever). Or, as you did, sink hole it.True, in this case, pfBlocker-NG isn't protected.
Yet ;)edit : btw : the fact pfBlcoker-NG using a web server to "serve" empty pub pages is a dead-end solution.
Nearly all traffic is https anyway, so the traffic send won't get accepted by our browsers anyway - because it's a pure MITM action.
Using a 0.0.0.0 address (Black holing) is the only solution. And simply ditching the "443" redirect. -
@Gertjan the DNSBL VIP is already supposed to be a sinkhole. I'm having to protect that sinkhole with another sinkhole. I shouldn't have to do that, especially since this sinkception breaks reporting. I didn't have to do that with pihole.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
the DNSBL VIP is already supposed to be a sinkhole
Im' not reading sink hole :
I'm reading "a web server".
Sending traffic to a web server is not sink holing it ....edit :
If you know which DNSBL feed is blocking your device, switch it here :
or better : make your own DNSBL feed file, with only the domain nam, the one the device is using, have it loaded at the top, and sink that one exclusivity..
-
@Gertjan A sinkhole is a destination address that unwanted traffic is redirected to using things like false DNS replies. Pfblockerng is designed to listen on its sinkhole address, which is why it is vulnerable where pihole does not. So I have to create a second sinkhole address to protect the first one.
-
@Gertjan said in PfBlockerNG high CPU:
The thing is, the device (an app ?) uses a 443 port, which is normally a port used by a web browser to connect to a web server. The traffic will get redirected to the pfBlocker-NG's web server. And it keeps hammering, as it doesn't accept "no" for an answer.
Hello!
Well, you are not really telling the device "no", you are returning a "blocked" page that isn't speaking its secret "language".
If you want to tell that particular device "NO" when it hits a blocked domain you could modify the pfsense file at /usr/local/www/pfblockerng/www/dnsbl_default.php and stick some variation of code like this at the top :
<?php
$deviant_ips = array ("192.168.0.6", "192.168.0.7", "192.168.0.8");
if (in_array ($ptype ['REMOTE_ADDR'], $deviant_ips))
{
http_response_code (404);
die();
}
?>Save it as a different name in the same folder and then load it on Firewall -> pfBlockerNG -> DNSBL -> DNSBL Configuration -> Blocked Webpage. You have to save it and reload it because it will copy it to the file dnsbl_active.php, which is the file actually used on the block page.
Of course, it still might not take "no" for an answer...
John
-
@serbus I'll try that next to see what impact it has. I don't want to be forced to sacrifice logging to solve this.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
I don't want to be forced to sacrifice logging to solve this
You don't need to : see what I proposed above.
@lunaticfringe80 said in PfBlockerNG high CPU:
A sinkhole is a destination address that unwanted traffic is redirected ...
And that already far to much. The word sink-hole is misleading.
When the DNS returns, 0.0.0.0 for a domain, the device will not initiate a connection. 0.0.0.0 isn't a valid routable address anyway. Nothing will move on the wire so pfSense doesn't get bothered - smacked in the face xx hundred times a second. The device will recognise this as a domain does not - "does not exist" and that will force to stop sending useless requests.
When the DNS returns a 10.10.10.1 (the pfBlockerNG-devel default setting) we all presume that a web browser will use port 80 - (or the useless port 443) so it can serve an "empty : non pub here page" - or whatever dummy page you uploaded.
The one @serbus proposes is very interesting. Put in the array the devices that behave badly and see what happens. -
Hello!
This might give more insight into how to deal with the shield/amplitude or other similar devices/services...
https://developers.amplitude.com/docs/http-api-v2
Pfb will probably block the link, but it is interesting...at least as long as you are the vampire and not the "donor".
John
-
@Gertjan said in PfBlockerNG high CPU:
@lunaticfringe80 said in PfBlockerNG high CPU:
I don't want to be forced to sacrifice logging to solve this
You don't need to : see what I proposed above.
Your suggestion above was quite literally to disable logging. See the word just to the left of what you suggested I switch? Also, I'm reading both sinkholed and web server.
@serbus Your suggestion to use a custom block page that sends a 404 appears to be working. So far so good.
-
@lunaticfringe80 said in PfBlockerNG high CPU:
Also, I'm reading both sinkholed and web server.
Important note : 'English' is not my native language. More somewhat between second and third.
I'm probably confounding "Black hole" with "Sink hole"'.And you're right : When you put the/a DNSBL in own it's DNDBL feed file - a manually created feed - prioritize it and then set Logging to "Disabled" for this DNSBL file then there are no more logs. Because the device will not access the network, because it can't use a obtained 0.0.0.0 address.
This is where I consider that 0.0.0.0 is a black hole (== nothing comes in (into pfSense) because nothing comes out (of the device).
Logging won't work at that moment for very obvious reasons.If you keep the VIP as the "Sink hole" then the device will "hit" the pfBlockerNG-devel web server - and leave a log trace at that moment.
With all the related consequences ..... as the device won't take no for an answer == what created the problem raised in this thread. -
I know I'm a bit late to the party, but I had a similar issue. I'm posting this to document the cause and the solution.
In my case, I correlated the high CPU load to an unusually high amount of LAN traffic. I isolated it to my Windows-10 desktop machine. Further isolation using the Windows 'Resource Monitor' app identified the activity on the LAN causing the high CPU load; it was Malwarebytes attempting to send its telemetry back to the mothership. It was doggedly persistent. Once I allowed the traffic, all was back to normal. Unfortunately, the clever 404 PHP solution in post 75 didn't work.
-
@Spacecase I can confirm the issue, I had one machine that was trying to reach out to MalwareBytes telemetry and killing the CPU on my pfSense box.
Once I disabled telemetry, issue resolved. I turned it on again and saw the issue, turned it off again and problem gone.
-
Problem:
High CPU attributed to pfBlockerNG.
Issue was traced to a single user with Malwarebytes by examining the pfBlockerNG DNSBL.LOGSolution:
Put ".malwarebytes.com" in the DNSBL Whitelist.Effect:
CPU returned to its usual value of about 5% (J1900)
from 33% (one core almost fully used) -
@RVAIBC
Worked for me, thank you! -
Thanks everyone for the helpful post.
I wanted to share my story here for the next person from the future that finds it.
This process was previously taking 80% of the CPU, without ever letting up:
78554 root 20 0 9016K 5576K kqread 0 0:02 0.78% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.confI was concerned because that kept the temps on my SG-3100 higher than I was comfortable with.
The key insight for me was that
lighttpd_pfbis probably pfBlocker's web server that sends out the "blocked" web page, so something on my network must be spamming it.Root Cause
A device on my local network occasionally tries to phone home tosome.service.com(that's the fake FQDN I'll use). It doesn't do this all the time, but when the attempt fails due to DNSBL, it re-tries repeatedly, causing the CPU spike.Solution
I whitelisted the domain in DNSBL, similar to what others have done. However, I'm not simply giving in to a app's persistent whining to reach a service that I'd prefer it didn't. So, I created a host alias forsome.service.comand blocked it in the firewall.Now the DNS queries go through, and then the request is rejected. pfBlocker's web server is not needed for this, so the CPU load is back to normal.
