pfBlockerNG-devel v3.0.0_15

DaddyGo

@bbcan177 said in pfBlockerNG-devel v3.0.0_15:

Is this a challenge :)

Like the life :)

pftdm007

Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

I see tons of these... Seems to happen with IPv4 lists....

See attached file for log output, the SPAM bot wont let me post the 20 odd lines snippet without flagging me as spam..... What a PITA.

log.txt

Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

@pftdm007 You need more upvotes. Let's help them out.

BBcan177

@pftdm007 said in pfBlockerNG-devel v3.0.0_15:

Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

BGPView is rate-limiting:
https://twitter.com/BBcan177/status/1357161876812087297

Reduce the frequency of ASN Updates.

Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

See:
https://redmine.pfsense.org/issues/11398

@bbcan177 Yeah, it is ;)

        try:
            manager = urllib3.PoolManager()
            # bgpview throttles queries
            time.sleep(0.5)     <----------------
            response = manager.request(
                "GET", "https://api.bgpview.io/asn/" + self.asn + "/prefixes")
        except (urllib3.exceptions.HTTPError, urllib3.exceptions.PoolError):
            return

Edited to add:
In case anyone cares, an alternative to BGPView is this for getting the routed prefixes for an ASN:

        elif data_provider == "radb":
            result = ""
            try:
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.connect(("whois.radb.net", 43))
                sock.send(bytes("-i origin " + self.asn + "\r\n", "utf-8"))
                while True:
                    data = sock.recv(1024)
                    if not data:
                        break
                    result += data.decode("utf-8")
                sock.close()
            except OSError:
                sock.close()
                return
            if result:
                output = result.splitlines()
                # The error message if the ASN returns no data is:
                # "%  No entries found for the selected source(s)."
                if output[0].startswith("% "):
                    return
                routes4 = [line.removeprefix("route:").strip()
                           for line in output if line.startswith("route:")]
                routes6 = [line.removeprefix("route6:").strip()
                           for line in output if line.startswith(("route6:"))]

BBcan177

@jwj said in pfBlockerNG-devel v3.0.0_15:

whois.radb.net

Radb.net isn't as accurate as BGPview

@bbcan177 Thanks. I didn't know that.

BBcan177

@jwj said in pfBlockerNG-devel v3.0.0_15:

bbcan177 Thanks. I didn't know that.

I am going to test adding a UA string with a random gnid number for each ASN download, and see if this makes it better.

If you can test this code block would be great.

ua="pfSense/pfBlockerNG cURL download agent-"
guid="$(/usr/sbin/gnid)"
ua_final="${ua}${guid}"

bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
"${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"

@bbcan177 ~~I put that into a shell script. I can bang away with that in a loop and it doesn't error. Nice!~~

Edited: OK, maybe I'm not getting something. gnid always returns the same string. How is that fixing the throttling/rate limiting? Maybe I'm just having a moment, but I'm confused.

Tested again and it does fail. Not sure what is going on. Worked and now fails.

Edited again: Must have been that the last run was successful and that's why I thought all was good. My bad... Today's not my day... I'm giving up on the day, cocktail time!

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.1</center>
</body>
</html>

~~I'm going to look at how I've been doing those calls from python with urllib3 and see if I can come up with a similar workaround.~~

Thanks! You're the best!

#!/bin/sh
for i in 1 2 3 4 5 6 7 8 9 10
do
        ua="pfSense/pfBlockerNG cURL download agent-"
        guid="$(/usr/sbin/gnid)"
        ua_final="${ua}${guid}"
        asntemp="tmpdata"
        pathcurl="/usr/local/bin/curl"
        asn="AS32934"
        bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
        "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
        echo $i
done

~~Well that was easy:~~

>>> import urllib3
>>> user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 6.3; rv:36.0) ..'}
>>> http = urllib3.PoolManager(10, headers=user_agent)

I generate a user agent string like this:

{'user-agent': 'Blocklist/HTTP download agent-659027266803267.1'}
503
{'user-agent': 'Blocklist/HTTP download agent-6668964514238788.0'}
200
{'user-agent': 'Blocklist/HTTP download agent-5613465811201082.0'}
503

the last part is random and I still get rate limiting...

BBcan177

@jwj
They are behind CloudFlare, so I can't say exactly what tools/mechanisms they use to rate-limit. But my guess, is that that you are hitting them too hard, and potentially rate-limiting the IP.

@bbcan177 Yup, for sure. Having a short wait is not an issue for my application. This all started for me when I was asked to come up with a quick way to block some stuff (Twitter/Facebook) during school hours only. I wrote a little python script to grab the routed prefixes and do the block on a schedule (ubiquity edgerouter, not pfsense). Easy enough.

Then it kind of grew out of control. It's been a few years since I had done anything with python, I do real work in Rust and Julia, so I just kept adding stuff. Kinda fun. I'm more or less over it now, but python is fun.

pftdm007

@bbcan177 The ASN update frequency was already at 24hrs, I changed it to 1 week. Lets see if this will help!

For the other issue (unbound not restarting automatically), issue 11398 seems to indicate the install of pfblocker halts and never finishes (or I misread the ticket?) which is not my case. For me it seems that pfblockerNG is successfully installed, IP blocklists work but unbound doesnt restart automatically and need to be "jump started" manually... Not a big issue IMO since I do package updates "manually" and I always do a walk around to see if all the services are up & running.

10101000

Basic ASN update (using bgpview.io) is broken for me without a sleep (especially with more than a few). The frequency doesn't matter in this case, nor does changing my public IP. Failures result in the ip_placeholder addresses. Can I request that this be configurable in the GUI? It would also be nice if there was better alerting to failures. Thanks for your hard work @BBcan177

--- /usr/local/pkg/pfblockerng/pfblockerng.sh.2021-01-17        2021-01-07 15:09:37.000000000 -0700
+++ /usr/local/pkg/pfblockerng/pfblockerng.sh   2021-01-17 12:20:47.367047000 -0700
@@ -755,6 +755,7 @@
                        bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
                        "${pathcurl}" -s1 "${bgp_url}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig"
                fi
+                sleep 1
        done
 }

BBcan177

@10101000

Have been working on this, download the patched file:

curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/3aabea5edf7b40554d93085bff380b6f/raw"

As best I can tell the rate-limiting is just how it is:

        try:
            # BGPView rate-limits, try 5 times then give up
            re_try = 0
            while re_try < 5:
                response = ASN.manager.request(
                    "GET", "https://api.bgpview.io/asn/" + self.asn + "/prefixes")
                if response.status == 503:
                    re_try += 1
                    sleep(re_try / 2)
                else:
                    break
        except (HTTPError, PoolError):
            ASN.manager.clear()
            return

        if response.status != 200:
            ASN.manager.clear()
            return

Sleeps for 0.5 seconds on the first re-try, that appears to be sufficient.

10101000

@bbcan177 said in pfBlockerNG-devel v3.0.0_15:

@10101000

Have been working on this, download the patched file:
curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/3aabea5edf7b40554d93085bff380b6f/raw"

It works perfectly, thanks!

jvamos

I updated and rebooted and seem to get some packetloss. 2-5%
I removed the package and the problem was resolved.
Pinging the router was fine but sites visited at the WAN really have a degradation in performance.

Gertjan

@jvamos pfBlokcerNG uses very few CPU resources.

But, if loaded up with many feeds (thousands of IP's, thousands of hostnames) unbound can start stressing.
Which could impact overall system performances.
Equally distributes over all NICs, not only "WAN".

Packet loss normally means : invest in the WAN part of your network wiring => must often this means : call your ISP.

Xentrk

@bbcan177

Unable to filter Alerts by the Source IP field. After entering the Source IP, I select the "Apply Filter" button. Search results are not filtered and web page "spins". Have to press the "X" in the browser to make the web page usable.

Gertjan

@xentrk

Like this :

.....

and way below, a couple of thousand lines lower :

It took far less then a second to generate the 'page'.

Btw : My Samsung TV insist on calling 8.8.8.8, even when it's off (sleep mode). Still wonder what it has to tell to Google .....