pfBlockerNG-devel v3.0.0_15

johnpoz

Hmmm - I am only showing
pfBlockerNG-devel net 3.0.0_10

bmeeks

There are two different and distinct respositories for packages. One repo is for CE and the other is for pfSense+. Been that way since "Factory Edition" appeared a while back.

The Netgate developer team has to manually merge Pull Requests into the pfSense+ repo and then kick off a package build there. Until that is done, new packages that appear for CE users don't show up for pfSense+ users.

RIght now, looking in the two repos, I see pfSense-2.5.0 CE has version .14 of pfBlockerNG-devel while pfSense+ is still showing only version .10.

MORGiON

@bmeeks Thankyou for the clarification, I was beginning to think something was wrong with my install

Gertjan

@bmeeks thank for the heads up.

I better get used to mention that I'm using the CE.
pfBlockerNG-devel is mostly if not all PHP scripts files, some python script files . No binaries, so basiclly, it's just a pure copy from CE (bbcan is using CE I guess) to +.

bmeeks

@gertjan said in pfBlockerNG-devel v3.0.0_15:

@bmeeks thank for the heads up.

I better get used to mention that I'm using the CE.
pfBlockerNG-devel is mostly if not all PHP scripts files, some python script files . No binaries, so basiclly, it's just a pure copy from CE (bbcan is using CE I guess) to +.

Yeah, discovered the existence of the two distinct repos shortly after "Factory Edition" emerged when trying to figure out why my "latest" Snort package update was not showing up for Netgate appliances but showed up for me on CE in my test virtual machines at the time.

johnpoz

_15 is available for 21.02p1 now - just updated to it.

Gertjan

Also for the CE.

Gertjan

What ?
No dot 16 this morning ??

fireodo

@gertjan said in pfBlockerNG-devel v3.0.0_15:

What ?
No dot 16 this morning ??

No

BBcan177

@gertjan said in pfBlockerNG-devel v3.0.0_15:

What ?
No dot 16 this morning ??

Is this a challenge :)

DaddyGo

@bbcan177 said in pfBlockerNG-devel v3.0.0_15:

Is this a challenge :)

Like the life :)

pftdm007

Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

I see tons of these... Seems to happen with IPv4 lists....

See attached file for log output, the SPAM bot wont let me post the 20 odd lines snippet without flagging me as spam..... What a PITA.

log.txt

Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

A Former User

@pftdm007 You need more upvotes. Let's help them out.

BBcan177

@pftdm007 said in pfBlockerNG-devel v3.0.0_15:

Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

BGPView is rate-limiting:
https://twitter.com/BBcan177/status/1357161876812087297

Reduce the frequency of ASN Updates.

Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

See:
https://redmine.pfsense.org/issues/11398

A Former User

@bbcan177 Yeah, it is ;)

        try:
            manager = urllib3.PoolManager()
            # bgpview throttles queries
            time.sleep(0.5)     <----------------
            response = manager.request(
                "GET", "https://api.bgpview.io/asn/" + self.asn + "/prefixes")
        except (urllib3.exceptions.HTTPError, urllib3.exceptions.PoolError):
            return

Edited to add:
In case anyone cares, an alternative to BGPView is this for getting the routed prefixes for an ASN:

        elif data_provider == "radb":
            result = ""
            try:
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.connect(("whois.radb.net", 43))
                sock.send(bytes("-i origin " + self.asn + "\r\n", "utf-8"))
                while True:
                    data = sock.recv(1024)
                    if not data:
                        break
                    result += data.decode("utf-8")
                sock.close()
            except OSError:
                sock.close()
                return
            if result:
                output = result.splitlines()
                # The error message if the ASN returns no data is:
                # "%  No entries found for the selected source(s)."
                if output[0].startswith("% "):
                    return
                routes4 = [line.removeprefix("route:").strip()
                           for line in output if line.startswith("route:")]
                routes6 = [line.removeprefix("route6:").strip()
                           for line in output if line.startswith(("route6:"))]

BBcan177

@jwj said in pfBlockerNG-devel v3.0.0_15:

whois.radb.net

Radb.net isn't as accurate as BGPview

A Former User

@bbcan177 Thanks. I didn't know that.

BBcan177

@jwj said in pfBlockerNG-devel v3.0.0_15:

bbcan177 Thanks. I didn't know that.

I am going to test adding a UA string with a random gnid number for each ASN download, and see if this makes it better.

If you can test this code block would be great.

ua="pfSense/pfBlockerNG cURL download agent-"
guid="$(/usr/sbin/gnid)"
ua_final="${ua}${guid}"

bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
"${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"

A Former User

@bbcan177 ~~I put that into a shell script. I can bang away with that in a loop and it doesn't error. Nice!~~

Edited: OK, maybe I'm not getting something. gnid always returns the same string. How is that fixing the throttling/rate limiting? Maybe I'm just having a moment, but I'm confused.

Tested again and it does fail. Not sure what is going on. Worked and now fails.

Edited again: Must have been that the last run was successful and that's why I thought all was good. My bad... Today's not my day... I'm giving up on the day, cocktail time!

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.1</center>
</body>
</html>

~~I'm going to look at how I've been doing those calls from python with urllib3 and see if I can come up with a similar workaround.~~

Thanks! You're the best!

#!/bin/sh
for i in 1 2 3 4 5 6 7 8 9 10
do
        ua="pfSense/pfBlockerNG cURL download agent-"
        guid="$(/usr/sbin/gnid)"
        ua_final="${ua}${guid}"
        asntemp="tmpdata"
        pathcurl="/usr/local/bin/curl"
        asn="AS32934"
        bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
        "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
        echo $i
done

~~Well that was easy:~~

>>> import urllib3
>>> user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 6.3; rv:36.0) ..'}
>>> http = urllib3.PoolManager(10, headers=user_agent)

I generate a user agent string like this:

{'user-agent': 'Blocklist/HTTP download agent-659027266803267.1'}
503
{'user-agent': 'Blocklist/HTTP download agent-6668964514238788.0'}
200
{'user-agent': 'Blocklist/HTTP download agent-5613465811201082.0'}
503

the last part is random and I still get rate limiting...

BBcan177

@jwj
They are behind CloudFlare, so I can't say exactly what tools/mechanisms they use to rate-limit. But my guess, is that that you are hitting them too hard, and potentially rate-limiting the IP.