Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfBlockerNG-devel v3.0.0_15

    Scheduled Pinned Locked Moved pfBlockerNG
    39 Posts 15 Posters 4.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @johnpoz
      last edited by Gertjan

      Also for the CE.

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      GertjanG 1 Reply Last reply Reply Quote 0
      • GertjanG
        Gertjan @Gertjan
        last edited by

        What ?
        No dot 16 this morning ??

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        fireodoF BBcan177B 2 Replies Last reply Reply Quote 0
        • fireodoF
          fireodo @Gertjan
          last edited by

          @gertjan said in pfBlockerNG-devel v3.0.0_15:

          What ?
          No dot 16 this morning ??

          No 😁 😂

          Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
          SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
          pfsense 2.8.0 CE
          Packages: Apcupsd, Cron, Iftop, Iperf, LCDproc, Nmap, pfBlockerNG, RRD_Summary, Shellcmd, Snort, Speedtest, System_Patches.

          1 Reply Last reply Reply Quote 0
          • BBcan177B
            BBcan177 Moderator @Gertjan
            last edited by

            @gertjan said in pfBlockerNG-devel v3.0.0_15:

            What ?
            No dot 16 this morning ??

            Is this a challenge :)

            "Experience is something you don't get until just after you need it."

            Website: http://pfBlockerNG.com
            Twitter: @BBcan177  #pfBlockerNG
            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

            DaddyGoD 1 Reply Last reply Reply Quote 2
            • DaddyGoD
              DaddyGo @BBcan177
              last edited by

              @bbcan177 said in pfBlockerNG-devel v3.0.0_15:

              Is this a challenge :)

              Like the life :)

              Cats bury it so they can't see it!
              (You know what I mean if you have a cat)

              1 Reply Last reply Reply Quote 0
              • P
                pftdm007
                last edited by pftdm007

                Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

                I see tons of these... Seems to happen with IPv4 lists....

                See attached file for log output, the SPAM bot wont let me post the 20 odd lines snippet without flagging me as spam..... What a PITA.

                log.txt

                Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

                ? BBcan177B 2 Replies Last reply Reply Quote 2
                • ?
                  A Former User @pftdm007
                  last edited by

                  @pftdm007 You need more upvotes. Let's help them out.

                  1 Reply Last reply Reply Quote 1
                  • BBcan177B
                    BBcan177 Moderator @pftdm007
                    last edited by

                    @pftdm007 said in pfBlockerNG-devel v3.0.0_15:

                    Anybody gets "parse error: Invalid numeric literal" while manually doing an update (probably also during automatic updates) with 3.0.0_15 ?

                    BGPView is rate-limiting:
                    https://twitter.com/BBcan177/status/1357161876812087297

                    Reduce the frequency of ASN Updates.

                    Also any of you guys noticed Unbound does NOT restart automatically after pfBlockerNG package update?

                    See:
                    https://redmine.pfsense.org/issues/11398

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    ? P 2 Replies Last reply Reply Quote 0
                    • ?
                      A Former User @BBcan177
                      last edited by A Former User

                      @bbcan177 Yeah, it is ;)

                              try:
                                  manager = urllib3.PoolManager()
                                  # bgpview throttles queries
                                  time.sleep(0.5)     <----------------
                                  response = manager.request(
                                      "GET", "https://api.bgpview.io/asn/" + self.asn + "/prefixes")
                              except (urllib3.exceptions.HTTPError, urllib3.exceptions.PoolError):
                                  return
                      

                      Edited to add:
                      In case anyone cares, an alternative to BGPView is this for getting the routed prefixes for an ASN:

                              elif data_provider == "radb":
                                  result = ""
                                  try:
                                      sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                                      sock.connect(("whois.radb.net", 43))
                                      sock.send(bytes("-i origin " + self.asn + "\r\n", "utf-8"))
                                      while True:
                                          data = sock.recv(1024)
                                          if not data:
                                              break
                                          result += data.decode("utf-8")
                                      sock.close()
                                  except OSError:
                                      sock.close()
                                      return
                                  if result:
                                      output = result.splitlines()
                                      # The error message if the ASN returns no data is:
                                      # "%  No entries found for the selected source(s)."
                                      if output[0].startswith("% "):
                                          return
                                      routes4 = [line.removeprefix("route:").strip()
                                                 for line in output if line.startswith("route:")]
                                      routes6 = [line.removeprefix("route6:").strip()
                                                 for line in output if line.startswith(("route6:"))]
                      
                      BBcan177B 1 Reply Last reply Reply Quote 0
                      • BBcan177B
                        BBcan177 Moderator @A Former User
                        last edited by

                        @jwj said in pfBlockerNG-devel v3.0.0_15:

                        whois.radb.net

                        Radb.net isn't as accurate as BGPview

                        "Experience is something you don't get until just after you need it."

                        Website: http://pfBlockerNG.com
                        Twitter: @BBcan177  #pfBlockerNG
                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                        ? 1 Reply Last reply Reply Quote 0
                        • ?
                          A Former User @BBcan177
                          last edited by

                          @bbcan177 Thanks. I didn't know that.

                          BBcan177B 1 Reply Last reply Reply Quote 0
                          • BBcan177B
                            BBcan177 Moderator @A Former User
                            last edited by

                            @jwj said in pfBlockerNG-devel v3.0.0_15:

                            bbcan177 Thanks. I didn't know that.

                            I am going to test adding a UA string with a random gnid number for each ASN download, and see if this makes it better.

                            If you can test this code block would be great.

                            ua="pfSense/pfBlockerNG cURL download agent-"
                            guid="$(/usr/sbin/gnid)"
                            ua_final="${ua}${guid}"
                            
                            bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
                            "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
                            

                            "Experience is something you don't get until just after you need it."

                            Website: http://pfBlockerNG.com
                            Twitter: @BBcan177  #pfBlockerNG
                            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                            ? 1 Reply Last reply Reply Quote 0
                            • ?
                              A Former User @BBcan177
                              last edited by A Former User

                              @bbcan177 I put that into a shell script. I can bang away with that in a loop and it doesn't error. Nice!

                              Edited: OK, maybe I'm not getting something. gnid always returns the same string. How is that fixing the throttling/rate limiting? Maybe I'm just having a moment, but I'm confused.

                              Tested again and it does fail. Not sure what is going on. Worked and now fails.

                              Edited again: Must have been that the last run was successful and that's why I thought all was good. My bad... Today's not my day... I'm giving up on the day, cocktail time!

                              <html>
                              <head><title>503 Service Temporarily Unavailable</title></head>
                              <body bgcolor="white">
                              <center><h1>503 Service Temporarily Unavailable</h1></center>
                              <hr><center>nginx/1.14.1</center>
                              </body>
                              </html>
                              
                              

                              I'm going to look at how I've been doing those calls from python with urllib3 and see if I can come up with a similar workaround.

                              Thanks! You're the best!

                              #!/bin/sh
                              for i in 1 2 3 4 5 6 7 8 9 10
                              do
                                      ua="pfSense/pfBlockerNG cURL download agent-"
                                      guid="$(/usr/sbin/gnid)"
                                      ua_final="${ua}${guid}"
                                      asntemp="tmpdata"
                                      pathcurl="/usr/local/bin/curl"
                                      asn="AS32934"
                                      bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
                                      "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
                                      echo $i
                              done
                              
                              

                              Well that was easy:

                              >>> import urllib3
                              >>> user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 6.3; rv:36.0) ..'}
                              >>> http = urllib3.PoolManager(10, headers=user_agent)
                              

                              I generate a user agent string like this:

                              {'user-agent': 'Blocklist/HTTP download agent-659027266803267.1'}
                              503
                              {'user-agent': 'Blocklist/HTTP download agent-6668964514238788.0'}
                              200
                              {'user-agent': 'Blocklist/HTTP download agent-5613465811201082.0'}
                              503
                              

                              the last part is random and I still get rate limiting...

                              BBcan177B 1 Reply Last reply Reply Quote 0
                              • BBcan177B
                                BBcan177 Moderator @A Former User
                                last edited by

                                @jwj
                                They are behind CloudFlare, so I can't say exactly what tools/mechanisms they use to rate-limit. But my guess, is that that you are hitting them too hard, and potentially rate-limiting the IP.

                                "Experience is something you don't get until just after you need it."

                                Website: http://pfBlockerNG.com
                                Twitter: @BBcan177  #pfBlockerNG
                                Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                ? 1 Reply Last reply Reply Quote 0
                                • ?
                                  A Former User @BBcan177
                                  last edited by

                                  @bbcan177 Yup, for sure. Having a short wait is not an issue for my application. This all started for me when I was asked to come up with a quick way to block some stuff (Twitter/Facebook) during school hours only. I wrote a little python script to grab the routed prefixes and do the block on a schedule (ubiquity edgerouter, not pfsense). Easy enough.

                                  Then it kind of grew out of control. It's been a few years since I had done anything with python, I do real work in Rust and Julia, so I just kept adding stuff. Kinda fun. I'm more or less over it now, but python is fun.

                                  1 Reply Last reply Reply Quote 0
                                  • P
                                    pftdm007 @BBcan177
                                    last edited by

                                    @bbcan177 The ASN update frequency was already at 24hrs, I changed it to 1 week. Lets see if this will help!

                                    For the other issue (unbound not restarting automatically), issue 11398 seems to indicate the install of pfblocker halts and never finishes (or I misread the ticket?) which is not my case. For me it seems that pfblockerNG is successfully installed, IP blocklists work but unbound doesnt restart automatically and need to be "jump started" manually... Not a big issue IMO since I do package updates "manually" and I always do a walk around to see if all the services are up & running.

                                    1 Reply Last reply Reply Quote 0
                                    • 1
                                      10101000
                                      last edited by

                                      Basic ASN update (using bgpview.io) is broken for me without a sleep (especially with more than a few). The frequency doesn't matter in this case, nor does changing my public IP. Failures result in the ip_placeholder addresses. Can I request that this be configurable in the GUI? It would also be nice if there was better alerting to failures. Thanks for your hard work @BBcan177

                                      --- /usr/local/pkg/pfblockerng/pfblockerng.sh.2021-01-17        2021-01-07 15:09:37.000000000 -0700
                                      +++ /usr/local/pkg/pfblockerng/pfblockerng.sh   2021-01-17 12:20:47.367047000 -0700
                                      @@ -755,6 +755,7 @@
                                                              bgp_url="https://api.bgpview.io/asn/${asn}/prefixes"
                                                              "${pathcurl}" -s1 "${bgp_url}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig"
                                                      fi
                                      +                sleep 1
                                              done
                                       }
                                      
                                      BBcan177B 1 Reply Last reply Reply Quote 0
                                      • BBcan177B
                                        BBcan177 Moderator @10101000
                                        last edited by BBcan177

                                        @10101000

                                        Have been working on this, download the patched file:

                                        curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/3aabea5edf7b40554d93085bff380b6f/raw"
                                        

                                        "Experience is something you don't get until just after you need it."

                                        Website: http://pfBlockerNG.com
                                        Twitter: @BBcan177  #pfBlockerNG
                                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                        1 1 Reply Last reply Reply Quote 1
                                        • ?
                                          A Former User
                                          last edited by A Former User

                                          As best I can tell the rate-limiting is just how it is:

                                                  try:
                                                      # BGPView rate-limits, try 5 times then give up
                                                      re_try = 0
                                                      while re_try < 5:
                                                          response = ASN.manager.request(
                                                              "GET", "https://api.bgpview.io/asn/" + self.asn + "/prefixes")
                                                          if response.status == 503:
                                                              re_try += 1
                                                              sleep(re_try / 2)
                                                          else:
                                                              break
                                                  except (HTTPError, PoolError):
                                                      ASN.manager.clear()
                                                      return
                                          
                                                  if response.status != 200:
                                                      ASN.manager.clear()
                                                      return
                                          

                                          Sleeps for 0.5 seconds on the first re-try, that appears to be sufficient.

                                          1 Reply Last reply Reply Quote 1
                                          • 1
                                            10101000 @BBcan177
                                            last edited by

                                            @bbcan177 said in pfBlockerNG-devel v3.0.0_15:

                                            @10101000

                                            Have been working on this, download the patched file:

                                            curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/3aabea5edf7b40554d93085bff380b6f/raw"
                                            

                                            It works perfectly, thanks!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.