Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfBlockerNG ASN downloads only contain a header

    Scheduled Pinned Locked Moved pfBlockerNG
    70 Posts 10 Posters 11.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • fireodoF
      fireodo @jrey
      last edited by

      @jrey said in pfBlockerNG ASN downloads only contain a header:

      so now it appears for you the patch is not working although the parse error is gone?

      Thats correct!

      You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

      I'll do that.

      also try the above commands from a shell

      That too ...

      Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
      SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
      pfsense 2.8.0 CE
      Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

      1 Reply Last reply Reply Quote 0
      • fireodoF
        fireodo @jrey
        last edited by fireodo

        @jrey said in pfBlockerNG ASN downloads only contain a header:

        You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

        THAT was the solution! With @BBcan177 s Patch and deleting the old files now the ASN Lists are populated as they should - that should be noticed somewhere when pfblockerNG 3.2.0_6 is available!

        Below the line I can say the culprit was the user agent and the curl parameter - maybe the user agent logic should be reconsidered ... if necessary.

        Thanks @jrey @BBcan177
        fireodo

        Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
        SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
        pfsense 2.8.0 CE
        Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

        1 Reply Last reply Reply Quote 0
        • M
          mbentley @jrey
          last edited by

          @jrey - Looks like your initial curl issue you mention with the spaces in the user agent is a lack of quotes in the script around the user agent string. I see quotes in the patch that @BBcan177 provided so I'd have to guess there is something funky going on with any modifications that were made which is why a user-agent with no spaces works.

          The bgpview.io API is definitely blocking based on user agent string which is why the default curl and pfSense strings are getting those 403s. If they're not intentionally blocking those user-agent strings, maybe it's cloudflare blocking heavily utilized user-agent strings - pure speculation though.

          J 1 Reply Last reply Reply Quote 1
          • J
            jrey @Bob.Dig
            last edited by

            @Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

            in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here?

            I don't think so, because the code in pfblockerng itself doesn't have any logic around this.
            the "agent-" is alway part of the string being built.

            ua="pfSense/pfBNG cURL download agent-"
            guid="$(/usr/sbin/gnid)" <-- the device ID is pulled here and immediately used in the next line to build the ua_final parameter used in the call
            ua_final="${ua}${guid}"

            so unless the setting you mention changes the setting for /usr/sbin/gnid so it returns nothing I would think the setting has no effect.
            even if it did and the guid is empty (it would build the agent string as "pfSense/pfBNG cURL download agent-"

            easy to confirm
            I changed the setting and tried it both ways. the logged command (because logging what is running is important !!!) still contains the device ID

            the setting makes no difference here, on or off, you still get the device id in the generated curl agent string.

            JR

            1 Reply Last reply Reply Quote 1
            • fireodoF
              fireodo @Bob.Dig
              last edited by

              @Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

              Netgate Device ID - Do NOT send Netgate Device ID with user agent
              in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here? Is this the real usecase for that Netgate Device ID? Or is it better to not have it? Will the other side still be able to see "pfSense" if i disable the option?

              I dont think so - this concerns (as far as I know) only the communication with Netgate Servers ...

              Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
              SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
              pfsense 2.8.0 CE
              Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

              J 1 Reply Last reply Reply Quote 0
              • J
                jrey @fireodo
                last edited by

                @fireodo said in pfBlockerNG ASN downloads only contain a header:

                this concerns (as far as I know) only the communication with Netgate Servers ...

                correct, verified with logging ;) - and the setting on the page referenced for this setting is in the section labelled "Installation Feedback" implies the same.. to me anyway.

                1 Reply Last reply Reply Quote 1
                • fireodoF
                  fireodo @jrey
                  last edited by

                  @jrey said in pfBlockerNG ASN downloads only contain a header:

                  I took a look at the dashboard part as well, and it seem the notice to the dashboard only looks at the errlog for the word "FAIL"
                  point here is that the download with or without the subsequent "parse error" isn't a download failure as such so it doesn't write to the error log.

                  Maybe that would also be nice if such failures as empty ASN-Files would be reported on the Dashboard Widget too - without reading in the Forum I didn't even noticed that something was going wrong ...

                  Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                  SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                  pfsense 2.8.0 CE
                  Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                  1 Reply Last reply Reply Quote 0
                  • J
                    jrey @mbentley
                    last edited by

                    @mbentley said in pfBlockerNG ASN downloads only contain a header:

                    with any modifications that were made which is why a user-agent with no spaces works

                    that's the thing no modifications where made, and it works sometime

                    @mbentley said in pfBlockerNG ASN downloads only contain a header:

                    The bgpview.io API is definitely blocking based on user agent string which is why the default curl and pfSense strings are getting those 403s.

                    I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
                    Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.

                    @mbentley said in pfBlockerNG ASN downloads only contain a header:

                    If they're not intentionally blocking those user-agent strings, maybe it's cloudflare blocking heavily utilized user-agent strings

                    thing is I don't believe the user-agent string is heavily used. at least in my case 10 or so AS queries 6 times a day. not to mention every device using the ASN feeds would have a different agent string (based on the device id) therefore cloudflare blocking base on this would mean they are "parsing" the agent string for everything that passes through and ignoring only the part of the string that contains agent-(device id) I think that seems highly unlikely.

                    So what if cloudflare has a lame cloud server somewhere and depending on the route to the destination the results are different. (again as long as we are speculating) after installing the patch late yesterday, it ran a couple of times with the schedule then started to fail. I re-added the debug line only this morning,

                    I agree, I think the patch is generally good (needs more logging, that won't hurt anything) and a notification on the dashboard when things do go south. Still the randomness of the failure (with the patch) does imply something else is still in play.

                    M 1 Reply Last reply Reply Quote 1
                    • M
                      mbentley
                      last edited by

                      One thing that I have noticed just working with the bgpview.io API is that it could be helpful in validating the response by parsing the returned .status from the json before processing it with jq and possibly catching the error in some way to bring the attention to the user:

                      $ curl -sSL -A "dont-block-me" "https://api.bgpview.io/asn/396017/prefixes" | jq .
                      {
                        "status": "ok",
                        "status_message": "Query was successful",
                        "data": {
                          "ipv4_prefixes": [
                            {
                              "prefix": "50.169.100.0/24",
                              "ip": "50.169.100.0",
                              "cidr": 24,
                              "roa_status": "Valid",
                              "name": "BAWA-CCS-1",
                              "description": "Comcast Cable Communications, LLC",
                              "country_code": "US",
                              "parent": {
                                "prefix": "50.128.0.0/9",
                                "ip": "50.128.0.0",
                                "cidr": 9,
                                "rir_name": "ARIN",
                                "allocation_status": "unknown"
                              }
                            },
                            {
                              "prefix": "50.225.242.0/24",
                              "ip": "50.225.242.0",
                              "cidr": 24,
                              "roa_status": "Valid",
                              "name": "CCCH3-4",
                              "description": "Comcast Cable Communications, LLC",
                              "country_code": "US",
                              "parent": {
                                "prefix": "50.128.0.0/9",
                                "ip": "50.128.0.0",
                                "cidr": 9,
                                "rir_name": "ARIN",
                                "allocation_status": "unknown"
                              }
                            }
                          ],
                          "ipv6_prefixes": []
                        },
                        "@meta": {
                          "time_zone": "UTC",
                          "api_version": 1,
                          "execution_time": "14.23 ms"
                        }
                      }
                      
                      1 Reply Last reply Reply Quote 0
                      • M
                        mbentley @jrey
                        last edited by

                        @jrey said in pfBlockerNG ASN downloads only contain a header:

                        I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
                        Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.

                        Here's a pretty good example of how you can see they're checking the user agent. I can reliably get the same responses.

                        # no user agent
                        $ curl -sSL "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 403
                        
                        # previous user agent the script used
                        $ curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 403
                        
                        # just with pfBlockerNG
                        $ curl -sSL -A "pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 418
                        
                        # both pfSense & pfBlockerNG
                        $ curl -sSL -A "pfSense/pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 418
                        
                        # misspelling of pfBlockerNG
                        $ curl -sSL -A "pfBlockerrNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 200
                        
                        # random string
                        $ curl -sSL -A "this-is-random" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 200
                        
                        # random with spaces
                        $ curl -sSL -A "this is random with spaces" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
                        HTTP/2 200
                        

                        I do appreciate the 418, I'm a teapot, http code usage.

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          jrey @mbentley
                          last edited by

                          @mbentley said in pfBlockerNG ASN downloads only contain a header:

                          curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'

                          Interesting - now try this (just let it dump to the screen) - no grep on just the headers so just this:

                          curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"

                          notice the title of the return
                          <title>Just a moment...</title>
                          and a little further in
                          <span id="challenge-error-text">Enable JavaScript and cookies to continue</span>

                          whereas (I dropped the NG from pfBlocker) and ran

                          curl -sSL -A "pfSense/pfBlocker cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"
                          returns, the good stuff

                          {"status":"ok","status_message":"Query was successful","data"

                          they likely think that pfSense/pfBlockerNG is a browser and want to detect javascript and cookies
                          whereas pfSense/pfBlocker or the new format in the patch being used pfSense/pfBNG or any other string is not a browser and the results work

                          go figure

                          M I 2 Replies Last reply Reply Quote 1
                          • M
                            mbentley @jrey
                            last edited by

                            @jrey

                            Yeah, so when I set my browser's user-agent to pfSense/pfBlockerNG cURL download agent-testvaluehere and accessed the site, it returned the typical "Cloudflare checking if this connection is secure" sort of page when I access "https://api.bgpview.io/asn/396017/prefixes" and then it returned a 418 again once it got past that. That's something I wouldn't expect from an API endpoint since an API client shouldn't be expected to need cookies and javascript.

                            Hopefully they respond to @BBcan177 so they can acknowledge the problem as it seems like issues with Cloudflare is one thing, the other being that they are trying to block certain clients via user agent. I'm sure it would be nice to understand what rules they are expecting us to play by with their API as they haven't published much in their FAQ besides that it's free to use.

                            J 1 Reply Last reply Reply Quote 1
                            • J
                              jrey @mbentley
                              last edited by

                              @mbentley said in pfBlockerNG ASN downloads only contain a header:

                              so when I set my browser's user-agent to pfSense/pfBlockerNG cURL download agent-testvaluehere and accessed the site, it returned the typical "Cloudflare checking if this connection is secure" sort of page when I access "https://api.bgpview.io/asn/396017/prefixes" and then it returned a 418 again once it got past that.

                              I did exactly the same thing and yes (it cycled past (briefly displayed) the cloudflare page didn't ask me to do anything, but then returned the right stuff -- no error, proper values displayed for the uri provided in your post ---

                              Browser test.png

                              Next thing was to delete the existing bgpview.io cookie, then set the browser to disable all cookies and try it again

                              the response in the browser this time as expected "Enable JavaScript and cookies to continue"

                              bgpcookie.png

                              seems they really just want a cookie -- at least with that agent string.

                              1 Reply Last reply Reply Quote 0
                              • I
                                iain @jrey
                                last edited by

                                @jrey This should check if the status is anything other than "ok" and if not, sets unavailable which is used as a flag later on as part of the error recovery. Also adds "challenge-error-text" to the strings caught to again follow the error-recovery path.

                                --- /usr/local/pkg/pfblockerng/pfblockerng.sh.orig  2023-08-13 22:08:41.956403000 -0400
                                +++ /usr/local/pkg/pfblockerng/pfblockerng.sh   2023-08-13 22:10:03.107682000 -0400
                                @@ -766,9 +766,10 @@
                                    unavailable=''
                                    for i in 1 2 3 4 5; do
                                        printf "."
                                -       "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
                                +       "${pathcurl}" -A "secret-agent-v1" -sS1 "${bgp_url}" > "${asntemp}"
                                
                                        if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then
                                             printf "."
                                -            unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})"
                                +            unavailable="$(grep 'Service Temporarily Unavailable\|Server Error\|Forbidden\|challenge-error-text' ${asntemp})"
                                +            if [ "$(jq -r .status < ${asntemp})" !=  "ok" ] ; then unavailable="NOT OK" ; fi
                                					if [ -z "${unavailable}" ]; then
                                @@ -823,8 +823,9 @@
                                    unavailable=''
                                    found=false
                                    for i in 1 2 3 4 5; do
                                -       "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
                                +       "${pathcurl}" -A "secret-agent-v1" -sS1 "${bgp_url}" > "${asntemp}"
                                
                                        if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then
                                -            unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})"
                                +            unavailable="$(grep 'Service Temporarily Unavailable\|Server Error\|Forbidden\|challenge-error-text' ${asntemp})"
                                +            if [ "$(jq -r .status < ${asntemp})" !=  "ok" ] ; then unavailable="NOT OK" ; fi
                                					if [ -z "${unavailable}" ]; then```
                                J 1 Reply Last reply Reply Quote 1
                                • J
                                  jrey @iain
                                  last edited by

                                  @iain said in pfBlockerNG ASN downloads only contain a header:

                                  "secret-agent-v1"

                                  nice -- I like using "bobisyouruncle"

                                  the code should likely go through @BBcan177 so it can get into the package

                                  that said you are providing the diff based on the original code, which a few of us have changed based on BBcan177's "try this" earlier -- no big deal at this point I can revert back to original make the change and then force a failure to test.

                                  Although it looks fine, I likely won't change this today, I've got other "real" work to do. tee off in 40 minutes (priorities)

                                  Thanks

                                  BBcan177B 1 Reply Last reply Reply Quote 1
                                  • BBcan177B
                                    BBcan177 Moderator @jrey
                                    last edited by

                                    I made some changes to my gist. Pls redownload and see how it goes. Should show a dashboard error on a fail now

                                    "Experience is something you don't get until just after you need it."

                                    Website: http://pfBlockerNG.com
                                    Twitter: @BBcan177  #pfBlockerNG
                                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                    1 Reply Last reply Reply Quote 3
                                    • Bob.DigB
                                      Bob.Dig LAYER 8
                                      last edited by Bob.Dig

                                      With 3.2.0_6 I see now this.

                                      [ AS202425_v4 ]			 Downloading update [ 08/14/23 22:42:23 ] .
                                        Downloading ASN: 202425... completed
                                      parse error: Invalid numeric literal at line 1, column 10
                                      . completed ..
                                        Empty file, Adding '127.1.7.7' to avoid download failure.
                                        ------------------------------
                                        Original Master     Final     
                                        ------------------------------
                                        0        0          0           [ Pass ] 
                                        -----------------------------------------------------------------
                                      

                                      Deleting files in /var/db/pfblockerng didn't helped.

                                      BBcan177B 1 Reply Last reply Reply Quote 1
                                      • BBcan177B
                                        BBcan177 Moderator @Bob.Dig
                                        last edited by

                                        @Bob-Dig

                                        Netgate approved my changes a bit early, as I am still waiting on a reply to my support ticket with BGPview/Security Trails/Recorded Future.

                                        The API is probably still blocking the agent string as it contains "pfBlockerNG"

                                        So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.

                                        Will keep everyone updated as I can

                                        "Experience is something you don't get until just after you need it."

                                        Website: http://pfBlockerNG.com
                                        Twitter: @BBcan177  #pfBlockerNG
                                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                        J fireodoF 3 Replies Last reply Reply Quote 1
                                        • J
                                          jrey @BBcan177
                                          last edited by

                                          @BBcan177

                                          the version you placed on the test URL for
                                          the first ua assignment
                                          is the old "pfSense/pfBlockerNG .... "
                                          the second one further down is the new
                                          "pfSense/pfBNG.....

                                          I didn't see anything appear on the dashboard when it failed but the IP count sure dropped on the dashboard.
                                          Downloading ASN: (masked) ........... completed ..
                                          Empty file, Adding '127.1.7.7' to avoid download failure.

                                          I changed the first ua back to pfSense/pfBNG as it was in the previous round of testing and of course it downloaded fine.
                                          I'll leave it until morning and look at it more.

                                          1 Reply Last reply Reply Quote 1
                                          • fireodoF
                                            fireodo @BBcan177
                                            last edited by fireodo

                                            @BBcan177 said in pfBlockerNG ASN downloads only contain a header:

                                            So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.

                                            That's what I have done after update to 3.2.0_6 and everything OK here ...
                                            (The decisive factor seems to be the user agent.)

                                            Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                                            SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                                            pfsense 2.8.0 CE
                                            Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.