pfBlockerNG ASN downloads only contain a header

jrey

@fireodo said in pfBlockerNG ASN downloads only contain a header:

It seams also to be a API issue at bgpview .

I didn't see that -- it worked as expected from every other platform I test pulled from and has been consistently returning results
You can even drop the uri into a browser and it will respond --- so from the client side any agent string works, they don't appear to tie the result success to the content of the agent string.

I'm thinking more that curl changed with a recent update and as a result the call with all the "-H" stuff started running into issues (outright failing actually) as demonstrated from the command line test.
the api also failed at the pgpview end because it no longer saw an "agent" and returned a response to the tmp/file that said as much. That response file ended up being an "orig" file with only a header. The agent string from my testing is required by the api but the agent string can be anything so "bobisyouruncle" as provided in my test strings works and the api didn't complain. I therefore didn't change the code so it still sends the Netgate device ID for the agent with the -A but all the other stuff that was originally in the -H parameter was obviously removed from the changed -A parameter.

Thanks for your feedback...
JR

BBcan177

@jrey

Thanks for digging in. Looks like that was the issue.. Sorry about that!

I made a patch which can be downloaded here as there are two lines that needed to be changed:

curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/1c1fee14759bc574350a3bc85b63a57e/raw"

Will get this into a PR soon.

UPDATE:
I believe that BGPview is rate limiting any Agent string that contains "pfBlockerNG", I changed it to "pfBNG" for now and will update once I get feedback from them,

BBcan177

@jrey @fireodo If you guys can test the patch would be appreciated Thanks!

jrey

@BBcan177 said in pfBlockerNG ASN downloads only contain a header:

I believe that BGPview is rate limiting any Agent string that contains "pfBlockerNG", I changed it to "pfBNG" for now and will update once I get feedback from them,

interesting, I'm likely not hitting a rate limit as I use less than 15 ASN feeds (and clearly they didn't prevent me from only hitting with an agent string like bobisyouruncle.

the second case you changed (around line 826) would only be used for IP to ASN right ? (so in the case where ASN Reporting is enabled)

also I did a mv to capture the current file, then got your new file -- forgot to set the execute permissions LOL that was fun -- whoops
see the permissions on file which
seems to be working on a force -- I'll let it run on the schedule for a bit.

I'd honestly still like to see that actual call / parameters logged (it is always helpful at some point) and perhaps something that notifies the dashboard if the list ends up with just a "header" and no IP addresses as was the case here.

Thanks,
JR

mbentley

I ended up creating a couple of patches that I could apply through the Patches package:

https://gist.github.com/mbentley/a3f93643de57a0f325fbee7bf34afbad

One where the user agent string is fixed from -H to -A and another where it isn't yet fixed (where you're running the original files directly from the package.

Nice thing about the patch there is that it could make it easy to have a unique user agent to try to not get whatever string is being set flagged for rate limiting as even the adding in the netgate device ID to the UA doesn't really change anything.

I've also noticed that the service was REALLY slow before they started blocking. I have to imagine their API was getting hammered and them blocking a ton of bots has helped their APIs be quicker - they're lightning fast right now.

*edit: I noticed that I actually missed one place where the user-agent was set when I was creating the patch file. Updated the patch file at https://gist.github.com/mbentley/a3f93643de57a0f325fbee7bf34afbad#file-pfblockerng-sh-patch

Bob.Dig

As a side question: If I disable

Netgate Device ID - Do NOT send Netgate Device ID with user agent

in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here? Is this the real usecase for that Netgate Device ID? Or is it better to not have it? Will the other side still be able to see "pfSense" if i disable the option?

fireodo

@BBcan177 said in pfBlockerNG ASN downloads only contain a header:

If you guys can test the patch would be appreciated Thanks!

Hi,

sorry but I have after the patch the same empty ASN Lists. (BTW. I have only 7 Lists) (I dont get the parsing error anymore)
@jrey The thumbs up was as a compliment for your intensive Work, but (here) it is not successful (I change the user agent and also the curl parameter)
There must be something else too ...
Edit: I look in the log and I saw that it begun on the night from 12 to 13.08. (no changes on pfblockerNG for a long time)

my 2 cents,
fireodo

jrey

@BBcan177

riddle me this.

appears are still using a technically failing command. I put back the printf to capture the command being run, because seeing what is happening is important.

once a cycle completed I pulled the command from the log

This is what is being executed from the "patch" - and at the command line it fails.

[23.05.1-RELEASE][mask]: /usr/local/bin/curl -A pfSense/pfBNG cURL download agent-MASKED -sS1 https://api.bgpview.io/MASKED/prefixes > /tmp/pfbtemp10_7790.
curl: (6) Could not resolve host: cURL
curl: (6) Could not resolve host: download
curl: (6) Could not resolve host: agent-MASKED

changing it at the command line so that the -A (agent parameter) only includes the agent-part of the string (No Errors at command line)
[23.05.1-RELEASE][mask]: /usr/local/bin/curl -A agent-MASKED -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_7790.
[23.05.1-RELEASE][mask]:

this works equally as well (I removed the redirect of the output away from the tmp file and just let it spew to the screen (still works, every time)
[23.05.1-RELEASE][mask]: /usr/local/bin/curl -A agent-bobisyouruncle -sS1 https://api.bgpview.io/asn/MASKED/prefixes
{"status":"ok","status_message":"Query was successful","......

Now from BPGView API page

they "suggest / try" with only the --include option
this by itself fails from a command line
/usr/local/bin/curl --include https://api.bgpview.io/asn/MASK/prefixes
returns a Forbidden

HTTP/2 403
date: Mon, 14 Aug 2023 11:29:10 GMT
content-type: text/html
cf-cache-status: DYNAMIC
...

However adding an agent string even "agent-bob" as shown here works just fine.
/usr/local/bin/curl -A agent-bob --include https://api.bgpview.io/asn/MASK/prefixes

this also works (took the spaces out of your agent string replace with -

/usr/local/bin/curl -A pfSense/pfBNG-cURL-download-agent-MASKED -sS1 https://api.bgpview.io/asn/MASKED/prefixes

clearly the version of curl doesn't like spaces in the agent string,
So the documentation for curl "suggests" that the -A string be enclose in quotes if it contains spaces so doing that of course works.
shows this
curl -A "user-agent-name-here" [URL]
and an example like this
curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" https://example.com/

therefore
/usr/local/bin/curl -A "pfSense/pfBNG cURL download agent-MASKED" -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_7790.
also works

and as for BGPView it doesn't appear they care what the agent string is, as long as there is one.

@fireodo
so now it appears for you the patch is not working although the parse error is gone?
You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

also try the above commands from a shell

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

so now it appears for you the patch is not working although the parse error is gone?

Thats correct!

You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

I'll do that.

also try the above commands from a shell

That too ...

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

THAT was the solution! With @BBcan177 s Patch and deleting the old files now the ASN Lists are populated as they should - that should be noticed somewhere when pfblockerNG 3.2.0_6 is available!

Below the line I can say the culprit was the user agent and the curl parameter - maybe the user agent logic should be reconsidered ... if necessary.

Thanks @jrey @BBcan177
fireodo

mbentley

@jrey - Looks like your initial curl issue you mention with the spaces in the user agent is a lack of quotes in the script around the user agent string. I see quotes in the patch that @BBcan177 provided so I'd have to guess there is something funky going on with any modifications that were made which is why a user-agent with no spaces works.

The bgpview.io API is definitely blocking based on user agent string which is why the default curl and pfSense strings are getting those 403s. If they're not intentionally blocking those user-agent strings, maybe it's cloudflare blocking heavily utilized user-agent strings - pure speculation though.

jrey

@Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here?

I don't think so, because the code in pfblockerng itself doesn't have any logic around this.
the "agent-" is alway part of the string being built.

ua="pfSense/pfBNG cURL download agent-"
guid="$(/usr/sbin/gnid)" <-- the device ID is pulled here and immediately used in the next line to build the ua_final parameter used in the call
ua_final="${ua}${guid}"

so unless the setting you mention changes the setting for /usr/sbin/gnid so it returns nothing I would think the setting has no effect.
even if it did and the guid is empty (it would build the agent string as "pfSense/pfBNG cURL download agent-"

easy to confirm
I changed the setting and tried it both ways. the logged command (because logging what is running is important !!!) still contains the device ID

the setting makes no difference here, on or off, you still get the device id in the generated curl agent string.

JR

fireodo

@Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

Netgate Device ID - Do NOT send Netgate Device ID with user agent
in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here? Is this the real usecase for that Netgate Device ID? Or is it better to not have it? Will the other side still be able to see "pfSense" if i disable the option?

I dont think so - this concerns (as far as I know) only the communication with Netgate Servers ...

jrey

@fireodo said in pfBlockerNG ASN downloads only contain a header:

this concerns (as far as I know) only the communication with Netgate Servers ...

correct, verified with logging ;) - and the setting on the page referenced for this setting is in the section labelled "Installation Feedback" implies the same.. to me anyway.

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

I took a look at the dashboard part as well, and it seem the notice to the dashboard only looks at the errlog for the word "FAIL"
point here is that the download with or without the subsequent "parse error" isn't a download failure as such so it doesn't write to the error log.

Maybe that would also be nice if such failures as empty ASN-Files would be reported on the Dashboard Widget too - without reading in the Forum I didn't even noticed that something was going wrong ...

jrey

@mbentley said in pfBlockerNG ASN downloads only contain a header:

with any modifications that were made which is why a user-agent with no spaces works

that's the thing no modifications where made, and it works sometime

@mbentley said in pfBlockerNG ASN downloads only contain a header:

The bgpview.io API is definitely blocking based on user agent string which is why the default curl and pfSense strings are getting those 403s.

I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.

@mbentley said in pfBlockerNG ASN downloads only contain a header:

If they're not intentionally blocking those user-agent strings, maybe it's cloudflare blocking heavily utilized user-agent strings

thing is I don't believe the user-agent string is heavily used. at least in my case 10 or so AS queries 6 times a day. not to mention every device using the ASN feeds would have a different agent string (based on the device id) therefore cloudflare blocking base on this would mean they are "parsing" the agent string for everything that passes through and ignoring only the part of the string that contains agent-(device id) I think that seems highly unlikely.

So what if cloudflare has a lame cloud server somewhere and depending on the route to the destination the results are different. (again as long as we are speculating) after installing the patch late yesterday, it ran a couple of times with the schedule then started to fail. I re-added the debug line only this morning,

I agree, I think the patch is generally good (needs more logging, that won't hurt anything) and a notification on the dashboard when things do go south. Still the randomness of the failure (with the patch) does imply something else is still in play.

mbentley

One thing that I have noticed just working with the bgpview.io API is that it could be helpful in validating the response by parsing the returned .status from the json before processing it with jq and possibly catching the error in some way to bring the attention to the user:

$ curl -sSL -A "dont-block-me" "https://api.bgpview.io/asn/396017/prefixes" | jq .
{
  "status": "ok",
  "status_message": "Query was successful",
  "data": {
    "ipv4_prefixes": [
      {
        "prefix": "50.169.100.0/24",
        "ip": "50.169.100.0",
        "cidr": 24,
        "roa_status": "Valid",
        "name": "BAWA-CCS-1",
        "description": "Comcast Cable Communications, LLC",
        "country_code": "US",
        "parent": {
          "prefix": "50.128.0.0/9",
          "ip": "50.128.0.0",
          "cidr": 9,
          "rir_name": "ARIN",
          "allocation_status": "unknown"
        }
      },
      {
        "prefix": "50.225.242.0/24",
        "ip": "50.225.242.0",
        "cidr": 24,
        "roa_status": "Valid",
        "name": "CCCH3-4",
        "description": "Comcast Cable Communications, LLC",
        "country_code": "US",
        "parent": {
          "prefix": "50.128.0.0/9",
          "ip": "50.128.0.0",
          "cidr": 9,
          "rir_name": "ARIN",
          "allocation_status": "unknown"
        }
      }
    ],
    "ipv6_prefixes": []
  },
  "@meta": {
    "time_zone": "UTC",
    "api_version": 1,
    "execution_time": "14.23 ms"
  }
}

mbentley

@jrey said in pfBlockerNG ASN downloads only contain a header:

I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.

Here's a pretty good example of how you can see they're checking the user agent. I can reliably get the same responses.

# no user agent
$ curl -sSL "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 403

# previous user agent the script used
$ curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 403

# just with pfBlockerNG
$ curl -sSL -A "pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 418

# both pfSense & pfBlockerNG
$ curl -sSL -A "pfSense/pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 418

# misspelling of pfBlockerNG
$ curl -sSL -A "pfBlockerrNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 200

# random string
$ curl -sSL -A "this-is-random" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 200

# random with spaces
$ curl -sSL -A "this is random with spaces" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
HTTP/2 200

I do appreciate the 418, I'm a teapot, http code usage.

jrey

@mbentley said in pfBlockerNG ASN downloads only contain a header:

curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'

Interesting - now try this (just let it dump to the screen) - no grep on just the headers so just this:

curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"

notice the title of the return
<title>Just a moment...</title>
and a little further in
<span id="challenge-error-text">Enable JavaScript and cookies to continue</span>

whereas (I dropped the NG from pfBlocker) and ran

curl -sSL -A "pfSense/pfBlocker cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"
returns, the good stuff

{"status":"ok","status_message":"Query was successful","data"

they likely think that pfSense/pfBlockerNG is a browser and want to detect javascript and cookies
whereas pfSense/pfBlocker or the new format in the patch being used pfSense/pfBNG or any other string is not a browser and the results work

go figure

mbentley

@jrey

Yeah, so when I set my browser's user-agent to pfSense/pfBlockerNG cURL download agent-testvaluehere and accessed the site, it returned the typical "Cloudflare checking if this connection is secure" sort of page when I access "https://api.bgpview.io/asn/396017/prefixes" and then it returned a 418 again once it got past that. That's something I wouldn't expect from an API endpoint since an API client shouldn't be expected to need cookies and javascript.

Hopefully they respond to @BBcan177 so they can acknowledge the problem as it seems like issues with Cloudflare is one thing, the other being that they are trying to block certain clients via user agent. I'm sure it would be nice to understand what rules they are expecting us to play by with their API as they haven't published much in their FAQ besides that it's free to use.