pfBlockerNG ASN downloads only contain a header
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
I took a look at the dashboard part as well, and it seem the notice to the dashboard only looks at the errlog for the word "FAIL"
point here is that the download with or without the subsequent "parse error" isn't a download failure as such so it doesn't write to the error log.Maybe that would also be nice if such failures as empty ASN-Files would be reported on the Dashboard Widget too - without reading in the Forum I didn't even noticed that something was going wrong ...
-
@mbentley said in pfBlockerNG ASN downloads only contain a header:
with any modifications that were made which is why a user-agent with no spaces works
that's the thing no modifications where made, and it works sometime
@mbentley said in pfBlockerNG ASN downloads only contain a header:
The bgpview.io API is definitely blocking based on user agent string which is why the default curl and pfSense strings are getting those 403s.
I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.@mbentley said in pfBlockerNG ASN downloads only contain a header:
If they're not intentionally blocking those user-agent strings, maybe it's cloudflare blocking heavily utilized user-agent strings
thing is I don't believe the user-agent string is heavily used. at least in my case 10 or so AS queries 6 times a day. not to mention every device using the ASN feeds would have a different agent string (based on the device id) therefore cloudflare blocking base on this would mean they are "parsing" the agent string for everything that passes through and ignoring only the part of the string that contains agent-(device id) I think that seems highly unlikely.
So what if cloudflare has a lame cloud server somewhere and depending on the route to the destination the results are different. (again as long as we are speculating) after installing the patch late yesterday, it ran a couple of times with the schedule then started to fail. I re-added the debug line only this morning,
I agree, I think the patch is generally good (needs more logging, that won't hurt anything) and a notification on the dashboard when things do go south. Still the randomness of the failure (with the patch) does imply something else is still in play.
-
One thing that I have noticed just working with the bgpview.io API is that it could be helpful in validating the response by parsing the returned
.statusfrom the json before processing it withjqand possibly catching the error in some way to bring the attention to the user:$ curl -sSL -A "dont-block-me" "https://api.bgpview.io/asn/396017/prefixes" | jq . { "status": "ok", "status_message": "Query was successful", "data": { "ipv4_prefixes": [ { "prefix": "50.169.100.0/24", "ip": "50.169.100.0", "cidr": 24, "roa_status": "Valid", "name": "BAWA-CCS-1", "description": "Comcast Cable Communications, LLC", "country_code": "US", "parent": { "prefix": "50.128.0.0/9", "ip": "50.128.0.0", "cidr": 9, "rir_name": "ARIN", "allocation_status": "unknown" } }, { "prefix": "50.225.242.0/24", "ip": "50.225.242.0", "cidr": 24, "roa_status": "Valid", "name": "CCCH3-4", "description": "Comcast Cable Communications, LLC", "country_code": "US", "parent": { "prefix": "50.128.0.0/9", "ip": "50.128.0.0", "cidr": 9, "rir_name": "ARIN", "allocation_status": "unknown" } } ], "ipv6_prefixes": [] }, "@meta": { "time_zone": "UTC", "api_version": 1, "execution_time": "14.23 ms" } } -
@jrey said in pfBlockerNG ASN downloads only contain a header:
I don't think this is the case, the only time you get the 403 is when the agent string has spaces and is not quoted or the agent string is missing completely.
Any agent string will return a result so I don't think bgpview api is blocking or even looking at it.Here's a pretty good example of how you can see they're checking the user agent. I can reliably get the same responses.
# no user agent $ curl -sSL "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 403 # previous user agent the script used $ curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 403 # just with pfBlockerNG $ curl -sSL -A "pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 418 # both pfSense & pfBlockerNG $ curl -sSL -A "pfSense/pfBlockerNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 418 # misspelling of pfBlockerNG $ curl -sSL -A "pfBlockerrNG" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 200 # random string $ curl -sSL -A "this-is-random" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 200 # random with spaces $ curl -sSL -A "this is random with spaces" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP' HTTP/2 200I do appreciate the 418, I'm a teapot, http code usage.
-
@mbentley said in pfBlockerNG ASN downloads only contain a header:
curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes" --head | grep '^HTTP'
Interesting - now try this (just let it dump to the screen) - no grep on just the headers so just this:
curl -sSL -A "pfSense/pfBlockerNG cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"
notice the title of the return
<title>Just a moment...</title>
and a little further in
<span id="challenge-error-text">Enable JavaScript and cookies to continue</span>whereas (I dropped the NG from pfBlocker) and ran
curl -sSL -A "pfSense/pfBlocker cURL download agent-testvaluehere" "https://api.bgpview.io/asn/396017/prefixes"
returns, the good stuff{"status":"ok","status_message":"Query was successful","data"
they likely think that pfSense/pfBlockerNG is a browser and want to detect javascript and cookies
whereas pfSense/pfBlocker or the new format in the patch being used pfSense/pfBNG or any other string is not a browser and the results workgo figure
-
Yeah, so when I set my browser's user-agent to
pfSense/pfBlockerNG cURL download agent-testvaluehereand accessed the site, it returned the typical "Cloudflare checking if this connection is secure" sort of page when I access "https://api.bgpview.io/asn/396017/prefixes" and then it returned a 418 again once it got past that. That's something I wouldn't expect from an API endpoint since an API client shouldn't be expected to need cookies and javascript.Hopefully they respond to @BBcan177 so they can acknowledge the problem as it seems like issues with Cloudflare is one thing, the other being that they are trying to block certain clients via user agent. I'm sure it would be nice to understand what rules they are expecting us to play by with their API as they haven't published much in their FAQ besides that it's free to use.
-
@mbentley said in pfBlockerNG ASN downloads only contain a header:
so when I set my browser's user-agent to pfSense/pfBlockerNG cURL download agent-testvaluehere and accessed the site, it returned the typical "Cloudflare checking if this connection is secure" sort of page when I access "https://api.bgpview.io/asn/396017/prefixes" and then it returned a 418 again once it got past that.
I did exactly the same thing and yes (it cycled past (briefly displayed) the cloudflare page didn't ask me to do anything, but then returned the right stuff -- no error, proper values displayed for the uri provided in your post ---
Next thing was to delete the existing bgpview.io cookie, then set the browser to disable all cookies and try it again
the response in the browser this time as expected "Enable JavaScript and cookies to continue"
seems they really just want a cookie -- at least with that agent string.
-
@jrey This should check if the status is anything other than "ok" and if not, sets unavailable which is used as a flag later on as part of the error recovery. Also adds "challenge-error-text" to the strings caught to again follow the error-recovery path.
--- /usr/local/pkg/pfblockerng/pfblockerng.sh.orig 2023-08-13 22:08:41.956403000 -0400 +++ /usr/local/pkg/pfblockerng/pfblockerng.sh 2023-08-13 22:10:03.107682000 -0400 @@ -766,9 +766,10 @@ unavailable='' for i in 1 2 3 4 5; do printf "." - "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}" + "${pathcurl}" -A "secret-agent-v1" -sS1 "${bgp_url}" > "${asntemp}" if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." - unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})" + unavailable="$(grep 'Service Temporarily Unavailable\|Server Error\|Forbidden\|challenge-error-text' ${asntemp})" + if [ "$(jq -r .status < ${asntemp})" != "ok" ] ; then unavailable="NOT OK" ; fi if [ -z "${unavailable}" ]; then @@ -823,8 +823,9 @@ unavailable='' found=false for i in 1 2 3 4 5; do - "${pathcurl}" -H "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}" + "${pathcurl}" -A "secret-agent-v1" -sS1 "${bgp_url}" > "${asntemp}" if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then - unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})" + unavailable="$(grep 'Service Temporarily Unavailable\|Server Error\|Forbidden\|challenge-error-text' ${asntemp})" + if [ "$(jq -r .status < ${asntemp})" != "ok" ] ; then unavailable="NOT OK" ; fi if [ -z "${unavailable}" ]; then``` -
@iain said in pfBlockerNG ASN downloads only contain a header:
"secret-agent-v1"
nice -- I like using "bobisyouruncle"
the code should likely go through @BBcan177 so it can get into the package
that said you are providing the diff based on the original code, which a few of us have changed based on BBcan177's "try this" earlier -- no big deal at this point I can revert back to original make the change and then force a failure to test.
Although it looks fine, I likely won't change this today, I've got other "real" work to do. tee off in 40 minutes (priorities)
Thanks
-
I made some changes to my gist. Pls redownload and see how it goes. Should show a dashboard error on a fail now
-
With 3.2.0_6 I see now this.
[ AS202425_v4 ] Downloading update [ 08/14/23 22:42:23 ] . Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 . completed .. Empty file, Adding '127.1.7.7' to avoid download failure. ------------------------------ Original Master Final ------------------------------ 0 0 0 [ Pass ] -----------------------------------------------------------------Deleting files in /var/db/pfblockerng didn't helped.
-
Netgate approved my changes a bit early, as I am still waiting on a reply to my support ticket with BGPview/Security Trails/Recorded Future.
The API is probably still blocking the agent string as it contains "pfBlockerNG"
So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.
Will keep everyone updated as I can
"Experience is something you don't get until just after you need it."
Website: http://pfBlockerNG.com
Twitter: @BBcan177 #pfBlockerNG
Reddit: https://www.reddit.com/r/pfBlockerNG/new/ -
the version you placed on the test URL for
the first ua assignment
is the old "pfSense/pfBlockerNG .... "
the second one further down is the new
"pfSense/pfBNG.....I didn't see anything appear on the dashboard when it failed but the IP count sure dropped on the dashboard.
Downloading ASN: (masked) ........... completed ..
Empty file, Adding '127.1.7.7' to avoid download failure.I changed the first ua back to pfSense/pfBNG as it was in the previous round of testing and of course it downloaded fine.
I'll leave it until morning and look at it more. -
@BBcan177 said in pfBlockerNG ASN downloads only contain a header:
So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.
That's what I have done after update to 3.2.0_6 and everything OK here ...
(The decisive factor seems to be the user agent.) -
So this code works --- I removed the
the check for the "error" strings if the file is html (is not needed IMHO) as the only check needed is that you have successful json file ie:
"Query was successful"you could optionally add an addition look here to see if the file also contains "prefix" as not having any "prefix" in the response would likely imply that they returned an AS the has no IP's associated (not sure that would/could happen) - so that for another day.
I've also changed the grep to only return a "count" -c of the "Query was successful" there should alway only be 1. If the json every came back with more, that would be bad. If the file has 0, it is not that json you expect. Also using count, the entire response string doesn't need to be loaded into the variable (which ultimately saves memory assignment/reassigment)
move the "break" inside the successful part of the if (so the intended retries work) and only break out on success
if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." successful="$(grep -c 'Query was successful' ${asntemp})" if [ ${successful} == 1 ]; then found=true echo ". completed" echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig" cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig" break else echo ". Failed to download ASN" echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" touch "${pfborig}${alias}.fail" fi else sleep_val="$((i * 2))" sleep "${sleep_val}" fiwith the the ua="pfSense/pfBlockerNG cURL download agent-"
it now tries the 5 times intended and ends up on the dashboard as intended
the associated trash can does clear the error from the dashboard
the associate pfblockerng.log shows the appropriate reties (and in my case yes I logged the command called ;-) )
Downloading ASN: masked.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
. completed ..Now at this point it still has an empty file so it
Empty file, Adding '127.1.7.7' to avoid download failure.
(this can be fixed so that if the empty file is a result of 5 failed download attempts it doesn't replace it, which may be "better" )so next was to change the ua back to the new (temporary perhaps) norm.
ua="pfSense/pfBNG cURL download agent-"Downloading ASN: (masked).
.. completed
. completed ..which it gets on the first try and the resulting files contain the IPs as expected. All good.
final comment -
the only issue with pfsense/pfBlockerNG is the cookie/javascript request which returns the HTML of the moment.
a "standard" build of curl should have the option to handle the cookie and it would work. The pfSense build of curl isn't dare we say standard (ie is missing these options)
on another box the ua="pfSense/pfBlockerNG .... works just fine, because the cookie is processed.Cheers
Edit -- actually playing with curl on the netgate - the options for it to handle cookies are in fact available on the command line (not in the curl --help) I can create the cookie jar, but for some reason it is not responding to the cookie request like it does on another box. so it might in fact just be a "bug" in the build on Netgate. I'm not going to chase this, and the issue is there should be no cookie response from a json api in the first place. why BGPView is only making the request for "pfSense/pfBlockerNG in the agent string, hope they provide you some insight.
just out of curiosity I changed the ua to "cURL download pfSense/pfBlockerNG agent-" and it works.. so they really just don't want pfSense/pfBlockerNG at the start of the agent string. go figure
-
Another tweak would be to only have it write and update the error if all the attempts fail
currently if try 1,2 3 fail and 4 works (simply because of download issues) there will still be 3 failed attempt in the error file.
which would show on the dashboard,
writing the error and touching the "failed" file only on the 5 and final failure would be appropriateso
echo ". Failed to download ASN" if [ ${i} == 5 ]; then echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" touch "${pfborig}${alias}.fail" firesulting in one notice and only if all 5 attempts fail.
-
B Bob.Dig referenced this topic on
-
@manilx said in No ASN shown in reports any longer:
There is a patch mentioned "curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/1c1fee14759bc574350a3bc85b63a57e/raw"
Should I apply this? Run from the pfsense CLI I suppose?
For me, it is still failing with that patch. I guess we have to wait.
-
yes because line 761 in that file is still wrong and then the "if" statement in the raw "patch" just masks the fact.
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.
-
I believe that the "final" patch should come from dev source.
Edit: if you are applying this patch, make sure you set the Path Strip Count and Base Directory as shown in the screen capture.
However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th
System -> Patches
add a custom patch that looks like this(just what I called it)
in the patch contents field paste this:
(you can then apply and/or revert if you don't like it)--- /usr/local/pkg/pfblockerng/pfblockerng.sh 2023-08-14 13:37:26.000000000 -0400 +++ /usr/local/pkg/pfblockerng/pfblockerng.sh.jrey 2023-08-17 06:56:23.376553000 -0400 @@ -739,6 +739,7 @@ # Backup previous orig file if [ -e "${pfborig}${alias}.orig" ]; then + printf "\n mv ${pfborig}${alias}.orig ${pfborig}${alias}.bk\n" mv "${pfborig}${alias}.orig" "${pfborig}${alias}.bk" fi @@ -758,7 +759,7 @@ asn="$(echo ${host} | tr -d 'AaSs')" printf " Downloading ASN: ${asn}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}" @@ -767,28 +768,30 @@ for i in 1 2 3 4 5; do printf "." "${pathcurl}" -A "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}" + printf "\n ${pathcurl} -A \"${ua_final}\" -sS1 ${bgp_url} > ${asntemp} \n" if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." - unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})" - if [ -z "${unavailable}" ]; then + successful="$(grep -c 'Query was successful' ${asntemp})" + if [ ${successful} == 1 ]; then found=true - echo ". completed" + echo ". completed (Download Valid)" echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig" cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig" break else + echo ". Failed to download ASN" + if [ ${i} == 5 ]; then + echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" + fi + fi + else sleep_val="$((i * 2))" sleep "${sleep_val}" - fi fi done - if [ ! -z "${unavailable}" ]; then - echo ". Failed to download ASN" - touch "${pfborig}${alias}.fail" - fi - if [ "${multiple}" -gt 0 ]; then sleep 1 fi @@ -798,23 +801,27 @@ # Restore previous orig file if [ "${found}" == false ]; then if [ -e "${pfborig}${alias}.bk" ]; then + printf "restore bk file\n" mv "${pfborig}${alias}.bk" "${pfborig}${alias}.orig" else + printf "making an empty orig file\n" echo > "${pfborig}${alias}.orig" + echo " [ AS${asn}: ${host} ] Made Empty Original file ${pfborig}${alias}.orig [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" fi else if [ -e "${pfborig}${alias}.bk" ]; then + printf "removing bk file\n" rm -f "${pfborig}${alias}.bk" fi fi } - # Function to convert IP to ASN iptoasn() { host="${alias}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}"
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
System -> Patches
add a custom patch that looks like this(just what I called it)
Hi I just reinstalled 3.2.0_6 and created a patch as you described. Run Force Reload Task - All and got the follwoing errors in the log:
[ Manually_blocked_Domains_custom_v4 ] Reload Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 50360... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 204428... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 44446... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 398324... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 23528... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 29470... completed parse error: Invalid numeric literal at line 1, column 10 . completed ..Netgate 8200max
