pfBlockerNG ASN downloads only contain a header
-
Netgate approved my changes a bit early, as I am still waiting on a reply to my support ticket with BGPview/Security Trails/Recorded Future.
The API is probably still blocking the agent string as it contains "pfBlockerNG"
So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.
Will keep everyone updated as I can
"Experience is something you don't get until just after you need it."
Website: http://pfBlockerNG.com
Twitter: @BBcan177 #pfBlockerNG
Reddit: https://www.reddit.com/r/pfBlockerNG/new/ -
the version you placed on the test URL for
the first ua assignment
is the old "pfSense/pfBlockerNG .... "
the second one further down is the new
"pfSense/pfBNG.....I didn't see anything appear on the dashboard when it failed but the IP count sure dropped on the dashboard.
Downloading ASN: (masked) ........... completed ..
Empty file, Adding '127.1.7.7' to avoid download failure.I changed the first ua back to pfSense/pfBNG as it was in the previous round of testing and of course it downloaded fine.
I'll leave it until morning and look at it more. -
@BBcan177 said in pfBlockerNG ASN downloads only contain a header:
So in the short term, after upgrading to the latest version, re-download the patch that I posted in this thread.
That's what I have done after update to 3.2.0_6 and everything OK here ...
(The decisive factor seems to be the user agent.) -
So this code works --- I removed the
the check for the "error" strings if the file is html (is not needed IMHO) as the only check needed is that you have successful json file ie:
"Query was successful"you could optionally add an addition look here to see if the file also contains "prefix" as not having any "prefix" in the response would likely imply that they returned an AS the has no IP's associated (not sure that would/could happen) - so that for another day.
I've also changed the grep to only return a "count" -c of the "Query was successful" there should alway only be 1. If the json every came back with more, that would be bad. If the file has 0, it is not that json you expect. Also using count, the entire response string doesn't need to be loaded into the variable (which ultimately saves memory assignment/reassigment)
move the "break" inside the successful part of the if (so the intended retries work) and only break out on success
if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." successful="$(grep -c 'Query was successful' ${asntemp})" if [ ${successful} == 1 ]; then found=true echo ". completed" echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig" cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig" break else echo ". Failed to download ASN" echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" touch "${pfborig}${alias}.fail" fi else sleep_val="$((i * 2))" sleep "${sleep_val}" fiwith the the ua="pfSense/pfBlockerNG cURL download agent-"
it now tries the 5 times intended and ends up on the dashboard as intended
the associated trash can does clear the error from the dashboard
the associate pfblockerng.log shows the appropriate reties (and in my case yes I logged the command called ;-) )
Downloading ASN: masked.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
.
.. Failed to download ASN
. completed ..Now at this point it still has an empty file so it
Empty file, Adding '127.1.7.7' to avoid download failure.
(this can be fixed so that if the empty file is a result of 5 failed download attempts it doesn't replace it, which may be "better" )so next was to change the ua back to the new (temporary perhaps) norm.
ua="pfSense/pfBNG cURL download agent-"Downloading ASN: (masked).
.. completed
. completed ..which it gets on the first try and the resulting files contain the IPs as expected. All good.
final comment -
the only issue with pfsense/pfBlockerNG is the cookie/javascript request which returns the HTML of the moment.
a "standard" build of curl should have the option to handle the cookie and it would work. The pfSense build of curl isn't dare we say standard (ie is missing these options)
on another box the ua="pfSense/pfBlockerNG .... works just fine, because the cookie is processed.Cheers
Edit -- actually playing with curl on the netgate - the options for it to handle cookies are in fact available on the command line (not in the curl --help) I can create the cookie jar, but for some reason it is not responding to the cookie request like it does on another box. so it might in fact just be a "bug" in the build on Netgate. I'm not going to chase this, and the issue is there should be no cookie response from a json api in the first place. why BGPView is only making the request for "pfSense/pfBlockerNG in the agent string, hope they provide you some insight.
just out of curiosity I changed the ua to "cURL download pfSense/pfBlockerNG agent-" and it works.. so they really just don't want pfSense/pfBlockerNG at the start of the agent string. go figure
-
Another tweak would be to only have it write and update the error if all the attempts fail
currently if try 1,2 3 fail and 4 works (simply because of download issues) there will still be 3 failed attempt in the error file.
which would show on the dashboard,
writing the error and touching the "failed" file only on the 5 and final failure would be appropriateso
echo ". Failed to download ASN" if [ ${i} == 5 ]; then echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" touch "${pfborig}${alias}.fail" firesulting in one notice and only if all 5 attempts fail.
-
B Bob.Dig referenced this topic on
-
@manilx said in No ASN shown in reports any longer:
There is a patch mentioned "curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/1c1fee14759bc574350a3bc85b63a57e/raw"
Should I apply this? Run from the pfsense CLI I suppose?
For me, it is still failing with that patch. I guess we have to wait.
-
yes because line 761 in that file is still wrong and then the "if" statement in the raw "patch" just masks the fact.
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.
-
I believe that the "final" patch should come from dev source.
Edit: if you are applying this patch, make sure you set the Path Strip Count and Base Directory as shown in the screen capture.
However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th
System -> Patches
add a custom patch that looks like this(just what I called it)
in the patch contents field paste this:
(you can then apply and/or revert if you don't like it)--- /usr/local/pkg/pfblockerng/pfblockerng.sh 2023-08-14 13:37:26.000000000 -0400 +++ /usr/local/pkg/pfblockerng/pfblockerng.sh.jrey 2023-08-17 06:56:23.376553000 -0400 @@ -739,6 +739,7 @@ # Backup previous orig file if [ -e "${pfborig}${alias}.orig" ]; then + printf "\n mv ${pfborig}${alias}.orig ${pfborig}${alias}.bk\n" mv "${pfborig}${alias}.orig" "${pfborig}${alias}.bk" fi @@ -758,7 +759,7 @@ asn="$(echo ${host} | tr -d 'AaSs')" printf " Downloading ASN: ${asn}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}" @@ -767,28 +768,30 @@ for i in 1 2 3 4 5; do printf "." "${pathcurl}" -A "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}" + printf "\n ${pathcurl} -A \"${ua_final}\" -sS1 ${bgp_url} > ${asntemp} \n" if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." - unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})" - if [ -z "${unavailable}" ]; then + successful="$(grep -c 'Query was successful' ${asntemp})" + if [ ${successful} == 1 ]; then found=true - echo ". completed" + echo ". completed (Download Valid)" echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig" cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig" break else + echo ". Failed to download ASN" + if [ ${i} == 5 ]; then + echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" + fi + fi + else sleep_val="$((i * 2))" sleep "${sleep_val}" - fi fi done - if [ ! -z "${unavailable}" ]; then - echo ". Failed to download ASN" - touch "${pfborig}${alias}.fail" - fi - if [ "${multiple}" -gt 0 ]; then sleep 1 fi @@ -798,23 +801,27 @@ # Restore previous orig file if [ "${found}" == false ]; then if [ -e "${pfborig}${alias}.bk" ]; then + printf "restore bk file\n" mv "${pfborig}${alias}.bk" "${pfborig}${alias}.orig" else + printf "making an empty orig file\n" echo > "${pfborig}${alias}.orig" + echo " [ AS${asn}: ${host} ] Made Empty Original file ${pfborig}${alias}.orig [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" fi else if [ -e "${pfborig}${alias}.bk" ]; then + printf "removing bk file\n" rm -f "${pfborig}${alias}.bk" fi fi } - # Function to convert IP to ASN iptoasn() { host="${alias}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}"
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
System -> Patches
add a custom patch that looks like this(just what I called it)
Hi I just reinstalled 3.2.0_6 and created a patch as you described. Run Force Reload Task - All and got the follwoing errors in the log:
[ Manually_blocked_Domains_custom_v4 ] Reload Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 50360... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 204428... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 44446... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 398324... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 23528... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 29470... completed parse error: Invalid numeric literal at line 1, column 10 . completed ..Netgate 8200max
-
For me it is also not working but I haven't reinstalled pfBlocker. -
@jrey said in pfBlockerNG ASN downloads only contain a header:
However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th
Have to correct myself, it is working great. Looks like
you really haveI had to reinstall pfBlocker. Thanks man!
-
This post is deleted! -
@jrey I had the "Path Strip Count" wrong (2).
Reinstalled pfblocker, applied the patch and all working.Thx!
-
@Bob-Dig said in pfBlockerNG ASN downloads only contain a header:
really have to reinstall pfBlocker.
you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first
just go from fresh 3.0.2_6 directly to the patch.
-
so you had to change it from 2 to 0 as shown in the screen shot I provided :-)
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first
just go from fresh 3.0.2_6 directly to the patch.
Now second instance is patched, thank you. Problem on first attempt was the original patch that couldn't fix it.
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
@Bob-Dig
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.Just a reflection - maybe the curl user agent should be created randomly ...?
Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
pfsense 2.8.0 CE
Packages: Apcupsd, Cron, Iftop, Iperf, LCDproc, Nmap, pfBlockerNG, RRD_Summary, Shellcmd, Snort, Speedtest, System_Patches. -
This post is deleted! -
Not sure that specifically is the issue..
the failure is very specific to an agent string the starts with pfSense/pfBlockerNGas mentioned you can change the order of the failing "agent" string and it will work.
However when failing if you capture the failed response page ie the "blocking" is typically from cloudflare, where BGPView is hosted. They are the ones that have to say why that specific (the original) agent string format (and then only those starting with pfSense/pfBlockerNG fail to pass). the responses I have captured are "we want a cookie and java script enabled"(ie you are a robot) and just an outright fail message (on purpose a bad request).Now some have suggested the using the Negate device ID as part of this "agent" string, is a bad thing. I'd suspect that BGPView would not require this, unless they are mining their log files looking "volume" from specific devices. (there doesn't seem to be anything about them doing or requiring this)
Subsequent to the patch provided and just for the purpose of messing around, I have changed the agent-(Netgate device ID) part of the string to agent-(numbers of "date")My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.Nothing bad has happened.
I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.
All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.I changed in the agent string also in something individual but not de device ID and it worked with no problems.
Nothing bad has happened.
So it was here too.
I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.
Thats what I suggested - something randomly generated.
All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"
Thats correct! But as you said - lets see what kind of answer BBcan177 gets from them - and then we can react. In the meantime the solution you and BBcan177 provided is good and functional.
Regards,
fireodo
