pfBlockerNG ASN downloads only contain a header

jrey

Another tweak would be to only have it write and update the error if all the attempts fail
currently if try 1,2 3 fail and 4 works (simply because of download issues) there will still be 3 failed attempt in the error file.
which would show on the dashboard,
writing the error and touching the "failed" file only on the 5 and final failure would be appropriate

so

					echo ". Failed to download ASN"
                                            if [ ${i} == 5 ]; then 
					  echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}"
					  touch "${pfborig}${alias}.fail"
                                            fi

resulting in one notice and only if all 5 attempts fail.

Screen Shot 2023-08-15 at 1.42.22 PM.png

Bob.Dig

@manilx said in No ASN shown in reports any longer:

There is a patch mentioned "curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/1c1fee14759bc574350a3bc85b63a57e/raw"

Should I apply this? Run from the pfsense CLI I suppose?

For me, it is still failing with that patch. I guess we have to wait.

jrey

@Bob-Dig

yes because line 761 in that file is still wrong and then the "if" statement in the raw "patch" just masks the fact.

I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.

jrey

@Bob-Dig

I believe that the "final" patch should come from dev source.

Edit: if you are applying this patch, make sure you set the Path Strip Count and Base Directory as shown in the screen capture.

However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th

System -> Patches
add a custom patch that looks like this(just what I called it)
Screen Shot 2023-08-20 at 7.43.55 AM.png

in the patch contents field paste this:
(you can then apply and/or revert if you don't like it)

--- /usr/local/pkg/pfblockerng/pfblockerng.sh	2023-08-14 13:37:26.000000000 -0400
+++ /usr/local/pkg/pfblockerng/pfblockerng.sh.jrey	2023-08-17 06:56:23.376553000 -0400
@@ -739,6 +739,7 @@
 
 	# Backup previous orig file
 	if [ -e "${pfborig}${alias}.orig" ]; then
+		printf "\n mv ${pfborig}${alias}.orig ${pfborig}${alias}.bk\n"
 		mv "${pfborig}${alias}.orig" "${pfborig}${alias}.bk"
 	fi
 
@@ -758,7 +759,7 @@
 			asn="$(echo ${host} | tr -d 'AaSs')"
 			printf "  Downloading ASN: ${asn}"
 
-			ua="pfSense/pfBlockerNG cURL download agent-"
+			ua="pfSense/pfBNG cURL download agent-"
 			guid="$(/usr/sbin/gnid)"
 			ua_final="${ua}${guid}"
 
@@ -767,28 +768,30 @@
 			for i in 1 2 3 4 5; do
 				printf "."
 				"${pathcurl}" -A "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}"
+                                printf "\n ${pathcurl} -A \"${ua_final}\" -sS1 ${bgp_url} > ${asntemp} \n"
 
 				if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then
 					printf "."
-					unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})"
-					if [ -z "${unavailable}" ]; then
+					successful="$(grep -c 'Query was successful' ${asntemp})"
+     					if [ ${successful} == 1 ]; then
 						found=true
-						echo ". completed"
+						echo ". completed (Download Valid)"
 						echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig"
 						cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig"
 						break
 					else
+						echo ". Failed to download ASN"
+                                                if [ ${i} == 5 ]; then 
+							echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}"
+							touch "${pfborig}${alias}.fail"
+                                                fi
+					fi
+				else
 						sleep_val="$((i * 2))"
 						sleep "${sleep_val}"
-					fi
 				fi
 			done
 
-			if [ ! -z "${unavailable}" ]; then
-				echo ". Failed to download ASN"
-				touch "${pfborig}${alias}.fail"
-			fi
-
 			if [ "${multiple}" -gt 0 ]; then
 				sleep 1
 			fi
@@ -798,23 +801,27 @@
 	# Restore previous orig file
 	if [ "${found}" == false ]; then
 		if [ -e "${pfborig}${alias}.bk" ]; then
+			printf "restore bk file\n"
 			mv "${pfborig}${alias}.bk" "${pfborig}${alias}.orig"
 		else
+			printf "making an empty orig file\n"
 			echo > "${pfborig}${alias}.orig"
+			echo " [ AS${asn}: ${host} ] Made Empty Original file ${pfborig}${alias}.orig [ ${now} ]" >> "${errorlog}"
+			touch "${pfborig}${alias}.fail"
 		fi
 	else
 		if [ -e "${pfborig}${alias}.bk" ]; then
+			printf "removing bk file\n"
 			rm -f "${pfborig}${alias}.bk"
 		fi
 	fi
 }
 
-
 # Function to convert IP to ASN
 iptoasn() {
 	host="${alias}"
 
-	ua="pfSense/pfBlockerNG cURL download agent-"
+	ua="pfSense/pfBNG cURL download agent-"
 	guid="$(/usr/sbin/gnid)"
 	ua_final="${ua}${guid}"

Screen Shot 2023-08-20 at 7.50.39 AM.png

manilx

@jrey said in pfBlockerNG ASN downloads only contain a header:

System -> Patches
add a custom patch that looks like this(just what I called it)

Hi I just reinstalled 3.2.0_6 and created a patch as you described. Run Force Reload Task - All and got the follwoing errors in the log:

[ Manually_blocked_Domains_custom_v4 ] Reload
  Downloading ASN: 202425... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 207812... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 50360... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 204428... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 202425... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 207812... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 44446... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 207812... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 398324... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 207812... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 202425... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 23528... completed
parse error: Invalid numeric literal at line 1, column 10
  Downloading ASN: 29470... completed
parse error: Invalid numeric literal at line 1, column 10
 . completed ..

Bob.Dig

~~For me it is also not working but I haven't reinstalled pfBlocker.~~

Bob.Dig

@jrey said in pfBlockerNG ASN downloads only contain a header:

However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th

Have to correct myself, it is working great. Looks like ~~you really have~~ I had to reinstall pfBlocker. Thanks man!

jrey

This post is deleted!

manilx

@jrey I had the "Path Strip Count" wrong (2).
Reinstalled pfblocker, applied the patch and all working.

Thx!

jrey

@Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

really have to reinstall pfBlocker.

you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first

just go from fresh 3.0.2_6 directly to the patch.

jrey

@manilx

so you had to change it from 2 to 0 as shown in the screen shot I provided :-)

Bob.Dig

@jrey said in pfBlockerNG ASN downloads only contain a header:

you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first

just go from fresh 3.0.2_6 directly to the patch.

Now second instance is patched, thank you. Problem on first attempt was the original patch that couldn't fix it.

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

@Bob-Dig
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.

Just a reflection - maybe the curl user agent should be created randomly ...?

NogBadTheBad

This post is deleted!

jrey

@fireodo

Not sure that specifically is the issue..
the failure is very specific to an agent string the starts with pfSense/pfBlockerNG

as mentioned you can change the order of the failing "agent" string and it will work.
However when failing if you capture the failed response page ie the "blocking" is typically from cloudflare, where BGPView is hosted. They are the ones that have to say why that specific (the original) agent string format (and then only those starting with pfSense/pfBlockerNG fail to pass). the responses I have captured are "we want a cookie and java script enabled"(ie you are a robot) and just an outright fail message (on purpose a bad request).

Now some have suggested the using the Negate device ID as part of this "agent" string, is a bad thing. I'd suspect that BGPView would not require this, unless they are mining their log files looking "volume" from specific devices. (there doesn't seem to be anything about them doing or requiring this)
Subsequent to the patch provided and just for the purpose of messing around, I have changed the agent-(Netgate device ID) part of the string to agent-(numbers of "date")

My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.

Nothing bad has happened.

I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.

All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.

I changed in the agent string also in something individual but not de device ID and it worked with no problems.

Nothing bad has happened.

So it was here too.

I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.

Thats what I suggested - something randomly generated.

All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"

Thats correct! But as you said - lets see what kind of answer BBcan177 gets from them - and then we can react. In the meantime the solution you and BBcan177 provided is good and functional.

Regards,
fireodo

Wolf666

@jrey
I have reintalled pfBlockerNG 3.2.0_6 on pfSense 23.05.1, I deleted old ASXXX files, applied the patch and force reloaded but files only contains IP 127.1.7.7.

[ AS14618_v4 ]			 Downloading update [ 08/23/23 00:00:45 ] .
  Downloading ASN: 14618... completed
parse error: Invalid numeric literal at line 1, column 10
. completed ..
  Empty file, Adding '127.1.7.7' to avoid download failure.

I will wait for an official fix. Thanks all for your efforts.

jrey

@Wolf666

~~so what is in the pfblockerng.log file?~~
sorry, I responded to this based on the email and noticed that you included the log snippet in the post online --

That doesn't look like the correct response and appears to be running the wrong code - when applying the patch did you change the Path Strip Count from the default value of 2 to a 0 (zero)
Screen Shot 2023-08-23 at 7.14.33 AM.png

~~specifically do you see this ?~~
you should specifically see this before the processing starts

.. completed (Download Valid)

anything in the error.log ?

is it possible the IP addresses are also in another list?
De-duplication on?
CIDR Aggregation on?

Not asking you to turn any of the above on or off, just how you have them set.

Thanks

Wolf666

@jrey
I uninstalled/reinstalled pfBlockerNG and applied the patch, now seems to work.

[ TWAS13414_v4 ]		 Downloading update .
  Downloading ASN: 13414.
 /usr/local/bin/curl -A &quot;pfSense/pfBNG cURL download agent-49de62cd6bb042f3ec1e&quot; -sS1 https://api.bgpview.io/asn/13414/prefixes &gt; /tmp/pfbtemp10_46283 
.. completed (Download Valid)
. completed ..

  Aggregation Stats:
  ------------------
  Original Final      
  ------------------
  53       14         
  ------------------

Thank you.

jrey

@Wolf666

bingo. Path Strip Count right. Glad it is working for you. Thanks