pfBlockerNG ASN downloads only contain a header
-
I believe that the "final" patch should come from dev source.
Edit: if you are applying this patch, make sure you set the Path Strip Count and Base Directory as shown in the screen capture.
However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th
System -> Patches
add a custom patch that looks like this(just what I called it)
in the patch contents field paste this:
(you can then apply and/or revert if you don't like it)--- /usr/local/pkg/pfblockerng/pfblockerng.sh 2023-08-14 13:37:26.000000000 -0400 +++ /usr/local/pkg/pfblockerng/pfblockerng.sh.jrey 2023-08-17 06:56:23.376553000 -0400 @@ -739,6 +739,7 @@ # Backup previous orig file if [ -e "${pfborig}${alias}.orig" ]; then + printf "\n mv ${pfborig}${alias}.orig ${pfborig}${alias}.bk\n" mv "${pfborig}${alias}.orig" "${pfborig}${alias}.bk" fi @@ -758,7 +759,7 @@ asn="$(echo ${host} | tr -d 'AaSs')" printf " Downloading ASN: ${asn}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}" @@ -767,28 +768,30 @@ for i in 1 2 3 4 5; do printf "." "${pathcurl}" -A "${ua_final}" -sS1 "${bgp_url}" > "${asntemp}" + printf "\n ${pathcurl} -A \"${ua_final}\" -sS1 ${bgp_url} > ${asntemp} \n" if [ -e "${asntemp}" ] && [ -s "${asntemp}" ]; then printf "." - unavailable="$(grep 'Service Temporarily Unavailable\|Server Error' ${asntemp})" - if [ -z "${unavailable}" ]; then + successful="$(grep -c 'Query was successful' ${asntemp})" + if [ ${successful} == 1 ]; then found=true - echo ". completed" + echo ". completed (Download Valid)" echo "### AS${asn}: ${host} ###" >> "${pfborig}${alias}.orig" cat "${asntemp}" | "${pathjq}" -r ".data.ipv${_bgp_type}_prefixes[].prefix" >> "${pfborig}${alias}.orig" break else + echo ". Failed to download ASN" + if [ ${i} == 5 ]; then + echo " [ AS${asn}: ${host} ] FAILED to download ASN [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" + fi + fi + else sleep_val="$((i * 2))" sleep "${sleep_val}" - fi fi done - if [ ! -z "${unavailable}" ]; then - echo ". Failed to download ASN" - touch "${pfborig}${alias}.fail" - fi - if [ "${multiple}" -gt 0 ]; then sleep 1 fi @@ -798,23 +801,27 @@ # Restore previous orig file if [ "${found}" == false ]; then if [ -e "${pfborig}${alias}.bk" ]; then + printf "restore bk file\n" mv "${pfborig}${alias}.bk" "${pfborig}${alias}.orig" else + printf "making an empty orig file\n" echo > "${pfborig}${alias}.orig" + echo " [ AS${asn}: ${host} ] Made Empty Original file ${pfborig}${alias}.orig [ ${now} ]" >> "${errorlog}" + touch "${pfborig}${alias}.fail" fi else if [ -e "${pfborig}${alias}.bk" ]; then + printf "removing bk file\n" rm -f "${pfborig}${alias}.bk" fi fi } - # Function to convert IP to ASN iptoasn() { host="${alias}" - ua="pfSense/pfBlockerNG cURL download agent-" + ua="pfSense/pfBNG cURL download agent-" guid="$(/usr/sbin/gnid)" ua_final="${ua}${guid}"
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
System -> Patches
add a custom patch that looks like this(just what I called it)
Hi I just reinstalled 3.2.0_6 and created a patch as you described. Run Force Reload Task - All and got the follwoing errors in the log:
[ Manually_blocked_Domains_custom_v4 ] Reload Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 50360... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 204428... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 44446... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 398324... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 207812... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 202425... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 23528... completed parse error: Invalid numeric literal at line 1, column 10 Downloading ASN: 29470... completed parse error: Invalid numeric literal at line 1, column 10 . completed ..Netgate 8200max
-
For me it is also not working but I haven't reinstalled pfBlocker. -
@jrey said in pfBlockerNG ASN downloads only contain a header:
However, if you install/reinstall 3.0.2_6 and then build a patch as shown below, you will get ASN files that update, dashboard notifications when it doesn't and extra logging to help you determine if/when things go wrong. I've been running this for days without further issue and turned it into a patch file on the 17th
Have to correct myself, it is working great. Looks like
you really haveI had to reinstall pfBlocker. Thanks man!
-
This post is deleted! -
@jrey I had the "Path Strip Count" wrong (2).
Reinstalled pfblocker, applied the patch and all working.Thx!
-
@Bob-Dig said in pfBlockerNG ASN downloads only contain a header:
really have to reinstall pfBlocker.
you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first
just go from fresh 3.0.2_6 directly to the patch.
-
so you had to change it from 2 to 0 as shown in the screen shot I provided :-)
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first
just go from fresh 3.0.2_6 directly to the patch.
Now second instance is patched, thank you. Problem on first attempt was the original patch that couldn't fix it.
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
@Bob-Dig
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.Just a reflection - maybe the curl user agent should be created randomly ...?
Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
pfsense 2.8.0 CE
Packages: Apcupsd, Cron, Iftop, Iperf, LCDproc, Nmap, pfBlockerNG, RRD_Summary, Shellcmd, Snort, Speedtest, System_Patches. -
This post is deleted! -
Not sure that specifically is the issue..
the failure is very specific to an agent string the starts with pfSense/pfBlockerNGas mentioned you can change the order of the failing "agent" string and it will work.
However when failing if you capture the failed response page ie the "blocking" is typically from cloudflare, where BGPView is hosted. They are the ones that have to say why that specific (the original) agent string format (and then only those starting with pfSense/pfBlockerNG fail to pass). the responses I have captured are "we want a cookie and java script enabled"(ie you are a robot) and just an outright fail message (on purpose a bad request).Now some have suggested the using the Negate device ID as part of this "agent" string, is a bad thing. I'd suspect that BGPView would not require this, unless they are mining their log files looking "volume" from specific devices. (there doesn't seem to be anything about them doing or requiring this)
Subsequent to the patch provided and just for the purpose of messing around, I have changed the agent-(Netgate device ID) part of the string to agent-(numbers of "date")My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.Nothing bad has happened.
I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.
All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"
-
@jrey said in pfBlockerNG ASN downloads only contain a header:
My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.I changed in the agent string also in something individual but not de device ID and it worked with no problems.
Nothing bad has happened.
So it was here too.
I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.
Thats what I suggested - something randomly generated.
All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"
Thats correct! But as you said - lets see what kind of answer BBcan177 gets from them - and then we can react. In the meantime the solution you and BBcan177 provided is good and functional.
Regards,
fireodo -
@jrey
I have reintalled pfBlockerNG 3.2.0_6 on pfSense 23.05.1, I deleted old ASXXX files, applied the patch and force reloaded but files only contains IP 127.1.7.7.[ AS14618_v4 ] Downloading update [ 08/23/23 00:00:45 ] . Downloading ASN: 14618... completed parse error: Invalid numeric literal at line 1, column 10 . completed .. Empty file, Adding '127.1.7.7' to avoid download failure.I will wait for an official fix. Thanks all for your efforts.
-
so what is in the pfblockerng.log file?
sorry, I responded to this based on the email and noticed that you included the log snippet in the post online --That doesn't look like the correct response and appears to be running the wrong code - when applying the patch did you change the Path Strip Count from the default value of 2 to a 0 (zero)
specifically do you see this ?
you should specifically see this before the processing starts.. completed (Download Valid)
anything in the error.log ?
is it possible the IP addresses are also in another list?
De-duplication on?
CIDR Aggregation on?Not asking you to turn any of the above on or off, just how you have them set.
Thanks
-
@jrey
I uninstalled/reinstalled pfBlockerNG and applied the patch, now seems to work.[ TWAS13414_v4 ] Downloading update . Downloading ASN: 13414. /usr/local/bin/curl -A "pfSense/pfBNG cURL download agent-49de62cd6bb042f3ec1e" -sS1 https://api.bgpview.io/asn/13414/prefixes > /tmp/pfbtemp10_46283 .. completed (Download Valid) . completed .. Aggregation Stats: ------------------ Original Final ------------------ 53 14 ------------------Thank you.
-
bingo. Path Strip Count right. Glad it is working for you. Thanks
-
@jrey It's been a month and still we're working around the issue. I wonder what is taking so long for a pfblocker package update......
-
@manilx said in pfBlockerNG ASN downloads only contain a header:
It's been a month
I don't think I would be too concerned at this point.
The patch provided currently resolves the issue and with 23.09 on the horizon, let's just wait until that is released and see what happens.I do know that when @BBcan177 emailed me a couple of weeks back, although he had heard back from the api provider and provided me a copy of their answer, it honestly wasn't an answer at that point.
So could be entirely possible that he just hasn't made any progress with them; or
that it is addressed in the next release; or
is just busy with family life.
(Note: the only additional change I have made was to randomize the NDI part of the agent string (as mentioned in this thread) with every api call and have actually increased the number of ASN's being queried. All running fine with no issues.)
-
@jrey Yes, we can only wait and see. Thx for your input.
