pfBlockerNG ASN downloads only contain a header

Bob.Dig

@jrey said in pfBlockerNG ASN downloads only contain a header:

you have to start from a clean 3.0.2_6 base, you can just reinstall that, (for clarity not a complete reinstall of pfblockerNG) and do not install the raw file first

just go from fresh 3.0.2_6 directly to the patch.

Now second instance is patched, thank you. Problem on first attempt was the original patch that couldn't fix it.

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

@Bob-Dig
I had emailed @BBcan177 a patch file that can be applied from system patches, but there has been no response or change thus far.

Just a reflection - maybe the curl user agent should be created randomly ...?

NogBadTheBad

This post is deleted!

jrey

@fireodo

Not sure that specifically is the issue..
the failure is very specific to an agent string the starts with pfSense/pfBlockerNG

as mentioned you can change the order of the failing "agent" string and it will work.
However when failing if you capture the failed response page ie the "blocking" is typically from cloudflare, where BGPView is hosted. They are the ones that have to say why that specific (the original) agent string format (and then only those starting with pfSense/pfBlockerNG fail to pass). the responses I have captured are "we want a cookie and java script enabled"(ie you are a robot) and just an outright fail message (on purpose a bad request).

Now some have suggested the using the Negate device ID as part of this "agent" string, is a bad thing. I'd suspect that BGPView would not require this, unless they are mining their log files looking "volume" from specific devices. (there doesn't seem to be anything about them doing or requiring this)
Subsequent to the patch provided and just for the purpose of messing around, I have changed the agent-(Netgate device ID) part of the string to agent-(numbers of "date")

My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.

Nothing bad has happened.

I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.

All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"

fireodo

@jrey said in pfBlockerNG ASN downloads only contain a header:

My agent string therefore now looks like this
"pfSense/pfBNG cURL download agent-210716112023"
and is of course is different with at least every cycle.

I changed in the agent string also in something individual but not de device ID and it worked with no problems.

Nothing bad has happened.

So it was here too.

I know BBcan177 is waiting for a response from them, however, without a specific requirement by them for identifying the devices. I would likely "vote" for the don't give them the device ID. But then at the same time I likely wouldn't use just the numbers of the date - and add something else random to it.

Thats what I suggested - something randomly generated.

All that said, you would think / expect that an API response page would not go through the "normal" are you a robot / cookie / have java?" poke and probe. At least let me respond to that with "of course I'm a robot - I'm hitting an API"

Thats correct! But as you said - lets see what kind of answer BBcan177 gets from them - and then we can react. In the meantime the solution you and BBcan177 provided is good and functional.

Regards,
fireodo

Wolf666

@jrey
I have reintalled pfBlockerNG 3.2.0_6 on pfSense 23.05.1, I deleted old ASXXX files, applied the patch and force reloaded but files only contains IP 127.1.7.7.

[ AS14618_v4 ]			 Downloading update [ 08/23/23 00:00:45 ] .
  Downloading ASN: 14618... completed
parse error: Invalid numeric literal at line 1, column 10
. completed ..
  Empty file, Adding '127.1.7.7' to avoid download failure.

I will wait for an official fix. Thanks all for your efforts.

jrey

@Wolf666

~~so what is in the pfblockerng.log file?~~
sorry, I responded to this based on the email and noticed that you included the log snippet in the post online --

That doesn't look like the correct response and appears to be running the wrong code - when applying the patch did you change the Path Strip Count from the default value of 2 to a 0 (zero)
Screen Shot 2023-08-23 at 7.14.33 AM.png

~~specifically do you see this ?~~
you should specifically see this before the processing starts

.. completed (Download Valid)

anything in the error.log ?

is it possible the IP addresses are also in another list?
De-duplication on?
CIDR Aggregation on?

Not asking you to turn any of the above on or off, just how you have them set.

Thanks

Wolf666

@jrey
I uninstalled/reinstalled pfBlockerNG and applied the patch, now seems to work.

[ TWAS13414_v4 ]		 Downloading update .
  Downloading ASN: 13414.
 /usr/local/bin/curl -A &quot;pfSense/pfBNG cURL download agent-49de62cd6bb042f3ec1e&quot; -sS1 https://api.bgpview.io/asn/13414/prefixes &gt; /tmp/pfbtemp10_46283 
.. completed (Download Valid)
. completed ..

  Aggregation Stats:
  ------------------
  Original Final      
  ------------------
  53       14         
  ------------------

Thank you.

jrey

@Wolf666

bingo. Path Strip Count right. Glad it is working for you. Thanks

manilx

@jrey It's been a month and still we're working around the issue. I wonder what is taking so long for a pfblocker package update......

jrey

@manilx said in pfBlockerNG ASN downloads only contain a header:

It's been a month

I don't think I would be too concerned at this point.
The patch provided currently resolves the issue and with 23.09 on the horizon, let's just wait until that is released and see what happens.

I do know that when @BBcan177 emailed me a couple of weeks back, although he had heard back from the api provider and provided me a copy of their answer, it honestly wasn't an answer at that point.

So could be entirely possible that he just hasn't made any progress with them; or

that it is addressed in the next release; or

is just busy with family life.

(Note: the only additional change I have made was to randomize the NDI part of the agent string (as mentioned in this thread) with every api call and have actually increased the number of ASN's being queried. All running fine with no issues.)

manilx

@jrey Yes, we can only wait and see. Thx for your input.

musivian

Good day, how do you guys make API with multiple ASN and result in plain text (all prefixes) using PHP. Thank you.

Bob.Dig

@jrey said in pfBlockerNG ASN downloads only contain a header:

I don't think I would be too concerned at this point.

I noticed with another problem with pfBlockerNG that the non-devil version was fixed and the devil was not. Both have the same version number... So I have to ask, are we sure that the ASN-problem isn't fixed in the non-devil version too?

jrey

@Bob-Dig

I can't speak to the -dev version - however,

I just updated to 23.09 beta and marked both of my local patches for (ASN) as "auto apply" before running the upgrade to 23.09.

After the update, the pfBlockerNG package is still showing current at 3.2.0_6,(ie there is no update tied specifically to 23.09) my patches are still applied and it still works. System has already been through 1 set of ASN updates with no issues at this time.

Bob.Dig

@jrey said in pfBlockerNG ASN downloads only contain a header:

I can't speak to the -dev version - however,

No need to, that was what I was running so it is not fixed for both. Good to know.

I have read your news on the beta.