pfblockerNG ASN bgpview trouble

Patch

@jrey
I tried un-installing nmap v1.4.4_7
un-installing pfBlockerNG then re installing -> no difference
The only other package I have installed is System_Patches v2.2.11_15

@jrey said in pfblockerNG ASN bgpview trouble:

Please show me the screen where you have the feed set up.

pfblockerng ASN Netflix.jpg

@jrey said in pfblockerNG ASN bgpview trouble:

Can you also include the first part of the response from the curl you did manually

{"status":"ok","status_message":"Query was successful","data":{"ipv4_prefixes":[{"prefix":"45.57.8.0\/23","ip":"45.57.8.0","cidr":23,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.8.0\/24","ip":"45.57.8.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.9.0\/24","ip":"45.57.9.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.40.0\/23","ip":"45.57.40.0","cidr":23,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.120.152.0\/22","ip":"45.120.152.0","cidr":22,"rir_name":"APNIC","allocation_status":"unknown"}},{"prefix":"45.57.40.0\/24","ip":"45.57.40.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.41.0\/24","ip":"45.57.41.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.155.40.0\/22","ip":"45.155.40.0","cidr":22,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"45.57.86.0\/23","ip":"45.57.86.0","cidr":23,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.86.0\/24","ip":"45.57.86.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.87.0\/24","ip":"45.57.87.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.90.0\/23","ip":"45.57.90.0","cidr":23,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.90.0\/24","ip":"45.57.90.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"45.57.91.0\/24","ip":"45.57.91.0","cidr":24,"roa_status":"Valid","name":"SS-CDN-4","description":"Netflix Streaming Services Inc.","country_code":"US","parent":{"prefix":"45.57.0.0\/17","ip":"45.57.0.0","cidr":17,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"207.45.72.0\/24","ip":"207.45.72.0","cidr":24,"roa_status":"Valid","name":null,"description":null,"country_code":null,"parent":{"prefix":null,"ip":null,"cidr":null,"rir_name":null,"allocation_status":"unknown"}},{"prefix":"207.45.72.0\/23","ip":"207.45.72.0","cidr":23,"roa_status":"Valid","name":null,"description":null,"country_code":null,"parent":{"prefix":null,"ip":null,"cidr":null,"rir_name":null,"allocation_status":"unknown"}},{"prefix":"207.45.73.0\/24","ip":"207.45.73.0","cidr":24,"roa_status":"Valid","name":"DVD-NETFLIX","description":"Netflix, Inc","country_code":"US","parent":{"prefix":"207.45.72.0\/22","ip":"207.45.72.0","cidr":22,"rir_name":"ARIN","allocation_status":"unknown"}},{"prefix":"207.45.74.0\/23","ip":"207.45.74.0","cidr":23,"roa_status":"Valid","name":null,"description":null,"country_code":null,"parent":{"prefix":null,"ip":null,"cidr":null,"rir_name":null,"allocation_status":"unknown"}}],"ipv6_prefixes":[{"prefix":"2a00:86c0:2008::\/48","ip":"2a00:86c0:2008::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2008::\/47","ip":"2a00:86c0:2008::","cidr":47,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2009::\/48","ip":"2a00:86c0:2009::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2040::\/48","ip":"2a00:86c0:2040::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2040::\/47","ip":"2a00:86c0:2040::","cidr":47,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2041::\/48","ip":"2a00:86c0:2041::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2042::\/48","ip":"2a00:86c0:2042::","cidr":48,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2042::\/47","ip":"2a00:86c0:2042::","cidr":47,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2043::\/48","ip":"2a00:86c0:2043::","cidr":48,"roa_status":"Valid","name":"NET6-2A00-86C-3","description":"NET6 2A00 86C 3","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2052::\/47","ip":"2a00:86c0:2052::","cidr":47,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2086::\/48","ip":"2a00:86c0:2086::","cidr":48,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2086::\/47","ip":"2a00:86c0:2086::","cidr":47,"roa_status":"Valid","name":"US-NETFLIX1-20120130","description":"Netflix Inc","country_code":"GB","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2087::\/48","ip":"2a00:86c0:2087::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2090::\/47","ip":"2a00:86c0:2090::","cidr":47,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2090::\/48","ip":"2a00:86c0:2090::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}},{"prefix":"2a00:86c0:2091::\/48","ip":"2a00:86c0:2091::","cidr":48,"roa_status":"Valid","name":null,"description":null,"country_code":"US","parent":{"prefix":"2a00:86c0::\/32","ip":"2a00:86c0::","cidr":32,"rir_name":"RIPE","allocation_status":"unknown"}}]},"@meta":{"time_zone":"UTC","api_version":1,"execution_time":"28.06 ms"}}

Which Beyond compare suggests is identical to yours for the data shown.

Patch

I also tried

disabling pfblockerNG ASN cache,
deleting the /var/db/pfblockerng/original/ files
Deleting the /var/db/pfblockerng/native/ files
Rerunning Firewall / pfBlockerNG / Update -> Reload, IP

But still only empty files result

pfblockerng ASN Cache disable.jpg

Bob.Dig

@Patch I tried it myself, making this alias and failed. I see the same as you. Older ASN-aliases aren't affected as far as I can tell. So you are not alone. I guess I will try another ASN next.
Edit: Also no luck, old ones work.

jrey

@Bob-Dig

and yet it works for me (still) (Don't read any of this as directed at you. I'm just tagging you because:

You might recall, as I think you where part of a thread, I'm guessing about a year ago, with the same OMG it is creating 127... Empty files

You have to ask "why does it work for me ?"
They patch I created then still works. However words of
WARNING: That version no longer applies directly on 3.0.2_9 (I had to tweak it a bit because of an underlying file change. On my end that tweak took less than five minutes. Unfortunately I can't just provide my updated patch because I have "fixed/added" several other things that are "not important" to either the developer and/or the public.. (or so I'm told)
WARNING: Not sure that the older version of the patch specifically addresses this issue, which I know I've tweaked it a couple of times over the past year. I'm to lazy to look at, code change log to find out. So, the patch you might still find floating around might only address the specific case at the time and not additional cases I've encountered in daily use since then.

I can clearly prove it on both my 2.7.2 test box or 23.03 production box, simple to do - revert the patch watch it fail as being discussed, apply the patch watch it work. Clearly it is NOT the feed at this point as demonstrated by the direct curl command test I suggested earlier in this thread.

After the last go around and being told (by several) that the patch was "silly", "not needed" and not going to be incorporated because what is there, "works fine" I just moved on, hence the I can't release an updated patch I'm currently using. In part, some of those other changes are direct updating to remote syslog in real time, for example.

When this all started going down again last week, my graylog immediately started informing me of the download issue, (filtered on level 2) (critical enough but not notifications.)

Screen Shot 2024-08-18 at 9.18.42 AM.png

part of that most recent outage was the opportunity for me to simulate the case of the system creating an empty file --- to you know -- "test the emergency broadcast system." I had the OMG (filtered on level 1) failure event email within seconds of creating the "empty file manually" --

Screen Shot 2024-08-18 at 9.21.26 AM.png

Sorry, not sure what else I can do to help. I've given up on trying to create tickets, discuss with the developers, suggest improvements etc Now I just "twist" the "silly" into the code so that it suits my application, requirements and use of the device... I been through a few OS upgrades and other than the small adjustment I had to make to one patch, going to 24.03 and pf 3. _9 I have never had a problem

Moving on..

Bob.Dig

@jrey said in pfblockerNG ASN bgpview trouble:

You might recall, as I think you where part of a thread, I'm guessing about a year ago, with the same OMG it is creating 127... Empty files

Sure, I do remember and am following you for that.

What I didn't remembered, that the patch is still needed. And I can confirm, once again, that your patch is working. I applied it a few seconds ago, thanks again @jrey .

jrey

@Bob-Dig

Cool for the record can you just confirm what versions of things you are currently running ?

Edit: because what I couldn't remember is what specific change I may have made in that patch, caused the specific hiccup when I upgraded to 24.03 and 3.0.2_9 came with it. I just recall having to tweak my current version of the patch to make it apply. (I guessing it might have been something else I've changed in the patch since that earlier one)

Bob.Dig

This post is deleted!

Bob.Dig

24.03-RELEASE, pfBlockerNG 3.2.0_10
2.7.2-RELEASE, pfBlockerNG 3.2.0_8

@jrey When I tested with CE, it failed with one AS for me (AS8881). I then retested with another and this one was working (AS1299)...
I have not tested (AS8881) on Plus though. So there might be a problem still.

jrey

@Bob-Dig

Great thanks - so then the version of the patch I originally provided you still applies to both. good to know (I guess)

Means something I changed since that earlier version is what sent me down the path won't apply path when I upgrade to 24.03 --

Oh darn, I just shut my 2.7.2 virtual network down. Let me fire it back up and look at those two ASnumbers you provided ..

Thanks

Bob.Dig

@jrey Retesting this ASN again, now works everywhere. So I think we are good. Maybe it was just a hiccup.

jrey

@Bob-Dig

Interesting, can you share the log snippet where it failed ?

I tried 2.7.2 and for me it failed on the first attempt but picked it up on the auto retry.

[ AS40027_v4 ]			 Downloading update [ 08/18/24 11:01:12 ] .
  Downloading ASN: 40027.
.. completed (Download Valid)
. completed ..

[ AS8881_v4 ]			 Downloading update .
  Downloading ASN: 8881.
.. Failed to download ASN
.
.. completed (Download Valid)
. completed ..

[ AS1299_v4 ]			 Downloading update [ 08/18/24 11:01:13 ] .
  Downloading ASN: 1299.
.. completed (Download Valid)
. completed ..

Bob.Dig

@jrey said in pfblockerNG ASN bgpview trouble:

Interesting, can you share the log snippet where it failed ?

Where is that exactly, sry still not an experienced pfBlocker User.
Also I thought I hadn't this bevor on that firewall but I guess I am wrong and it was there so it could be an old list...

jrey

@Bob-Dig

In the log here -- you will have to scroll through to find the one that failed (don't need the whole file just the part for the one that failed.

Screen Shot 2024-08-18 at 11.32.47 AM.png

Bob.Dig

@jrey

  Restoring previously downloaded file contents... [ 08/9/24 20:45:57 ]
[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS8881 [ VERSATEL, DE ] ] [ 08/17/24 18:21:15 ]
[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS8881 ] [ 08/17/24 18:21:32 ]

jrey

@Bob-Dig

That's different.

Doesn't look related to the actual "download" of the file. but rather something bad left over from when it didn't download -- so it though it needed to restore the previous one 08/9/24 and then tried to process it.

if the latest download worked (the file in question has IPs listed) and "worked" as you stated above "Retesting this ASN again, now works everywhere. So I think we are good. Maybe it was just a hiccup." would likely be a good caption for the event.

if the [PFB_FILTER - 2] error shows up again, that might be something else to look at, I've never logged one of those and it is a completely different block of code.

Cheers

Bob.Dig

@jrey Sry, reading the log again, the first line I posted is not related to this but something before that.

And yes the file is working fine. Maybe I was to fast klicking reload again or something, it is an install on a VPS.

BBcan177

The problem is with BGPview.io which is now owned by Recorded Future. They are rate limiting their service due to some users who hit their services too much. The current code in pfB contains a User Agent Header which is being blocked en masse by them. I have requested that if they rate limit to block specific Agent strings as the suffix of the Agent String is unique to the user.

I have been emailing their support team for several weeks and they are saying it's low priority. I have also asked for their usage policy which they say they will add to their FYI page but it's low on their to do.

So I assume if it's working for some users, it could be they changed the Agent String. But if that were to happen en masse, it's back to the same result.

I am looking at an alternative ASN source as BGPview.io and previously Hurricane Electric don't want to support Open Source projects. Most likely will try to use ipinfo if their free options are accurate enough.

Popolou

Came across the same issue on one of our boxes. For the time being, have disabled the ASN download and copied over the relevant original files to reload into the affected system.