pfBlockerNG-devel 3.2.0_18

emikaadeo

Hi,
anyone try to activate new Spamhaus URL feed under PRI1 Group?
https://www.spamhaus.org/drop/drop_v4.json

I get this:

[ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK
[PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjason|0]

 [ pfB_PRI1_v4 - Spamhaus_Drop_v4 ] Download FAIL [ 10/3/24 15:15:46 ]
  DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.

  Restoring previously downloaded file contents... completed ..

dotsch

@jrey said in pfBlockerNG-devel 3.2.0_18:

g

thank you.

web gui is time out.
No, had not, not added ipinfo accounted as I expected that there is any error handling...

Got the following logs...
Downloading [ IPinfo databases ] [ 10/03/24 15:26:51 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:26:54 ]
..
Downloading [ IPinfo databases ] [ 10/03/24 15:27:32 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:27:36 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:27:40 ]
..

EDIT:
GUI is avialable, try to recover....
IP Info Lookups was enabled by default - Ups, expected not so...

jrey

@dotsch

So you need to register with ipinfo get the Token and enter it on the GUI
(what page was timing out -- the dashboard?)

also read the second, or my post here:

https://forum.netgate.com/topic/190009/pfblockerng-devel-v3-2-0_15

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

_18 is really the new _15 everything in between was a dream ..

dotsch

Thank you very much!

Got it back and everything is working again.

There was some few problems...

The ASN Reporing was enabled by default. Got a ton of errors in pfblocker.log, extra logs due the permantent download. System was to bussy, as web GUI timed out with "504 Gateway Time-out" and login was not possible.
I expexted that I can add the missing key later.

Next problem was, that ipinfo is blocked within snort:
1:2025331 ET POLICY Possible External IP Lookup Domain Observed in SNI (ipinfo. io)
1:2054168 ET INFO External IP Lookup Domain in DNS Lookup (ipinfo .io)
Both was default the snort "connectivity" rules.

jrey

@emikaadeo

I can't comment on this because that list has been json for a while, not specifically related to or because of _18. The change was announced that the txt was being discontinued some time back. The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work. there are other threads on this already (so you are not alone), and now that the .txt version is returning a 404 (the discontinuation is complete)

preemptively however, when that announcement was made, my solution was not wait and simply download the json file on another system using a scheduled script

then process the json on that system to create the txt file format,
and then just have pfblocker pull the feed as a txt from from there ..

Been working fine for months or since the day the announcement was made.

that script looks like --

echo -e "-- Spamhaus Drop"
uri="https://www.spamhaus.org/drop/drop_v4.json"
lfile="drop_v4.json"
bkfile="drop_v4.orig"
mv "$lfile" "$bkfile"
if test -e "$lfile"
then zflag="-z '$lfile'"
else zflag=
fi
http_code=$(curl -w "%{http_code}\n" --output "$lfile" $zflag "$uri" --silent)
echo -e "$http_code"
if [ $http_code == "200" ]; then
   echo -e "-- Process File"
   cat "$lfile" | jq .cidr | tr -d '"' | tr -d 'null' > drop_v4.txt
   echo -e "-- drop_v4.txt created"
else
   echo -e "-- Restore Backup"
   mv "$bkfile" "$lfile"
fi
echo -e "--Done"
# end of file.

then I just pull the txt version of the file from the system that script runs on
Screen Shot 2024-10-03 at 10.01.49 AM.png

pfblocker can deal with txt files nicely - and if the format changes nothing needs to be changed in pfblocker.

so for me moving the source download away from pfblockerng - put it where I have more control and still provides the same data needed without the hassle or sitting around waiting for the world to change.

I use similar techniques for building my own GeoIP (not using maxmind at all) and my geoip data is far more granular and updated once daily not every 3 or 4 days or whatever maxmind is using for their free level these days. My GeoIP data is all pulled and processed on the other system. Then pfblocker just gets a txt file reference to the locally hosted system. (the added "bonus" here is that I can point multiple internal systems to the internal data, and I'm not downloading the same thing from source multiple times. just once from external)

I'm sure it will get fixed, either by patch or new code at some point.

Meanwhile, there is always a way.

emikaadeo

@jrey said in pfBlockerNG-devel 3.2.0_18:

The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work

I see this now https://www.patreon.com/posts/pfblockerng-v3-2-111306340
Zrzut ekranu 2024-10-3 o 16.57.43.png
Well, just wondering - what's the point then for changing URL and download a .JSON file which pfBlockerNG can't handle ?
It can be confusing for users.

BBcan177

@emikaadeo

Didn't catch that before. Will add a fix

You can manualy edit

/usr/local/pkg/pfblockerng/pfblockerng.inc

https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

After line 266, add this line:

    'application/x-ndjason',

jrey

@emikaadeo

That post you reference was made long after the announcement of the change by Spamhaus -

Because I moved past the whole json issue as described a long ago - I don't think I can answer your question except to say it is -devel code are there are issues - that's why it is devel and not production.

and as I'm typing this @BBcan177 has just provided you a "patch" - so there you go .

emikaadeo

@BBcan177 said in pfBlockerNG-devel 3.2.0_18:

@emikaadeo

Didn't catch that before. Will add a fix

You can manualy edit

/usr/local/pkg/pfblockerng/pfblockerng.inc

https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

After line 266, add this line:
    'application/x-ndjason',

Hi @BBcan177
I edit file manually, then:

Firewall -> pfBlockerNG -> Update ->  Select 'Force' option -> Update

Logs looks ok now:

[ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK. completed ..

jrey

@BBcan177

The default Talos link now provided (which I think is the same as before) is also now wanting to make a stop through a terms and conditions page. Which of course pfb is not "accepting" and therefore you get

[ Talos_BL_v4 ]			 Downloading update .. 200 OK. completed ..
[ pfB_PRI1_v4 Talos_BL_v4 ] No IPs found! Ensure only IP based Feeds are used! ]

Tracked this post down from 7 days ago, it has no responses, but pretty much sums up what is going on.

https://forum.netgate.com/topic/190285/changes-to-snort-org-talos-intel-ip-block-list-affecting-pfblockerng?_=1727956822880

tman222

For some reason in my situation ASN information from IPInfo isn't even attempting to download after enabling the ASN setting and entering a token. Could I be missing a setting somewhere?

https://forum.netgate.com/topic/190395/pfblockerng-devel-not-downloading-asn-information

jrey

@tman222

so this maybe.

what is in your extras.log ? that is where the download is logged --

@jrey said in pfBlockerNG-devel 3.2.0_18:

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

tman222

@jrey said in pfBlockerNG-devel 3.2.0_18:

@tman222

so this maybe.

what is in your extras.log ? that is where the download is logged --

@jrey said in pfBlockerNG-devel 3.2.0_18:

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

Thanks @jrey for the response. I'll continue the conversation regarding this issue in the other thread I posted.

jrey

@BBcan177

after going to _20

isn't jason is a horror movie reference ?

as it appears in the _20 release code as
'application/x-ndjason',

The list fails
Spamhaus_Drop_v4 ] Downloading update .. 200 OK
[PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjson|0]

now s/b json the correct way.

'application/x-ndjson',

if it was working with ndjason (and it was for a while), seems they may have fixed a typo upstream
previously it was:
[ Spamhaus_Drop_v4 ] Downloading update .. 200 OK
[PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjason|0]