pfBlockerNG-devel 3.2.0_18

jrey

Heads up:

on 2.7.2 where I have been testing in advance of release _18

if you use any ASN with international characters in the string and here is one example AS29571 (or use any other with international characters in the name)

AS29571 [ Orange Côte d'Ivoire ]

with _18 it will still fail to save the ASN to the config, you will get this error

Screen Shot 2024-10-03 at 6.18.30 AM.png

in conversation with @BBcan177 last week - we got here:
"the devs found the issue. To continue with the new code they either have to add a System Patch or wait for the next version of pfSense."

you will find a vague reference to the issue (24.08) here -

@marcosm said in New 24.08 snapshot!:

there's also been significant changes to the general configuration backend

and what caused this:
https://forum.netgate.com/topic/190240/pfblockerng_devel-commit-reverse

so not sure how this will be addressed in 2.7.2. it is possible to change the pfblocker code to address this, but I also realize there is benefit to having the code the same across the platforms.

Right now however, if you use ASN with international characters it is likely going to be problem, you should probably hold off. There are other somewhat minor issues as well but this may be a show stopper for some.

it is -devel code so proceed with caution.

BBcan177

Thanks @jrey for finding that issue.

Until this is fixed, all the user needs to do is when adding an ASN using the ASN lookup process, is to remove any special characters before saving the changes.

jrey

@BBcan177

you can, but you can't - unless you have changed the code since last week --
because the next time it refreshes the international characters are returned - as the field is refreshed it will return whatever is in the updated asn data -- "thinking it has changed" since the last time the asn data was pulled.

Needs more fix -- IMHO. - the method I provided would circumvent that (I think) and should remedy the issue regardless of what happens to the underlying "config" system.

J

BBcan177

@jrey ok you can also add those particular ASNs to the Customlist at the bottom of your IP Alias and select the AS/Domain checkbox.

I couldn't add the fix for that yet as there was too many moving parts with all the different branches of the code.

jrey

@BBcan177 said in pfBlockerNG-devel 3.2.0_18:

there was too many moving parts with all the different branches of the code.

i get it --

again just providing the information as heads up for those that might get caught by this.

dotsch

got a lot of errors in pfblocker.log : "ASN Token not defined. Terminating Download." and web gui is not accessable anymore.

jrey

@dotsch

did you setup the ipinfo.io key ?

? Screen Shot 2024-10-03 at 9.18.42 AM.png

when you say the web gui is not acccessable anymore - so the dashboard?

can you go directly to any other page ?
https://(your ip/name here)/pfblockerng/pfblockerng_ip.php

can you ssh in and see if the asn data has been downloaded.

cat /var/log/pfblockerng/extras.log

or if you can get to other webpages --
https://(your ip/name here)/pfblockerng/pfblockerng_log.php
then pick logs, extras.log

should look like (showing my last three days only) :

Download Process Starting [ 10/1/24 07:45:00 ]
 /usr/local/share/GeoIP/asn.mmdb		200 OK
 /usr/local/share/GeoIP/asn.csv.gz		200 OK
ASN Lookup Table has been updated [ 10/01/24 07:45:04 ]
Download Process Ended [ 10/1/24 07:45:07 ]


Download Process Starting [ 10/2/24 07:45:00 ]
 /usr/local/share/GeoIP/asn.mmdb		200 OK
 /usr/local/share/GeoIP/asn.csv.gz		200 OK
ASN Lookup Table has been updated [ 10/02/24 07:45:05 ]
Download Process Ended [ 10/2/24 07:45:07 ]


Download Process Starting [ 10/3/24 07:45:00 ]
 /usr/local/share/GeoIP/asn.mmdb		200 OK
 /usr/local/share/GeoIP/asn.csv.gz		200 OK
ASN Lookup Table has been updated [ 10/03/24 07:45:04 ]
Download Process Ended [ 10/3/24 07:45:06 ]

emikaadeo

Hi,
anyone try to activate new Spamhaus URL feed under PRI1 Group?
https://www.spamhaus.org/drop/drop_v4.json

I get this:

[ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK
[PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjason|0]

 [ pfB_PRI1_v4 - Spamhaus_Drop_v4 ] Download FAIL [ 10/3/24 15:15:46 ]
  DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.

  Restoring previously downloaded file contents... completed ..

dotsch

@jrey said in pfBlockerNG-devel 3.2.0_18:

g

thank you.

web gui is time out.
No, had not, not added ipinfo accounted as I expected that there is any error handling...

Got the following logs...
Downloading [ IPinfo databases ] [ 10/03/24 15:26:51 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:26:54 ]
..
Downloading [ IPinfo databases ] [ 10/03/24 15:27:32 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:27:36 ]
Downloading [ IPinfo databases ] [ 10/03/24 15:27:40 ]
..

EDIT:
GUI is avialable, try to recover....
IP Info Lookups was enabled by default - Ups, expected not so...

jrey

@dotsch

So you need to register with ipinfo get the Token and enter it on the GUI
(what page was timing out -- the dashboard?)

also read the second, or my post here:

https://forum.netgate.com/topic/190009/pfblockerng-devel-v3-2-0_15

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

_18 is really the new _15 everything in between was a dream ..

dotsch

Thank you very much!

Got it back and everything is working again.

There was some few problems...

The ASN Reporing was enabled by default. Got a ton of errors in pfblocker.log, extra logs due the permantent download. System was to bussy, as web GUI timed out with "504 Gateway Time-out" and login was not possible.
I expexted that I can add the missing key later.

Next problem was, that ipinfo is blocked within snort:
1:2025331 ET POLICY Possible External IP Lookup Domain Observed in SNI (ipinfo. io)
1:2054168 ET INFO External IP Lookup Domain in DNS Lookup (ipinfo .io)
Both was default the snort "connectivity" rules.

jrey

@emikaadeo

I can't comment on this because that list has been json for a while, not specifically related to or because of _18. The change was announced that the txt was being discontinued some time back. The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work. there are other threads on this already (so you are not alone), and now that the .txt version is returning a 404 (the discontinuation is complete)

preemptively however, when that announcement was made, my solution was not wait and simply download the json file on another system using a scheduled script

then process the json on that system to create the txt file format,
and then just have pfblocker pull the feed as a txt from from there ..

Been working fine for months or since the day the announcement was made.

that script looks like --

echo -e "-- Spamhaus Drop"
uri="https://www.spamhaus.org/drop/drop_v4.json"
lfile="drop_v4.json"
bkfile="drop_v4.orig"
mv "$lfile" "$bkfile"
if test -e "$lfile"
then zflag="-z '$lfile'"
else zflag=
fi
http_code=$(curl -w "%{http_code}\n" --output "$lfile" $zflag "$uri" --silent)
echo -e "$http_code"
if [ $http_code == "200" ]; then
   echo -e "-- Process File"
   cat "$lfile" | jq .cidr | tr -d '"' | tr -d 'null' > drop_v4.txt
   echo -e "-- drop_v4.txt created"
else
   echo -e "-- Restore Backup"
   mv "$bkfile" "$lfile"
fi
echo -e "--Done"
# end of file.

then I just pull the txt version of the file from the system that script runs on
Screen Shot 2024-10-03 at 10.01.49 AM.png

pfblocker can deal with txt files nicely - and if the format changes nothing needs to be changed in pfblocker.

so for me moving the source download away from pfblockerng - put it where I have more control and still provides the same data needed without the hassle or sitting around waiting for the world to change.

I use similar techniques for building my own GeoIP (not using maxmind at all) and my geoip data is far more granular and updated once daily not every 3 or 4 days or whatever maxmind is using for their free level these days. My GeoIP data is all pulled and processed on the other system. Then pfblocker just gets a txt file reference to the locally hosted system. (the added "bonus" here is that I can point multiple internal systems to the internal data, and I'm not downloading the same thing from source multiple times. just once from external)

I'm sure it will get fixed, either by patch or new code at some point.

Meanwhile, there is always a way.

emikaadeo

@jrey said in pfBlockerNG-devel 3.2.0_18:

The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work

I see this now https://www.patreon.com/posts/pfblockerng-v3-2-111306340
Zrzut ekranu 2024-10-3 o 16.57.43.png
Well, just wondering - what's the point then for changing URL and download a .JSON file which pfBlockerNG can't handle ?
It can be confusing for users.

BBcan177

@emikaadeo

Didn't catch that before. Will add a fix

You can manualy edit

/usr/local/pkg/pfblockerng/pfblockerng.inc

https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

After line 266, add this line:

    'application/x-ndjason',

jrey

@emikaadeo

That post you reference was made long after the announcement of the change by Spamhaus -

Because I moved past the whole json issue as described a long ago - I don't think I can answer your question except to say it is -devel code are there are issues - that's why it is devel and not production.

and as I'm typing this @BBcan177 has just provided you a "patch" - so there you go .

emikaadeo

@BBcan177 said in pfBlockerNG-devel 3.2.0_18:

@emikaadeo

Didn't catch that before. Will add a fix

You can manualy edit

/usr/local/pkg/pfblockerng/pfblockerng.inc

https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

After line 266, add this line:
    'application/x-ndjason',

Hi @BBcan177
I edit file manually, then:

Firewall -> pfBlockerNG -> Update ->  Select 'Force' option -> Update

Logs looks ok now:

[ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK. completed ..

jrey

@BBcan177

The default Talos link now provided (which I think is the same as before) is also now wanting to make a stop through a terms and conditions page. Which of course pfb is not "accepting" and therefore you get

[ Talos_BL_v4 ]			 Downloading update .. 200 OK. completed ..
[ pfB_PRI1_v4 Talos_BL_v4 ] No IPs found! Ensure only IP based Feeds are used! ]

Tracked this post down from 7 days ago, it has no responses, but pretty much sums up what is going on.

https://forum.netgate.com/topic/190285/changes-to-snort-org-talos-intel-ip-block-list-affecting-pfblockerng?_=1727956822880

tman222

For some reason in my situation ASN information from IPInfo isn't even attempting to download after enabling the ASN setting and entering a token. Could I be missing a setting somewhere?

https://forum.netgate.com/topic/190395/pfblockerng-devel-not-downloading-asn-information

jrey

@tman222

so this maybe.

what is in your extras.log ? that is where the download is logged --

@jrey said in pfBlockerNG-devel 3.2.0_18:

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

tman222

@jrey said in pfBlockerNG-devel 3.2.0_18:

@tman222

so this maybe.

what is in your extras.log ? that is where the download is logged --

@jrey said in pfBlockerNG-devel 3.2.0_18:

regarding the change regarding the setting for ASN Reporting which must be enabled now.
The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

Thanks @jrey for the response. I'll continue the conversation regarding this issue in the other thread I posted.