pfBlockerNG IP Reputation

[Gertjan]

@NollipfSense said in pfBlockerNG IP Reputation:

Then, when reinstalling pfBlockerNG, some how IP reputation.php is not included.

Yep, I confirm.
It's not included in the package file - neither referenced in the manifest.
It's a package build issue.
Some one should inform @BBcan177.

Maybe the reputation functionality was removed from pfBlocker ?

I can rename any file in that directory, re install pfBlockerNG, and the file will reapaer.
Except the pfblockerng_reputation.php file.

@serbus said in pfBlockerNG IP Reputation:

I think that file might be dynamically generated at some point during that process.

It's a static web page.
pfBlockerNG does not generate this page.
An exception exists : the /usr/local/www/pfblockerng/www/dnsbl_active.php page, which is copied from the dnsbl_default.php file by pfBlockerNG itself. This is the way we select the default "Blocked Webpage" - made our own one.

[serbus]

@Gertjan said in pfBlockerNG IP Reputation:

It's a static web page.
pfBlockerNG does not generate this page.

Hello!

I am new to pfsense and still working through much of the code. Maybe you can help.

How and when are the pfblocker reputation and country xml files generated?
What is the purpose of the pkg.php system?
Does pkg ever save/cache the php it dynamically generates from xml for performance or other reasons?

Thanks!

John

[serbus]

@NollipfSense said in pfBlockerNG IP Reputation:

No one else reproduced the above result? Is it that IP reputation had been removed from the package?

Hello!

You could try running :

php /usr/local/www/pfblockerng/pfblockerng.php gc

To recreate the reputation and country php files.

John

[NollipfSense]

@serbus said in pfBlockerNG IP Reputation:

I think that file might be dynamically generated at some point during that process.

I have been thinking along those lines because I noticed also that Maxmind data had not loaded despite providing my key.

[NollipfSense]

@Gertjan Do you also have the MaxMind feed loaded?

[NollipfSense]

@serbus said in pfBlockerNG IP Reputation:

php /usr/local/www/pfblockerng/pfblockerng.php gc

Hey John, I am reporting that worked. However, the command result shows geolocation files not found confirmed that IP reputation list is derived from geolocation feed. Just discovered making the page had produced a crash

[Gertjan]

@NollipfSense :

[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: php /usr/local/www/pfblockerng/pfblockerng.php gc
 Creating pfBlockerNG Continent PHP files
 IPv4 Africa                     [ 05/25/20 07:16:41 ]
 IPv6 Africa                     [ 05/25/20 07:16:43 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 05/25/20 07:16:46 ]
 IPv4 Europe                     [ 05/25/20 07:16:47 ]
 IPv6 Europe                     [ 05/25/20 07:16:55 ]
 IPv4 North America              [ 05/25/20 07:16:58 ]
 IPv6 North America              [ 05/25/20 07:17:04 ]
 IPv4 Oceania                    [ 05/25/20 07:17:05 ]
 IPv6 Oceania                    [ 05/25/20 07:17:06 ]
 IPv4 South America
 IPv6 South America              [ 05/25/20 07:17:07 ]
 IPv4 Proxy and Satellite
 IPv6 Proxy and Satellite        [ 05/25/20 07:17:08 ]
 IPv4 Top Spammers
 IPv6 Top Spammers
 pfBlockerNG Reputation Tab
Country Code Update Ended

and

[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: php /usr/local/www/pfblockerng/pfblockerng.php ugc
Country code update Start [ 05/25/20 07:42:27 ]
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 05/25/20 07:42:55 ]
 Creating pfBlockerNG Continent PHP files
 IPv4 Africa                     [ 05/25/20 07:43:05 ]
 IPv6 Africa                     [ 05/25/20 07:43:06 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 05/25/20 07:43:09 ]
 IPv4 Europe                     [ 05/25/20 07:43:10 ]
 IPv6 Europe                     [ 05/25/20 07:43:19 ]
 IPv4 North America              [ 05/25/20 07:43:21 ]
 IPv6 North America              [ 05/25/20 07:43:27 ]
 IPv4 Oceania                    [ 05/25/20 07:43:29 ]
 IPv6 Oceania
 IPv4 South America
 IPv6 South America              [ 05/25/20 07:43:30 ]
 IPv4 Proxy and Satellite        [ 05/25/20 07:43:31 ]
 IPv6 Proxy and Satellite
 IPv4 Top Spammers
 IPv6 Top Spammers               [ 05/25/20 07:43:32 ]
 pfBlockerNG Reputation Tab
Country Code Update Ended

@serbus said in pfBlockerNG IP Reputation:

Did you run a full update of pfbng? I think that file might be dynamically generated at some point during that process.

Well .... @servus is right.
I was wrong.
This file "reputation" IS actually regenerated out of /usr/local/www/pfblockerng/pfblockerng.php ...

@NollipfSense : Yes, I have an activated MaxiMind account.

@NollipfSense : How many files - what ype of files do you have here /usr/local/share/GeoIP/cc/ ?

I have more the 1500 files - it depends probably on which regions I've selected, I guess.
Some of them have a time stamp like 03/09/2019 - others 07/05/2020 - and the better part was downloaded just today : 25/05/2020 - 07h17.

Also : /usr/local/share/GeoIP/ ? This directory gets filled with files from MaxMind - if you have an account with them.

So : files actually get downloaded , No file system full ? Run a "fsck" just to be sure.

Btw : The Diagnostic > Command propmpt : never use that one. It could hide stuff. Keyboard command belong on a real command line. It's one click away with Putty or any other SSH client. Better get used to it ^^

[NollipfSense]

@Gertjan said in pfBlockerNG IP Reputation:

How many files - what ype of files do you have here /usr/local/share/GeoIP/cc/ ?

Appeared empty!

@Gertjan said in pfBlockerNG IP Reputation:

Also : /usr/local/share/GeoIP/ ?

Also, appeared empty!

[serbus]

Hello!

php /usr/local/www/pfblockerng/pfblockerng.php dc

should re-download the maxmind files. They should appear in the /usr/local/share/GeoIP folder.

the "dc" command will also run the "ugc" functions after the download.

John

[Gertjan]

@NollipfSense : do not use the GUI for this. Use the console/SSH.

There is a little surprise here, see the last two lines :

[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: php /usr/local/www/pfblockerng/pfblockerng.php dc

Download Process Starting [ 05/27/20 09:48:21 ]
 /usr/local/share/GeoIP/GeoLite2-Country.tar.gz         200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip                200 OK
Download Process Ended [ 05/27/20 09:48:26 ]

Country code update Start
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 05/27/20 09:48:55 ]
 Creating pfBlockerNG Continent PHP files
 IPv4 Africa                     [ 05/27/20 09:49:03 ]
 IPv6 Africa                     [ 05/27/20 09:49:04 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 05/27/20 09:49:08 ]
 IPv4 Europe
 IPv6 Europe                     [ 05/27/20 09:49:17 ]
 IPv4 North America              [ 05/27/20 09:49:20 ]
 IPv6 North America              [ 05/27/20 09:49:25 ]
 IPv4 Oceania                    [ 05/27/20 09:49:27 ]
 IPv6 Oceania
 IPv4 South America              [ 05/27/20 09:49:28 ]
 IPv6 South America
 IPv4 Proxy and Satellite        [ 05/27/20 09:49:29 ]
 IPv6 Proxy and Satellite        [ 05/27/20 09:49:30 ]
 IPv4 Top Spammers
 IPv6 Top Spammers
 pfBlockerNG Reputation Tab
Country Code Update Ended

[NollipfSense]

@serbus and @Gertjan I used the CLI and here is the result:

[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: php /usr/local/www/pfblockerng/pfblockerng.php dc

Download Process Starting [ 05/27/20 10:55:35 ]
/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 401 Unauthorized

Failed to Download GeoLite2-Country.mmdb
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 401 Unauthorized

Failed to Download
Download Process Ended [ 05/27/20 10:55:36 ]

[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root:

What I don't understand is I have a registered key; so, not sure what the unauthorized is all about nor what to do to resolve.

[RonpfS]

Goto Maxminds and check your account and Download History.

[NollipfSense]

@RonpfS Last download was on May 5, 2020 at 14.56pm ... so, I guess I'll have to wait for June. I had to reinstall a fresh pfSense 2.5-dev so may explain why I haven't got the feed since its once per month.

[RonpfS]

It changes every 6 days, do you see the md5 download every day ?

[serbus]

Hello!

Is there a limit to the number of times per month you can download the files from maxmind?

I use the same license key in a number of different routers and routinely download "off schedule" when setting things up or troubleshooting.

Maybe you could create a new license key at the maxmind site and try that in your router.

John

[jdeloach]

@NollipfSense said in pfBlockerNG IP Reputation:

@RonpfS Last download was on May 5, 2020 at 14.56pm ... so, I guess I'll have to wait for June. I had to reinstall a fresh pfSense 2.5-dev so may explain why I haven't got the feed since its once per month.

Run this command from the command prompt to force Maxmind to update: php /usr/local/www/pfblockerng/pfblockerng.php dc . This should force the Maxmind.com database to update.

[Gertjan]

@jdeloach said in pfBlockerNG IP Reputation:

Run this command from the command prompt to force Maxmind to update: php /usr/local/www/pfblockerng/pfblockerng.php dc . This should force the Maxmind.com database to update.

He did ( see above ) :

@NollipfSense said in pfBlockerNG IP Reputation:

[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: php /usr/local/www/pfblockerng/pfblockerng.php dc
Download Process Starting [ 05/27/20 10:55:35 ]

He wasn't authorized.

/usr/local/share/GeoIP/GeoLite2-Country.tar.gz 401 Unauthorized
Failed to Download GeoLite2-Country.mmdb
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 401 Unauthorized
Failed to Download
Download Process Ended [ 05/27/20 10:55:36 ]

[NollipfSense]

So, I contacted MaxMind support that confirmed that somehow when I did the force the update, it kept downloading last month's (April) database ... which is not available ... hence, the unauthorized message.

Support suggested "If you alter your download URL and remove the 'date' parameter entirely, that will make it download the most recent database available rather than a specific database version. Alternatively, you can use the 'Get permalinks' link in your Download Files page to get a permanent download URL that you can use."

So, m question: where would I find the download file to change or replace with "permalinks."

[RonpfS]

Maybe it's time to move this topic to pfblockerNG forum.

[NollipfSense]

@RonpfS That's okay with me ... admin.