[SOLVED] Ntop GEO MAP

[dragoangel]

How-to fix:
[UPDATED]:
Get API key from MaxMind by register at https://www.maxmind.com/en/geolite2/signup
After this - change CHANGE_ME from #5 in 2 links to your actual API key.

1. Install System_Patches from Packages.
2. Go to System => Patches
3. Add new patch
4. Give it Description, like: PFSENSE-9211 Fix GeoIP DB
5. In Patch Contents copy-paste text:

--- /usr/local/pkg/ntopng.inc
+++ /usr/local/pkg/ntopng.inc
@@ -241,16 +241,12 @@
 function ntopng_update_geoip() {
 	global $config;
 	$fetchcmd = "/usr/bin/fetch";
-	$geolite_city = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz";
-	$geolite_city_v6 = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz";
-	$geoip_asnum = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz";
-	$geoip_asnum_v6 = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz";
+	$geolite_city = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=CHANGE_ME&suffix=tar.gz";
+	$geoip_asnum = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=CHANGE_ME&suffix=tar.gz";
 	$output_dir = "/usr/local/share/ntopng";
-
+	
 	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city}");
-	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city_v6}");
 	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum}");
-	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum_v6}");
 
 	ntopng_fixup_geoip();
 
@@ -271,16 +267,15 @@
 
 	safe_mkdir($target_dir, 0755);
 
-	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
+	foreach(glob("{$source_dir}/Geo*.tar.gz") as $geofile) {
 		/* Decompress if needed. */
-		if (substr($geofile, -3, 3) == ".gz") {
-			// keep -f here, otherwise the files will not get updated
-			mwexec("/usr/bin/gzip -d -f " . escapeshellarg($geofile));
+		if (substr($geofile, -7, 7) == ".tar.gz") {			
+			mwexec("tar -C {$source_dir} -f {$geofile} --strip 1 -xz '*.mmdb'");
 		}
 	}
 
 	/* Use a separate glob since the filenames could have changed since the last run */
-	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
+	foreach(glob("{$source_dir}/Geo*.mmdb") as $geofile) {
 		$target_file = $target_dir . '/' . basename($geofile);
 		if (!file_exists($target_file)) {
 			symlink($geofile, $target_file);

6. Set Path Strip Count to 0
7. Save it and Click Test
8. If test done successfully - Apply button will appear.
9. Apply patch
10. Go to Status = >Services and Stop Ntop NG
11. Got to Diagnostics => ntopng Settings and Remove checkbox from Enable ntopng
12. Go to bottom of ntopng Settings and Update GeoIP Data
13. Enable ntopng and Save - You will receive error, ignore it.
14. Go to Status = >Services and Start Ntop NG

P.S. Revert patch can be done from same way Test -> Revert at System Patches

[gogglespisano]

I had to set the Path Strip Count to 1.

That will ignore the a/ and b/ path prefixes.

--- a/usr/local/pkg/ntopng.inc
+++ b/usr/local/pkg/ntopng.inc

[gogglespisano]

Line 203: function exec_bg undefined

[dragoangel]

@gogglespisano updated How-to, in your case try reinstall package from scratch and then apply fix

[gogglespisano]

@dragoangel Reinstalling and re-applying worked. Thanks for the Patch!

[recue]

@manjotsc Finally, Thank you

[navjot]

@manjotsc it's been a while I was screwing my head with this thanksss

[gacpac]

Omg This was awesomeee.

Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

There's some automated ones like netflix and that kind of stuff.

[amarcino]

Thanks very much for this fix.

[feerab]

Thanks a lot @dragoangel

[Hans from Berlin]

Thank you very much dragoangel!

[robvanhooren]

@gacpac said in [SOLVED] Ntop GEO MAP:

Omg This was awesomeee.

Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

There's some automated ones like netflix and that kind of stuff.

yes.

plex should actually be recognized by ntopng (as of v3.9) but the pfS pkg is 3.8 at the moment.

for now you have to do custom protocols by hand.

see redmine #9912 for a bit of a howto.

(that will get the protocols 'known'; afterwards, you can set them to an appropriate category in the ntop gui).

[gacpac]

@robvanhooren sorry but redmine #9912 where?

[robvanhooren]

the bugtracker is in the pfSense menu (top right corner of the GUI)

it opens the redmine site

or, you can go directly to request #9912 here

hope that helps?

R.

[gniting]

@dragoangel After applying this patch, I am seeing a flood of msgs in the system log with the following text:
As of two days ago, I am seeing a ton of entries in the system log with the following msg:

Attack from "192.168.7.1" on service 100 with danger 10.
Did not receive identification string from 192.168.7.1 port 3736

192.168.7.1 the IP of the pfsense box itself. If I turn off ntopng, the msgs go away. Any ideas?

[dragoangel]

@ibbetsion this logs not related to patch. They logged without it too if you enable ntopng and do not configure it. Disable alerts in ntopng itself, and please look to all setting).

[dragoangel]

This now broken again due MaxMind require LicenseKeys usage now https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

[manjotsc]

@dragoangel

[manjotsc]

@dragoangel As temporary solution I uploaded latest files on web server, and It seems to be working fine again.

ntopng.inc.txt

[dragoangel]

@manjotsc I understand that I can simply put files even on pfsense with geoip. Question about that this plugins must already be officially fixed by netgate to support new API with authorization