[SOLVED] Ntop GEO MAP

[manjotsc]

I have generated API key, also enabled Maps Java API Key.
But there is nothing populating on geo map.

[gfeiner]

I have the same problem. The map only shows your local host location and no flows of the traffic/connections of the selected host. I did some googling and found the issue has already been reported here: Bug9211. It seems the cause is that ntopng can't download geolocation data because the source it uses has discontinued that particular ip database. It appears the fix (using a new ip database) is in ntopng CE 3.8 (pfsense 2.4 uses v3.6). Seems we just need to wait until the ntopng package for pfsense is updated.

[manjotsc]

Step by step tutorial to fix geomap Youtube Video

[dragoangel]

@manjotsc thanks for video, Im waiting for update about a year already

[manjotsc]

@dragoangel no problem,

[dragoangel]

How-to fix:
[UPDATED]:
Get API key from MaxMind by register at https://www.maxmind.com/en/geolite2/signup
After this - change CHANGE_ME from #5 in 2 links to your actual API key.

1. Install System_Patches from Packages.
2. Go to System => Patches
3. Add new patch
4. Give it Description, like: PFSENSE-9211 Fix GeoIP DB
5. In Patch Contents copy-paste text:

--- /usr/local/pkg/ntopng.inc
+++ /usr/local/pkg/ntopng.inc
@@ -241,16 +241,12 @@
 function ntopng_update_geoip() {
 	global $config;
 	$fetchcmd = "/usr/bin/fetch";
-	$geolite_city = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz";
-	$geolite_city_v6 = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz";
-	$geoip_asnum = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz";
-	$geoip_asnum_v6 = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz";
+	$geolite_city = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=CHANGE_ME&suffix=tar.gz";
+	$geoip_asnum = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=CHANGE_ME&suffix=tar.gz";
 	$output_dir = "/usr/local/share/ntopng";
-
+	
 	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city}");
-	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city_v6}");
 	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum}");
-	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum_v6}");
 
 	ntopng_fixup_geoip();
 
@@ -271,16 +267,15 @@
 
 	safe_mkdir($target_dir, 0755);
 
-	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
+	foreach(glob("{$source_dir}/Geo*.tar.gz") as $geofile) {
 		/* Decompress if needed. */
-		if (substr($geofile, -3, 3) == ".gz") {
-			// keep -f here, otherwise the files will not get updated
-			mwexec("/usr/bin/gzip -d -f " . escapeshellarg($geofile));
+		if (substr($geofile, -7, 7) == ".tar.gz") {			
+			mwexec("tar -C {$source_dir} -f {$geofile} --strip 1 -xz '*.mmdb'");
 		}
 	}
 
 	/* Use a separate glob since the filenames could have changed since the last run */
-	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
+	foreach(glob("{$source_dir}/Geo*.mmdb") as $geofile) {
 		$target_file = $target_dir . '/' . basename($geofile);
 		if (!file_exists($target_file)) {
 			symlink($geofile, $target_file);

6. Set Path Strip Count to 0
7. Save it and Click Test
8. If test done successfully - Apply button will appear.
9. Apply patch
10. Go to Status = >Services and Stop Ntop NG
11. Got to Diagnostics => ntopng Settings and Remove checkbox from Enable ntopng
12. Go to bottom of ntopng Settings and Update GeoIP Data
13. Enable ntopng and Save - You will receive error, ignore it.
14. Go to Status = >Services and Start Ntop NG

P.S. Revert patch can be done from same way Test -> Revert at System Patches

[gogglespisano]

I had to set the Path Strip Count to 1.

That will ignore the a/ and b/ path prefixes.

--- a/usr/local/pkg/ntopng.inc
+++ b/usr/local/pkg/ntopng.inc

[gogglespisano]

Line 203: function exec_bg undefined

[dragoangel]

@gogglespisano updated How-to, in your case try reinstall package from scratch and then apply fix

[gogglespisano]

@dragoangel Reinstalling and re-applying worked. Thanks for the Patch!

[recue]

@manjotsc Finally, Thank you

[navjot]

@manjotsc it's been a while I was screwing my head with this thanksss

[gacpac]

Omg This was awesomeee.

Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

There's some automated ones like netflix and that kind of stuff.

[amarcino]

Thanks very much for this fix.

[feerab]

Thanks a lot @dragoangel

[Hans from Berlin]

Thank you very much dragoangel!

[robvanhooren]

@gacpac said in [SOLVED] Ntop GEO MAP:

Omg This was awesomeee.

Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

There's some automated ones like netflix and that kind of stuff.

yes.

plex should actually be recognized by ntopng (as of v3.9) but the pfS pkg is 3.8 at the moment.

for now you have to do custom protocols by hand.

see redmine #9912 for a bit of a howto.

(that will get the protocols 'known'; afterwards, you can set them to an appropriate category in the ntop gui).

[gacpac]

@robvanhooren sorry but redmine #9912 where?

[robvanhooren]

the bugtracker is in the pfSense menu (top right corner of the GUI)

it opens the redmine site

or, you can go directly to request #9912 here

hope that helps?

R.

[gniting]

@dragoangel After applying this patch, I am seeing a flood of msgs in the system log with the following text:
As of two days ago, I am seeing a ton of entries in the system log with the following msg:

Attack from "192.168.7.1" on service 100 with danger 10.
Did not receive identification string from 192.168.7.1 port 3736

192.168.7.1 the IP of the pfsense box itself. If I turn off ntopng, the msgs go away. Any ideas?