[SOLVED] Ntop GEO MAP



  • I have generated API key, also enabled Maps Java API Key.
    But there is nothing populating on geo map.
    0_1551673092734_CaptureNtop.PNG
    0_1551846403381_CaptureNtop Log.PNG



  • I have the same problem. The map only shows your local host location and no flows of the traffic/connections of the selected host. I did some googling and found the issue has already been reported here: Bug9211. It seems the cause is that ntopng can't download geolocation data because the source it uses has discontinued that particular ip database. It appears the fix (using a new ip database) is in ntopng CE 3.8 (pfsense 2.4 uses v3.6). Seems we just need to wait until the ntopng package for pfsense is updated.



  • Step by step tutorial to fix geomap https://youtu.be/soxD194Dpsw



  • @manjotsc thanks for video, Im waiting for update about a year already ☹



  • @dragoangel no problem,



  • How-to fix:
    [UPDATED]:
    Get API key from MaxMind by register at https://www.maxmind.com/en/geolite2/signup
    After this - change CHANGE_ME from #5 in 2 links to your actual API key.

    1. Install System_Patches from Packages.
    2. Go to System => Patches
    3. Add new patch
    4. Give it Description, like: PFSENSE-9211 Fix GeoIP DB
    5. In Patch Contents copy-paste text:
    --- /usr/local/pkg/ntopng.inc
    +++ /usr/local/pkg/ntopng.inc
    @@ -241,16 +241,12 @@
     function ntopng_update_geoip() {
     	global $config;
     	$fetchcmd = "/usr/bin/fetch";
    -	$geolite_city = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz";
    -	$geolite_city_v6 = "https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz";
    -	$geoip_asnum = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz";
    -	$geoip_asnum_v6 = "https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz";
    +	$geolite_city = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=CHANGE_ME&suffix=tar.gz";
    +	$geoip_asnum = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=CHANGE_ME&suffix=tar.gz";
     	$output_dir = "/usr/local/share/ntopng";
    -
    +	
     	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city}");
    -	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geolite_city_v6}");
     	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum}");
    -	mwexec("{$fetchcmd} -o {$output_dir} -T 5 {$geoip_asnum_v6}");
     
     	ntopng_fixup_geoip();
     
    @@ -271,16 +267,15 @@
     
     	safe_mkdir($target_dir, 0755);
     
    -	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
    +	foreach(glob("{$source_dir}/Geo*.tar.gz") as $geofile) {
     		/* Decompress if needed. */
    -		if (substr($geofile, -3, 3) == ".gz") {
    -			// keep -f here, otherwise the files will not get updated
    -			mwexec("/usr/bin/gzip -d -f " . escapeshellarg($geofile));
    +		if (substr($geofile, -7, 7) == ".tar.gz") {			
    +			mwexec("tar -C {$source_dir} -f {$geofile} --strip 1 -xz '*.mmdb'");
     		}
     	}
     
     	/* Use a separate glob since the filenames could have changed since the last run */
    -	foreach(glob("{$source_dir}/Geo*.dat*") as $geofile) {
    +	foreach(glob("{$source_dir}/Geo*.mmdb") as $geofile) {
     		$target_file = $target_dir . '/' . basename($geofile);
     		if (!file_exists($target_file)) {
     			symlink($geofile, $target_file);
    
    
    1. Set Path Strip Count to 0
    2. Save it and Click Test
    3. If test done successfully - Apply button will appear.
    4. Apply patch
    5. Go to Status = >Services and Stop Ntop NG
    6. Got to Diagnostics => ntopng Settings and Remove checkbox from Enable ntopng
    7. Go to bottom of ntopng Settings and Update GeoIP Data
    8. Enable ntopng and Save - You will receive error, ignore it.
    9. Go to Status = >Services and Start Ntop NG

    P.S. Revert patch can be done from same way Test -> Revert at System Patches



  • I had to set the Path Strip Count to 1.

    That will ignore the a/ and b/ path prefixes.

    --- a/usr/local/pkg/ntopng.inc
    +++ b/usr/local/pkg/ntopng.inc



  • Line 203: function exec_bg undefined



  • @gogglespisano updated How-to, in your case try reinstall package from scratch and then apply fix



  • @dragoangel Reinstalling and re-applying worked. Thanks for the Patch!



  • @manjotsc Finally, Thank you



  • @manjotsc it's been a while I was screwing my head with this thanksss



  • Omg This was awesomeee.

    Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

    There's some automated ones like netflix and that kind of stuff.



  • Thanks very much for this fix.



  • Thanks a lot @dragoangel



  • Thank you very much dragoangel!



  • @gacpac said in [SOLVED] Ntop GEO MAP:

    Omg This was awesomeee.

    Also, my plex server is getting flows categorized as unknown application. Is there a way to create my category for Plex?

    There's some automated ones like netflix and that kind of stuff.

    yes.

    plex should actually be recognized by ntopng (as of v3.9) but the pfS pkg is 3.8 at the moment.

    for now you have to do custom protocols by hand.

    see redmine #9912 for a bit of a howto.

    (that will get the protocols 'known'; afterwards, you can set them to an appropriate category in the ntop gui).



  • @robvanhooren sorry but redmine #9912 where?



  • the bugtracker is in the pfSense menu (top right corner of the GUI)

    it opens the redmine site

    or, you can go directly to request #9912 here

    hope that helps?

    R.



  • @dragoangel After applying this patch, I am seeing a flood of msgs in the system log with the following text:
    As of two days ago, I am seeing a ton of entries in the system log with the following msg:

    Attack from "192.168.7.1" on service 100 with danger 10.
    Did not receive identification string from 192.168.7.1 port 3736
    

    192.168.7.1 the IP of the pfsense box itself. If I turn off ntopng, the msgs go away. Any ideas?



  • @ibbetsion this logs not related to patch. They logged without it too if you enable ntopng and do not configure it. Disable alerts in ntopng itself, and please look to all setting).



  • This now broken again due MaxMind require LicenseKeys usage now https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/





  • @dragoangel As temporary solution I uploaded latest files on web server, and It seems to be working fine again.

    ntopng.inc.txt



  • @manjotsc I understand that I can simply put files even on pfsense with geoip. Question about that this plugins must already be officially fixed by netgate to support new API with authorization



  • @dragoangel I think they are waiting for PfSense 2.5 release, because I tried installing Ntopng 3.8 manually, but ntopng 3.8 needs some package dependencies that only supported in FreeBSD12.



  • @dragoangel Problem Solved Working Now

    Youtube Video

    alt text



  • @manjotsc devel version of pfsense 2.5 has same ntopng. Thanks for video.



  • @dragoangel hmmm, sorry maybe I am confused.



  • @manjotsc yep, I doesn't know, but looks like pkg maintainer simply lost.
    Direct link by simple parameters works so I updated https://forum.netgate.com/topic/141150/solved-ntop-geo-map/6 as well



  • @dragoangel Atleast this time it got solved quickly as compared to before, which was broken for months.



  • I'm not sure if anyone else is running into issues with the above patch and the new access key requirement from maxmind, but mine still wouldn't update the GeoIP database..

    It seems like fetch has an issue with the new URL, so changed the /usr/local/pkg/ntopng.inc file to use curl.. These changes are made after applying the above mentioned patch:

    function ntopng_update_geoip() {
    global $config;
    $curlcmd = "/usr/local/bin/curl";

        $geolite_city = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOURKEYHERE&suffix=tar.gz";
        $geoip_asnum = "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=YOURKEYHERE&suffix=tar.gz";
    
        $output_dir_city = "/usr/local/share/ntopng/GeoCity.tar.gz";
        $output_dir_asn = "/usr/local/share/ntopng/GeoASN.tar.gz";
    
        mwexec("{$curlcmd} --output {$output_dir_city} \"{$geolite_city}\"");
        mwexec("{$curlcmd} --output {$output_dir_asn} \"{$geoip_asnum}\"");
    
        ntopng_fixup_geoip();
    
        /* Do not (re)start services on package (re)install, only on manual GeoIP updates via the GUI */
        if ($_POST['Submit'] == "Update GeoIP Data") {
                init_config_arr(array('installedpackages', 'ntopng', 'config', 0));
                $ntopng_config = $config['installedpackages']['ntopng']['config'][0];
                ntopng_services_stop();
                if ($ntopng_config['enable'] == "on") {
                        start_service("ntopng");
                }
        }
    

    }


Log in to reply