4. Ntopng development

Ntopng development

  • R

    Hi All,

    I reached out to Luca Deri from ntop.org a while back in the hope that there may be some way to license his awesome ntopng software for use with some of our clients. I mentioned that we use pfSense in a few locations and he said he was keen to try and get ntopng to work with pfsense WITH packet filtering.

    In the last batch of emails his questions went well beyond my own capabilities, so I said I would raise it on the forum and see if anyone could assist him.

    For reference his email is deri (@) ntop (.) org

    Here's a trail of the conversation.

    Would ntopng integration with pfSense (e.g. mark packets that you can then discard in pfSense based on L7 protocol) be what you are looking for?

    ntopng compiles on the latest FreeBSD: what version are you using?

    We have started to do some integration with pfSense in a similar way other apps do and it seems not too difficult. We would like to use ntopng in a way that packets are marked by ntopng and you can drop them in pfSense to avoid duplication of roles.

    as of of sense, please read https://github.com/ntop/ntopng/blob/dev/doc/README.pfsense and let me know your comments. I am no too familiar with pfsense and your opinion is valuable. For instance can you please send me a merge request for the readme where you describe, step by step, how to configure ALTQ queues where I can send classified flows?

    in the meantime I have implemented some pfSense support in ntopng (see https://github.com/ntop/ntopng/blob/dev/doc/README.pfsense). I hope is what you need.

    If you can find a sponsor for finishing/polishing this work that would be great

    Regards Luca

    0
  • Rebel Alliance Developer Netgate

    It may compile on FreeBSD in that case but the FreeBSD port needs fixed: https://www.freshports.org/net/ntopng/

    Once the port is fixed, then we can get it back into a package, but the port has to come first.

    0
  • Administrator

    It's been moving along and Luca is in the loop: https://github.com/ntop/ntopng/issues/297#issuecomment-198017871

    0
  • R

    Awesome. I wasn't sure if anyone on here had been in contact with him. I reached out in January, but I'm not a developer.

    I'd gladly pay a license to be able to use pfSense with nTopng+nDPI.

    We have a number of sites where pfSense is the perfect solution, but lack of application filtering (and cumbersome web filtering) has resulted in us needing to put more expensive 'commercial' solutions in place.

    0
  • A

    https://redmine.pfsense.org/issues/6204

    hopefully issue will b fixed soon

    0
  • C

    Any news?

    0
  • A

    it seems they are testing it internally

    https://github.com/ntop/ntopng/issues/297

    
@Andrew17856 Hi, I'm working on the FreeBSD port.

I'm almost done with that, I'm waiting for feedback from a pair of persons who are helping me test it.

I'm going to commit it as soon as I'm sure it works fine.

If you want to test the FreeBSD port you can grab what I have done here:

http://www.madpilot.net/~mad/ntopng_port.txz

Please note that this also needs adding a user in /usr/ports/UIDs and /usr/ports/GIDs to work:

> grep ntop UIDs GIDs 
UIDs:ntopng:*:288:288::0:0:ntopng daemon user:/nonexistent:/usr/sbin/nologin
GIDs:ntopng:*:288:
    0
  • T

    Looks like we have a port now:
    http://www.freshports.org/net/ntopng/

    0
  • T

    Hi All

    Is there any update on when we can expect the ntop-ng package to be released? I see there is a post about a failed install (https://forum.pfsense.org/index.php?topic=113173.0) but I don't see the package in the available list on pfSense yet.

    Thanks to all for the work to make it available.

    Been using pfSense since it was 0. something Beta on various sites/configurations. Awesome to see where it has got to.

    0
  • A

    ntopng is back in the 2.3.2 snapshot,  see https://redmine.pfsense.org/issues/6443

    However, the ability to install from custom package repository urls was removed in 2.3.x as far as I can tell, so I'm not aware of an easy way to install it on the current 2.3.1 release.  (Happy to be corrected on that if someone can point me in the right direction).

    0
  • T

    @Andrew453 Thanks for the details.

    I see that 2.3.2 is not a stable build yet.
    Guess I'll be waiting a little longer unless someone can correct you  :)

    so I'm not aware of an easy way to install it on the current 2.3.1 release. (Happy to be corrected on that if someone can point me in the right direction).

    0
  • A

    … if you're happy to run off a development snapshot, you can specify the development branch in the update settings in pfSense, but that will update your entire system.

    0
  • E

    I've been following the developments very closely. There isn't any way ntopng is going to be included in 2.3.1 update 2, is there? Or will we need to wait until the stable release of 2.3.2?

    0
  • Rebel Alliance Developer Netgate

    If it proves stable enough on 2.3.2, it may be made available elsewhere. It's still being tested, though.

    0
  • I

    great. I think there's a lot of ppl waiting for it.

    just because it's an excellent interface to monitor realtime bandwidth usage on the fw and I don't seem to be able to find a good alternative to it.

    0
  • P

    Thank you for adding this package! It is working well for me locally, but I am having issues with setting up ntopng over HTTPS via NGINX.

    I have tried setting up a proxy_pass directive, but I cannot get past the login screen. I also tried editing the /usr/local/etc/rc.d/ntopng.sh file to add –http-prefix="/ntopng" to the startup strings, but unfortunately I get the same issue.

    Can we integrate SSL certificates into ntopng or allow for native nginx https proxy through pfSense's nginx setup?

    Thank you!

    0
  • P

    temporarily to fix the authentication/login issue through NGINX, I have added –disable-login '1' to the /usr/local/pkg/ntopng.inc file in the DNS Mode string:

            /* DNS Mode */
        if (is_numeric($ntopng_config['dns_mode']) && ($ntopng_config['dns_mode'] >= 0) && ($ntopng_config['dns_mode'] <= 3)) {
                $dns_mode = "--disable-login '1' --dns-mode " . escapeshellarg($ntopng_config['dns_mode']);
        }

    I enabled htaccess password protection via nginx.

    Everything is working great now! Thanks!

    0
  • S

    @jimp:

    If it proves stable enough on 2.3.2, it may be made available elsewhere. It's still being tested, though.

    Is there a chance that it will be included as an alpha release in 2.3.2?

    0
  • Rebel Alliance Developer Netgate

    It's already in 2.3.2. If you install a 2.3.2 snapshot you can use it now (or at least once I get this fix pushed to correct the password handling)

    When 2.3.2 releases (probably next week) you'll have access to it.

    0

  • any chance of pulling in the 2.4 version of ntopng that is available in FreeBSD ports?

    The changelog is long, but the first 2 items alone seem enough to make it worth it:

    • Memory-management, stability and speed have been fundamentally improved

    • We have kept an eye on security and hardened the code to prevent privileges escalation and XSS

    0
  • Rebel Alliance Developer Netgate

    @luckman212:

    any chance of pulling in the 2.4 version of ntopng that is available in FreeBSD ports?

    The changelog is long, but the first 2 items alone seem enough to make it worth it:

    • Memory-management, stability and speed have been fundamentally improved

    • We have kept an eye on security and hardened the code to prevent privileges escalation and XSS

    After we release 2.3.2 we can look into that.

    0
  • J

    Great package, haven't used ntop in years and its great to get this level of detail back, thank you! Looking forward to ntopng 2.4

    0
  • T

    Hi Guys

    Thanks to everyone involved in getting the ntopng package back into pfsense with version 2.3.2. Its great and works well.

    Could I possibly ask for one, hopefully minor, improvement?

    For the historical data could we specify the max period that the data is kept for eg. 30 days.
    This is should be supported by ntop but I can't find the option to specify the limit before I enable the Historical Data storage
    Ref: http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/

    Much appreciated.

    0
  • Rebel Alliance Developer Netgate

    The screen shown on that page is for a MySQL database. That isn't what is in use on pfSense. Eventually the package could grow the ability to export to an external MySQL server, but it wouldn't ever be using a MySQL database on the firewall itself.

    0
  • R

    How are people accessing the ntopng dashboard?  I get an SSL error every I try to access it.  I'm pretty sure it's because pfsense is using HSTS, and thus my browser always wants to connect via HTTPS instead of HTTP, but I'm not really sure how to work around this problem.

    0
  • Rebel Alliance Developer Netgate

    Use HTTPS for the GUI and ntopng

    0

  • HTTPS for ntopng isn't working for me. I receive a SSL protocol error when attempting to access. I haven't tracked it down yet. HTTP via IP address works fine.

    @jimp:

    Use HTTPS for the GUI and ntopng

    0

  • Wasn't too hard to find. HTTPS is disabled because there is no certificate installed. From ntopng.log:

    
27/Jul/2016 21:45:14 [HTTPserver.cpp:464] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
27/Jul/2016 21:45:14 [HTTPserver.cpp:466] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
27/Jul/2016 21:45:14 [HTTPserver.cpp:509] Web server dirs [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts]
27/Jul/2016 21:45:14 [HTTPserver.cpp:512] HTTP server listening on port 3000
    0
  • T

    Thanks Jimp

    Using an external MySQL instance would be a great option.

    For now though, if we enable the historical usage, where is it going to store the data?
    If its in daily files I could just add a cronjob to wipe the older files after x days. Just a thought

    The screen shown on that page is for a MySQL database. That isn't what is in use on pfSense. Eventually the package could grow the ability to export to an external MySQL server, but it wouldn't ever be using a MySQL database on the firewall itself.

    0
  • H

    @dennypage:

    Wasn't too hard to find. HTTPS is disabled because there is no certificate installed. From ntopng.log:

    
27/Jul/2016 21:45:14 [HTTPserver.cpp:464] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
27/Jul/2016 21:45:14 [HTTPserver.cpp:466] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
27/Jul/2016 21:45:14 [HTTPserver.cpp:509] Web server dirs [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts]
27/Jul/2016 21:45:14 [HTTPserver.cpp:512] HTTP server listening on port 3000

    Have the same problem. Is there an easy fix (like symlink the real certificate)?  Should/can this be fixed by the package maintainer?

    0
  • Rebel Alliance Developer Netgate

    I was getting my packages mixed up there. The ntopng package doesn't have an HTTPS option yet. It will need to be added to the package. I can look into it, but I have no idea when I'll be able to get around to adding that as a feature. If someone wants to make a pull request, have a look at the lightsquid package which has a cert selection for HTTPS that would be very similar.

    0
  • R

    Have they managed to get application filtering/control working in nTopng with pfsense?

    That was what my original discussions with Luca Deri were about.

    It's great to have nTopng working again from an analysis/reporting fashion, but the real power of the application is to be able to control applications instead of trying to do a half arsed job with Snort….

    0

  • I don't think certificate selection code is necessary. The ntopng service is another port on the firewall itself. The certificate for ntopng would need to have the same common name and alternate names as the firewall itself. In other words, the certificate for ntopng is the same certificate used by the pfSense webgui.

    I'll have a look at it when time permits.

    0

  • Here you go:

    https://github.com/pfsense/FreeBSD-ports/pull/172

    Note that if you change between http and https for the webgui, either the ntopng settings will need to be re-saved or the system rebooted. Given how rare the switch is, I don't think this will be too much of an issue.

    0
  • C

    @Tram:

    Thanks Jimp

    Using an external MySQL instance would be a great option.

    For now though, if we enable the historical usage, where is it going to store the data?
    If its in daily files I could just add a cronjob to wipe the older files after x days. Just a thought

    The screen shown on that page is for a MySQL database. That isn't what is in use on pfSense. Eventually the package could grow the ability to export to an external MySQL server, but it wouldn't ever be using a MySQL database on the firewall itself.

    I haven't used ntopng in the last few months and looking over the doc, looks like sqlite support isn't there anymore for historical data.  Looks like mysql or es flow-dumps are the only option now.

    There is an error when 'Historical Data Storage' is enabled Im going to change the syntax to use the mySQL i have running

    
18/Aug/2016 08:49:31 [Prefs.cpp:792] WARNING: Discarding -F -i: value out of range

    http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/

    0
  • C

    I was able to get mysql historical data to work.

    I changed file /usr/local/pkg/ntopng.inc line 123 to something like this

    
	/* Historical Data Storage, Dump expired flows */
	if ($ntopng_config['dump_flows'] == "on") {
		$dump_flows = "-F 'mysql;mysql.server;ntopng;flows;ntopng;password'";
	}


    
'mysql;mysql.server;ntopng;flows;ntopng;password'

    mysql - letting it know you are using a mysql db
    mysql.server - mysql server ip or hosename
    ntopng - mysql DB name, you have to create a DB so i used ntopng for its name
    flows - tablename prefix, it will create flowsv4 and flowsv6
    ntopng - mysql username that needs write access to the db it will be inserting data into
    password - password to the mysql user you are using to access you mysql server

    I also swapped the syntax order on line 139. I put the interfaces before the historical data option

    
	$start .= "\t/usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -e {$disable_alerts} {$ifaces} {$dump_flows} {$dns_mode} {$aggregations} {$local_networks} &\n";

    Only LAN is keeping historical data. Not sure why but my other interfaces are vlans so that may have something to do with it.  Needs more testing to confirm.

    I noticed the 'Local Networks' option hasn't been updated to grab IPv6 addresses from interfaces. I manually updated line 117 to include my IPv6 subnet and also some of the multicast subnets to keep them local instead of remote within ntopng

    PS What happen to the preference menu item?

    0

  • Version 0.8.4_1 with HTTPS support is rolling out now.

    0
  • V

    @dennypage:

    Version 0.8.4_1 with HTTPS support is rolling out now.

    Is this package update version 2.4 of ntopng, or still the previous version?

    0

  • It's the same version of ntopng, with the addition of support for HTTPS in the pfSense package.

    0
  • C

    I found bug with the ssl version. You may want to remove "\n" from the new code.

    
	if ($config['system']['webgui']['protocol'] == "https") {
		$cert =& lookup_cert($config['system']['webgui']['ssl-certref']);
		ntopng_write_cert_file("/usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem", $cert);
		$http_args = "-w 0 -W 3000\n";
	} else {
		unlink_if_exists("/usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem");
		$http_args = "-w 3000\n";
	}

    It created the the startup file incorrectly

    
rc_start() {
		/usr/local/bin/redis-server --dir /var/db/ntopng/ --dbfilename ntopng.rdb &
	/usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000
    -i 'em2' --dns-mode '0'  --local-networks 'fe80::/10,192.168.0.0/24,2000:0000:0000:2400::/64' &
    0
 

Products

Applications

Support

Services

News

Resources

Company

Our Mission

As host of the pfSense open source firewall project, Netgate believes in enhancing network connectivity that maintains both security and privacy. We also believe everyone should be able to afford it.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy