Ntopng development

dennypage

Here you go:

https://github.com/pfsense/FreeBSD-ports/pull/172

Note that if you change between http and https for the webgui, either the ntopng settings will need to be re-saved or the system rebooted. Given how rare the switch is, I don't think this will be too much of an issue.

Cino

@Tram:

Thanks Jimp

Using an external MySQL instance would be a great option.

For now though, if we enable the historical usage, where is it going to store the data?
If its in daily files I could just add a cronjob to wipe the older files after x days. Just a thought

The screen shown on that page is for a MySQL database. That isn't what is in use on pfSense. Eventually the package could grow the ability to export to an external MySQL server, but it wouldn't ever be using a MySQL database on the firewall itself.

I haven't used ntopng in the last few months and looking over the doc, looks like sqlite support isn't there anymore for historical data. Looks like mysql or es flow-dumps are the only option now.

There is an error when 'Historical Data Storage' is enabled Im going to change the syntax to use the mySQL i have running


18/Aug/2016 08:49:31 [Prefs.cpp:792] WARNING: Discarding -F -i: value out of range

http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/

Cino

I was able to get mysql historical data to work.

I changed file /usr/local/pkg/ntopng.inc line 123 to something like this


	/* Historical Data Storage, Dump expired flows */
	if ($ntopng_config['dump_flows'] == "on") {
		$dump_flows = "-F 'mysql;mysql.server;ntopng;flows;ntopng;password'";
	}


'mysql;mysql.server;ntopng;flows;ntopng;password'

mysql - letting it know you are using a mysql db
mysql.server - mysql server ip or hosename
ntopng - mysql DB name, you have to create a DB so i used ntopng for its name
flows - tablename prefix, it will create flowsv4 and flowsv6
ntopng - mysql username that needs write access to the db it will be inserting data into
password - password to the mysql user you are using to access you mysql server

I also swapped the syntax order on line 139. I put the interfaces before the historical data option


	$start .= "\t/usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -e {$disable_alerts} {$ifaces} {$dump_flows} {$dns_mode} {$aggregations} {$local_networks} &\n";

Only LAN is keeping historical data. Not sure why but my other interfaces are vlans so that may have something to do with it. Needs more testing to confirm.

I noticed the 'Local Networks' option hasn't been updated to grab IPv6 addresses from interfaces. I manually updated line 117 to include my IPv6 subnet and also some of the multicast subnets to keep them local instead of remote within ntopng

PS What happen to the preference menu item?

dennypage

Version 0.8.4_1 with HTTPS support is rolling out now.

MikeV7896

@dennypage:

Version 0.8.4_1 with HTTPS support is rolling out now.

Is this package update version 2.4 of ntopng, or still the previous version?

dennypage

It's the same version of ntopng, with the addition of support for HTTPS in the pfSense package.

Cino

I found bug with the ssl version. You may want to remove "\n" from the new code.


	if ($config['system']['webgui']['protocol'] == "https") {
		$cert =& lookup_cert($config['system']['webgui']['ssl-certref']);
		ntopng_write_cert_file("/usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem", $cert);
		$http_args = "-w 0 -W 3000\n";
	} else {
		unlink_if_exists("/usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem");
		$http_args = "-w 3000\n";
	}

It created the the startup file incorrectly


rc_start() {
		/usr/local/bin/redis-server --dir /var/db/ntopng/ --dbfilename ntopng.rdb &
	/usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000
    -i 'em2' --dns-mode '0'  --local-networks 'fe80::/10,192.168.0.0/24,2000:0000:0000:2400::/64' &

dennypage

@Cino:

I found bug with the ssl version. You may want to remove "\n" from the new code.

You are quite correct. Not sure how I missed that in testing.

PR created.

Thanks

garyd9

dennypage, is there some trick to getting someone to accept pull requests? I submitted one a couple weeks ago to add ipv6 addresses to "local-networks" in the ntopng command line, but it's gone idle as far as I can see…

(I'm just trying to figure out if I missed a step in the submission or not.)

Thanks
Gary

dennypage

@garyd9:

dennypage, is there some trick to getting someone to accept pull requests? I submitted one a couple weeks ago to add ipv6 addresses to "local-networks" in the ntopng command line, but it's gone idle as far as I can see…

No special trick, but I went to look for your PR and couldn't find it. What is the PR number?

garyd9

@dennypage:

No special trick, but I went to look for your PR and couldn't find it. What is the PR number?

https://github.com/pfsense/pfsense-packages/pull/1262

dennypage

I believe you want this repo here:

https://github.com/pfsense/FreeBSD-ports

which corresponds to packages in pfSense 2.3 and later.

Look in net/pfSense-pkg-ntopng.

garyd9

@dennypage:

I believe you want this repo here:

https://github.com/pfsense/FreeBSD-ports

which corresponds to packages in pfSense 2.3 and later.

Look in net/pfSense-pkg-ntopng.

Okay, thank you. I'll redo my patch tomorrow morning in that other repo.

r_e_d_b_a_r_o_n

@dennypage:

@Cino:

I found bug with the ssl version. You may want to remove "\n" from the new code.

You are quite correct. Not sure how I missed that in testing.

PR created.

Thanks

Hello, I updated ntopng package to 0.8.4_2. From changelog it fixes problem with \n in startup script. BUT after upgrade still same config and startup file is generated. Edited it manually, removed newline but still not starting with error : "[HTTPserver.cpp:503] ERROR: Unable to start HTTP server (IPv4) on ports 3000s: Invalid argument". It only started after removing -W argument (HTTPS).
New bug or my problem?

Modified ntopng.sh

rc_start() {
/usr/local/bin/redis-server –dir /var/db/ntopng/ --dbfilename ntopng.rdb &
/usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i 'vmx1' -i 'vmx2' --dns-mode '0' --local-networks '192.168.0.0/16,172.16.0.0/12,10.0.0.0/8' &

}

dennypage

If you are using HTTPS enabled in the pfSense gui, you will need to visit the ntopng settings page (Diagnostics / ntopng Settings) and press save. This will cause the certificate to be set up in order to use HTTPS with ntopng.

luckman212

@dennypage:

It's the same version of ntopng, with the addition of support for HTTPS in the pfSense package.

Denny, do you know what the main reason is that the 2.4 version of ntopng isn't available in pfSense? I see the FreeBSD port has been ready for some time…
https://github.com/freebsd/freebsd-ports/blob/master/net/ntopng/Makefile

kesawi

I've just upgraded to ntopng 0.8.4_2 on pfSense 2.3.2 and get the following error message in the System Log:

ntopng 		[HTTPserver.cpp:503] ERROR: Unable to start HTTP server (IPv4) on ports 3000s: Invalid argument

The ntopng service is showing as stopped in the service status and the ntopng process isn't running (ie not appearing when I run ps aux or sockstat.

I've tried uninstalling, reinstalling, and unchecking the "Keep Data/Settings" options, but my settings seem to be kept, and the error remains. I've tried changing the settings in the GUI and re-saving as suggested a couple of posts up.

I do have a certificate (ntopng-cert.pem) in /usr/local/share/ntopng/httpdocs/ssl which gets regenerated each time I update the settings through the GUI.

dennypage

Would you mind checking the sum on ntopng.inc please?

The command is this: sum /usr/local/pkg/ntopng.inc

The expected output is this: 51966 12 /usr/local/pkg/ntopng.inc

Thanks

dennypage

@luckman212:

Denny, do you know what the main reason is that the 2.4 version of ntopng isn't available in pfSense?

I don't have direct knowledge, but if I had to guess I would say that it's a fair bit of work and nobody has volunteered or been assigned for it.

luckman212

Hmm ok I didn't realize it was much work after the FreeBSD port. Ok, I wish I could help somehow. I looked at the "Developing Packages" page but tbh I wouldn't know where to start :(