A Guide to current NtopNG on pfSense 2.6/22.01 :-)

keyser

@cyphonsqr said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

Thanks for the great write up @keyser. :)
I had an error with Redis (not being started), but it was fixed by rebooting the box (Netgate 4860 desktop).
System: 22.01 Plus (zfs).

Regarding RAM Disk, my box only has 8GB RAM and 32GB eMMC and I would like to store historic ntopng data without excessive wear on the disk. I'm happy to lose ~12hours of ntop data (from reboots, power failures, etc.), if that helps minimise undue wear on the disk.

Do you have any suggestions for RamDisk settings to achieve this?

Unfortunately I won't be installing larger RAM/eMMC.

Unfortunately I have very little experience using RAMdisk on pfSense. But I do know that retaining data/settings in NtopNG will not work if you enable RAM disk - even if you enable some of the “flush xxx dir to disk” features in the RAMdisk setup. Thats because /var/db/ntopng is not included in any of the built in flush/restore scripts of specific directories to/from RAMdisk.

So as I see it you have three options:

1: Get a bigger SSD or Attempt to minimize write I/O to a level you are comfortable with on your eMMC
2: Do some experimenting with the process launch settings in the ntopng.inc file where you added the --community setting. There is a -d option set that explains where NtopNG should store its data/settings (default /var/db/ntopng). You could attempt to store that in one of the directories that are covered by the RAMdisk periodic flush/restore at startup feature. I have no Idea if it will work though.
3: Create your own periodic cron job that flushes /var/db/ntopng to disk, and importantly, restores it at bootup and before NtopNG service start.

cyphonsqr

@keyser
Thanks for all that.

I wonder if I could use a cheap 64GB USB drive as the RAMDisk, and not need to worry about wearing out the onboard eMMC.
If you have any thoughts/experience with that, I'd be open to hearing them.

Alternatively, I have a NAS online on my LAN 24/7... could I send the Ntopng data there instead?

keyser

@cyphonsqr said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser
Thanks for all that.

I wonder if I could use a cheap 64GB USB drive as the RAMDisk, and not need to worry about wearing out the onboard eMMC.
If you have any thoughts/experience with that, I'd be open to hearing them.

Alternatively, I have a NAS online on my LAN 24/7... could I send the Ntopng data there instead?

You definitely can, but it will require some manual adaptations to have a script mount the drive at bootup, and change the path where NtopNG stores data.

Just out of curiosity, why not just install a cheap SSD instead? Its not like they cost an arm and a leg :) I got my 512Gb SSD for less than a 100 dollars, and I could have gotten a 128Gb one for about 50 dollars. It needs not be a particular fast model, just one that comes with either a slightly above average write enduance, or capacity enough that the endurance is given because of the size.

keyser

Just a small update here. You can get a fairly current 5.2 build on your ARM based appliances as well by installing the latest port from the FreeBSD ports repository. It works equally beautiful as the fix above for AMD64 based boxes.
And this is far more helpful as the current NtopNG pfsense package actually installs a v4 community edition even though it's named v5.0.xxxxx

If anyone want further details then just post here, and I'll see if I can create a guide for aarch64 based appliances.

aduzsardi

@keyser that would be awesome , please add some info for arm based appliances as well
thank you

cyphonsqr

@keyser I like the idea.. but my pfSense box is an official NetGate 4860.. ~~which I don't think can take any upgrades.~~
I stand corrected.. it can take an mSATA drive : https://docs.netgate.com/pfsense/en/latest/solutions/sg-4860/msata-installation.html

keyser

@cyphonsqr said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser I like the idea.. but my pfSense box is an official NetGate 4860.. ~~which I don't think can take any upgrades.~~
I stand corrected.. it can take an mSATA drive : https://docs.netgate.com/pfsense/en/latest/solutions/sg-4860/msata-installation.html

Yea, and that’s exactly what I would do - get a >=256Gb mSata drive, it’s a very small price to both allow yourself log to log whatever you want, and prevent wearing your eMMC to death regardless of settings in NtopNG, pfBlocker and what not :-)

keyser

@aduzsardi said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser that would be awesome , please add some info for arm based appliances as well
thank you

I have now updated the guide (first post) so it contains both a AMD64 version and a ARM/aarch64 version :-)

Please confirm if it works as expected. I wrote it down after completing my own device :-)

netblues

@keyser I'm sorry to report that even if this works, it has serious issues when/if upgrades come.

Various important things will never get updated, so one has to reinstall from scratch.

I suspect the issue is that pkg prefers to fetch things from general bsd repos and not pf, thus producing unexpected results.

Certainly not worth the effort/ uncertainty it produces in the long run

keyser

@netblues said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser I'm sorry to report that even if this works, it has serious issues when/if upgrades come.

Various important things will never get updated, so one has to reinstall from scratch.

I suspect the issue is that pkg prefers to fetch things from general bsd repos and not pf, thus producing unexpected results.

Certainly not worth the effort/ uncertainty it produces in the long run

It’s true that it will not work without a “redo” of the process after a pfsense version upgrade.
Thats because pfsense after a upgrade will install the pfsense NtopNG package from scratch.
But if you disable NtopNG before upgrading, you can immediately redo the above process after a pfsense version upgrade, and thus reinstall the latest NtopNG - before the old buggy version included in the package starts and maybe causes corruption to the stored timeseries data.

Personally this makes NtopNG usable and a Very valued Daily tool, so it’s worth the trouble for me :-) I guess you dont really have a need for NtopNG because the buggy version included in the current package is completely useless and impossible to “live” with.

Here’s hoping that 22.05/2.7 will include an updated NtopNG in the official package so this “hack” is not needed. In that case it might even upgrade “and just work” with the current NtopNG data and settings from your hacked install.

netblues

@keyser Well, yes and no
The problem isn't contained to ntopng per se
Pfsense can't update it self properly.
and you end up with a system that misses critical components, such as mpd , needed for pppoe, and if this is your only way to connect, then you are left with no means to manualy fix things.

I doubt that disabling ntopng will make the upgrade process work as expected again.

The safe way is to install from scratch and then restore configuration.

22.-5 rc is on ntopng v.5.2.220602 (FreeBSD 12.3)
and since we are approaching release, I doubt it will change.
Same package version (08.13_10) also appears at the 2.7 dev for now.

keyser

@netblues said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser Well, yes and no
The problem isn't contained to ntopng per se
Pfsense can't update it self properly.
and you end up with a system that misses critical components, such as mpd , needed for pppoe, and if this is your only way to connect, then you are left with no means to manualy fix things.

I doubt that disabling ntopng will make the upgrade process work as expected again.

The safe way is to install from scratch and then restore configuration.

22.-5 rc is on ntopng v.5.2.220602 (FreeBSD 12.3)
and since we are approaching release, I doubt it will change.
Same package version (08.13_10) also appears at the 2.7 dev for now.

Strange - when I update my test machine it works fine using that process. Are you using AMD64 or ARM64?

Anyhow, IF the included NtopNG version with 22.05/2.7 really is 5.2.220602, then I see no reason for using my hack as it has reached a reasonably mature and stable NtopNG version. I hope that goes for both AMD64 and ARM64.
ARM64 was left behind on a v4.0 release in 22.01/2.6 even though it said 5.0.xxxxx in the package manager.

If this is the case I think I will be doing clean install of 22.05 to remove any leftovers from the hacked install.

netblues

@keyser I'm talking about pf 22.05 rc, upgraded from 22.01 with the ntopng hack, on amd64

And yes, ntopng does seem to work nicely too.

keyser

@netblues said in A Guide to current NtopNG on pfSense 2.6/22.01 :-):

@keyser I'm talking about pf 22.05 rc, upgraded from 22.01 with the ntopng hack, on amd64

And yes, ntopng does seem to work nicely too.

Okay that’s Nice to know. I will clean install then, restore the config, and after the official NtopNG pf package has installed (5.2.2206), I will restore my “in NtopNG” settings (rules, checks and so forth). No need for for my hack anymore

netblues

@keyser Ι have restored from ntopng 5.3 to 5.2 and nothing broke.
on 22.05rc after clean install
However not much customisation had been done.