Heavy traffic monitoring at service or application level

[JT40]

Hello,

I need to understand what's using my network so heavily time to time (300GB in a short time but only every 2-3 months).

Is there any proper ready made solution for this scenario?
I need to log the following info:

• service/process name
• domain to where is connected
• VLAN_ID possibly (I use PVLANs but I don't think it's an issue at all), or or some sort of identifier like IP and Device_name all together.

Other info like port and protocol are secondary, but a nice have for further investigation.

As you can understand, I need to dump these info somewhere, I have a lot of space (180GB) on the SSD, but I'm not planning to fill it up :D .
Same way, if I have a huge amount of data to ingest in any application or package in Pfsense, then it will be quite hard to process such amount of data...
Whatever tool I use, it can't be like Wireshark, I need to have a report, not dumping all the traffic, I hope I explained it well :) .

[Gertjan]

@jt40

Looks like you could consider using System > Package Manager > Available Packages > ntopng

ntopng builds 'html' pages to show, so no php that parses huge logs files, and goes "time out" doing so.
ntopng can't see what LAN devices 'consume' per process, of course. It can only see : what IP, what ports used.

I'm not using ntopng myself.
Warning : don't install ntopng an walk away. Check disk space used, process power used, etc. daily.

[JT40]

@gertjan said in Heavy traffic monitoring at service or application level:

@jt40

Looks like you could consider using System > Package Manager > Available Packages > ntopng

ntopng builds 'html' pages to show, so no php that parses huge logs files, and goes "time out" doing so.
ntopng can't see what LAN devices 'consume' per process, of course. It can only see : what IP, what ports used.

I'm not using ntopng myself.
Warning : don't install ntopng an walk away. Check disk space used, process power used, etc. daily.

Thanks for the suggestions, but I see that it's not a ready-made solution:

1. I need to install a couple of things and configure them, I have skill to follow those procedures, but I really don't like to mess up in BSD, I don't know that extremely well :D , same for the distro on top...

2. It will weight on my system, I have 16GB of Ram and 8 threads, pretty recent CPU, but I don't think it's enough for what I need, not over a couple of months of HA required to catch these network spikes... Eventually, I should set up another machine for that, but it's gonna be another expense...

3. I've seen that Redis is a DB in memory, I hope it doesn't run in that way in Pfsense as a package, or that at least I can change the configuration.

Is there anything else you would recommend?
I can search online, but it's better to get advices on my specific requirements.