How to run the packet capture on pfsense and ntopng on a different VM?
-
After hours of playing around I need to give up and pass my question to the community.
I noticed that running the ntopng on my pfSense always causes a lot of trouble, although if has 12GB of RAM (<25% used) and plenty of processing power (Dell R210ii). Swap filling up to 100% in 2.3 days even with a RAM usage of <25% is one point, slow reaction of the web UI the other.
I have a VM running an enterprise version of ntopng, capturing the traffic via a mirroring port on the switch where also the pfSense machine is connected to. But that's a really ugly workaround.
So, the next attempt is exporting the traffic data via softflowd to nprobe running on the ntopng machine which then passes the data to ntopng via ZMQ connector. In priciple this works like a charm, but there is one issue which I can't get over in the last 3 hours:
When ntopng runs on pfsense itself and I start i.a. a download on PC 10.1.2.3 from the server 12.34.56.78, the flow appears in ntopng. The flow is updated constantly, and if the download is running for 30 minutes, I still have one flow with a duration of 30 minutes and all needed data in the flow details. That's what I want.
But with the "remote capturing" via softflowd, I have some issues:
-
No data seems to be sent from softflowd before the expiry time was reached. So, having a timeout of 60 seconds configured in softflowd, I need to wait 60 seconds until the flow is visible in nprobe/ntopng. This is a thing I can live with - that the update rate is only every 60 seconds.
-
The main issue is: That the flow will not be kept and being "updated" in ntopng. With a cache flow idle timeout setting of 300 seconds in ntopng I would expect that a 30 minutes download is still a flow of 30 minutes duration, but now the values are only updated every 60 seconds (as new data for that flow is sent by softflowd every 60 seconds).
But his does not work. I have no flow in ntopng with more that one minute duration at all, and clicking on a flow detail, in 95% of the cases there is a red message telling me that the flow was likely purged from memory.
So, it looks like a 30 minute single download will be chopped into 30 single, independent flows in ntopng which are living only one minute - that makes it completely impossible to get a picture what connections are currently active, for how long and how many data they use.
Do I oversee any configuration here? Can I get a single flow which exists for infinite time in ntopng as long as it's updated, having softflowd just sending "updates" ever 30 seconds? And only if no further "update" was received by ntopng for 5 minutes, the flow is considered idle and deleted from memory?
In case this softflowd model does not allow this (what is the sense with it, then?), how can I get the interface packets captured in pfsense, but ntopng running in a different VM - giving me the same "flow display experience" as if it would run locally on pfsense itself?
-
-
@jacotec said in How to run the packet capture on pfsense and ntopng on a different VM?:
ntopng
I think ntopng has a memory leak somewhere. Mine was also using 100% swap. I disabled the service and all the issues disappeared. Everything is on the latest community version.
-
@jacotec I know this is an old thread, but I don’t think Softflowd can quite deliver what you are asking. If you want a report of one 30 min flow in NtopNG, you have to ask softflowD not to expire flows on a timer - you need to leave the expiration field empty. (Otherwise it effectively reports flows that is not completed as unique flows).
Then you get what you want, but the problem is then that you cannot see the flow while it is in progress. You have to wait 30 min.What we really need is a Nprobe package for pfSense that allows us to install Nprobe (and license it). That way it can do the actual capture/analyze part - including the much more insightfull DPI features compared to standard netflow, and report all of it to NtopNG on another machine via ZMQ.
