Ntop History Details
-
I'm hoping someone can clear up some confusion for me. I'm trying to figure out how to use Ntop to dig into traffic history details, but coming up with mostly live data. I'm trying to look back over the past 1-2 weeks to determine what is causing large surges in Internet traffic.
I found this Ntop blog post describing where to find exactly what I'm looking for:
http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng-part-2/
The Flows, Talkers and Protocols tabs look like what I need, but they are missing from the Ntop instance in pfSense (or at least in my install). The Chart tab is the only one I see on that page.
Is there some way to turn on those views, or is that just a limitation of the Ntop package (or only available in the Ntop Pro version…)?
Thanks!
-
This looks like much the same question as I was asking in the thread two down for yours: https://forum.pfsense.org/index.php?topic=125906.0, except that I did not find the chart feature particularly useful because you cannot have it show you consistent time periods for comparison (all the periods are "to now").
The Chart tab should be available by clicking on the "Interfaces" drop down and choosing your interface, then clicking on the little graph icon. After that you can choose the displayed duration (to now).
Note that you can show different things on the chart by clicking on the "Timeseries" button for a drop down list. But note this list only shows items available in the currently displayed data. For example if the graph is showing just the last hour then Facebook doesn't show in the drop down because the Facebook user on my network isn't here. If I expand the display to 12h then Facebook shows on the dropdown because my Facebook user was here earlier.
I do NOT see any of the additional tabs next to "Chart" (IP v4 Flows, etc.), I suspect they might be professional version only. (Your linked article says "With the advances made in the latest ntopng Pro Small Business it is possible to drill-down historical flows and obtain, among other things:" - emphasis mine). But I don't see them as being particularly useful unless you plan to export data for analysis outside ntopng.
-
There is no such feature available in the pfSense package (plus as noted, it needs to be exported to MySQL or ELK.)
https://github.com/ntop/ntopng/wiki/03-MySQL-FAQ
https://github.com/ntop/ntopng/wiki/02-Elasticsearch-FAQThe only thing that's available is what can be configured in Preferences - On-Disk Timeseries.
-
Ok, thank you both. I thought that might be the case, but I was hoping I had missed something.
Does anyone have a recommendation for a way (built in tools, 3rd party system, open to anything…) to analyze historical traffic data? The core capability I'm looking for is to be able to identify the LAN side source of large surges in Internet traffic from 15-30 days back. Ideally I'd like to be able to see source, dest and port/protocol with amount of traffic.
I've played around with softflowd and some of the free netflow software out there, but nothing has had it quite figured out. GeoffW, it does sound like you're looking for pretty much the same thing...
-
xawen, yes it sounds like we are both looking for the same thing.
As noted in my thread, the only (temporary) solution I have found is to read the ntop data at midnight and then clear its logs so I can read it the next day. I've been doing this manually at the moment, just to try and get a hint at what's going on, and to get some base figures. Obviously a manual approach is no good for anything more than a few days.
I'm pretty certain ntop already has everything needed to give us what we want EXCEPT the user interface support to show it. It's quite frustrating. I know it is "cute" to see current flows, but I can't figure how why there don't seem to be more people interested in historical views of this data: except in times of crisis, current flows are pretty useless, trends (using comparable periods of data) are much more interesting (or so it seems to me).