Identifying traffic
-
Hi all, (I'm new here, be gentle etc.)
I think this Traffic Monitoring forum is the most appropriate to my question, so here goes:
I'm on a wireless plan that gives me 25GB (up and down) which until recently my small network was staying under fairly comfortably. Then January hit with 35GB and February is headed the same way (and I get charged $10/GB when I go over the limit.)
I'm using pfSense v2.3.2-p1 (updated from v2.3.1 just a few nights ago), along with squid, squidGuard and lightsquid_web, and recently installed ntopng and Traffic Totals packages as well. The last two are because I am now trying to work out who/what is generating between 40% and 60% of the traffic on my Internet connection.
I know that Windows 10 caused at least 4GB of the extra in both months (systems forcing the Anniversary update on me whether I want it or not). But I can see that download on the lightsquid reports, including which the days they occurred. But lightsquid only showed 10.8GB for Jan, it's showing 9.4GB for Feb so far. … In an effort to let lightsquid see more I have now turned on SSL interception in squid, but early signs are that this will only explain a small portion - taking me up to around 50% of what the ISP says I am using, and it appears that Traffic Totals is confirming the ISP totals (I haven't been running it long enough to say it's an exact match, just long enough to see that it is reporting almost double what lightsquid sees).
So I installed ntopng, but most of its interface seems to concentrate on live monitoring, and its few historical interfaces are all moving targets (1day to now, 12h to now and so on), making it almost impossible to use it to analyse what happened 00:00..23:59 yesterday (for example) - so I cannot easily compare to Traffic Totals or ISP reports. ntopng also makes it difficult when it describes an application as "Facebook" but does not explain what it means when it also shows http and http_proxy (so it's hard to see which figures in lightsquid correspond with figures from ntopng).
I do use a VPN (client is inside my network, not in pfSense, server outside) for RDP connections, but I can see impact of this and see that it isn't anywhere near big enough to explain the difference. Similarly DNS, ICMP etc.
So my question is: What tools can/should I be using to help me try to identify what is generating all this traffic on a historical basis (so I can compare day to day with my ISP, Traffic Totals, lightsquid etc.) ?
-
Try ntopng YMMV
-
Try ntopng YMMV
Read again… :) Maybe you have some tips on using it?
So I installed ntopng, but most of its interface seems to concentrate on live monitoring, and its few historical interfaces are all moving targets (1day to now, 12h to now and so on), making it almost impossible to use it to analyse what happened 00:00..23:59 yesterday (for example) - so I cannot easily compare to Traffic Totals or ISP reports. ntopng also makes it difficult when it describes an application as "Facebook" but does not explain what it means when it also shows http and http_proxy (so it's hard to see which figures in lightsquid correspond with figures from ntopng).
I want to get into using ntopng myself… but I'm running the pfSense 2.4 beta and I'm hoping they'll update ntopng to a newer build that has fixed some bugs with the version currently available for pfSense 2.4... so for the moment I'm holding off and won't be much help unfortunately.
-
I want to get into using ntopng myself… but I'm running the pfSense 2.4 beta and I'm hoping they'll update ntopng to a newer build that has fixed some bugs with the version currently available for pfSense 2.4... so for the moment I'm holding off and won't be much help unfortunately.
Thanks for your reply.
While I was signing up here I saw v2.3.3 had come out, so I updated pfSense again, and the ntopng package is now updated to v2.4 … but it doesn't actually change much with regard to being able compare figures.
The closest I have come to getting comparable figures is to purge the ntopng data at midnight - so that the cumulative totals that it shows match what I get from Traffic Totals and my ISP. I may try that for a few nights and see what I come up with.
-
I'm sure there's some way to get what you're looking for. I'm pretty sure that ntop keeps lots of historical data, so there should be some way to limit its reporting scope to a certain period of time.
-
ntop definitely has the data, what it lacks is the user interface to present it the way I need it. It is possible the paid version might have something, but I can find nothing in the user manual (which includes paid version details).
Of course, since it is open source I guess I could try to add what I need … but not sure if I have the necessary skills, and I definitely don't have the time at the moment. The only other alternative would appear to be to export its totals and collate them in some other package.
I discovered another complication. It looks like lightSquid's "day" is (if you update every hour) is actually 23:00..22:59 (the update they run at midnight goes into the next day (despite the fact that it is obviously counting traffic from the previous day). So it just gets harder and harder to compare figures. (My network is small enough I could probably run lightsquid's update more often and reduce the problem.)
Deleting ntopng data just before midnight (disable the service, hit the delete data button, restart the service, then restart it again before it will actually work) clears most of its history (but not traffic totals). And using this I've been able to see that ntop's totals come close enough to the "Traffic Totals" package, both of which are close to the ISP totals for the day.
Without having collected the history going back to December or early January I cannot see for certain what has blown out in my network use, but since ntop's "Facebook" application is something like 40% to the total I am guessing it must be a significant part of it (none of the other totals are large enough to hide the approx 300MB a day that the traffic has increased - up from around 700MB/day to 1000MB/day).
So ntopng has helped (or offers hints), it's just a shame it doesn't offer consistent, comparable historical analysis from its current user interface.