Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Identifying traffic

    Scheduled Pinned Locked Moved Traffic Monitoring
    6 Posts 3 Posters 1.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      GeoffW
      last edited by

      Hi all,  (I'm new here, be gentle etc.)

      I think this Traffic Monitoring forum is the most appropriate to my question, so here goes:

      I'm on a wireless plan that gives me 25GB (up and down) which until recently my small network was staying under fairly comfortably.  Then January hit with 35GB and February is headed the same way (and I get charged $10/GB when I go over the limit.)

      I'm using pfSense v2.3.2-p1 (updated from v2.3.1 just a few nights ago), along with squid, squidGuard and lightsquid_web, and recently installed ntopng and Traffic Totals packages as well.  The last two are because I am now trying to work out who/what is generating between 40% and 60% of the traffic on my Internet connection.

      I know that Windows 10 caused at least 4GB of the extra in both months (systems forcing the Anniversary update on me whether I want it or not).  But I can see that download on the lightsquid reports, including which the days they occurred.  But lightsquid only showed 10.8GB for Jan, it's showing 9.4GB for Feb so far.  … In an effort to let lightsquid see more I have now turned on SSL interception in squid, but early signs are that this will only explain a small portion - taking me up to around 50% of what the ISP says I am using, and it appears that Traffic Totals is confirming the ISP totals (I haven't been running it long enough to say it's an exact match, just long enough to see that it is reporting almost double what lightsquid sees).

      So I installed ntopng, but most of its interface seems to concentrate on live monitoring, and its few historical interfaces are all moving targets (1day to now, 12h to now and so on), making it almost impossible to use it to analyse what happened 00:00..23:59 yesterday (for example) - so I cannot easily compare to Traffic Totals or ISP reports.  ntopng also makes it difficult when it describes an application as "Facebook" but does not explain what it means when it also shows http and http_proxy (so it's hard to see which figures in lightsquid correspond with figures from ntopng).

      I do use a VPN (client is inside my network, not in pfSense, server outside) for RDP connections, but I can see impact of this and see that it isn't anywhere near big enough to explain the difference.  Similarly DNS, ICMP etc.

      So my question is:  What tools can/should I be using to help me try to identify what is generating all this traffic on a historical basis (so I can compare day to day with my ISP, Traffic Totals, lightsquid etc.) ?

      1 Reply Last reply Reply Quote 0
      • NogBadTheBadN
        NogBadTheBad
        last edited by

        Try ntopng YMMV

        Andy

        1 x Netgate SG-4860 - 3 x Linksys LGS308P - 1 x Aruba InstantOn AP22

        1 Reply Last reply Reply Quote 0
        • MikeV7896M
          MikeV7896
          last edited by

          @NogBadTheBad:

          Try ntopng YMMV

          Read again… :) Maybe you have some tips on using it?

          @GeoffW:

          So I installed ntopng, but most of its interface seems to concentrate on live monitoring, and its few historical interfaces are all moving targets (1day to now, 12h to now and so on), making it almost impossible to use it to analyse what happened 00:00..23:59 yesterday (for example) - so I cannot easily compare to Traffic Totals or ISP reports.  ntopng also makes it difficult when it describes an application as "Facebook" but does not explain what it means when it also shows http and http_proxy (so it's hard to see which figures in lightsquid correspond with figures from ntopng).

          I want to get into using ntopng myself… but I'm running the pfSense 2.4 beta and I'm hoping they'll update ntopng to a newer build that has fixed some bugs with the version currently available for pfSense 2.4... so for the moment I'm holding off and won't be much help unfortunately.

          The S in IOT stands for Security

          1 Reply Last reply Reply Quote 0
          • G
            GeoffW
            last edited by

            @virgiliomi:

            I want to get into using ntopng myself… but I'm running the pfSense 2.4 beta and I'm hoping they'll update ntopng to a newer build that has fixed some bugs with the version currently available for pfSense 2.4... so for the moment I'm holding off and won't be much help unfortunately.

            Thanks for your reply.

            While I was signing up here I saw v2.3.3 had come out, so I updated pfSense again, and the ntopng package is now updated to v2.4 … but it doesn't actually change much with regard to being able compare figures.

            The closest I have come to getting comparable figures is to purge the ntopng data at midnight - so that the cumulative totals that it shows match what I get from Traffic Totals and my ISP.  I may try that for a few nights and see what I come up with.

            1 Reply Last reply Reply Quote 0
            • MikeV7896M
              MikeV7896
              last edited by

              I'm sure there's some way to get what you're looking for. I'm pretty sure that ntop keeps lots of historical data, so there should be some way to limit its reporting scope to a certain period of time.

              The S in IOT stands for Security

              1 Reply Last reply Reply Quote 0
              • G
                GeoffW
                last edited by

                ntop definitely has the data, what it lacks is the user interface to present it the way I need it.  It is possible the paid version might have something, but I can find nothing in the user manual (which includes paid version details).

                Of course, since it is open source I guess I could try to add what I need … but not sure if I have the necessary skills, and I definitely don't have the time at the moment.  The only other alternative would appear to be to export its totals and collate them in some other package.

                I discovered another complication.  It looks like lightSquid's "day" is (if you update every hour) is actually 23:00..22:59 (the update they run at midnight goes into the next day (despite the fact that it is obviously counting traffic from the previous day).  So it just gets harder and harder to compare figures.  (My network is small enough I could probably run lightsquid's update more often and reduce the problem.)

                Deleting ntopng data just before midnight (disable the service, hit the delete data button, restart the service, then restart it again before it will actually work) clears most of its history (but not traffic totals).  And using this I've been able to see that ntop's totals come close enough to the "Traffic Totals" package, both of which are close to the ISP totals for the day.

                Without having collected the history going back to December or early January I cannot see for certain what has blown out in my network use, but since ntop's "Facebook" application is something like 40% to the total I am guessing it must be a significant part of it (none of the other totals are large enough to hide the approx 300MB a day that the traffic has increased - up from around 700MB/day to 1000MB/day).

                So ntopng has helped (or offers hints), it's just a shame it doesn't offer consistent, comparable historical analysis from its current user interface.

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.