Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    New Traffic Totals package available for testing

    Traffic Monitoring
    37
    121
    31299
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jdillard
      jdillard last edited by

      It uses the vnstat database (just takes the JSON output to make the graphs).

      go forth and break things. (or just play around with it and have fun)

      Known issues:

      #1) There is an issue with the stacked bar graph. Documented here: https://github.com/novus/nvd3/issues/341#issuecomment-222370991

      #2) If you have vnstat already running it will most likely override your options and also install the new database files in a different location than where yours currently are. This scenario is likely to produce the most edge cases.

      #3) The start day defaults to 1 and is stored in the vnstat config file at the time of saving, but I don't have a way to get it back out of the config currently, so it shows up as blank. [fixed]

      #4) Which ties into number four, there isn't a way to save the graph options at this time. [fixed]

      #5) The cron job that grabs the data runs every 5 minutes and the interval isn't documented at this time. [fix pending]

      #6a) units should be in KiB/MiB/etc and the Y axis label needs to be changed to reflect totals not speed. [fixed]

      #6b) y axis tickmarks need custom formatting. [fixed]

      #7) lists PHP as a dependency when it doesn't need to. [fixed]

      #8) Top 10 Days only works when one interface is selected. I need to throw an error if multiple interfaces are selected. [fixed]

      #9a) NanoBSD image is untested at this time, but should work. Data loss after a reboot needs to also be tested (actually, I may be able to tie the data into backups).

      #9b) vnStat databases aren't being backed up.

      #10a) Some interfaces are showing up blank. [fixed]

      #10b) It looks like vnStat doesn't support virtual and aliased interfaces, so not all interfaces will show up. [not entirely true]

      #11) match graph legend to interface names [fixed]

      #12) the sortable table doesn't work after you update the graph or change the time period.

      #13) looks like crap on mobile

      #14) package needs to remove cron/config on uninstall

      1 Reply Last reply Reply Quote 0
      • M
        MyKroFt last edited by

        Is there any way it can save the last graph settings for each of the 4 output options?

        Thanks
        Myk

        1 Reply Last reply Reply Quote 0
        • jdillard
          jdillard last edited by

          Ah no, I forgot about that. I haven't figured out a good way to save config options for this package just yet. I'll take another look at that as well.

          1 Reply Last reply Reply Quote 0
          • M
            MyKroFt last edited by

            also

            On the daily stats, can we have to option to reverse the order, newest on top - so you dont have to scroll to the bottom of the list.  Thats the way the old vnstat2 did it.

            Thanks again for this package!

            Myk

            1 Reply Last reply Reply Quote 0
            • M
              MyKroFt last edited by

              @jdillard:

              Ah no, I forgot about that. I haven't figured out a good way to save config options for this package just yet. I'll take another look at that as well.

              one of my docker containers, rutorrent front end, saves selected columns, width, columns sortings etc via a json file that is save - maybe that is a simple option

              or maybe something on the user/browser side?

              Thanks
              Myk

              1 Reply Last reply Reply Quote 0
              • jdillard
                jdillard last edited by

                @MyKroFt:

                On the daily stats, can we have to option to reverse the order, newest on top - so you dont have to scroll to the bottom of the list.

                I'll see about reversing the table order for all of them. Also, possibly making it a sortable table.

                1 Reply Last reply Reply Quote 0
                • MikeV7896
                  MikeV7896 last edited by

                  Installed 2.3.2 just for this… was going to try and wait until release, but now you did this. :)

                  So far in favor of all mentioned items... save default graph settings, sortable table, and most recent period at the top.

                  My comment is that it seems to be showing an average speed for the period rather than the actual amount of data. Maybe make that an option as well? Some people might care about the average speed for the period... but most are more interested in the actual amount of data transferred instead.

                  Additionally, it would be good to see vnstatd added to the Services widget on the dashboard, so we can make sure it's running and properly tracking traffic.

                  The S in IOT stands for Security

                  1 Reply Last reply Reply Quote 0
                  • jdillard
                    jdillard last edited by

                    @virgiliomi:

                    My comment is that it seems to be showing an average speed for the period rather than the actual amount of data. Maybe make that an option as well? Some people might care about the average speed for the period… but most are more interested in the actual amount of data transferred instead.

                    What leads you to believe it is average? It should be totals.

                    @virgiliomi:

                    Additionally, it would be good to see vnstatd added to the Services widget on the dashboard, so we can make sure it's running and properly tracking traffic.

                    It uses a cron job not vnstatd, I thought it would be more reliable. You can see the cron job if you install the cron package. I also haven't added the functionality that lets you change how often the cron job runs yet.

                    1 Reply Last reply Reply Quote 0
                    • MikeV7896
                      MikeV7896 last edited by

                      @jdillard:

                      @virgiliomi:

                      My comment is that it seems to be showing an average speed for the period rather than the actual amount of data. Maybe make that an option as well? Some people might care about the average speed for the period… but most are more interested in the actual amount of data transferred instead.

                      What leads you to believe it is average? It should be totals.

                      I've been watching a 5Mbps+ video stream for the past 45 minutes… yet for this hour (25 minutes so far), it's not showing anywhere near the amount of traffic showing it should be showing... Unless the numbers are really megabytes? The unit on the graph says "bits per second"... that's clearly not right.


                      The S in IOT stands for Security

                      1 Reply Last reply Reply Quote 0
                      • jdillard
                        jdillard last edited by

                        I believe I have the units represented wrong, it looks like it should be KiB/MiB/etc. Thanks for catching that!

                        1 Reply Last reply Reply Quote 0
                        • M
                          MyKroFt last edited by

                          ya, mine shows today aprox 10mb, old graph shows right at 9.xg

                          Myk

                          1 Reply Last reply Reply Quote 0
                          • MikeV7896
                            MikeV7896 last edited by

                            Ok, that's what screwed me up then. :)

                            As for cron versus service… It's not a big deal to me either way... I just wasn't sure how it was running. I did quickly pick up that it was updating every 5 minutes though.

                            The S in IOT stands for Security

                            1 Reply Last reply Reply Quote 0
                            • M
                              MyKroFt last edited by

                              Also on the side of the graphs - Traffic (Bits per Second) - does not work - its total traffic - no speed

                              Myk

                              1 Reply Last reply Reply Quote 0
                              • jdillard
                                jdillard last edited by

                                Yea, I think that was an artifact of using the Status > Monitoring as a starting point. I'll try and get some of these fixes in tomorrow.

                                1 Reply Last reply Reply Quote 0
                                • MikeV7896
                                  MikeV7896 last edited by

                                  Just wondering about why PHP is listed as a dependency for the package. Isn't PHP part of the whole pfSense ecosystem anyway? Or is that for future planning if it goes away? Does this potentially cause problems if PHP is upgraded by pfSense core, but the package isn't updated to reflect the new version of PHP, when someone goes to install the package?

                                  The S in IOT stands for Security

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    athurdent last edited by

                                    @jdillard:

                                    It uses a cron job not vnstatd, I thought it would be more reliable.

                                    Why is the vnstatd daemon unreliable on FreeBSD? Had it running for years on a Linux box with no problems at all.

                                    1 Reply Last reply Reply Quote 0
                                    • MikeV7896
                                      MikeV7896 last edited by

                                      @athurdent:

                                      @jdillard:

                                      It uses a cron job not vnstatd, I thought it would be more reliable.

                                      Why is the vnstatd daemon unreliable on FreeBSD? Had it running for years on a Linux box with no problems at all.

                                      He didn't say that it was unreliable on FreeBSD… just that he thought it would be more reliable as a cron job. Using the cron job, it's only running every 5 minutes rather than a constantly running process that could be stopped at any time leaving holes in the data.

                                      I see pluses and minuses to both ways... I personally have no preference. I just want it to work and accurately track my usage. It doesn't need to be minute-by-minute... updating every 5 minutes is fine by me. Most of what I care about (and probably most people) is monthly usage more than anything.

                                      The S in IOT stands for Security

                                      1 Reply Last reply Reply Quote 0
                                      • R
                                        robi last edited by

                                        Hmmm…

                                        Is this topic about the new graphing system on 2.3 which replaced RRD? Or is it an addition to that? Or is it just a similar graphing to what's already built-in?
                                        Why as a separate package?

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          athurdent last edited by

                                          @virgiliomi:

                                          He didn't say that it was unreliable on FreeBSD… just that he thought it would be more reliable as a cron job. Using the cron job, it's only running every 5 minutes rather than a constantly running process that could be stopped at any time leaving holes in the data.

                                          Cron jobs are run by a daemon that could also be stopped at any time ;) So the question remains, why not use the vnstat daemon? Don't get me wrong, I just want to understand it, don't want to offend anyone :)

                                          1 Reply Last reply Reply Quote 0
                                          • jdillard
                                            jdillard last edited by

                                            @virgiliomi:

                                            Just wondering about why PHP is listed as a dependency for the package. Isn't PHP part of the whole pfSense ecosystem anyway? Or is that for future planning if it goes away? Does this potentially cause problems if PHP is upgraded by pfSense core, but the package isn't updated to reflect the new version of PHP, when someone goes to install the package?

                                            I had a line in the Makefile that didn't need to be there. I'll be taking it out and it should remove the PHP dependancy.

                                            1 Reply Last reply Reply Quote 0
                                            • jdillard
                                              jdillard last edited by

                                              @athurdent:

                                              @jdillard:

                                              It uses a cron job not vnstatd, I thought it would be more reliable.

                                              Why is the vnstatd daemon unreliable on FreeBSD? Had it running for years on a Linux box with no problems at all.

                                              It's not unreliable on FreeBSD, just harder to tie into all the necessary GUI bits last time I looked. Maybe I was looking at it wrong.

                                              1 Reply Last reply Reply Quote 0
                                              • jdillard
                                                jdillard last edited by

                                                @robi:

                                                Hmmm…

                                                Is this topic about the new graphing system on 2.3 which replaced RRD? Or is it an addition to that? Or is it just a similar graphing to what's already built-in?
                                                Why as a separate package?

                                                TL;DR: RRDTool graphing dependancies got too large so we cut them out. Part of that was the function that provided the period totals. Period totals using RRD data aren't that accurate to begin with since they get averaged out over time. vnStat handles the collection of historical traffic totals better and is more accurate, hence the addition. Also, not everyone cares about traffic totals so no need to bundle it in core and having it as a package allows it to be revved separate from core.

                                                1 Reply Last reply Reply Quote 0
                                                • R
                                                  robi last edited by

                                                  @jdillard:

                                                  Period totals using RRD data aren't that accurate to begin with since they get averaged out over time. vnStat handles the collection of historical traffic totals better and is more accurate, hence the addition. Also, not everyone cares about traffic totals so no need to bundle it in core and having it as a package allows it to be revved separate from core.

                                                  In that case, replacing Period totals with vnStat in the future would make many people happy.

                                                  1 Reply Last reply Reply Quote 0
                                                  • G
                                                    grandrivers last edited by

                                                    under interfaces 2 of my 4 interfaces dont show but if i click the blank area datas there

                                                    pfsense 2.4 super micro A1SRM-2558F
                                                    C2558 8gig ECC  60gig SSD
                                                    tripple Wan dual pppoe

                                                    1 Reply Last reply Reply Quote 0
                                                    • T
                                                      timotl last edited by

                                                      I don't see anything in the interface selection.
                                                      I can click through the blank space and it does follow
                                                      I'm using VLANS on a single LAGG of two physical interfaces.
                                                      The totals do show up for my physical and ovpn interfaces, but none of those are assigned labels.

                                                      1 Reply Last reply Reply Quote 0
                                                      • jdillard
                                                        jdillard last edited by

                                                        Ok I'll have to look into the code that generates the interface names, I have a feeling it is making the option element, but not populating it with content. Grandrivers, do you happen to know what makes those 2 interfaces that aren't showing up different from the others (based on type, etc)?

                                                        1 Reply Last reply Reply Quote 0
                                                        • G
                                                          grandrivers last edited by

                                                          one is a pppoe WAN  the other is a non assigned opt interface but had not noticed the neither of my HE ipv6 tunnels show up at all might be nice to have those as well

                                                          pfsense 2.4 super micro A1SRM-2558F
                                                          C2558 8gig ECC  60gig SSD
                                                          tripple Wan dual pppoe

                                                          1 Reply Last reply Reply Quote 0
                                                          • MikeV7896
                                                            MikeV7896 last edited by

                                                            I wonder if there's limits to what vnstat can track usage for… because interfaces that seem to be showing up are physical interfaces, while non-physical ones - VLANs, tunnels, etc. - don't seem to be showing up.

                                                            Also, the interfaces selected show their pfSense names (WAN, LAN, GUEST), but in the graph and table they're identified as the actual physical interface (igb0, igb1, igb2). I also have a blank entry, which is my unassigned igb3 interface. This is part of what leads me to the above suggestion that vnstat may only be able to track physical interfaces.

                                                            The S in IOT stands for Security

                                                            1 Reply Last reply Reply Quote 0
                                                            • jdillard
                                                              jdillard last edited by

                                                              Just bumped the version to 0.2. It should include misc fixes as well as fixes for:

                                                              #6) The correct units and axis label. Although, I have to figure out how to make a custom axis format as it uses binary prefixes and not the SI prefixes.

                                                              #7) PHP shouldn't be listed as a dependency anymore. This may take some more kicking to get to work.

                                                              #8) Don't allow TOP 10 when multiple interfaces are selected.

                                                              1 Reply Last reply Reply Quote 0
                                                              • jdillard
                                                                jdillard last edited by

                                                                @virgiliomi:

                                                                I wonder if there's limits to what vnstat can track usage for… because interfaces that seem to be showing up are physical interfaces, while non-physical ones - VLANs, tunnels, etc. - don't seem to be showing up.

                                                                Also, the interfaces selected show their pfSense names (WAN, LAN, GUEST), but in the graph and table they're identified as the actual physical interface (igb0, igb1, igb2). I also have a blank entry, which is my unassigned igb3 interface. This is part of what leads me to the above suggestion that vnstat may only be able to track physical interfaces.

                                                                It could also have to do with the get_interface_list() function I am using. I might need to use a different function or a combination of them, I am in the process of testing get_configured_interface_with_descr().

                                                                Edit: Bad news bears, this is what the vnStat man page says:

                                                                Virtual and aliased interfaces cannot be monitored because the kernel doesn't provide traffic information for that type of interfaces. Such interfaces are usually named eth0:0, eth0:1, eth0:2 etc. where eth0 is the actual interface being aliased.

                                                                1 Reply Last reply Reply Quote 0
                                                                • jdillard
                                                                  jdillard last edited by

                                                                  For those that have blank interface issues can you view the page source (or inspect the element if you know how) and grab the HTML for the select element with id="interfaces".

                                                                  It should look something like:

                                                                   <select class="form-control" id="interfaces" name="interfaces" multiple=""><option value="em0" selected="">wan</option></select>
                                                                  
                                                                  1 Reply Last reply Reply Quote 0
                                                                  • D
                                                                    Dazog last edited by

                                                                    As requested:

                                                                    <select class="form-control" id="interfaces" name="interfaces" multiple=""><option value="igb3" selected="">lan</option></select>

                                                                    1 Reply Last reply Reply Quote 0
                                                                    • G
                                                                      grandrivers last edited by

                                                                      <select class="form-control" id="interfaces" name="interfaces" multiple=""><option value="igb1" selected="">opt1</option><option value="igb2" selected="">lan</option></select>

                                                                      pfsense 2.4 super micro A1SRM-2558F
                                                                      C2558 8gig ECC  60gig SSD
                                                                      tripple Wan dual pppoe

                                                                      1 Reply Last reply Reply Quote 0
                                                                      • M
                                                                        MorpheusRO last edited by

                                                                        <select class="form-control" id="interfaces" name="interfaces" multiple=""><option value="igb0" selected="">wan</option><option value="igb1" selected="">opt3</option></select>

                                                                        1 Reply Last reply Reply Quote 0
                                                                        • jdillard
                                                                          jdillard last edited by

                                                                          Thanks! I was finally able to replicate the blank interface issue, looking into it now.

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • A
                                                                            athurdent last edited by

                                                                            Will the vnstat database be included in the XML backup similar to RRD?

                                                                            1 Reply Last reply Reply Quote 0
                                                                            • jdillard
                                                                              jdillard last edited by

                                                                              In theory yes, but I haven't messed with that code yet.

                                                                              1 Reply Last reply Reply Quote 0
                                                                              • jdillard
                                                                                jdillard last edited by

                                                                                I just pushed version 0.4 and it should fix:

                                                                                #6) Units are more accurate and consistent. it was mostly fixed before, but there were a couple small errors.

                                                                                #10a) Interfaces shouldn't show up blank anymore, but it is recommended to reset the graphing data after updating the package.

                                                                                #10b) not really a fix, but it doesn't look as bad as previously thought. I can pull in VLANs, IPSec, and PPPOE. More interface types will need to be tested. If you want to see which interfaces vnStat can capture data for you can run the command 'vnstat –iflist'.

                                                                                1 Reply Last reply Reply Quote 0
                                                                                • M
                                                                                  MyKroFt last edited by

                                                                                  still have a blank line - it used to show as my LAN interface

                                                                                  Myk


                                                                                  1 Reply Last reply Reply Quote 0
                                                                                  • MikeV7896
                                                                                    MikeV7896 last edited by

                                                                                    I don't think that the update was released last night yet… I had updated my pfSense build maybe 15-20 minutes before your post time and the package manager still showed the .3 version of this package.

                                                                                    The S in IOT stands for Security

                                                                                    1 Reply Last reply Reply Quote 0
                                                                                    • First post
                                                                                      Last post