PfSense 2.3 new Traffic Graph doesn't include total Traffic in time period
-
@cmb:
Because it's a function of rrdtool graph, which has absurdly large dependencies in latest rrdtool. jdillard's working on a replacement with vnstat.
It worked.
-
+1 !!
Please bring it back!! -
The data it displays is not even close to correct now, its not just total inpass is no longer a sum of all data for the period
ntopng and rrd on the same system are way off, ntopng has some issue where you must select traffic even though its the default otherwise the min and max pull reported are averaged but ntopng is correct
rrd still needs to be fixed, it was very useful in proving packet loss and latency issues to ISPs, countless times we have been able to get ISP to escalate tickets when they were telling us "no issue"
ntop is over the head of most of the ISP support people on packet loss and it does not do latency at all (ntopng lacks nprobe)
in case of Time Warner, their high up techs have access to rrd tools monitoring the modem, and the rrd reporting use to match, now it does not
-
I don't believe this.
The one reason I switched to pfsense was cause it didn't loose total traffic on reboot compared to my plastic router, and now don't even show it ?
Just when Comcast roll it's 1TB data cap for home users and now somebody decide it's not no longer relevant feature to have, really ? After so much advanced features pfsense has, it can't even show you if you are over you data cap ? I am very disappointed, I even support the pf-sense project for future development.
If we are willing to revert to much older version of pf-sense just to get this crucial functionality back(disregarding any other changes for the last year as not as important as this), how come this is happening.
P.S. This is so bad. I can only speculate why , but I guess in something internally changed significant in pf-sense team, perhaps the whole thing switched hands. I had my 2.2.6 running for a year, and finally decide to upgrade. What a shame. I am glad I didn't donate as I was intending to, but upgrade first.
-
THe RRD_Summary package shows this and last months WAN traffic in the time period and is accurate enough to check how near a usage cap I am. Good enough for me although not exactly the presentation of data which used to be available. Many ISP's will send an E-mail when usage is near the cap.
J
-
This is so bad. I can only speculate why.
You were told exactly why:
"we cut the graphing utility out because it was going to double the total size of the pfSense image."
Double. Just for that one function.
-
THe RRD_Summary package shows this and last months WAN traffic in the time period and is accurate enough to check how near a usage cap I am. Good enough for me although not exactly the presentation of data which used to be available. Many ISP's will send an E-mail when usage is near the cap.
J
Thanks for the suggestion. I already install the package. It does the job till some future update made the package obsolete , but it works for now. Thanks
As far is "Many ISP's will send an E-mail when usage is near the cap." in this case you rather get their router and be done with it, but I don't think pf-sense is about that type of users.
-
This is so bad. I can only speculate why.
You were told exactly why:
"we cut the graphing utility out because it was going to double the total size of the pfSense image."
Double. Just for that one function.
The question is : Do you really believe that ?!?
Functionality to create graphs out the data is already there. We are talking about the functionally to count the data passing through the WAN interface. Is that what is going to double the size of the image ? I don't think one need to be familiar with coding to guess how much code it will take to do that.
-
perhaps you should just install the "Status_Traffic_Totals' package ?
-
perhaps you should just install the "Status_Traffic_Totals' package ?
Thanks for the suggestion.I did that and I'll have to admit is looking and working great.
My only worry is this: Since it's a package now, next few updates of the pfsense itself may broke it's functionality and all the accumulated data going back to the drain. -
The question is : Do you really believe that ?!?
Yes. cmb and jdillard both said the same thing.
pfSense did not change. The rrdtool dependencies did. Why is that so hard for you to believe? Why be so caustic?
Everything is not a conspiracy against you.
-
The question is : Do you really believe that ?!?
Yes. cmb and jdillard both said the same thing.
pfSense did not change. The rrdtool dependencies did. Why is that so hard for you to believe? Why be so caustic?
Everything is not a conspiracy against you.
I could be wrong , but if a small package like Status_Traffic_Totals could bring the functionality back , how come pfsense need to double it's size to have it as a part of pf-sense like before ?!?!?!?!?!
"Everything is not a conspiracy against you."
Thank god it's not just me. If I was the only one seeing purpose of data usage feature , I doubt Status_Traffic_Totals will be written just for me. :-)))
It was a quite outrage for this feature being removed from what I read, and I agree there people who don't even care about it. I just think that the creators of the software should know better than everybody, that's all.P.S. I love to be wrong :-))
-
Here's some news to you: PfSense developers have no control over RRDTool and what its developers do with it and how they see what is an acceptable amount of installed dependencies. RRDTool is fine on a full FreeBSD system where the excessive dependencies don't matter because they would be installed anyway but on pfSense it's a dead end. If you can point out a replacement system that is capable of what RRDTool was and has an acceptable amount of dependencies we are all ears. Anything?
-
I can't suggest anything, but alternative solution that will have the same functionality like the optional package made later on just for that purpose.
I don't understand dynamics behind it, but I can tell for sure when the end result is better or worse. I wish there was a way to update but not upgrade.
-
The old totals from rrdtool were calculated by rrdtool's GRAPH code and they were not accurate. With rrdtool GRAPH code disabled so it only handles the database side of things, there is no way to get those (inaccurate) totals from rrdtool. Now the graphing part is handled client-side in JS and not using rrdtool's GRAPH functionality, which is where the extra dependencies came from. What rrdtool does, basically, is to multiply average traffic transferred over a period attempting to turn that back into a total transferred, but while that is OK in theory in reality it doesn't always come close.
We could have wasted a bunch of time writing code to replicate rrdtool's code to generate inaccurate totals, but it wasn't worth the time. rrdtool loses accuracy over time as it consolidates, normalizes, and averages data and any attempt to extract meaningful totals is only an estimate at best. rrdtool is great at what it does, but it isn't intended for that. There is the rrd summary package which makes an attempt to get some totals but search a bit and you'll find many complaints about its accuracy, too.
The Status_Traffic_Totals package uses vnstat and tracks totals, sidestepping the problems inherent in relying on rrdtool to do such things.
So we were faced with a few problems:
- rrdtool had a massive shift in dependencies that made newer versions huge and caused it to pull in parts of X11/xorg that we didn't feel comfortable with in the base install.
- We could no longer keep using the outdated-but-small rrdtool version that had been deprecated, due to security concerns and the fact that we needed features from newer rrdtool versions for other purposes, such as ntopng.
- Complaints about inaccurate totals on graphs, which were valid concerns
- rrdtool graphs were looking out-of-place/dated compared to the rest of the redesigned GUI
So we could push forward with a new rrdtool, increasing the size of the install image and base OS by a significant amount, and still be stuck with dated graphs that have inaccurate totals, or we could do what we did and use rrdtool to manage the data with client-side graph handling, and opt to move the totals to a more appropriate package that was better suited to the task.
-
The old totals from rrdtool were calculated by rrdtool's GRAPH code and they were not accurate. With rrdtool GRAPH code disabled so it only handles the database side of things, there is no way to get those (inaccurate) totals from rrdtool. Now the graphing part is handled client-side in JS and not using rrdtool's GRAPH functionality, which is where the extra dependencies came from. What rrdtool does, basically, is to multiply average traffic transferred over a period attempting to turn that back into a total transferred, but while that is OK in theory in reality it doesn't always come close.
We could have wasted a bunch of time writing code to replicate rrdtool's code to generate inaccurate totals, but it wasn't worth the time. rrdtool loses accuracy over time as it consolidates, normalizes, and averages data and any attempt to extract meaningful totals is only an estimate at best. rrdtool is great at what it does, but it isn't intended for that. There is the rrd summary package which makes an attempt to get some totals but search a bit and you'll find many complaints about its accuracy, too.
The Status_Traffic_Totals package uses vnstat and tracks totals, sidestepping the problems inherent in relying on rrdtool to do such things.
So we were faced with a few problems:
- rrdtool had a massive shift in dependencies that made newer versions huge and caused it to pull in parts of X11/xorg that we didn't feel comfortable with in the base install.
- We could no longer keep using the outdated-but-small rrdtool version that had been deprecated, due to security concerns and the fact that we needed features from newer rrdtool versions for other purposes, such as ntopng.
- Complaints about inaccurate totals on graphs, which were valid concerns
- rrdtool graphs were looking out-of-place/dated compared to the rest of the redesigned GUI
So we could push forward with a new rrdtool, increasing the size of the install image and base OS by a significant amount, and still be stuck with dated graphs that have inaccurate totals, or we could do what we did and use rrdtool to manage the data with client-side graph handling, and opt to move the totals to a more appropriate package that was better suited to the task.
I appreciate you explaining the situation in details. What you are saying it does make sense and indeed changes the prospect of things. It's seems it was not just overlooked feature, that "just felt of the table" and nobody bother to pick it up.
As I said I love to wrong, especially since I love pfsense I was hoping that the pf-sense project go strong in the same direction for many years in a future , before it go south like everything else.:-)
But I am glad to find out that day is not today , therefore I'll continue to contribute to this great project.I don't know how good the JS graphing will feel, but I hope the overhaul will not bring new security or reliability challenges.
-
I'd add an old issue: with a 2 DSL in fail-over/load balancing mode configuration, monitoring data are confused (since pfsense 2.2.4)
Please compare the Dashboard (real traffic) with monitoring graphs (confused) below.
I think, that should not depend from displaying tools but from raw collected data…
-
I was also very surprised and anxious to not find the totals.
Thank you all for requesting, developing and overall improving the project.I suppose the historical data is imposible to recover, is it? as a one-time csv or something.
Anyway, my question is: How does vnStat affect performance? Various articles mention it's very lightweight. What are your thoughts.
Also, I don't understand the usage of enabling the graphics, as it wont show anything without it activated.
I also deactivated and reactivated graphics and the historical data seems to have been lost. Is this expected or a bug?Thank you again
-
Hi,
I installed this traffic totals script (or plugin?) and now on the latest PFsense it doesnt work & I get an error:
Error: {"vnstatversion":"1.15","jsonversion":"1","interfaces":[Error: Database load failed even when using backup (No such file or directory). Aborting.
How do I uninstall this?
Thanks,
Rich
