RRD graphs throughput but not totals

icemanncsu

Anything I can do?

charliem

Anything I can do?

Take a look at the shell code that updates the rrd database: /var/db/rrd/updaterrd.sh. In there you can see that the basic data is generated by parsing the output of a few different pfctl commands; and the one you are most interested in is "/sbin/pfctl -vvsI -i <your_interface>".

From one of my systems:

[2.2.1-DEVELOPMENT][root@pfsense.localdomain]/root: date
Mon Mar  9 15:01:54 EDT 2015
[2.2.1-DEVELOPMENT][root@pfsense.localdomain]/root: /sbin/pfctl -vvsI -i bge1
bge1
        Cleared:     Fri Mar  6 10:54:20 2015
        References:  54
        In4/Pass:    [ Packets: 14644508           Bytes: 19738865157        ]
        In4/Block:   [ Packets: 84578              Bytes: 10523842           ]
        Out4/Pass:   [ Packets: 6882554            Bytes: 827972742          ]
        Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
        In6/Pass:    [ Packets: 2638425            Bytes: 3263746038         ]
        In6/Block:   [ Packets: 648                Bytes: 505363             ]
        Out6/Pass:   [ Packets: 1605617            Bytes: 206398431          ]
        Out6/Block:  [ Packets: 0                  Bytes: 0                  ]

I would first check the output of that command, once before a big transfer, and once after it. If pfctl output rolls over or otherwise screws up, best you can do then is report a bug with a method to reproduce the problem.

If pfctl output looks good, ie no rollover, then the problem lies in the awk code that parses the pfctl output, or in the rrd database structure (32bit integers?), or in the code that calculates the maximum, average, 95th percentile, etc. The code that generates the page lives in /usr/local/www/status_rrd_graph.php. I assume the period values are calculated in there, but I have not looked into it.</your_interface>

churchi

Thanks Charliem for your suggestions.

I have had the same issues in another thread. https://forum.pfsense.org/index.php?topic=90726.0

Charliem, do you know if 64bit counters are being used in pfSense RRD graphs? If they are 32bit ones, then i think we may have found the issue.

I have had this before when i have used cacti to graph higher 100MB+ links.

Whats the easiest way to confirm if the counters are 32bit or 64bit?

Have you been able to look into this any more?

charliem

@churchi:

Charliem, do you know if 64bit counters are being used in pfSense RRD graphs? If they are 32bit ones, then i think we may have found the issue.

AFAIK it follows the architecture of pfSense: 32bit rrd for i386 pfSense, and 64bit rrd for amd64 pfSense. That said, rrdtool is supposed to detect a wrap for the counter data type and compensate accordingly (counter data types are not allowed to decrease; if a decrease is seen then a wrap is assumed).

Are you on 32bit pfSense? Remember that php is either 32 or 64 bit too, so there could be an overflow in the calculations in /usr/local/www/status_rrd_graph_img.php, if you are on 32bit.

churchi

Ok great thanks.

I am running the 64bit version of pfsense.

2.2.1-RELEASE (amd64)
built on Fri Mar 13 08:16:49 CDT 2015
FreeBSD 10.1-RELEASE-p6

That said, whats the easiest way to see if my rrd graphs are in 32bit or 64bit counters? and if they are 32bit can they be upgraded to 64bit graphs?

Can i just look at the .rrd file itself or do i have to look at the php code to check?

BTW, i have upgraded this system on that same architecture from previous 2.x releases. Hopefully that shouldn't matter with the way the rrd graphs were created.

charliem

@churchi:

That said, whats the easiest way to see if my rrd graphs are in 32bit or 64bit counters?

They are certainly 64bit if you are on amd64 pfsense. To verify use this:```

[2.2.1-DEVELOPMENT][root@pfsense.localdomain]/var/db/rrd: file ./wan-traffic.rrd
./wan-traffic.rrd: RRDTool DB version 0003 64bit aligned little-endian 64bit long (alpha/amd64/ia64)


And since php is 64 bit as well, the problem may lie elsewhere.  Maybe there's an issue with the rrd version?  By default pfSense 2.2 comes with rrdtool v1.2.30, which is quite old (last copyright notice 2008).  I noticed that the ntopng package includes a more recent version of rrdtool, 1.4.8 (last copyright 2013).  The rrdtool changelog is huge, so maybe some change in between has fixed the issue.  But to look any further needs more time than I have, sorry.

Perhaps you should file a bug in redmine (search first to be sure there's nothing there for this already), and make your rrd database available to the developers.  You can also point to this thread.

churchi

Cheers thanks Charliem.

Here is my output:
[2.2.1-RELEASE][root@server]/var/db/rrd: file ./wan-traffic.rrd
./wan-traffic.rrd: RRDTool DB version 0003 64bit aligned little-endian 64bit long (alpha/amd64/ia64)
[2.2.1-RELEASE][root@server]/var/db/rrd:

So it looks good to me. Looks like i have 64bit counters in place already. hmm.

So with this ntopng package, if i install this will it update the default RRD on the system?

Maybe i need to ask to have the RRD version upgraded to a later release in which some issues may be fixed.

churchi

Hi charliem,

i have installed ntopng and that has not made a difference to the graphs. I tested a 1GB download and still the graph increased but the total down the bottom is still broken.

I guess i will need to log a bug on the tracker.

charliem

@churchi:

So with this ntopng package, if i install this will it update the default RRD on the system?

No, it is installed in parallel. Each package includes its own set of binaries and libraries, so versions can differ without causing problems with the base install.

The newer version is /usr/pbi/ntopng-amd64/local/bin/rrdtool. If you are comfortable coding and experimenting, ~~you could edit a single line in /usr/local/www/status_rrd_graph_img.php, about line 144, to point to the newer rrdtool~~. But be ready for things to break! Try it in a test VM first.

Edit: Don't do that, it breaks the font rendering. The newer rrdtool is not a drop-in replacement!

churchi

ok cool thanks for the clarification and the explanation as to why the other package didnt update the base system.

So i guess i should just log a bug now and see if we can get the pfSense team to see what the issue is and maybe update the RRD version in the base image?

Do you think this is the best next step?

Do you think the pfsense maintainers/admins will be likely to read this thread?

charliem

Do you have traffic shaper enabled? The rrd graph generation code does some data validation before plotting, discarding data values that lie outside the range of 0 to $speedlimit. If the traffic shaper is enabled, the value for $speedlimit is taken from bandwidth defined for the queue, otherwise it's set to 12,500,000

If you increased your wan bandwidth significantly, you need to adjust your interface bandwidth definitions in the traffic shaper.

To check what $speedlimit is currently set to, you can add this to /usr/local/www/status_rrd_graph_img.php around line 253:

$speedlimit = ($upstream + $downstream); <===== Existing line
log_error("speedlimit $speedlimit, upstream $upstream, downstream $downstream"); <===== Add this line

Then view an rrd graph and check the result with:

[2.2.1-DEVELOPMENT][root@pfsense.localdomain]/usr/local/www: clog /var/log/system.log | grep speedlimit |tail
Mar 21 09:28:15 pfsense php-fpm[97229]: /status_rrd_graph_img.php: sspeedlimit 248320, upstream 124160, downstream 124160

Don't forget to go back and comment or remove the debugging line when you're done.

churchi

Sorry for the delay in getting back to you.

I have done what you asked and here is the results.

Mar 24 23:04:01 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:01 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[823]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[823]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:04:02 gw1 php-fpm[89472]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400

churchi

Ok so i did another test. When my Limiter changed at 11:30pm to 950Kbps up from 450Kbps on the earlier test, this is the results.

Mar 24 23:57:38 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:38 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:38 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400
Mar 24 23:57:39 gw1 php-fpm[88321]: /status_rrd_graph_img.php: speedlimit 460800, upstream 230400, downstream 230400

No change.

So it looks like the speed limit is being set somewhere. Where do you think that limit will be set?

How can i go about resetting this to correct values so that the data is not purged/disguarded by the graphing?

The actual graphs are fine, just the numeral calculations down the bottom of the graph that are not adding up correctly.

Thanks.

charliem

@churchi:

Ok so i did another test. When my Limiter changed at 11:30pm … No change.

Not the limiter, but the traffic shaper I think.

So it looks like the speed limit is being set somewhere. Where do you think that limit will be set?

How can i go about resetting this to correct values so that the data is not purged/disguarded by the graphing?

$speedlimit is calculated in the very file you just modified. If you suspect it could be causing an issue, just set it to 12500000 and try it, though the limits you posted should be OK.

The alternative is for you to find the rrdtool man pages, and try to analyze the RPN expressions in the "$graphcmd" stanzas in that file (/usr/local/www/status_rrd_graph_img.php).

With your graphs and the others posted here, it seems that the average is being calculated correctly. That is: (printed average throughput) * (time span of graph) == (about what you think is the correct total traffic for the period). Can you confirm that?

If you really need to generate a periodic throughput summary, I'd suggest you script up an alternative using the pfctl command I listed early in the thread, rather than going through the web interface.

churchi

Thanks Charlie.

$speedlimit is calculated in the very file you just modified. If you suspect it could be causing an issue, just set it to 12500000 and try it, though the limits you posted should be OK.
Where abouts in the .php file do i manually set the 12500000 to?

And where is it automatically getting the speedlimit 460800 from? I have not set this anywhere in my pfSense config at all? Seems like a random number for me. Then it is splitting it equally to upstream and down stream.

Thanks.

churchi

Hi Charlie,

i have checked out my other graphs, like the LAN interface and other VLAN interfaces and i can see the same issue. There is a speed limit set somewhere in my config that has affected the RRD reporting on the graphs.

Here is another example.

![pfsense WAN traffic pic -03.png](/public/imported_attachments/1/pfsense WAN traffic pic -03.png)
![pfsense WAN traffic pic -03.png_thumb](/public/imported_attachments/1/pfsense WAN traffic pic -03.png_thumb)

charliem

@churchi:

Where abouts in the .php file do i manually set the 12500000 to?

Honestly, if you cannot follow the code in that file then you shouldn't be poking around in it; sorry for the suggestions.

File a bug.

icemanncsu

Can someone please open a bug on this… Us fiber users obviously exceed the normal limits on the WAN interface.

Harvy66

So again, the "period" in-pass is obviously not correct. The claimed 224.78Mb/s over the 8 hour period is about 750GB, not 21.71GB as claimed.

Question for OP. Have you fried a fresh install? Obviously something is wrong, but you seem to be the only person complaining about this issue. Not to say others don't have it, just that you're the only one in here talking about it.

icemanncsu

@Harvy66:

So again, the "period" in-pass is obviously not correct. The claimed 224.78Mb/s over the 8 hour period is about 750GB, not 21.71GB as claimed.

Question for OP. Have you fried a fresh install? Obviously something is wrong, but you seem to be the only person complaining about this issue. Not to say others don't have it, just that you're the only one in here talking about it.

I have installed pfSense several times, and this is the case with each install, this one being v2.2.2.