Request: 95th percentile calculation in RRD

clarknova

Yeah, that would be the horizontal red line through the linked graph, below which 95% of the samples would fall.

AhnHEL

According to that link you posted, it states billing in that format is more than twice as expensive. If you have a possibility of moving to that billing format, I would think you would opt out if you could.

clarknova

It states that it's more than twice the cost of average-based billing. Guess what billing method nobody sells around here.

The available alternative to 95th percentile billing is fixed rate, which is essentially 100th percentile. With that model, you either go big to handle your peak times or you go small to be affordable, and your users feel the congestion. At least 95th percentile billing gives some relief compared to the flat model, which is where many of us are at now.

Suppose that linked RRD graph was yours and typical. What are your choices?

1. 95th percentile billing, where you pay for 2.62 mbit
2. flat rate billing, where you pay for 6.28 mbit and experience no congestion
3. flat rate billing, where you pay for something less, say 4 mbit, and experience congestion sometimes.

In my case, I have enough users that I don't see many outliers on my rrd graph, where any delta between option 2 and option 3 is manifest in significant increases in congestion time.

joebarnhart

clarknova is right. The billing model is imposed by the seller of bandwidth here, it's not a choice we can elect to take or not. All of our "wholesale" transport is priced this way. I have only one question, tho… Wouldn't we need to input the "window" over which the 95th percentile is calculated? In order to match our billing?

cmb

there is a feature request open on this.
http://redmine.pfsense.org/issues/796

@joebarnhart:

I have only one question, tho… Wouldn't we need to input the "window" over which the 95th percentile is calculated? In order to match our billing?

Yes, though I believe custom periods with RRD can accomplish that

bardelot

Simple patch to include the feature, no configuration or anything like that. Use and modify as you wish.

rrdcolors.inc.txt
status_rrd_graph_img.txt

databeestje

Cool, I'll see if I can get this worked in.

clarknova

Cool. Thank you. I can't wait to try it.

bardelot

fixed the SI magnitude. %sb/s instead of %Sb/s in both GPRINT lines.

databeestje

Code committed to the git repo. It should appear in snapshots half a day from now, please test.

podilarius

Thank you for this. It looks really good. I have to revert back to an older build and I will miss the functionality.

Good work.

Podilarius

clarknova

I finally had a chance to update and try this out. Beautiful. Thank you both and anyone else who contributed to having this included.

clarknova

One minor problem, it appears the "out" line for 95th %ile is drawn on the wrong side of 0 on the x axis.

95th.png_thumb

databeestje

oops

geeknik

I'm pretty sure this commit broke my RRD traffic graphs. Any way to fix this w/o deleting all of my data history?

http://forum.pfsense.org/index.php/topic,34715.0.html

status_rrd_graph_img.php.png
status_rrd_graph_img.php.png_thumb

databeestje

the color is missing for the theme, nothing to do with the data.

Bill48105

@geeknik:

I'm pretty sure this commit broke my RRD traffic graphs. Any way to fix this w/o deleting all of my data history?

http://forum.pfsense.org/index.php/topic,34715.0.html

Howdy,
Yeah I see the same thing when I choose traffic. Logs show:

php: /status_rrd_graph_img.php: Failed to create graph with error code 1, the error is: ERROR: Unknown function string '95,PERCENT,-1,*' in VDEF 'opt1-out_bits_95'/usr/bin/nice -n20 /usr/local/bin/rrdtool graph /tmp/opt1-traffic.rrd-4year.png --start 1175097181 --end 1301586781 --vertical-label "bits/sec" --color SHADEA#eeeeee --color SHADEB#eeeeee --title "`hostname` - TCWIRELESS :: Traffic - 4 years - 1 day average" --height 200 --width 620 DEF:opt1-in_bytes_pass=/var/db/rrd/opt1-traffic.rrd:inpass:AVERAGE DEF:opt1-out_bytes_pass=/var/db/rrd/opt1-traffic.rrd:outpass:AVERAGE DEF:opt1-in_bytes_block=/var/db/rrd/opt1-traffic.rrd:inblock:AVERAGE DEF:opt1-out_bytes_block=/var/db/rrd/opt1-traffic.rrd:outblock:AVERAGE CDEF:"opt1-in_bits_pass=opt1-in_bytes_pass,8,*" CDEF:"opt1-out_bits_pass=opt1-out_bytes_pass,8,*" CDEF:"opt1-in_bits_block=opt1-in_bytes_block,8,*" CDEF:"opt1-out_bits_block=opt1-out_bytes_block,8,*" CDEF:"opt1-in_bytes=opt1-in_bytes_pass,opt1-in_bytes_block,+" CDEF:"opt1-out_byte

Seems it doesn't like something about the option string. I'm not familiar enough with RRD to know for sure why or how to fix it.
As a quick fix I edited /usr/local/www/status_rrd_graph_img.php to make 'out' the same as 'in' (removing ,$multiplier,*) to end up with:


        $graphcmd .= "VDEF:\"$curif-in_bits_95=$curif-in_bits,95,PERCENT\" ";
        $graphcmd .= "VDEF:\"$curif-out_bits_95=$curif-out_bits,95,PERCENT\" ";

I'm guessing that will break the scaling of the 95th line but hey it works for now at least. :D
Bill
Ps. Great work to everyone involved working on pfsense! In process of migrating from m0n0wall (need multiwan mostly) and aside from a few issues, (such as this graphing thing and a PPPoE/interface issue I need to report when I get time), it has been working great.

clarknova

Ok, the out line is on the correct (bottom) side of the graph now, but it looks like a 5th percentile calculation instead of a 95th.

Screenshot.png_thumb

dzeanah

I just want to chime in and say that I really appreciate the addition of this feature.

Really nice for those of us in datacenters who get billed at the 95% rate.