Q{wan,lan}Root bandwith speed bonded T1

  • I have a bonded T1 which is 3Mbs symetrical.  Since the incoming and outgoing data is not being congested over the same physical connection (assuming I understand the difference between symetric and asymetric), do I still need to lower my q{wan,lan}Root by 10% for overhead?  Or is the overhead because of the shaping itself and not related to the T1? With shaping disabled I get 3Mbs down and up, and most of the posts I have read refer to ADSL connections.

    Thanks in advance for the clarification.

  • Hey Vince,

    A symmetrical link, or symmetrical bandwidth in this case, is essentially a squared total throughput value.  Being squared, that means your traffic is the same up, AND down.  Asymmetrical is almost always the case with residential ISP customers, because most people want fast downloads, and that's what they care about.  So the ISP throttles the upstream.  This is especially important to a provider.  In most cases, they are giving you just enough upstream to give you decent ACK throughput.

    That's essentially what symmetrical/asymmetrical is, when discussing throughput.  This doesn't mean you can't have two symmetrical T1's, each on it's own interface.  It also doesn't mean you can't have a T1 dedicated to up, and a T1 dedicated to down.  If the total throughput value of your capacity is squared, then the link is symmetrical.  Throughput is throughput.  ACKS are ACKS.  Even if it's separated in your network, it will still flow through routers upon routers.  (hops)  Your upstream still has to acknowledge your downstream packets.  (And vice versa)  Just because the link(s) are symmetrical, doesn't mean you shouldn't be "realistic" about specifying your maximum capacity on your root queues.

    So, "pushing the limit" on your ROOT WAN/LAN queue is a bad idea.  There are three reasons for this:

    1.) The most important reason.  ALTQ (the shaper in pfSense), heavily depends on a REASONABLE max capacity values to properly shape your throughput with the queues you create.  A lot of people with severe packet drops on all or most of their queues don't stop to realize this sometimes.  If you "ride the line" or go over your actual max capacity.. by the time you get done creating all of your queues with it being divided up appropriately, it doesn't matter anymore.  RED or no RED, you simply do not have enough REAL throughput to handle the throughput ALTQ thinks it has; so it drops/loses packets.

    2.) In the case that you don't have an ACK queue.  Without an ACK queue, your TCP acknowledgements are being shared.  Not only is this bad practice to begin with, but now you wont have enough bandwidth available for ACKS when your pipe is saturated.  However, even with an ACK queue (and the traffic shaper wizard creates one for you by default), you can still lose/drop acks if the ROOT value is too high.  This is almost always the problem when people are experiencing ACK queue drops.  They either haven't reserved enough for the ACK queue itself, or the root capacity is a little too ambitious.

    3.) Your throughput is (obviously) totally dependent on your ISP.  In most cases, a provider will provide services at X speed.  But it's pretty common these days to assume you are getting 10% less that value.  And in a lot of cases, (especially where residential consumers are concerned) this throughput can be shared and very dynamic.  It can fluctuate drastically, or in most cases… it will almost always vary between 90%-95% of "ISP promised" capacity.  And if your paired T1's are dynamic, (your voice/phone trunks share the available channels), then this gets real ugly.  Because not only is your max capacity dependent on the above; it also depends on how many voice data channels are being used.  Which means your root capacity is a lie.  Which also means your internet bandwidth slows down, and the traffice shaper drops/loses packets when people start using the phones.  In these cases, it's especially intelligent to set your root capacity to values you KNOW you will ALWAYS have available.

    In cases where packets are lost/dropped, it doesn't mean you are dead in the water.  ALTQ is designed to handle this to the best of it's abilities.  It has a lot of help from the nature of the TCP/IP protocol.  But what this does mean; a lot of lost packets, a lot more ECN's sent out, RED forcefully dropping packets at random, and an overall degradation of link throughput.  This can drop an entire state in the firewall, which in turn means a user has to refresh a failed GET request on a website, or redownload a file.  It can definitely get ugly.

    I have a very reliable symmetrical T1 assigned to our WAN interface in pfSense, and created a rule to block all traffic but my own.  I would do this for about 5 minutes every 2 hours, for 8 hours every day.  I would do this during the day and around peak times to get an idea how "reliable" my capacity was.  While traffic to your WAN isn't totally silent, at least packets are being dropped and it gives me a pretty good result on AVERAGING my capacity.  Don't assume that all links are created equal.  If you want reliable throughput, do the tests.

    I found that my average for a T1 1536Kbps link over a week worth of these tests was around 1430Kbps up and down.   That's 7%!!  Pretty close to the golden 10% catch phrase thrown around these days?  :)  So, while my throughput can exceed 1430Kbps up or down, it's not consistent enough to create traffic shaping queues based on it.

    I highly recommend doing tests to get an average for a realistic value.  If you can't do this, or simply don't want to, then start with 10% less your promised capacity.  You wont even notice it.  So while 7% sounds like a lot.. it's really only around 26KBp/s.  You aren't going to miss 26KBp/s out of your theoretical 384KBp/s throughput capacity.

    You'll thank yourself for it.  So will your queues.  And your traffic shaper will be much more efficient and reliable because of it.

    The next step is configuring your queues correctly.  And that's a whole different subject and thread.  ;)

    I hope this helps!  Good luck!

    edit: grammar and missed typos!

  • Wow, thank you very much for such a thorough response.  I have set my root at 2700kbps (3000 - 10%) and so far I have seen no drops in ACK.

  • No problem!  I hope it helps!  One of my favorite speed testing sites is http://www.speedtest.net
    It's very reliable, and it stores results for your IP.  You set your download AND upload to 2700?  This is a good starting point, and you may be able to fine tune it some more by averaging your capacity over time.  Again, if this is a business you are setting this up for, make sure you also know if the phones use the paired T1's.

    A good stress test on your shaper is to ALLOW P2P traffic and attempt to download multiple Ubuntu (for example) torrents.  The goal is to saturate your pipe and to start sending out as many ACKs as possible to make sure you have everything tuned correctly.  You have the added benefit of also testing out other queue reservations, to make sure the saturation isn't bringing everything else down to it's knees.  (Like system related traffic, mail server, web server, vpn server, etc)  P2P traffic is a good way to do this, since you are connected to so many peers with a lot of data coming down.

    Then when you're done, PENALIZE P2P traffic in it's lower priority queue.  :)

Log in to reply