Traffic Shaping Performance issues



  • Hey all,

    So I am brand new to traffic shaping, but wanted to give it a try in order to better manage traffic on my network.

    I went through and filled out the wizard in the traffic shaper, and didn't modify anything else manually.

    The connection bench tests at ~152Mbit  down and 160Mbit  up normally, using Speedtest.net so as recommended in many guides, I set the limits to 97% of this on the first page of the wizard.

    Followed by this I set vonage for VOIP, and reserved 90kbit both up and down for it.  I also added deprioritization for peer to peer networks, enabling all of them, and prioritized game traffic, adding all of them.

    I am familiar with, and understand the concept of QoS, and how you need to create the local limit lower than your total bandwidth in order to be able to control the traffic, and was expecting a slight speed test drop, in line with the 97% values I entered in the wizard, but my speed tests after, with the traffic shaper wizard enabled are abysmal.  I get about 70Mbit/s down, and 102Mbit/s up, much lower than the 97%*152mbit and 97%*160mbit I expected.

    Just to make sure I wasn't CPU limited (which I didn't expect, it's a haswell core dual core, with a base clock of 2.9 ghz and a turbo clock of up to 3.6ghz) I signed on via SSH and looked at the overall CP usage in top while performing the speed test.  The CPI Idle figure never dropped below 87%, which suggests to me I never went above ~13% CPU usage in total.

    Then I though maybe PowerD was to blame, so I tried disabling it and rerunning the test, but it made no difference.

    The system is built around a Supermicro mini-ITX server board with dual intel gigabit network adapters (i210at and i217)  One of them uses the igb driver, and the other uses the em driver (though I haven't been able to figure out which is which).  Not knowing if there would be any advantage to the configuration either way, I randomly set em0 as my WAN and igb0 as my LAN.

    Could the adapters be to blame?  Could toggling various adapter offloads on and off make a difference here?  Hardware offload of QoS (if they support it) is great but not if the hardware offload is too slow, and the CPU can do it faster, right?

    Or am I going about this all wrong?  Are my poor speed test results just the traffic shaper doing it's job, detecting a bandwidth hog and slowing it down?  That can't be the case though, right?  I had thought the way the traffic shaping was supposed to work was to allow even a low priority category to use all of the bandwidth, at times when no one else needs it (and I intentionally disconnected everything else using bandwidth for the test)

    I intentionally over dimensioned this system because I wanted to run QoS on it and I am a bit disappointed right now.  Does anyone have any thoughts about what I might try?

    Much obliged,
    Matt



  • Which version(s) of pfSense?



  • @Nullity:

    Which version(s) of pfSense?

    
    2.3-RELEASE (amd64) 
    built on Mon Apr 11 18:10:34 CDT 2016 
    FreeBSD 10.3-RELEASE 
    
    The system is on the latest version.
    


  • The default buffer sizes are only 50 when you enabled shaping. These are way too small for most 10Mb+ networks. Even when I had a 50Mb connection, my bandwidth was only about 30Mb/s with a buffer of 50, and it was up and down and all over. Be warned, too large of a buffer will add bloat. I recommend just enabling CoDel, which has a large buffer, but fights bloat. /guessing at the issue



  • @Harvy66:

    The default buffer sizes are only 50 when you enabled shaping. These are way too small for most 10Mb+ networks. Even when I had a 50Mb connection, my bandwidth was only about 30Mb/s with a buffer of 50, and it was up and down and all over. Be warned, too large of a buffer will add bloat. I recommend just enabling CoDel, which has a large buffer, but fights bloat. /guessing at the issue

    Thank you, I appreciate this suggestion, and will try it when I get home.

    Which buffer are you talking about though?  Is this a buffer specific to the traffic shaping queue?  Where do I change it?

    Also, What is CoDel, and where is that setting?  I did sopme googling but this was unsatisfiactory.  Do you have any links to good reading material on what CoDel is and how it works?  Is it just an alternative ot HFSC?

    Thanks again,
    Matt



  • @Harvy66:

    Be warned, too large of a buffer will add bloat.

    Also,

    What do you mean by bloat in this context?  Are we just talking about added RAM use?

    My pfSense box has 2GB of RAM, only because I wanted to take advantage of the dual channel performance, so I needed two sticks of RAM, and it turns out you just can't buy anything smaller than a 1GB stick these days :p

    With 2GB of RAM, I've never seen pfSense use more than ~5%.  I think I can probably live with a little RAM bloat if that's the extent of it.



  • @mattlach:

    @Harvy66:

    Be warned, too large of a buffer will add bloat.

    Also,

    What do you mean by bloat in this context?  Are we just talking about added RAM use?

    My pfSense box has 2GB of RAM, only because I wanted to take advantage of the dual channel performance, so I needed two sticks of RAM, and it turns out you just can't buy anything smaller than a 1GB stick these days :p

    With 2GB of RAM, I've never seen pfSense use more than ~5%.  I think I can probably live with a little RAM bloat if that's the extent of it.

    He means bufferbloat: latency caused by oversized network buffers.

    Regarding your original question, if you put in 150Mbit then you should get practically 150Mbit. If you are not, then something is wrong. Honestly, I usually only see this strange behavior when someone is new to pfSense, so I assume misconfiguration, but I could be wrong. Double-check your setting and perhaps show pictures of your queues. Make sure to reset states between queue/firewall changes.

    Plenty of us use pfSense and when we setup a 10Mbit queue, that queue moves 10Mbit. Long-time users would quickly recognize if the values were skewed.



  • @Nullity:

    He means bufferbloat: latency caused by oversized network buffers.

    Regarding your original question, if you put in 150Mbit then you should get practically 150Mbit. If you are not, then something is wrong. Honestly, I usually only see this strange behavior when someone is new to pfSense, so I assume misconfiguration, but I could be wrong. Double-check your setting and perhaps show pictures of your queues. Make sure to reset states between queue/firewall changes.

    Plenty of us use pfSense and when we setup a 10Mbit queue, that queue moves 10Mbit. Long-time users would quickly recognize if the values were skewed.

    Thank you.  I'll do some poking around and post some screenshots.

    While I am not new to pfSense at all (been using it since ~2010) I AM new to traffic shaping.  I tried to set it up once a few years back, but it got complicated and I gave up.

    While I understand the basics, I've always gotten myself confused by how the firewall rules work, and how that interacts with queues etc. I have no problem blocking or opening certain ports and setting up port forwards, but any more complicated than that, and I've quickly become confused and I've never found a good write-up that explains the whole thing adequately.

    I'll happy accept recommendations for further reading :)

    When I built this router box last week, I simply saved my old configuration from the pfSense install I had running as a guest on my ESXi server.  Hopefully there are no old settings in there screwing things up.  Maybe I should try a clean install.  It doesn't take THAT long to set up my port forwards and static DHCP leases…



  • @mattlach:

    @Nullity:

    He means bufferbloat: latency caused by oversized network buffers.

    Regarding your original question, if you put in 150Mbit then you should get practically 150Mbit. If you are not, then something is wrong. Honestly, I usually only see this strange behavior when someone is new to pfSense, so I assume misconfiguration, but I could be wrong. Double-check your setting and perhaps show pictures of your queues. Make sure to reset states between queue/firewall changes.

    Plenty of us use pfSense and when we setup a 10Mbit queue, that queue moves 10Mbit. Long-time users would quickly recognize if the values were skewed.

    Thank you.  I'll do some poking around and post some screenshots.

    While I am not new to pfSense at all (been using it since ~2010) I AM new to traffic shaping.  I tried to set it up once a few years back, but it got complicated and I gave up.

    While I understand the basics, I've always gotten myself confused by how the firewall rules work, and how that interacts with queues etc. I have no problem blocking or opening certain ports and setting up port forwards, but any more complicated than that, and I've quickly become confused and I've never found a good write-up that explains the whole thing adequately.

    I'll happy accept recommendations for further reading :)

    When I built this router box last week, I simply saved my old configuration from the pfSense install I had running as a guest on my ESXi server.  Hopefully there are no old settings in there screwing things up.  Maybe I should try a clean install.  It doesn't take THAT long to set up my port forwards and static DHCP leases…

    The official pfSense wiki is good. Otherwise, there's the official pfSense book, "The book of pf", and of course Google.

    Firewall rules match and then assign said packets to a queue of your choosing. Then you setup the queue to have whatever traffic-shaping characterics you want.



  • @mattlach:

    @Harvy66:

    Be warned, too large of a buffer will add bloat.

    Also,

    What do you mean by bloat in this context?  Are we just talking about added RAM use?

    My pfSense box has 2GB of RAM, only because I wanted to take advantage of the dual channel performance, so I needed two sticks of RAM, and it turns out you just can't buy anything smaller than a 1GB stick these days :p

    With 2GB of RAM, I've never seen pfSense use more than ~5%.  I think I can probably live with a little RAM bloat if that's the extent of it.

    Bufferbloat is why people see high latency during congestion. Get rid of the bloat and you get rid of the latency.

    Example. Say set your buffer to hold a gigabit of packets, but you only have a 1Mb connection. If someone was to send you a bunch of data faster than 1Mb/s, your buffer would eventually fill up. Once your buffer is full, it will take at least 1,000 seconds to empty, and that's assuming no new data comes in. Now what happens if your buffer is nearly full and someone send a ping packet? The ping has to sit at the back of the line and wait 1,000 seconds before you see it. Now you have 1,000 seconds of latency.

    You want a large enough buffer to absorb a bursty traffic, but you want a small buffer to handle sustained traffic. It's very difficult to balance. CoDel doesn't use data sized buffers but time sized buffers. The default is 5ms. Once your buffer has 5ms of data, a packet will get dropped. The most likely packet to get dropped is a packet associated with a greedy flow, signally it to back-off. It's not guaranteed, but CoDel is biased towards dropping packets from high bandwidth flows and unlikely to drop small packets from low bandwidth flows, like VoIP, ping, games.

    To set your queue size, edit your traffic shaper queue in the "Queue Limit" setting. When not set, it's 50. I think the queue size is meaningless when CoDel is selected.




  • Thank you very much for the explanation.

    What is codel, and how do I set it?  I can't seem to find it anywhere in the traffic shaper menus?



  • Alright, So I did the test I suggested I was going to above.

    I reset pfsense to factory settings to make sure nothing old was hiding in the config.

    Ran bandwidth test without traffic shaping, got my full ~160/160

    Again, did the HFSC traffic shaper wizard, and wound up with the same results as before, very slow.

    So I went back again, to a saved config I had before doing the traffic shaper wizard, and instead went to the "by interface" screen, and added codelq there for both LAN and WAN.  It was confusing though, because this thread suggested no bandwidth needed to be entered for codelq, as it's operation is bandwidth independent and it adjusts on its own, but the screen would not let me apply the settings without applying a bandwidth.

    It was unclear to me which bandwidth this should be though.  The wizard game me an "upstream" and a "downstream" bandwidth.  Here it is just one per interface.  The tooltip said that "this is usually the interface bandwidth", so I set both to 1 Gbit/s.  Was this the right thing to do, or should I have chosen something closer to my external bandwidth (or just under, like with HFSC), and in that case which do I enter where, upstream and downstream wise?

    Anyway,  I did a bandwidth test after applying codelq, and it appears as if I have my full bandwidth, but how do I know it is actually working and doing it's magic in the background?

    Again, appreciate all your help.

    –Matt



  • @mattlach:

    Alright, So I did the test I suggested I was going to above.

    I reset pfsense to factory settings to make sure nothing old was hiding in the config.

    Ran bandwidth test without traffic shaping, got my full ~160/160

    Again, did the HFSC traffic shaper wizard, and wound up with the same results as before, very slow.

    So I went back again, to a saved config I had before doing the traffic shaper wizard, and instead went to the "by interface" screen, and added codelq there for both LAN and WAN.  It was confusing though, because this thread suggested no bandwidth needed to be entered for codelq, as it's operation is bandwidth independent and it adjusts on its own, but the screen would not let me apply the settings without applying a bandwidth.

    It was unclear to me which bandwidth this should be though.  The wizard game me an "upstream" and a "downstream" bandwidth.  Here it is just one per interface.  The tooltip said that "this is usually the interface bandwidth", so I set both to 1 Gbit/s.  Was this the right thing to do, or should I have chosen something closer to my external bandwidth (or just under, like with HFSC), and in that case which do I enter where, upstream and downstream wise?

    Anyway,  I did a bandwidth test after applying codelq, and it appears as if I have my full bandwidth, but how do I know it is actually working and doing it's magic in the background?

    Again, appreciate all your help.

    –Matt

    The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

    Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

    but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)



  • Shaping is bloody simple, but everything you read out there makes it seem hard for some reason. I'll see about creating some screenshots some time this weekend.

    There are only four things you need to do

    HSFC

    1. Set your interface bandwidth
    2. Create your queues
    3. Set your queue minimum bandwidths
    4. Assign traffic to your queues

    #1 seems to be the #1 reason for issue. I have no idea how this part is so hard. If you only have 10Mb of bandwidth, set your Interface to only have 9Mb.
    #2 seems to be the next issue. Stop creating complicated hierarchical queues, just create 3, high normal and idle
    #3 I can't explain this, same as #1. Stop thinking about priorities and blah blah blah. How much minimum bandwidth do you want for this queue
    #4 The wizard does create these for you. Unlike normal firewall rules, for floating, the last one wins

    P.S. Don't use realtime, it's confusing, seems to have little benefit in most cases, and is easy to mess up



  • @Harvy66:

    1. Set your interface bandwidth
      […]
      #1 seems to be the #1 reason for issue. I have no idea how this part is so hard.

    I do.

    I worked quite a while on this until I found out that the selection 'kbit/s' in the bandwidth setting was plain wrong. It should have been 'bit/s'. (Don't know whether this has been fixed in 2.3 but the bug is still in 2.2.6.)

    If as a consequence of this bug I set the bandwidth limit to effectively 10 bit/s (intending it to be 10 kbit/s) the performance was of course completely off …

    -flo-



  • Here is my setup. I'm only showing the WAN because the LAN is mostly identical, just slightly lower interface bandwidth. Some of my settings, like queue size when using CoDel, is probably not correct, and I have funny percentages, but this is exactly what I use. Take it, leave it, learn from it. Do whatever.


















  • @pf3000:

    @-flo-:

    I worked quite a while on this until I found out that the selection 'kbit/s' in the bandwidth setting was plain wrong. It should have been 'bit/s'. (Don't know whether this has been fixed in 2.3 but the bug is still in 2.2.6.)
    If as a consequence of this bug I set the bandwidth limit to effectively 10 bit/s (intending it to be 10 kbit/s) the performance was of course completely off …

    -flo-

    Thanks for pointing that out. So it's actually bit labeled as Kbit huh. I was confused when the speed test showed 10% of my actual download after setting in Kbit. I settled for Mbit and it's fine. However setting up bandwidth in Kbit via the wizard, it's okay. After that editing to Mbit and back to Kbit messes it up for me. It's like that on both 2.3 and 2.3.1

    Hmm… I have used pfSense since 2.1.x and kbit has always been kbit. With 2.3 I am using kbit with no problems.

    I extensively tested 2.2.x using mostly "kbit" and never experienced this bug. Are we sure it even exists? Has it been submitted to the pfSense bug tracker?



  • @Nullity:

    I extensively tested 2.2.x using mostly "kbit" and never experienced this bug. Are we sure it even exists?

    I already reported this nearly one year ago, see this thread: Noob guide to Traffic Shaping. You already commented on that thread back then.

    At least one user confirmed this problem, others didn't.

    @Nullity:

    Has it been submitted to the pfSense bug tracker?

    I didn't, should I?

    -flo-



  • @-flo-:

    @Nullity:

    Has it been submitted to the pfSense bug tracker?

    I didn't, should I?

    -flo-

    Absolutely.

    You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

    It might just be easier to post the info here and someone more experienced will submit a bug.



  • It's my mistake; I couldn't duplicate the issue. I knew that traffic wizard always worked, and my internet upload speed is 20% of my download.

    • While editing the shaper, I may have plugged in up/down bandwidth numbers in lan/wan instead of the other way round.

    • I also had "Disable hardware large receive offload" unchecked - it hinders my real speed test results.

    Hence I got a speed test result of about 10% of the download speed, leading me to think that kbits might be bits. Sorry for the confusion : D



  • @Nullity:

    The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

    Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

    but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

    Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

    In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
    Set bandwidth on WAN interface to ~97% of upstream bandwidth
    Set bandwidth on LAN interface to ~97% of downstream bandwidth.

    Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

    I may have to try this.

    It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good.  That's where I checked before I posted this thread, but it was lacking.  Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question.  I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.



  • @Nullity:

    You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

    I worked on this but right now I cannot reproduce this anymore. Originally the results were differing form the configured values by about factor 10. This is not the case in my setup anymore now.

    I'll keep an eye on this.

    -flo-



  • @-flo-:

    @Nullity:

    You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

    I worked on this but right now I cannot reproduce this anymore. Originally the results were differing form the configured values by about factor 10. This is not the case in my setup anymore now.

    I'll keep an eye on this.

    -flo-

    Always reset the states, or even just restart pfSense, when playing with traffic-shaping.

    Personally, my first few months playing with QoS & pfSense included lots of "why is it doing that?!" situations. Like you, I even thought that the bit/kbit/mbit might be inaccurate. Though, after lots reading & testing, eventually my configurations started to perform as I predicted.



  • @mattlach:

    @Nullity:

    The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

    Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

    but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

    Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

    In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
    Set bandwidth on WAN interface to ~97% of upstream bandwidth
    Set bandwidth on LAN interface to ~97% of downstream bandwidth.

    Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

    I may have to try this.

    It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good.  That's where I checked before I posted this thread, but it was lacking.  Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question.  I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.

    You have it pretty right. My upload connection is ~760Kbit link-rate, with a real-world bitrate of ~666Kbit, so I set my queue to 640Kbit. My ADSL is very stable, so 640 works well enough.

    CODELQ is dirt simple, but yeah, there is no fair queueing. You can use either HFSC (Hierarchical Fair Service Curve) or FAIRQ and select the "CoDel Active Queue" check-box to get both fair queueing & codel.

    Yeah… searching the forums can be annoying, but you gotta do it. We (pfSense forumites) are not very responsive when it's obvious that a poster has not searched for an answer.



  • @Nullity:

    @mattlach:

    @Nullity:

    The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

    Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

    but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

    Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

    In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
    Set bandwidth on WAN interface to ~97% of upstream bandwidth
    Set bandwidth on LAN interface to ~97% of downstream bandwidth.

    Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

    I may have to try this.

    It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good.  That's where I checked before I posted this thread, but it was lacking.  Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question.  I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.

    You have it pretty right. My upload connection is ~760Kbit link-rate, with a real-world bitrate of ~666Kbit, so I set my queue to 640Kbit. My ADSL is very stable, so 640 works well enough.

    CODELQ is dirt simple, but yeah, there is no fair queueing. You can use either HFSC (Hierarchical Fair Service Curve) or FAIRQ and select the "CoDel Active Queue" check-box to get both fair queueing & codel.

    Yeah… searching the forums can be annoying, but you gotta do it. We (pfSense forumites) are not very responsive when it's obvious that a poster has not searched for an answer.

    Hmm, so I finally got around to working on this today.

    I did several speed tests in a row to establish downstream at about 152Mb/s and upstream at about 161Mb/s.

    Here is how I set it up: (click for full size)

    WAN:

    WAN Child:

    LAN:

    LAN Child:

    After which I saved all my settings, and made sure to reload the firewall rules.

    Then, just as a sanity check I ran another speed test:

    This suggests that it isn't working, as it is allowing this single process to exceed the bandwidth limits (the 148 and 156 respectively)

    Any idea what I might be doing wrong?

    Thanks,
    Matt



  • Did you reset the firewall states?



  • @Nullity:

    Did you reset the firewall states?

    Yeah, the box at the top of the window after you apply the settings tells you to click here to reload the firewall rules in order for it to become effective.  I did that.

    Is there anything in addition to that I should be doing?



  • @mattlach:

    @Nullity:

    Did you reset the firewall states?

    Yeah, the box at the top of the window after you apply the settings tells you to click here to reload the firewall rules in order for it to become effective.  I did that.

    Is there anything in addition to that I should be doing?

    Try a temporary bitrate (like 20Mbit) that will not be as easily confused with your internet connection's bitrate. The official pfSense wiki's data should work.



  • @Nullity:

    Try a temporary bitrate (like 20Mbit) that will not be as easily confused with your internet connection's bitrate. The official pfSense wiki's data should work.

    Good idea, I'll try that, thank you.

    Meanwhile, could these alerts be related?

    Error occurred
    
    SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:45
    SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:46
    SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:47
    SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:48
    SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:49
    SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:50
    


  • You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.



  • @Harvy66:

    You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.

    Yeah, it looks like that might have been it.  I selected to make the child queues above the default, and I saw an effect.  Unfortunately it was more of an effect than I was hoping to see.  With this configuration I was down to 138/150mbit.

    At first I thought the fairq interface queue might be to blame, but when I deleted the child queues and set the interface queues to "codelq" i had the same performance loss.  Maybe this is how it works?  I'm not sure I want to give up THAT much bandwidth though.



  • @mattlach:

    @Harvy66:

    You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.

    Yeah, it looks like that might have been it.  I selected to make the child queues above the default, and I saw an effect.  Unfortunately it was more of an effect than I was hoping to see.  With this configuration I was down to 138/150mbit.

    At first I thought the fairq interface queue might be to blame, but when I deleted the child queues and set the interface queues to "codelq" i had the same performance loss.  Maybe this is how it works?  I'm not sure I want to give up THAT much bandwidth though.

    The interface bitrate needs to be lower than your real-world maximum, so tune it as close as possible.



  • You're set to 148Mb and getting 138Mb. 93% of your total bandwidth is pretty close. In theory, you cannot go above 148, so your average must be below it. The real question is what your bufferbloat is at. Check dslreports.com's speedtest. You can always increase your rates to see how it affects your traffic. The more stable your internet connection, the closer to 100% you can get without getting blips of latency or packetloss.



  • @Harvy66:

    You're set to 148Mb and getting 138Mb. 93% of your total bandwidth is pretty close. In theory, you cannot go above 148, so your average must be below it. The real question is what your bufferbloat is at. Check dslreports.com's speedtest. You can always increase your rates to see how it affects your traffic. The more stable your internet connection, the closer to 100% you can get without getting blips of latency or packetloss.

    How do we define stable in this case?

    The last mile service seems to be very stable. Anything within my ISP's network will saturate my links without a problem.  Depending on where I cross my ISP's networks edge routers, I'll have anywhere from excellent performance to OK performance, but the sum total of multiple connections from different locations seems to pretty much always max out my connection, at ~152Mbit down and ~161 Mbit up  (advertised as 150/150).

    Using Speedtest.net I tested all the test points in my state and a few in nearby neighboring ones (n~20).  The fastest one is not on my ISP's network, but on an adjacent one, and it is the one that consistently seems to give me 152/161Mbit, the highest I get anywhere.  I can make the figure go down by intentionally choosing less fast test servers, but this suggests that I am at least getting a consistent 152/161 total from all sources down my last mile pipe.

    So, individual connections may vary over time depending on routing and which peering point they cross to get where they are going, but the sum total is incredibly stable.

    Since traffic shaping is all about balancing multiple connections and making them able to coexist, by guess would be that the last mile total is the figure that is the most interesting for traffic shaping purposes, correct?



  • How do we define stable in this case?

    Loosely. Your description makes it sound "stable". TCP can be finicky, and small blips of loss could reduce your average quite a bit. In my case I say my connection is stable because I have a 100Mb connection, and if I don't do any shaping on my end, my sustained single TCP flow can maintain 99.5Mb/s +- 0.25Mb/s when looking RRD's 1min avg. By "maintain", I mean for hours on end during peak hours. I actually shape to 98.5Mb/s on my download and I get about 95-96Mb/s while staying around 1ms of bloat.

    An interesting tid-bit about my ISP is they don't shape my Ethernet port, they shape at their router. Their Cisco router, which is very new and shiny and full of 100Gb+ ports, does not police the bandwidth, but kind of "jitters" it, which works really really well. Instead of getting the standard TCP saw-tooth, I get their very mild fluctuation of +-0.1Mb/s while averaging my 100Mb for the most part. They used to use the built in ONT rate limiter which is a strict policer and gave the TCP saw-tooth pattern, which hurt my average performance by strictly dropping packets the instant my link tried to go over the provisioned rate.

    The TCP caw-tooth pattern is bad for average throughput during saturation with many flows. A dumb FIFO buffer causes this, and your max throughput will approach ~80%. With anti-buffer bloat, you can start to approach 95%. It's possible your ISP can supply you a "stable" 152Mb/s when averaged over a large window because of large buffers, but at any given moment, may be "unstable", which could hurt your performance when you're also doing your own shaping and setting yet an additional cap.

    Since traffic shaping is all about balancing multiple connections and making them able to coexist, by guess would be that the last mile total is the figure that is the most interesting for traffic shaping purposes, correct?

    Correct. The main issue of bufferbloat is to concern yourself about the bloat on your connection to your ISP, not the possible N peering routes your ISP may have with congestion.



  • @Harvy66:

    How do we define stable in this case?

    Loosely. Your description makes it sound "stable". TCP can be finicky, and small blips of loss could reduce your average quite a bit. In my case I say my connection is stable because I have a 100Mb connection, and if I don't do any shaping on my end, my sustained single TCP flow can maintain 99.5Mb/s +- 0.25Mb/s when looking RRD's 1min avg. By "maintain", I mean for hours on end during peak hours. I actually shape to 98.5Mb/s on my download and I get about 95-96Mb/s while staying around 1ms of bloat.

    An interesting tid-bit about my ISP is they don't shape my Ethernet port, they shape at their router. Their Cisco router, which is very new and shiny and full of 100Gb+ ports, does not police the bandwidth, but kind of "jitters" it, which works really really well. Instead of getting the standard TCP saw-tooth, I get their very mild fluctuation of +-0.1Mb/s while averaging my 100Mb for the most part. They used to use the built in ONT rate limiter which is a strict policer and gave the TCP saw-tooth pattern, which hurt my average performance by strictly dropping packets the instant my link tried to go over the provisioned rate.

    The TCP caw-tooth pattern is bad for average throughput during saturation with many flows. A dumb FIFO buffer causes this, and your max throughput will approach ~80%. With anti-buffer bloat, you can start to approach 95%. It's possible your ISP can supply you a "stable" 152Mb/s when averaged over a large window because of large buffers, but at any given moment, may be "unstable", which could hurt your performance when you're also doing your own shaping and setting yet an additional cap.

    Since traffic shaping is all about balancing multiple connections and making them able to coexist, by guess would be that the last mile total is the figure that is the most interesting for traffic shaping purposes, correct?

    Correct. The main issue of bufferbloat is to concern yourself about the bloat on your connection to your ISP, not the possible N peering routes your ISP may have with congestion.

    Ahh,  thanks for that, that is great info.

    So how do you go about measuring buffer bloat?  Just ping a known source (maybe the first hop outside my router in a traceroute?) and then load up the connection and see how the ping changes?



  • That is the simplest way without any external tools. DSLReports.com has a nice speedtest with a decent bufferbloat ability, which pretty much does what you said, except with websockets, which is technically less accurate than ICMP/UDP pings. It still gets within single millisecond precision and seems sane and reproducible, so I assume accurate.



  • @Harvy66:

    That is the simplest way without any external tools. DSLReports.com has a nice speedtest with a decent bufferbloat ability, which pretty much does what you said, except with websockets, which is technically less accurate than ICMP/UDP pings. It still gets within single millisecond precision and seems sane and reproducible, so I assume accurate.

    I checked out the DSLreports test.  It is pretty good.  I wish it would keep the max buffer bloat raw numbers for upstream and downstream rather than turn it into a grade though.

    So, with codel turned on, I only get 139.5 down and 146.4 up, but my buffer bloat is non-existent, (1-2 ms if watching the live chart)

    With codel disabled - however - I get 146.9 down and 163.6 up, but buffer bloat is horrific.  ~80ms during download and ~225ms during upload.

    So, the improved buffer bloat is what I was expecting, but at what cost?  A loss of ~10Mbps up and down?  That's more than some peoples entire connections…  I wonder if it is worth it.  I also wonder how I would perform with buffer bloat if I rather than using codel, just put a static bandwidth limit at the level of performance reported with codel on.  I wonder if it is actually the codel that is helping, or just the fact that the bandwidth is less utilized that reduces the buffer bloat...



  • @mattlach:

    @Harvy66:

    That is the simplest way without any external tools. DSLReports.com has a nice speedtest with a decent bufferbloat ability, which pretty much does what you said, except with websockets, which is technically less accurate than ICMP/UDP pings. It still gets within single millisecond precision and seems sane and reproducible, so I assume accurate.

    I checked out the DSLreports test.  It is pretty good.  I wish it would keep the max buffer bloat raw numbers for upstream and downstream rather than turn it into a grade though.

    So, with codel turned on, I only get 139.5 down and 146.4 up, but my buffer bloat is non-existent, (1-2 ms if watching the live chart)

    With codel disabled - however - I get 146.9 down and 163.6 up, but buffer bloat is horrific.  ~80ms during download and ~225ms during upload.

    So, the improved buffer bloat is what I was expecting, but at what cost?  A loss of ~10Mbps up and down?  That's more than some peoples entire connections…  I wonder if it is worth it.  I also wonder how I would perform with buffer bloat if I rather than using codel, just put a static bandwidth limit at the level of performance reported with codel on.  I wonder if it is actually the codel that is helping, or just the fact that the bandwidth is less utilized that reduces the buffer bloat...

    CoDel itself should not make that big of a difference, especially if you are using the "Codel Active Queue" check-box with HFSC, FAIRQ, CBQ, or PRIQ.

    If you are using CODELQ, tune your interfaces' bitrate.

    Regardless, traffic-shaping limits your max bandwidth. Personally, I limit my upload to 98% of my maximum (which I find tolerable), but I forego any download traffic-shaping because my bufferbloat only grows to ~50ms (during saturation) from ~15ms (idle), so I prefer maximum download speed.



  • @mattlach:

    So, the improved buffer bloat is what I was expecting, but at what cost?  A loss of ~10Mbps up and down?  That's more than some peoples entire connections…

    The loss of bandwidth is proportional to the stability of your connection. You may be "losing" 10Mb/s of your bandwidth to maintain a low bloat, but I only need to lose about 1.5Mb/s.

    And when you say "at what cost", it almost sounds sarcastic because the cost is so low. So you lose 7% of your bandwidth, but now your ping will be between 80ms and 200ms lower during saturation of your connection. You can still play games. And some connections are latency sensitive, like HTTPS. With 11 round trips, a 200ms ping increase will take 2.2 seconds before your web page starts to download. With a 2ms ping, it will take 22ms. That's the difference between instant and twiddling your thumbs.


Log in to reply