Trafic shaping for Streaming to twitch as well as streaming Netflix/YouTube



  • I have been plugging away at this for a few days now. I cannot seem to figure out the correct parameters for traffic shaping. I stream content to twitch whilst playing games for an average of about 6 hours a day. My fiancee needs something to do in those six hours. When i stream she either watches netflix or youtube. But when she does I get some latency issues and some lost packets. I have tried all 4 forms of traffic shaping with different settings (just random parameters, trying to get things to work). The most success i have reached has been using CBQ. I am a NOOB when it comes to traffic shaping. I used the wizard "traffic_shaper_wizard_multi_all.xml" to set up CBQ and under the Games child i have it set for 40% bandwidth. Twitch is not a P2P service so that is set at 5%. I do not know what qACK is so i left that at the default percentage of 15%.
    Please let me know if i need or can provide more information. I really would like to figure this out. My ISP is Comcast wouldn't be my first choice, but they are the best option for my area (they kinda blow chunks).





  • Try to run the wizard HFSC and enable codel on all queues



  • I will try this. I start my stream in a few hours. Thank you in advance!
    I will update this as soon as i know



  • If it still doesn't work, use FairQ as your shaper, set your interface to 90% of your actual speed, have one queue on your interface, make sure it's the default and enable Codel on it.



  • I'll bet that if you just use dummynet and enable fq_codel via CLI and use that for all your traffic then your problem will be solved.

    Fq_codel is like magic for latency sensitive traffic when the pipe is full.

    EDIT: This requires 2.4.x to be installed.



  • I attempted to run the wizard in HFSC with CODEL enabled on all queues and it fixed the problem for me on my end but slowed the WiFi down to the speed of dial-up. Which kind of defeats the purpose of what i want to do.  I am now going to try to run the wizard in CBQ and enable CODEL on all as well.

    @Harvy66:

    If it still doesn't work, use FairQ as your shaper, set your interface to 90% of your actual speed, have one queue on your interface, make sure it's the default and enable Codel on it.

    Can you help me more with this or explain to me what you mean by having one queue on my interface?

    thanks for all your help.



  • " but slowed the WiFi down to the speed of dialup"

    That is strange it has nothing to do with specifically slowing down the wifi speed, try restarting pfsense after the traffic shaping  setup. if that still does not work for you take Harvy66 advice, he  is one of the best in traffic shaping here you cant get better advice then he can give you



  • fq_codel is not the same as codel. You can only implement it on dummynet and only via CLI. You just turn it on for everything and it will almost certainly solve your problem.



  • @belt9:

    fq_codel is not the same as codel. You can only implement it on dummynet and only via CLI. You just turn it on for everything and it will almost certainly solve your problem.

    Dude we for sure know what FQ_CODEL is  and that it is not codel, but the OP has to install 2.4 RC first and install it  from the command line, so it is not the supported way to advice yet till it is officially in pfsense



  • fq_codel I agree has a good chance of solving this, but if you not comfortable in doing all this in the cli, then try this.

    HSFC for downstream and make sure to give the queue handling netflix realtime bandwidth allocation enough for it to stream optimally.

    FairQ for upstream.

    Enable codel on all queues.

    By default the queue size is 50, in certain scenarios this may need to be increased for proper performance, if you increase do it in small chunks. so dont e.g. jump from 50 to 500. A larger queue makes it less likely for packet drops in that queue, and less likely to have packet ordering issues.  However it also increases the possibility of buffer bloat.

    There is also RED (random early detection), what this does is drop packets in a queue before the queue fills up, in anticipation of the queue filling up, the aim of this is to slow down throughput in that queue before it gets full.

    The issue I think is very likely in the upstream, since streaming is uploading, its likely affecting the ack's from the netflix downstream traffic, so downstream shaping may not have much of an impact.  Fairq+codel combo seems best for me on upstream on altq.  Make sure acks for downstream traffic are properly been allocated to the 'ack' queue on the upstream.



  • @Chrismallia:

    @belt9:

    fq_codel is not the same as codel. You can only implement it on dummynet and only via CLI. You just turn it on for everything and it will almost certainly solve your problem.

    Dude we for sure know what FQ_CODEL is  and that it is not codel, but the OP has to install 2.4 RC first and install it  from the command line, so it is not the supported way to advice yet till it is officially in pfsense

    It is officially in pfSense…. You don't install it from the CLI, you just select it.



  • @belt9:

    @Chrismallia:

    @belt9:

    fq_codel is not the same as codel. You can only implement it on dummynet and only via CLI. You just turn it on for everything and it will almost certainly solve your problem.

    Dude we for sure know what FQ_CODEL is  and that it is not codel, but the OP has to install 2.4 RC first and install it  from the command line, so it is not the supported way to advice yet till it is officially in pfsense

    It is officially in pfSense…. You don't install it from the CLI, you just select it.

    Just to clarify, did you mean fq_codel is officially in pfSense or just Codel - the latter yes I would agree, but if the former, where do you select it?



  • You select it with ipfw

    ipfw sched 1 config pipe 1 type fq_codel
    

    So if you have dummynet setup with two pipes (up & down) and want to make fq_codel persistent, then in shellcmd ->

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    If you plug in a monitor on boot you'll see it load up fq_codel, no need to install anything at all, just select it on CLI, and if you want it to persist across reboots, use shellcmd.



  • @belt9:

    You select it with ipfw

    ipfw sched 1 config pipe 1 type fq_codel
    

    So if you have dummynet setup with two pipes (up & down) and want to make fq_codel persistent, then in shellcmd ->

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    If you plug in a monitor on boot you'll see it load up fq_codel, no need to install anything at all, just select it on CLI, and if you want it to persist across reboots, use shellcmd.

    That makes sense - my apologies, I misunderstood your original post and thought you meant selecting fq_codel in the UI vs the CLI.



  • Oh gotya, no worries. Stilla CLI selection, I was just saying it's all preinstalled and ready to go, no additional repositories or sideloading stuff into pfSense.



  • "It is officially in pfSense…. You don't install it from the CLI, you just select it."

    You are spreading  miss information. FQ_codel is officially in BSD  not officially implanted in  pfsense, it wil be officially implanted in pfsense when the pf team enable it and you can select it from the gui not playing around in cli.



  • Right, well it officially works. But sure, if nothing's official to you unless there's a box to check in a gui then by all means it isn't official.  ::)



  • "if nothing's official to you unless there's a box to check in a gui then by all means it isn't official.  ::)"

    Exactly. if it was fully official pfsense devs would have put in the checkbox. hekk 2.4 is not even  official release yet.    Oh and keep hitting the smite as much as you like two can play that game lol if you cant have a simple conversation you should get a real smite  lol



  • You keep on ignoring features unless someone builds a GUI box for you.

    fq_codel is still the solution to this problem, and it is part of pfSense 2.4 - and it works, and it's easier to setup than ALTQ for this problem, and it's more effective.



  • @belt9:

    You keep on ignoring features unless someone builds a GUI box for you.

    fq_codel is still the solution to this problem, and it is part of pfSense 2.4 - and it works, and it's easier to setup than ALTQ for this problem, and it's more effective.

    Thats not what I was saying, infect I used the cli also, all i said is that we know what fq_codel is and we know what  it can do and it solves most of the problems and yes  if the OP uses fq_codel his problems might be over, all I said is thet till 2.4 is fully released and fq_ is implemented I do not think it is a good Idea to suggest playing around in the cli to maybe newer people to pfsense or traffic shaping. Oh and looks like we are going to start having some fun with the smite game if you want it will do it :)



  • it's just copy paste into shellcmd - that has a GUI! not "playing around with cli".

    I would argue that it's a much better idea to simply pass along very simple instructions on how to solve a maybe newer users problem with the appropriate tool, rather than suggesting they do a complicated workaround with ALTQ / HFSC which few people really understand to begin with - just because it has a GUI.

    Anyways, if OP would like any help either method just ask.



  • "I would argue that it's a much better idea to simply pass along very simple instructions on how to solve a maybe newer users problem with the appropriate tool, rather than suggesting they do a complicated workaround with ALTQ / HFSC which few people really understand to begin with - just because it has a GUI. "

    Well everyone has his/her  opinion , but next time do not forget to tell them that  they have to install 2.4 RC  as you failed doing that



  • You're right, that was my bad. I'll modify that post.



  • I can pin a string in C# and modify the structure such that it thinks the size of the string is anything I want and can change the value of the string. I can "officially" change the value and length of strings.

    Just because you can't doesn't mean you should. fq_Codel technically working is an undefined implementation detail.



  • @Harvy66:

    Just because you can't doesn't mean you should. fq_Codel technically working is an undefined implementation detail.

    Would you care to elaborate on that please  ;)
    Are you saying that native FreeBSD queuing are undefined ?
    Or just not tested with pfSense ?

    I have installed the sudo package from the "BSD repos", because i saw a need for it.
    And still think it's ok to use the "right tool" for the job.

    But "fiddling under the hood" will of course VOID the warranty like everywhere else.

    /Bingo



  • I'm just saying, if my car lets me drive off a bridge, it doesn't mean that's an officially supported use of the vehicle. You take full responsibility for undocumented "features".



  • Ok. I stated before that I am a noob.
    I also think that some are misunderstanding what I am meaning by streaming. Yes Netflix is a streaming service, but I am talking about streaming content from my computer to a website upstream ( to twitch.tv) as well as allowing streaming downstream (from netflix.com) at the same time. I am lost with the what you guys are calling dummynet and I do not fully understand what some of the names and acronyms stand for such as these:
    qInternet
    qACK
    qDefault
                      I understand what the words default and internet mean, i just do not know what their purposes are if that makes sense.

    If I understand this correctly, "q" stands for queue followed by what is in the queue for being shaped? What is qACK?
    I also have found that the higher the number in the priority list = the higher priority i.e. 7= highest priority, correct?

    I have the bandwidth percentages set kind of odd to me in the sense that it doesn't add up. If I have the bandwidth percentages adding up to 100% across all queues NOTHING works. So currently, most queues sit at about 5%-15% qGames is the highest at 30%.

    I really would like to figure this out and I am sorry that i cannot figure it out on my own. You guys have been very helpful so far. I just am having difficulty making sense of the language.
    Thank you in advance.




  • @Harvy66:

    If it still doesn't work, use FairQ as your shaper, set your interface to 90% of your actual speed, have one queue on your interface, make sure it's the default and enable Codel on it.

    Where would I find FairQ in the wizard? I only get three options; HSFC CBQ and PRIQ



  • Correct those are different queues within the ALTQ shaper.

    pfSense uses two basic types of traffic shaping, ALTQ & Dummynet (dummynet is under the "Limiters" tab).

    ALTQ is what your screenshot is of. It is very effective, but you need to know what you are doing and even then it takes time to enter all of your queues.

    Dummynet is very simple, you just make two limiters. One for Download, the other for Upload.
    Set a queue within each limiter, set your subnet mask (probably /24) - ask if oyu need help with any of the other settings, there aren't many).
    You would limit both of these streams to some value slightly less than the slowest your internet ever gets.
    This is the step that most people mess up with traffic shaping - because no one wants to do it.

    But, put simply - your internet bandwidth is a series of funnels. For simplicity we'll just say there are two funnels - you and your ISP (there are more). If your funnel is larger than your ISP's, then your ISP funnel will limit your traffic - probably not in the way that it best for you.
    If you want control over how your traffic is limited then you MUST make your funnel smaller than your ISP's - therefore you MUST limit your bandwidth to less than the slowest it EVER gets.
    What this means is - if you pay for a 50Mbps down / 10 Mbps up line, but during peak traffic you are actually only getting 40/8, then you would set your limiters to 38,000Kbps / 7,600Kbps. If you decided to just set it to something like 49,000Kbps/9,000Kbps, then the limiters would do nothing for you during peak hours (when it matters most).

    Once the dummynet limiters are created, you just apply them to your firewall rules in the advanced section of each rule and you are done.

    You can use dummynet in pre 2.4.x, but you can't use fq_codel. I recommend using 2.4.0 (the Release version comes out tomorrow). Because you can set your dummynet to use fq_codel with the CLI and shellcmd package.

    fq_codel is just a very good algorithm that is exceptional at fairly sharing your bandwidth and greatly reducing latency at the same time. So it will generally solve all of your problems without a whole bunch of config.

    If you don't want to go that way, I recommend you watch this video on traffic shaping. It will show you how it works so you know what you are doing. After watching that you'll at least know what questions to ask. After watching that you'll probably also see why I recommend you just use dummynet for a home network that is simply trying to upload and download video content without bufferbloat.

    Youtube Video



  • Thank you sir! Story of my life. Here i am doing the advanced S*** where the next tab over is the thing I need that will work and be ok. LOL
    Thanks man.
    I will try this and I will also putts around learning with AltQ.



  • Anytime!

    Please ask if you have any other questions.



  • @belt9:

    Correct those are different queues within the ALTQ shaper.

    pfSense uses two basic types of traffic shaping, ALTQ & Dummynet (dummynet is under the "Limiters" tab).

    ALTQ is what your screenshot is of. It is very effective, but you need to know what you are doing and even then it takes time to enter all of your queues.

    Dummynet is very simple, you just make two limiters. One for Download, the other for Upload.
    Set a queue within each limiter, set your subnet mask (probably /24) - ask if oyu need help with any of the other settings, there aren't many).
    You would limit both of these streams to some value slightly less than the slowest your internet ever gets.
    This is the step that most people mess up with traffic shaping - because no one wants to do it.

    But, put simply - your internet bandwidth is a series of funnels. For simplicity we'll just say there are two funnels - you and your ISP (there are more). If your funnel is larger than your ISP's, then your ISP funnel will limit your traffic - probably not in the way that it best for you.
    If you want control over how your traffic is limited then you MUST make your funnel smaller than your ISP's - therefore you MUST limit your bandwidth to less than the slowest it EVER gets.
    What this means is - if you pay for a 50Mbps down / 10 Mbps up line, but during peak traffic you are actually only getting 40/8, then you would set your limiters to 38,000Kbps / 7,600Kbps. If you decided to just set it to something like 49,000Kbps/9,000Kbps, then the limiters would do nothing for you during peak hours (when it matters most).

    Once the dummynet limiters are created, you just apply them to your firewall rules in the advanced section of each rule and you are done.

    You can use dummynet in pre 2.4.x, but you can't use fq_codel. I recommend using 2.4.0 (the Release version comes out tomorrow). Because you can set your dummynet to use fq_codel with the CLI and shellcmd package.

    fq_codel is just a very good algorithm that is exceptional at fairly sharing your bandwidth and greatly reducing latency at the same time. So it will generally solve all of your problems without a whole bunch of config.

    If you don't want to go that way, I recommend you watch this video on traffic shaping. It will show you how it works so you know what you are doing. After watching that you'll at least know what questions to ask. After watching that you'll probably also see why I recommend you just use dummynet for a home network that is simply trying to upload and download video content without bufferbloat.

    Youtube Video

    Now by setting these things slower than my actual aren't I effectively bottlenecking my own internet? Also, the lingo for these limiters; for the destination tab - source is down while destination means up? (see attachment)




  • Down is destination

    Up is source

    Yes, you have to be your own bottleneck otherwise an upstream router will be queueing your traffic



  • Kind of like this?
    And what more options are there?

    I get 90 up and 25 down.






  • The opposite of that, downstream is destination, upstream is source



  • Lol
    That IS how I had it saved, Don't know why i screenshot it that way. I was more so asking about the funnels.



  • Funnels as in your bandwidth limits?

    That depends very much on your personal connection.

    Some people have WAN connections that are rock solid i.e., they pay for say 100/100 FiOS service, and actually get 100/100 or better 99%+ of the time. If that's you then you could safely set your bandwidth limits to 95%+ of your advertised bandwidth.

    Most people's connection would be some cable or dsl connection - let's say 50/10. During non-peak hours when everyon's at work, they actually get 50/10 or slightly better. But when everyone comes home from work they get something more like 48/9. Then in the evenings when everyone is streaming netflix, etc. they get something like 38/6.5.
    If this is you, then you would take the worst of those connections (38/6.5) and set your limits to maybe 95% of that, or 36,100Kbps/6,175Kbps. Then you have to live with cutting your internet connection down by 28%down/35%up all of the time just because your ISP sucks.
    -This is what all the stuff in my signature is trying to avoid.

    So, if you've accounted for that then you are good to go. But you have to measure your own network at different times, no one on here can accurately tell you exactly what to set it at.


Log in to reply