Playing with fq_codel in 2.4



  • I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    This is my experience as well. Only when I was beginning my traffic-shaping journey did I experience strange things like that. My assumption is that I was misconfiguring.

    I suppose it's possible that these algorithms incorrectly calculate bitrates but that is very unlikely since transmitting at the configured bitrate is perhaps the most fundamental aspect of any traffic-shaping algorithm.



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    I also have 150mb connection and am running an i5 mini PC with PFsense. It seems like a simple configuration so I'm not sure what could actually be misconfigured but I'm not ruling it out. Any ideas?



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    Full disclosure, I am running a VPN, but it pins at 147mb no matter what….until this config.



  • HeatmiserNYC
    So, with FQ_CODEL you have 130Mbps max, right? You said -20Mbps…
    The misconfiguration can be interference with other limiters or rules if you have used same limiter twice or more — I did not checked but it was possible in certain conditions.
    Also TS mentioned that this FQ_CODEL setup equalizes traffic and with VPN it can be a real problem if you have concurrent or even the same traffic on both.
    Anyway, I did tests some time ago and there was 1-2 Mbps difference with bandwidth limit, if we compare to traditional HFSC this is about twice less. Now I don't use bandwidth limit but delay limit that is set to 0ms, this causes FQ_CODEL scheduler to process all traffic by using only internal parameters, I think. Double check everything and if problem persists, please provide some configuration sample.



  • Cool, thanks for replying.

    Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

    I use the VPN for all outbound traffic, it's not a separate situation.

    I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

    Just need a successful example of this to get running…

    Again, thanks.



  • @HeatmiserNYC:

    Cool, thanks for replying.

    Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

    I use the VPN for all outbound traffic, it's not a separate situation.

    I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

    Just need a successful example of this to get running…

    Again, thanks.

    Perhaps your speed drop is related to overhead like VPN, TCP, etc. I assume you are referring to goodput bitrates?

    On downloads you will commonly see below the configured bitrate because each time you hit the limit pfSense will tell the sender to slow down below the limit. Personally, I found very little useful benefit by limiting downloads because my ISP has minimal bufferbloat and allowing them to do the rate-limiting gives me 100% speeds.



  • What about to try to move shaper/limiters from LAN side to VPN side firewall rules?



  • That's an idea, I'll give that a shot!



  • Yea, that didn't work.



  • Just for testing purpose, try to change bw limiting to delay limiting :

    pipe 1 config delay 0ms
    

    for both pipes



  • Sorry gone for a few days, vacation.

    I gave that a shot by changing the /tmp file, it doesn't seem to have an affect. I am only changing the /tmp file, maybe it needs to be rebooted and hardcoded into the file? The only reason I haven't done this is because I haven't seen the results everybody is reporting…



  • Yes it's need to be rebooted or reloaded with```
    /etc/rc.reload_all

    After you did that run the following command```
    ipfw sched show
    ```and you should see something like```
    00001: unlimited         0 ms burst 0
    ```for the both pipes you have.


  • @w0w:

    Yes it's need to be rebooted or reloaded with```
    /etc/rc.reload_all

    After you did that run the following command```
    ipfw sched show
    ```and you should see something like```
    00001: unlimited         0 ms burst 0
    ```for the both pipes you have.
    

    Yes, all relatively simple and you've been great at walking through the steps you put in place.

    I'm getting this for both pipes.

    00003: unlimited        0 ms burst 0

    00004: unlimited        0 ms burst 0

    Yet I can't get better than a B rating for bufferbloat, which is the same if I literally do nothing at all….



  • But what about VPN bandwidth? Are you still getting 120Mbps?



  • That part HAS improved, looks like it does get about 145-ish or so which is about right. It just does nothing for bufferbloat.



  • Can you post the full output of```
    ipfw sched show



  • I'd like to look at implementing this, but I was wondering

    Anyone know the status of pfsync + limiters?



  • @moscato359:

    I'd like to look at implementing this, but I was wondering

    Anyone know the status of pfsync + limiters?

    What was the last status you know?  :D



  • The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why



  • @moscato359:

    The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why

    This is actual. https://redmine.pfsense.org/issues/4310 have 0% progress.



  • @w0w:

    @moscato359:

    The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why

    This is actual. https://redmine.pfsense.org/issues/4310 have 0% progress.

    D=



  • Is there any chance fq_codel will make it into the 2.4 GUI in limiters?



  • Definitely not!
    They are keeping eyes on it, but currently no plans, no moves, AFAIK.



  • Darn.  I'm thinking about switching back to pfSense but I really want fq_codel.



  • fq_codel, the ZFS of AQMs, or nearly. Cake aims to be the "ZFS", but close enough.



  • Is the command of "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" the same if I only have 2 root limiters?  Both of them are root limiters one has a mask of source and the other has a mask of destination.

    I would like to try this out but wondering if the command is different for just root limiters without "child" queues.  Obviously I am highly dependent on the gui I am a bit confused with the ipfw command since it references both sched and pipe.

    Thanks for any reply!



  • TS sample is for the root limiters also, if you have  some troubles understanding, post the content of your /tmp/rules.limiter



  • here is the content of my /tmp/rules.limiter

    pipe 1 config  bw 100Mb mask dst-ip6 /128 dst-ip 0xffffffff

    pipe 2 config  bw 10Mb mask src-ip6 /128 src-ip 0xffffffff

    I need help with the ipfw command to enable fq_codel on pipes 1 and 2 because i don't have any child queues.

    thanks in advance



  • According to documentation posted in this thread you need to configure sheduler at least to make things work.

    
    pipe 1 config bw 100Mb mask dst-ip6 /128 dst-ip 0xffffffff
    sched 1 config pipe 1 type fq_codel 
    
    pipe 2 config bw 10Mb mask src-ip6 /128 src-ip 0xffffffff
    sched 2 config pipe 2 type fq_codel
    
    

    EDIT:
    Tested, it will not work. You need to configure child queues and use them in ruleset, exactly as described by TS. Default automatically created pipe queue always uses FIFO sheduler and I am not sure it is possible to change this.

    So after changes made in GUI also, you must edit and create your own rules.limiter that should look like this.

    
    pipe 1 config bw 100Mb 
    sched 1 config pipe 1 type fq_codel
    queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff
    
    pipe 2 config bw 10Mb mask 
    sched 2 config pipe 2 type fq_codel
    queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff
    
    

    So the right answer is no you can not shape with fq_codel using only root limiters.



  • Thats really too bad.  We use PFsense primarily to "specify bandwidth limits per host." for a small ISP.

    I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs.  And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

    Is it possible to combine and use ALTQ and Dummynet at the same time?  Has anyone tried that or have a config example?

    I guess I could use limiters on 2 PFsense boxes.  First one limiting each host to 5 Mbps using limiters with a destination/source mask.  And the second limiting the entire subnet to 100Mbs using limiters without a mask and changing the type from WF2Q+ to FQ_Codel by issuing the command "ipfw pipe 1 config bw 100Mb type fq_codel"

    I hope thats not too confusing.  Anyone have a more eloquent way of trying this?

    As always, thank you for any reply.



  • Yes it's possible, but  you will have some overheads and losses, you can try it at least, I think. Just set your per host limits on ALTQ shaper side and do your evenly shared FQ_CODEL enabled limiters exactly as TS described for you entire subnet.
    I am sure it is possible to build ipfw only shaper model that works like you want it to work, but it would be complicated not only with pfSense and can cause some errors on pfSense.



  • Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

    Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

    I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.

    My last speed test.



  • @cplmayo:

    Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

    Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

    I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.

    My last speed test.

    enable re-enable or disable re-enable or enable re-disable or .. ?



  • @Harvy66:

    fq_codel, the ZFS of AQMs, or nearly. Cake aims to be the "ZFS", but close enough.

    This is very interesting.

    Any chance someone(s) knowledgeable would be willing to put together a single post along the lines of this - https://forum.pfsense.org/index.php?topic=126597.0

    Kind of like an fq_codel one-stop shop for the layman?



  • ok am finally testing this and got it working.

    I had observed some iptv/vpn issues that seemed to only occur when my ingress altq config was active, so am now testing this configuration.  I have not yet tested if this is as effective as hsfc alt for keeping steam downloads in check, I had to set the dummynet limiter to 95% of downstream cap to even get a 6 threaded downstream test to stop causing packetloss, so not confident that will be enough for a 30+ stream steam download but will see.

    How granular is this? can I e.g. route steam etc. all through it but at the same time applying a limit less than 95% for steam download whilst keeping things like youtube able to burst higher.  All on dummynet.  As I have a feeling I will need to drop this to at least 90% to manage steam but I consider that too low for lighter threaded stuff.



  • The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

    It's a granular as firewall rules can be



  • @belt9:

    The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

    It's a granular as firewall rules can be

    shaping is nowhere near that simple.

    The reason steam is harder to shape is it opens so many threads.

    I always get 100% from my isp but it doesnt mean 95% will always work well for all types of traffic.

    Looking at dummynet configuration it looks like multiple specific rate limits cannot be set within one pipe however weighting can be applied so I can put steam downloads on a low weight and things like dns lookups and emails on a high weight, this is what I will look into on my config next. Thanks to the OP giving me a starting point. :)

    AltQ on PFSense is incredibly granular but of course someone has put the effort into integrating it all into the GUI, and AltQ itself allows children in a queue to have their own limits set.

    Already got some good results.

    When I setup the dummynet config (basic as in the OP) I had a iptv stream running to my STB and I can see from my ping monitoring on my connection, the peak latency has plummeted, it was an almost steady increase in peak latency, now its spikes instead of constant and the spikes been generally much lower.

    I will post back on how my steam downloads testing goes.



  • That's kind of the point of fq_codel, KISS. It is by design intended to be simple.

    Now you might be trying to get it to do something it wasn't designed to do, in which case yes you will have to do some weird stuff - bit more likely you should just look elsewhere.



  • I am a fan of simple, if steam works well in the current config then the current config stays, the weighting is a fallback plan if it doesnt work well.

    From what I understand all the weighting is dummynet side, it simply dynamically adjusts the throughput allowance of each thread based on the weight assigned, by default everything has the same weight.

    All fully documented on the dummynet man page, I dont think its a non supported feature.


Log in to reply