Playing with fq_codel in 2.4

Animosity022

I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

I have the Shell Command to create the proper queue setup:

https://i.imgur.com/k08PJQZ.png

I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

https://i.imgur.com/6JZTEXd.png

https://i.imgur.com/6cDzTe5.png

Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

https://i.imgur.com/HeMy45B.png

My rules examples are a bit big so I linked them a little different:

Default queue
http://i.imgur.com/CQDQGcf.png

http://i.imgur.com/CQDQGcf.png

Low priority rule
http://i.imgur.com/MDuvFFe.png

http://i.imgur.com/CADTE77.pn

For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

My speedtest results made me happy:

Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.

strangegopher

@Animosity022:

I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

I have the Shell Command to create the proper queue setup:

https://i.imgur.com/k08PJQZ.png

I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

https://i.imgur.com/6JZTEXd.png

https://i.imgur.com/6cDzTe5.png

Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

https://i.imgur.com/HeMy45B.png

My rules examples are a bit big so I linked them a little different:

Default queue
http://i.imgur.com/CQDQGcf.png

http://i.imgur.com/CQDQGcf.png

Low priority rule
http://i.imgur.com/MDuvFFe.png

http://i.imgur.com/CADTE77.pn

For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

My speedtest results made me happy:

Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.

Are you sure you need both in and out direction floating rules? Mine seems to work with just outbound floating rule.

Animosity022

I'm definitely not sure as I was trying to figure out a way to match my private lan traffic to drop a few hosts specifically into my 'low' queue.

I tried something a little different and deleted all my floating rules and made a few LAN rules and since they are PASS rules, they logged fine on my private IPs and looks to have dropped them in the proper queue.

I also changed my WAN NAT rules to have a proper queue for in/out as well.

I get the same results on speed tests and tried saturating my low traffic and my high/default worked fine.

Going to give that a try for a few days and see how it works and I might write that up instead as it just seems easier to me than the floating rules.

pete35

Hi,

just trying this on Pfsense 2.4.2 and 2.4.1 Release.

Even i install this with ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

the resulting scheduler type is WF2Q+ …

ipfw sched show

00001: 25.000 kbit/s 0 ms burst 0
sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 1
00002: 5.000 kbit/s 0 ms burst 0
sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 2

What im doing wrong ?

Animosity022

@pete35:

Hi,

just trying this on Pfsense 2.4.2 and 2.4.1 Release.

Even i install this with ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

the resulting scheduler type is WF2Q+ …

ipfw sched show

00001: 25.000 kbit/s 0 ms burst 0
sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 1
00002: 5.000 kbit/s 0 ms burst 0
sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 2

What im doing wrong ?

You should see an error on the command line when you type it in as Unix/Linux is case sensitive. Make sure to use lower case for FQ_Codel:


ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

pete35

Thank you, that worked!

darkcrucible

I setup fq_codel using floating rules on another system and the same IPv4 traceroute/ICMP problem I mentioned earlier occurs.

Anyone else who uses floating rules to match traffic for fq_codel, do you see IPv4 ICMP traceroute working properly?

Cardnyl

@darkcrucible:

@Cardnyl:

I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.

In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-for

Actually this article does help quite a bit. There was also mention of adjusting some of these tuneables by the OP; I just didn't realize how critical it was until now. Here is the list:


net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

The article you linked suggests the following for a slow connection like mine:
net.inet.ip.dummynet.fqcodel.limit < 1000
net.inet.ip.dummynet.fqcodel.quantum < 300
ECN OFF

Is the right way to make these adjustments by adding/modifying them in the GUI (System->Advanced->System Tunables)?

Harvy66

Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.

Cardnyl

@Harvy66:

Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.

What is the correct way to set these targets?

Harvy66

I assume the "net.inet.ip.dummynet.fqcodel" settings you mentioned in your just prior post.

tman222

@Cardnyl:

@Harvy66:

Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.

What is the correct way to set these targets?

So I experimented with this a little bit, and changing the values below either in the System Tunables section or by editing loader.conf.local didn't seem to work because these fq_codel defaults don't appear to show up until fq_codel is actually loaded. So if you are loading fq_codel using shellcmd this won't really work.


net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

You can make it work, but you would have to load fq_codel using ipfw so the defaults show up, change the default values to what you want them to be using sysctl and then load fq_codel again with the new values.

However, I think there is an easier way as you can change these values just by editing the ipfw command that loads fq_codel.

From a link posted earlier in this thread:

http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

From section 3, the syntax for loading fq_codel via ipfw is:


ipfw sched x config [...] type fq_codel [target t] [interval t] [ecn | noecn] 
        [quantum n] [limit n] [flows n]

 where
    t is time in seconds (s), milliseconds (ms) or microseconds (us). The 
        default interpretation is milliseconds.
    n is an integer number

The command to load fq_codel with the defaults (from earlier in the thread):


ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

Using similar syntax as above, in your case all you would have to change is:


ipfw sched 1 config pipe 1 type fq_codel target 15 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 15 noecn quantum 300 limit 600

This does the following:
Set target to 15ms (you'll need higher than default of 5ms given the low upload on your 15/1 connection)
Turn off ECN
Set fq_codel quantum to 300
Set queue size limit to 600 packets

(feel free to tweak these values as necessary)

Now of course, this all assumes that you have already created two limiters and queues underneath them.

You can issue the command above from the command line and make it persistent between reboots using shellcmd.

Hope this helps.

tman222

I also have a question for everyone following this thread:

The command to enable fq_codel that has been shown in this thread to be:


ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

However, looking at the examples in 3.3 here:
http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

Doesn't the command have to be appended to include setting the the scheduler on the queue, i.e.:


ipfw sched 1 config pipe 1 type fq_codel && ipfw queue 1 config sched 1 && ipfw sched 2 config pipe 2 type fq_codel ipfw queue 2 config sched 2

The above assumes only two queues, one under each limiter.

Thanks in advance for the help.

w0w

@tman222:

I also have a question for everyone following this thread:

The command to enable fq_codel that has been shown in this thread to be:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
However, looking at the examples in 3.3 here:
http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

Doesn't the command have to be appended to include setting the the scheduler on the queue, i.e.:
ipfw sched 1 config pipe 1 type fq_codel && ipfw queue 1 config sched 1 && ipfw sched 2 config pipe 2 type fq_codel ipfw queue 2 config sched 2
The above assumes only two queues, one under each limiter.

Thanks in advance for the help.

I have used second, official variant, but have also found that first variant also works just fine if you have already configured limiters via GUI.

atrotter01

I am trying to set this up on an SG-3100 with an asymmetrical gigabit connection. I setup two queues and am configuring them as follows:


ipfw pipe 1 config  bw 920Mb
ipfw sched 1 config pipe 1 type fq_codel
ipfw queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

ipfw pipe 2 config  bw 40232Kb
ipfw sched 2 config pipe 2 type fq_codel
ipfw queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff

On the firewall side I am using a floating rule to apply the queues. This all works, with the floating rule enabled I get A / A+ on the DSL Reports speed test, whereas without the rule I get a C at best for bufferbloat. The issue I am having is with the rule enabled, I get at most ~650Mb/s bandwidth downstream, I have no problem hitting 940 with the rule disabled. Am I running into a CPU limitation of the SG-3100? Is there anything I can try to tweak?

Animosity022

@atrotter01:

I am trying to set this up on an SG-3100 with an asymmetrical gigabit connection. I setup two queues and am configuring them as follows:
ipfw pipe 1 config  bw 920Mb
ipfw sched 1 config pipe 1 type fq_codel
ipfw queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

ipfw pipe 2 config  bw 40232Kb
ipfw sched 2 config pipe 2 type fq_codel
ipfw queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff
I'd love to have someone else chime in, but the way I "think" the masks work is that you are sharing bandwidth between source and destinations based on the "mask src/mask dst" rules going on. So that causes some dynamic queues to be setup and it'll share out bandwidth amongst those queues so assuming you have some other traffic going on even if it's only a little, it will limit the bandwidth out.

Maybe try without having the src and dst masks setup and just leave those blank and see how your results look. Also, you can check the CPU % or something? i'm not familiar with that on that particular to device to tell you how, but that would validate if you are having a CPU bottleneck instead.

On the firewall side I am using a floating rule to apply the queues. This all works, with the floating rule enabled I get A / A+ on the DSL Reports speed test, whereas without the rule I get a C at best for bufferbloat. The issue I am having is with the rule enabled, I get at most ~650Mb/s bandwidth downstream, I have no problem hitting 940 with the rule disabled. Am I running into a CPU limitation of the SG-3100? Is there anything I can try to tweak?

atrotter01

I tried leaving the masks off and that didn't seem to make a difference. Then I tried disconnecting everything from the firewall except the laptop that I am using to test with, so that I could ensure nothing else was on the network. That didn't make a difference either. When the limiters are not enabled I can run a sustained 60 second download test and hit gig speeds 95% of the time, other than occasional dips. When I enable the limiters I can't get over the 650ish mark.

I can run top from an SSH session and it looks to be pegging one of the cores both with and without fq_codel so I can't really tell one way or the other there.

johnpoz

If you have symmetrical gig gig, why are you setting 40232Kb on pipe2

So 40mb? Whats the amount of traffic for acks to hit 650 vs 940? Have to do the math, etc. But limiting your upload could have an effect on your max download.. Shoot 500mbps down would be like 8mbps or something up just in acks..

If your on gig/gig why would you want to limit your upload so much?

Not a qos guy, this was my real first test of any sort of qos in pfsense.. But I got asked to take a look, that is what jumps out at me to look odd.. In a few days when my 4860 is online I would be able to test your settings. But not going to be ready until the weekend for sure.. But will be able try and duplicate the problem next week say.

atrotter01

John thanks, it's asymmetrical, not symmetrical. :) The connection maxes out at about 940/42.

Animosity022

What's the CPU as well? I couldn't drive shaping until I did some upgrades. I'm overpowered now with an:

Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz (4 cores)

But I wanted to future proof a bit. I can drive gig both directions at the same time without breaking my CPU atm.