Playing with fq_codel in 2.4

[jasonraymundo31]

@uptownvagrant do you mind sharing your current fq_codel setup ?

[uptownVagrant]

This is for a very heavily utilized 100/100 Mbps circuit. Hardware is C2758, 8 GB RAM, igb interfaces (Intel I354).

In my testing I may move limiters to other interfaces without recreating them and renaming them. In my examples IN=ingress and OUT=egress. From the perspective of the WAN port, IN is traffic coming into the interface from the Internet and OUT is traffic leaving the interface to the Internet.

Create Limiters:

1.) Create "Out" limiter

• Tick "Enable"
• Name: FQ_CODEL_OUT
• Bandwidth: 96907 Kbit/s
• Mask: None
• Queue Management Algorithm: Tail Drop
• Scheduler: FQ_CODEL
  • target: 5
  • interval: 100
  • quantum: 300
  • limit: 10240
  • flows: 20480
• Click Save/Apply Changes

2.) Add "Out" queue

• Tick "Enable"
• Name: fq_codel_out_q
• Mask: None
• Queue Management Algorithm: Tail Drop
• Click Save/Apply Changes

3.) Create "In" limiter

• Tick "Enable"
• Name: FQ_CODEL_IN
• Bandwidth: 83886 Kbit/s
• Mask: None
• Queue Management Algorithm: Tail Drop
• Scheduler: FQ_CODEL
  • target: 5
  • interval: 100
  • quantum: 300
  • limit: 10240
  • flows: 20480
• Click Save/Apply Changes

4.) Add "In" queue

• Tick "Enable"
• Name: fq_codel_in_q
• Mask: None
• Queue Management Algorithm: Tail Drop
• Click Save/Apply Changes

Floating Rules:

1.) Add quick pass floating rule to handle ICMP traceroute. This rule matches ICMP traceroute packets so that they are not matched by the WAN-Out limiter rule that utilizes policy routing. Policy routing breaks traceroute.

• Action: Pass
• Quick: Tick Apply the action immediately on match.
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Traceroute
• Source: any
• Destination: any
• Description: policy routing traceroute workaround
• Click Save

2.) Add quick pass floating rule to handle ICMP echo-request and echo-reply. This rule matches ping packets so that they are not matched by the limiter rules. See bug 9024 for more info.

• Action: Pass
• Quick: Tick Apply the action immediately on match.
• Interface: WAN
• Direction: any
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Echo reply, Echo Request
• Source: any
• Destination: any
• Description: limiter drop echo-reply under load workaround
• Click Save

3.) Add a match rule for incoming state flows so that they're placed into the FQ-CoDel in/out queues

• Action: Match
• Interface: WAN
• Direction: in
• Address Family: IPv4
• Protocol: Any
• Source: any
• Destination: any
• Description: WAN-In FQ-CoDel queue
• Gateway: Default
• In / Out pipe: fq_codel_in_q / fq_codel_out_q
• Click Save

4.) Add a match rule for outgoing state flows so that they're placed into the FQ-CoDel out/in queues

• Action: Match
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: Any
• Source: any
• Destination: any
• Description: WAN-Out FQ-CoDel queue
• Gateway: WAN_DHCP
• In / Out pipe: fq_codel_out_q / fq_codel_in_q
• Click Save/Apply Changes

Update 2018/01/04: After additional research and testing, I have made changes to FQ-CoDel quantum, limit, and flows in my environment where a 500 Mbps symmetric circuit is in use. I have also made changes to state timeouts on a firewall that consistently averages over 55k filter states.

• Quantum - setting this to 300 will give some priority to smaller packet flows like VoIP. As a reference, a quantum of 300 is used in the OpenWRT sqm scripts and is noted on bufferbloat.net and the mailing list as a good option. Note: Setting quantum below 300 is not advised.
• Limit - if your system is not severely memory constrained, setting this to 20480 packets, which is the max, will further protect against the "fq_codel_enqueue over limit" error. Depending on the flows, in my testing this error typically fires right before pfSense starts to drop many packets and in some instances causes pfSense to become unstable and/or reboot. Note: Setting this over-large packet limit can lead to bad results during slow starts for certain flows.
• Flows - if your system is not severely memory constrained, setting this to 65535 will allow very good flow separation up to 65535 flows. The default 1024 is pretty low for a network with more than a few clients doing anything other than basic web browsing. Note: while the ipfw man page does specify that the maximum acceptable value is 65536, you will find that if you use this max value the firewall will enter a boot loop and you will have to restore a previous configuration. Also, I'm still testing and observing but it appears that on my hardware, each 10240 increase in flows equals about 1ms of added latency under load.
• State Timeouts - under System / Advanced / Firewall & NAT, I set 'Firewall Optimization Options' to Aggressive and 'TCP Established' to 86400.

[gsmornot]

I have a gigabit link on my SG-3100. This is what I'm using and get all A+ on DSLReports. No rules, just the shaper on my main WAN connection.

[wgstarks]

@askmyteapot said in Playing with fq_codel in 2.4:

I've managed to get around the dropped pings problem by creating a floating rule with:
MATCH on WAN, any direction, ICMP (any subtypes), default gateway and no In/Out Pipes.
Then having it as the very first rule in floating.

It doesnt fix the cause of the issue, but should at least allow pings to get through normally.

I’m still working on getting fq_codel configured properly and this thread has been a big help. Haven't completely read all the way through it yet though. Perhaps someone could confirm that this is the proper floating rule to work around the problem with ICMP?

I think I got it right, but am a little behind so perhaps it has been modified.

[markn6262]

@uptownvagrant said in Playing with fq_codel in 2.4:

ICMP subtypes: Traceroute

I find Win tracert does not work with your config. Ping does, however. I changed your two workaround rules to one ICMP any in/out & tracert then works.

[markn6262]

@wgstarks

I think you’ll need an ICMP quick pass not match or a lower rule may prevail.

[wgstarks]

@markn6262 said in Playing with fq_codel in 2.4:

@wgstarks

I think you’ll need an ICMP quick pass not match or a lower rule may prevail.

Like this?

[uptownVagrant]

@markn6262

Hmm, it works here.

[uptownVagrant]

@gsmornot I'm glad you're getting all A+ on dslreports.com/speedtest but you traffic shaper is not configured properly IMO. You are letting your ISP manage your WAN ingress buffers or the interface buffers on your firewall are being used. You are only shaping out from WAN.

I'll update this post with graphs to help illustrate my point.

[markn6262]

@uptownvagrant

The first tracert is what I get with ICMP rule on top of floating as pass quick any and the 2nd tracert is what I get with the same rule as match, quick, any. With match the host name even resolves differently. Perhaps you aren't showing all of your floating rules and their order.

[uptownVagrant]

@markn6262
1.) What you are seeing in the first example you posted looks to be a properly behaving icmp trace. If you look at my example you will notice that I created a pass quick ICMP Traceroute floating rule in the first position. Maybe you read it as match only?
2.) The second example shows the "loop" folks were talking about earlier in this thread. All of the hops are being listed as the end node. The TTL is not being decreased due to policy routing.
3.) I'm showing all of the floating rules in my example.

[markn6262]

@uptownvagrant My first reply was to @wgstarks. I further replied to your post because I thought your position was in agreement with @wgstarks that a match rule is what is required. Miscommunication.

[wgstarks]

@markn6262 said in Playing with fq_codel in 2.4:

@uptownvagrant My first reply was to @wgstarks. I further replied to your post because I thought your position was in agreement with @wgstarks that a match rule is what is required. Miscommunication.

Ok. Just because I am easily confused , I'ld like to confirm that this should be a pass quick and not a match quick?

[gsmornot]

@uptownvagrant said in Playing with fq_codel in 2.4:

@gsmornot I'm glad you're getting all A+ on dslreports.com/speedtest but you traffic shaper is not configured properly IMO. You are letting your ISP manage your WAN ingress buffers or the interface buffers on your firewall are being used. You are only shaping out from WAN.

I'll update this post with graphs to help illustrate my point.

I follow you. I do it on purpose because the ingress results are not an issue for me. I assume the slowest part of my link is upstream from me so no shaping needed. I only need control my upload.

[uptownVagrant]

@markn6262 Gotcha, yeah, I was replying to your response to me. The config I posted should work and the change you specified in your response should not be required.

@markn6262 said in Playing with fq_codel in 2.4:

@uptownvagrant said in Playing with fq_codel in 2.4:

ICMP subtypes: Traceroute

I find Win tracert does not work with your config. Ping does, however. I changed your two workaround rules to one ICMP any in/out & tracert then works.

[uptownVagrant]

@gsmornot you are very fortunate. :) I'm curious, what amount of bufferbloat were you seeing before implementing shapers?

[gsmornot]

@uptownvagrant said in Playing with fq_codel in 2.4:

@gsmornot you are very fortunate. :) I'm curious, what amount of bufferbloat were you seeing before implementing shapers?

I am not as technical with this as you guys have been but will say DSLReports gives me a D on average without the config I shared. The grade is all due to upload. Download has never been an issue here.

I have tried the config like you posted. On average it gives me a C with an occasional B. My guess is power in the 3100 but just a guess. Keeping it simple just works best for this hardware. I will say with a config like your on my older desktop I pull out time to time I would get As. But it has much more power and needs more power which is my reason for the 3100.

[wgstarks]

@uptownvagrant said in Playing with fq_codel in 2.4:

1.) Create "Out" limiter

Tick "Enable"
Name: FQ_CODEL_OUT
Bandwidth: 90 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL

target: 5
interval: 100
quantum: 1514
limit: 10240
flows: 1024

Click Save/Apply Changes

2.) Add "Out" queue

Tick "Enable"
Name: fq_codel_out_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply Changes

3.) Create "In" limiter

Tick "Enable"
Name: FQ_CODEL_IN
Bandwidth: 90 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL

target: 5
interval: 100
quantum: 1514
limit: 10240
flows: 1024

Click Save/Apply Changes

4.) Add "In" queue

Tick "Enable"
Name: fq_codel_in_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply Changes

I am attempting to duplicate your configuration, but wanted to confirm which limiter is upload and which is download so that I can set my bandwidths properly. I'm guessing that "In" is upload and "Out" is download?

[xciter327]

It depends on the perspective. I normally put myself in place of the device.

So If I apply on WAN interface, "IN" is download and "OUT" is upload. So for the first floating rule matching incomming in You do regular IN/OUT, but for the second rule which matches outgoing traffic You reverse the queues.

Still, the configuration of @uptownVagrant confuses me. I can confirm however that under load ping and traceroutes are not dropped.

[wgstarks]

@xciter327 said in Playing with fq_codel in 2.4:

It depends on the perspective.

Yes, I agree. By my perspective, traffic entering (IN) the WAN port would be leaving my local network (upload). Traffic exiting (OUT) the WAN port would be entering my local network (download). IIRC this was the same perspective used by Netgate in their recent hangout regarding fq_codel. It's all a little confusing, that's why I wanted to confirm with @uptownVagrant .