Playing with fq_codel in 2.4

dtaht · Nov 24, 2018, 7:00 PM

udp is being dropped also. Unless this is yet another badmodems.com bug?

markn6262 · Nov 24, 2018, 7:17 PM

Perhaps a packet size rather than protocol issue

cwagz · Nov 25, 2018, 12:16 AM

I have an APU2C4 and 150/150 FiOS. It looks like I hit about 30 - 40% CPU usage when several clients are downloading from steam using the fq_codel limiter setup mentioned earlier in the thread. I was looking at top and it appears like the load gets spread across all 4 cores of my processor.

Will this always be the case or could the limiters actually cause me to become CPU bound on one core in certain instances? I am moving to 200/200 speed soon. The APU2 runs at 1GHz. I was considering moving to a Celeron J3455 @ 2.3GHz but only if it was needed to handle running fq_codel.

I appreciate any feedback.

jasonraymundo31 · Nov 26, 2018, 12:39 PM

@rasool said in Playing with fq_codel in 2.4:

@strangegopher Excellent work!
I installed pfSense and was able to setup fq_codel correctly (without CoDel) using just the WebUI. Here are the steps:

1- Create "out" limiter

Tick Enable

Name: pipe_out

Set the bandwidth

Queue Management Algorithm: Tail Drop

Scheduler: FQ_CODEL

2- Add new Queue

Tick "Enable"

Name: queue_out

Queue Management Algorithm: Tail Drop

Save

3- Create "in" limiter

Tick Enable

Name: pipe_in

Set the bandwidth

Queue Management Algorithm: Tail Drop

Scheduler: FQ_CODEL

4- Add new Queue

Tick "Enable"

Name: queue_in

Queue Management Algorithm: Tail Drop

Save

5- Add limiter in firewall rule

Configure floating rule (as normal)

In / Out pipe: queue_in / queue_out

I believe these steps prevent "config_aqm Unable to configure flowset, flowset busy!" error and no need for rebooting pfSense.

Could you please test the above setup?

what's the normal way to configure floating rule? can I have more detail about it? like what direction and interface ? also Quick option should be tick ?

Rasool · Nov 26, 2018, 2:39 PM

@knowbe4
I meant by "as normal" as per the youtube video https://youtu.be/o8nL81DzTlU?t=801. Btw, I am not a pfSense expert. I learned how to use the basic features just to help in this forum ;)

jasonraymundo31 · Nov 26, 2018, 2:39 PM

@rasool Thank you very much, I really appreciate it. Keep up the good work

uptownVagrant · Nov 26, 2018, 8:23 PM

@pentangle I can't recreate your results with VoIP. I'm using a cloud provider for VoIP and signaling uses (UDP/5060), RTP (media) uses (UDP/10,000-30,000). DTMF Mode RFC 2833 Payload type 101 and G.711 a-law. I can run an rrul test across pfSense, placing all traffic, except for ICMP, in the 90Mbit in/out FQ-CoDel limiter queues and I place a call during the test. Call quality remains excellent during a 5 minute rrul test.

Did you see any "fq_codel_enqueue over limit" messages in syslog during the speedtest where VoIP and Sonos was cutting out?

Are your using the FQ-CoDel defaults of?
target: 5
interval: 100
quantum: 1514
limit: 10240
flows: 1024

jasonraymundo31 · Nov 27, 2018, 7:27 AM

@uptownvagrant do you mind sharing your current fq_codel setup ?

uptownVagrant · Jan 29, 2019, 5:25 PM

This is for a very heavily utilized 100/100 Mbps circuit. Hardware is C2758, 8 GB RAM, igb interfaces (Intel I354).

In my testing I may move limiters to other interfaces without recreating them and renaming them. In my examples IN=ingress and OUT=egress. From the perspective of the WAN port, IN is traffic coming into the interface from the Internet and OUT is traffic leaving the interface to the Internet.

Create Limiters:
login-to-view

1.) Create "Out" limiter

Tick "Enable"
Name: FQ_CODEL_OUT
Bandwidth: 96907 Kbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
- target: 5
- interval: 100
- quantum: 300
- limit: 10240
- flows: 20480
Click Save/Apply Changes

2.) Add "Out" queue

Tick "Enable"
Name: fq_codel_out_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply Changes

3.) Create "In" limiter

Tick "Enable"
Name: FQ_CODEL_IN
Bandwidth: 83886 Kbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
- target: 5
- interval: 100
- quantum: 300
- limit: 10240
- flows: 20480
Click Save/Apply Changes

4.) Add "In" queue

Tick "Enable"
Name: fq_codel_in_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply Changes

Floating Rules:

login-to-view

1.) Add quick pass floating rule to handle ICMP traceroute. This rule matches ICMP traceroute packets so that they are not matched by the WAN-Out limiter rule that utilizes policy routing. Policy routing breaks traceroute.

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Traceroute
Source: any
Destination: any
Description: policy routing traceroute workaround
Click Save

2.) Add quick pass floating rule to handle ICMP echo-request and echo-reply. This rule matches ping packets so that they are not matched by the limiter rules. See bug 9024 for more info.

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround
Click Save

3.) Add a match rule for incoming state flows so that they're placed into the FQ-CoDel in/out queues

Action: Match
Interface: WAN
Direction: in
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-In FQ-CoDel queue
Gateway: Default
In / Out pipe: fq_codel_in_q / fq_codel_out_q
Click Save

4.) Add a match rule for outgoing state flows so that they're placed into the FQ-CoDel out/in queues

Action: Match
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-Out FQ-CoDel queue
Gateway: WAN_DHCP
In / Out pipe: fq_codel_out_q / fq_codel_in_q
Click Save/Apply Changes

Update 2018/01/04: After additional research and testing, I have made changes to FQ-CoDel quantum, limit, and flows in my environment where a 500 Mbps symmetric circuit is in use. I have also made changes to state timeouts on a firewall that consistently averages over 55k filter states.

Quantum - setting this to 300 will give some priority to smaller packet flows like VoIP. As a reference, a quantum of 300 is used in the OpenWRT sqm scripts and is noted on bufferbloat.net and the mailing list as a good option. Note: Setting quantum below 300 is not advised.
Limit - if your system is not severely memory constrained, setting this to 20480 packets, which is the max, will further protect against the "fq_codel_enqueue over limit" error. Depending on the flows, in my testing this error typically fires right before pfSense starts to drop many packets and in some instances causes pfSense to become unstable and/or reboot. Note: Setting this over-large packet limit can lead to bad results during slow starts for certain flows.
Flows - if your system is not severely memory constrained, setting this to 65535 will allow very good flow separation up to 65535 flows. The default 1024 is pretty low for a network with more than a few clients doing anything other than basic web browsing. Note: while the ipfw man page does specify that the maximum acceptable value is 65536, you will find that if you use this max value the firewall will enter a boot loop and you will have to restore a previous configuration. Also, I'm still testing and observing but it appears that on my hardware, each 10240 increase in flows equals about 1ms of added latency under load.
State Timeouts - under System / Advanced / Firewall & NAT, I set 'Firewall Optimization Options' to Aggressive and 'TCP Established' to 86400.

gsmornot · Nov 28, 2018, 3:43 PM

I have a gigabit link on my SG-3100. This is what I'm using and get all A+ on DSLReports. No rules, just the shaper on my main WAN connection.
login-to-view login-to-view

wgstarks · Nov 28, 2018, 4:40 PM

@askmyteapot said in Playing with fq_codel in 2.4:

I've managed to get around the dropped pings problem by creating a floating rule with:
MATCH on WAN, any direction, ICMP (any subtypes), default gateway and no In/Out Pipes.
Then having it as the very first rule in floating.

It doesnt fix the cause of the issue, but should at least allow pings to get through normally.

I’m still working on getting fq_codel configured properly and this thread has been a big help. Haven't completely read all the way through it yet though. Perhaps someone could confirm that this is the proper floating rule to work around the problem with ICMP?

login-to-view

I think I got it right, but am a little behind so perhaps it has been modified.

markn6262 · Nov 28, 2018, 5:21 PM

@uptownvagrant said in Playing with fq_codel in 2.4:

ICMP subtypes: Traceroute

I find Win tracert does not work with your config. Ping does, however. I changed your two workaround rules to one ICMP any in/out & tracert then works.

markn6262 · Nov 28, 2018, 5:27 PM

@wgstarks

I think you’ll need an ICMP quick pass not match or a lower rule may prevail.

wgstarks · Nov 28, 2018, 5:57 PM

@markn6262 said in Playing with fq_codel in 2.4:

@wgstarks

I think you’ll need an ICMP quick pass not match or a lower rule may prevail.

Like this?
login-to-view

uptownVagrant · Nov 28, 2018, 6:12 PM

@markn6262

Hmm, it works here.

login-to-view

uptownVagrant · Nov 28, 2018, 6:39 PM

@gsmornot I'm glad you're getting all A+ on dslreports.com/speedtest but you traffic shaper is not configured properly IMO. You are letting your ISP manage your WAN ingress buffers or the interface buffers on your firewall are being used. You are only shaping out from WAN.

I'll update this post with graphs to help illustrate my point.

markn6262 · Nov 28, 2018, 8:08 PM

@uptownvagrant

The first tracert is what I get with ICMP rule on top of floating as pass quick any and the 2nd tracert is what I get with the same rule as match, quick, any. With match the host name even resolves differently. Perhaps you aren't showing all of your floating rules and their order.

login-to-view

uptownVagrant · Nov 28, 2018, 9:31 PM

@markn6262
1.) What you are seeing in the first example you posted looks to be a properly behaving icmp trace. If you look at my example you will notice that I created a pass quick ICMP Traceroute floating rule in the first position. Maybe you read it as match only?
2.) The second example shows the "loop" folks were talking about earlier in this thread. All of the hops are being listed as the end node. The TTL is not being decreased due to policy routing.
3.) I'm showing all of the floating rules in my example.

markn6262 · Nov 28, 2018, 10:00 PM

@uptownvagrant My first reply was to @wgstarks. I further replied to your post because I thought your position was in agreement with @wgstarks that a match rule is what is required. Miscommunication.

wgstarks · Nov 28, 2018, 10:17 PM

@markn6262 said in Playing with fq_codel in 2.4:

@uptownvagrant My first reply was to @wgstarks. I further replied to your post because I thought your position was in agreement with @wgstarks that a match rule is what is required. Miscommunication.

Ok. Just because I am easily confused , I'ld like to confirm that this should be a pass quick and not a match quick?