Playing with fq_codel in 2.4
-
@uptownvagrant said in Playing with fq_codel in 2.4:
@gsmornot you are very fortunate. :) I'm curious, what amount of bufferbloat were you seeing before implementing shapers?
I am not as technical with this as you guys have been but will say DSLReports gives me a D on average without the config I shared. The grade is all due to upload. Download has never been an issue here.
I have tried the config like you posted. On average it gives me a C with an occasional B. My guess is power in the 3100 but just a guess. Keeping it simple just works best for this hardware. I will say with a config like your on my older desktop I pull out time to time I would get As. But it has much more power and needs more power which is my reason for the 3100.
-
@uptownvagrant said in Playing with fq_codel in 2.4:
1.) Create "Out" limiter
Tick "Enable"
Name: FQ_CODEL_OUT
Bandwidth: 90 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODELtarget: 5
interval: 100
quantum: 1514
limit: 10240
flows: 1024Click Save/Apply Changes
2.) Add "Out" queue
Tick "Enable"
Name: fq_codel_out_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply Changes3.) Create "In" limiter
Tick "Enable"
Name: FQ_CODEL_IN
Bandwidth: 90 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODELtarget: 5
interval: 100
quantum: 1514
limit: 10240
flows: 1024Click Save/Apply Changes
4.) Add "In" queue
Tick "Enable"
Name: fq_codel_in_q
Mask: None
Queue Management Algorithm: Tail Drop
Click Save/Apply ChangesI am attempting to duplicate your configuration, but wanted to confirm which limiter is upload and which is download so that I can set my bandwidths properly. I'm guessing that "In" is upload and "Out" is download?
-
It depends on the perspective. I normally put myself in place of the device.
So If I apply on WAN interface, "IN" is download and "OUT" is upload. So for the first floating rule matching incomming in You do regular IN/OUT, but for the second rule which matches outgoing traffic You reverse the queues.
Still, the configuration of @uptownVagrant confuses me. I can confirm however that under load ping and traceroutes are not dropped.
-
@xciter327 said in Playing with fq_codel in 2.4:
It depends on the perspective.
Yes, I agree. By my perspective, traffic entering (IN) the WAN port would be leaving my local network (upload). Traffic exiting (OUT) the WAN port would be entering my local network (download). IIRC this was the same perspective used by Netgate in their recent hangout regarding fq_codel. It's all a little confusing, that's why I wanted to confirm with @uptownVagrant .
-
@wgstarks I've updated the post to better explain IN and OUT. From the perspective of WAN, IN is traffic coming into the interface from the Internet, aka download, and OUT is traffic leaving the interface and bound for the Internet, aka upload. Hope that clears things up.
-
@uptownvagrant
Thanks for clarifying.Also (minor issue), in rules creation step 3 I couldn’t select “Default” for gateway. Had to actually pick a gateway as you did in step 4.
-
@wgstarks Make sure you are choosing "Direction: In" on that rule. IN rules do not require a gateway be selected.
-
@uptownvagrant
Worked. Thanks again. -
For those interested, here is a quick comparison I ran today of ALTQ shapers using FAIRQ + CoDel and DummyNet limiters using FQ-CoDel. I set both ALTQ shapers and the limiters to 94 Mbit/s.
- The ALTQ shapers used a FAIRQ parent discipline and one CoDel child queue for WAN and a FAIRQ parent discipline and on CoDel child queue for LAN.
- The DummyNet limiters used a FQ-CoDel scheduler and one Tail Drop queue for WAN-OUT and a FQ-CoDel scheduler and one Tail Drop queue for WAN-IN. ICMP was passed before entering the limiter.
0_1543552183964_rrul-2018-11-29T175857.808604.rrul_C2758_pfSense2_4_4_100Mb_FairQ_CoDel_netperf2.flent.gz
0_1543552199424_rrul-2018-11-28T183120.881107.rrul_C2758_pfSense2_4_4_100Mb_FQ-CoDel_netperf2.flent.gz -
@uptownvagrant Thanks!
fq_codel is definitely better, especially for upload bandwidth, but fairq+codel is not that bad for how easy it is to setup.
-
This post is deleted! -
Substitua CoDel por TailDrop. Não use o CoDel AQM com FQ-CoDel.
My apologies if the English to Portuguese translation is incorrect. Replace CoDel with TailDrop. Don't use CoDel AQM with FQ-CoDel.
-
This post is deleted! -
@harvy66 Okay, thank you so much, it worked.
-
This config work perfect! for 1 wan
I have 2 adsl, with balance group (tier 1 both wans).
But dont know how to config the floating rules to use this balance group.
The wans have different speeds.Thanks!
-
I do not speak English, but google speaks for me KKK, create 2 floating rules one for each gateway this way :
-
@ricardox Thanks, I'm going to try it
This is what I have
WAN1 and WAN2Create 8 floating rules (following the example of @uptownVagrant )
but I think that is not the correct way to achieve the desired goal1.)
Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN1
Direction: out
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Traceroute
Source: any
Destination: anyAction: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN2
Direction: out
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Traceroute
Source: any
Destination: any3.)
Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN1
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround4.)
Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN2
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround5.)
Action: Match
Interface: WAN1
Direction: in
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Gateway: WAN1
In / Out pipe: fq_codel_WAN1_in_q / fq_codel_WAN1_out_q6.)
Action: Match
Interface: WAN2
Direction: in
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Gateway: WAN2
In / Out pipe: fq_codel_WAN2_in_q / fq_codel_WAN2_out_q7.)
Action: Match
Interface: WAN1
Direction: out
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Gateway: WAN1
In / Out pipe: fq_codel_WAN1_out_q / fq_codel_WAN1_in_q8.)
Action: Match
Interface: WAN2
Direction: out
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Gateway: WAN2
In / Out pipe: fq_codel_WAN2_out_q / fq_codel_WAN2_in_q -
Do the same for Upload
-
So an update from myself.
Still trying to perfect the interactive packets been dropped issue, I noticed pie is now implemented on the AQM.
So what I have done for now to test on my own network is changed droptail to pie for the pipe and queue, scheduler is still using fq_codel.
Also in regards to the dynamic flow sets, it seems something isnt quite right, as the OP pointed out, there is just one single flow created and always shows src and target ip as 0.0.0.0 so regardless of what mask is configured there will always only be one flow and I suspect if multiple flows can be made to work results will be better. The reason been if you have say two flows, one is a full speed saturated download over fast ftp and the other is from an interactive ssh session, the latter will never fill its flow queue and wont drop packets, while the first would fill and drop packets, as it is now both tcp streams share the same flow, so how does the shaper know which packets to drop? the only help on that seems to come from the quantum value which prevents smaller packets from been put at the end of the queue.
--update--
Ok I have managed to get the multiple flows working. Was a lot of experimentation but I got there after finally understanding what the man page for ipfw/dummynet is explaining. If you get it wrong dummynet stops passing traffic ;) but I am only testing on my home network so is fine.
--update2--
I chose to have a flow per internet ip rather than per lan device. Also this is only for ipv4 right now, I think if you configure it so it overloads the flow limit then it breaks, and obviously the ipv6 address space is way bigger.
root@PFSENSE home # ipfw sched show 00001: 69.246 Mbit/s 0 ms burst 0 q00001 500 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x000000ff/0x0000 -> 0x00000000/0x0000 sched 1 type FQ_CODEL flags 0x1 256 buckets 4 active FQ_CODEL target 5ms interval 30ms quantum 300 limit 1000 flows 1024 ECN mask: 0x00 0xffffff00/0x0000 -> 0x00000000/0x0000 Children flowsets: 1 BKT ___Prot___ _flow-id_ ______________Source IPv6/port_______________ _______________Dest. IPv6/port_______________ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0 ::/0 ::/0 6 288 0 0 0 17 ip 46.17.x.x/0 0.0.0.0/0 4 176 0 0 0 123 ip 208.123.x.x/0 0.0.0.0/0 5 698 0 0 0 249 ip 80.249.x.x/0 0.0.0.0/0 6 168 0 0 0 00002: 18.779 Mbit/s 0 ms burst 0 q00002 500 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 300ms quantum 300 limit 800 flows 1024 NoECN Children flowsets: 2--update 3--
Initial tests this is working sweet, on the below paste you can clearly see the bulk download flow and notice it only drops packets from that flow. I also tested on steam and zero dropped packets now on flows outside of steam, wow.
BKT ___Prot___ _flow-id_ ______________Source IPv6/port_______________ _______________Dest. IPv6/port_______________ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0 ::/0 ::/0 7 348 0 0 0 17 ip 46.17.x.x/0 0.0.0.0/0 3 132 0 0 0 58 ip 216.58.x.x/0 0.0.0.0/0 1 52 0 0 0 123 ip 208.123.x.x/0 0.0.0.0/0 5 698 0 0 0 135 ip 5.135.x.x/0 0.0.0.0/0 11919 17878500 3 4500 2 249 ip 80.249.x.x/0 0.0.0.0/0 6 168 0 0 0 254 ip 162.254.x.x/0 0.0.0.0/0 1 40 0 0 0 -
@chrcoluk said in Playing with fq_codel in 2.4:
So an update from myself.
Still trying to perfect the interactive packets been dropped issue, I noticed pie is now implemented on the AQM.
What interactive packets being dropped issue? Can you be specific? How are you measuring drops?
So what I have done for now to test on my own network is changed droptail to pie for the pipe and queue, scheduler is still using fq_codel.
When FQ-CoDel is used as the scheduler, the AQM you choose is not utilized. (fq_codel has separate enqueue/dequeue functions - AQM is ignored) https://forum.netgate.com/post/804118
Also in regards to the dynamic flow sets, it seems something isnt quite right, as the OP pointed out, there is just one single flow created and always shows src and target ip as 0.0.0.0 so regardless of what mask is configured there will always only be one flow and I suspect if multiple flows can be made to work results will be better. The reason been if you have say two flows, one is a full speed saturated download over fast ftp and the other is from an interactive ssh session, the latter will never fill its flow queue and wont drop packets, while the first would fill and drop packets, as it is now both tcp streams share the same flow, so how does the shaper know which packets to drop? the only help on that seems to come from the quantum value which prevents smaller packets from been put at the end of the queue.
What you are seeing using 'ipfw sched show' is not exposing the dynamic internal flows and sub-queues that FQ-CoDel is managing. It seems you are trying to create a visual representation, via 'ipfw sched show', of the mechanism that is already in place and operating within FQ-CoDel. See this post https://forum.netgate.com/post/803139
