Playing with fq_codel in 2.4
-
This has to be the best thread on the forum right now. Helped me a lot.
-
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codelI searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?
-
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codelI searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?
Refresh your browser window and under Services->Shell Command
-
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codelI searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?
Refresh your browser window and under Services->Shell Command
Doh. That's the one thing I didn't try. Many thanks!
-
Hi
It is possible to use queues (different weights) with fq_codel ? -
So I have a pretty odd situation with fq_codel.
I created a floating rule:Action - Match Interface - WAN Direction - out Address Family - IPv4 Protocol - Any In / Out pipe - lan wanNow, when I do traceroute from a LAN machine, all the hops show up as the destination! If I disable this floating rule, then traceroute works normally. If it's enabled and I do traceroute from pfsense itself, that traceroute is fine. ICMP ping itself seems oddly unreliable because every first ping is lost.
I have a regular firewall rule to allow IPv4 ICMP to the WAN address but disabling/enabling this does nothing. I also do manual outbound NAT but just the regular rules.
What's causing this? As soon as I disable the out floating rule, traceroute works normally. The bad traceroute even has latency corresponding to what the hops would normally be. traceroute through a site-to-site VPN works normally (no floating rules for VPN).
IPv6 traceroute is unaffected.
Does anyone else who uses a floating rule like mine see this? Any known solutions?
A LAN machine example traceroute:
# traceroute -I forum.pfsense.org traceroute to forum.pfsense.org (208.123.73.70), 30 hops max, 60 byte packets 1 uac.localdomain (192.168.112.1) 0.163 ms 0.143 ms 0.138 ms 2 * 208.123.73.70 (208.123.73.70) 14.600 ms 15.180 ms 3 208.123.73.70 (208.123.73.70) 15.136 ms 15.379 ms 15.528 ms 4 * * * 5 208.123.73.70 (208.123.73.70) 16.811 ms 16.837 ms 16.848 ms 6 208.123.73.70 (208.123.73.70) 17.816 ms 16.261 ms 17.627 ms 7 208.123.73.70 (208.123.73.70) 17.644 ms 19.906 ms 20.173 ms 8 208.123.73.70 (208.123.73.70) 20.262 ms 19.962 ms 19.813 ms 9 208.123.73.70 (208.123.73.70) 19.594 ms 19.867 ms 19.898 ms 10 208.123.73.70 (208.123.73.70) 60.034 ms 59.998 ms 60.353 ms 11 208.123.73.70 (208.123.73.70) 59.594 ms 55.994 ms 55.201 ms 12 208.123.73.70 (208.123.73.70) 55.441 ms 55.469 ms 56.421 ms 13 208.123.73.70 (208.123.73.70) 55.350 ms 57.401 ms 57.401 ms 14 208.123.73.70 (208.123.73.70) 54.101 ms 55.283 ms 62.990 ms 15 208.123.73.70 (208.123.73.70) 62.392 ms 62.218 ms * 16 * * * 17 * * * 18 208.123.73.70 (208.123.73.70) 61.501 ms 62.231 ms 59.980 ms 19 208.123.73.70 (208.123.73.70) 67.374 ms 68.414 ms 68.897 ms 20 208.123.73.70 (208.123.73.70) 63.797 ms 74.896 ms 70.074 msThe same traceroute from pfsense itself:
traceroute -I forum.pfsense.org traceroute to forum.pfsense.org (208.123.73.70), 64 hops max, 48 byte packets 1 173-228-88-1.dsl.dynamic.fusionbroadband.com (173.228.88.1) 14.262 ms 20.740 ms 17.791 ms 2 gig1-29.cr1.lsatca11.sonic.net (70.36.243.77) 17.422 ms 21.650 ms 14.505 ms 3 * * * 4 50.ae4.gw.pao1.sonic.net (50.0.2.5) 15.070 ms 21.048 ms 19.681 ms 5 ae6-102.cr1-pao1.ip4.gtt.net (69.22.130.85) 14.544 ms 13.726 ms 21.730 ms 6 xe-8-1-6.cr0-sjc1.ip4.gtt.net (89.149.142.18) 16.210 ms 14.649 ms 14.841 ms 7 as6461.ip4.gtt.net (216.221.158.110) 18.493 ms 18.531 ms 20.437 ms 8 ae16.cr1.sjc2.us.zip.zayo.com (64.125.31.12) 22.334 ms 20.133 ms 20.252 ms 9 ae27.cs1.sjc2.us.eth.zayo.com (64.125.30.230) 60.536 ms 79.274 ms 54.876 ms 10 ae2.cs1.lax112.us.eth.zayo.com (64.125.28.145) 55.359 ms 57.712 ms 63.770 ms 11 ae3.cs1.dfw2.us.eth.zayo.com (64.125.29.52) 59.466 ms 64.721 ms 63.456 ms 12 ae27.cr1.dfw2.us.zip.zayo.com (64.125.30.181) 60.663 ms 63.960 ms 59.337 ms 13 ae11.er1.dfw2.us.zip.zayo.com (64.125.20.66) 58.965 ms 64.998 ms 61.659 ms 14 ae8.er2.dfw2.us.zip.zayo.com (64.125.29.122) 64.904 ms 60.535 ms 59.003 ms 15 te-6-1-aus-core-11.zip.zayo.com (64.125.32.202) 65.484 ms 63.259 ms 63.947 ms 16 net64-20-229-170.static-customer.corenap.com (64.20.229.170) 65.048 ms 61.145 ms 59.658 ms 17 gw1.netgate.com (66.219.34.173) 67.752 ms 63.029 ms 63.543 ms 18 fw2.pfmechanics.com (208.123.73.4) 66.281 ms 68.793 ms 67.865 ms 19 208.123.73.70 (208.123.73.70) 66.834 ms 66.319 ms 66.047 ms -
Hi
It is possible to use queues (different weights) with fq_codel ?I'm unsure about exactly what you are asking but ALTQ (pfSense "queues") only supports codel and dummynet (pfSense "limiters") supports fq_codel.
You should probably create your own thread, including more details about your setup & intentions, if you need more help.
-
I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.
Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any. Put a rule above to allow access to your other vlans without the pipe's applied.
These settings changed my bufferbloat tests on dslreports to A..
Hello to all!
I have followed these instructions, together with the fq_codel trick on /etc/inc/shaper.inc and /root/rules.limiter . All my tests on dslreports are A or A+ . Also the download speed between my computers is shared evenly but with a serios issue:
When i download one file at one computer and one file at another computer, both from HTTP servers and a different server per computer, each computer takes the half speed which is what i want.
When any of the two computers start downloading a second file, from the same server or another, then it gets more speed and the second computer download speed slows down. If it starts a third download the other computers gets even less speed.
For example, when i download one file at each of the two computers, each one gets about 1.5 MB/s (my line maxes out at about 3 MB/s to 3.5 MB/s) . When any one starts a second download, the other computer gets about 1.2 MB/s and the other with the two downloads about 2.3 MB/s.
I am searching for a lot of days and have not found a solution. What i need to achieve, is to evenly share the speed between my computers no mater how many simultaneous downloads each of them runs.
-
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codelI searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?
Refresh your web browsers and it's under Services->Shellcmd
-
I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.
I have the Shell Command to create the proper queue setup:
https://i.imgur.com/k08PJQZ.png
I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.
https://i.imgur.com/6JZTEXd.png
https://i.imgur.com/6cDzTe5.png
Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.
I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:
https://i.imgur.com/HeMy45B.png
My rules examples are a bit big so I linked them a little different:
Default queue
http://i.imgur.com/CQDQGcf.pnghttp://i.imgur.com/CQDQGcf.png
Low priority rule
http://i.imgur.com/MDuvFFe.pngFor floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.
My speedtest results made me happy:
Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.
-
I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.
I have the Shell Command to create the proper queue setup:
https://i.imgur.com/k08PJQZ.png
I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.
https://i.imgur.com/6JZTEXd.png
https://i.imgur.com/6cDzTe5.png
Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.
I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:
https://i.imgur.com/HeMy45B.png
My rules examples are a bit big so I linked them a little different:
Default queue
http://i.imgur.com/CQDQGcf.pnghttp://i.imgur.com/CQDQGcf.png
Low priority rule
http://i.imgur.com/MDuvFFe.pngFor floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.
My speedtest results made me happy:
Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.
Are you sure you need both in and out direction floating rules? Mine seems to work with just outbound floating rule.
-
I'm definitely not sure as I was trying to figure out a way to match my private lan traffic to drop a few hosts specifically into my 'low' queue.
I tried something a little different and deleted all my floating rules and made a few LAN rules and since they are PASS rules, they logged fine on my private IPs and looks to have dropped them in the proper queue.
I also changed my WAN NAT rules to have a proper queue for in/out as well.
I get the same results on speed tests and tried saturating my low traffic and my high/default worked fine.
Going to give that a try for a few days and see how it works and I might write that up instead as it just seems easier to me than the floating rules.
-
Hi,
just trying this on Pfsense 2.4.2 and 2.4.1 Release.
Even i install this with ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL
the resulting scheduler type is WF2Q+ …
ipfw sched show
00001: 25.000 kbit/s 0 ms burst 0
sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 1
00002: 5.000 kbit/s 0 ms burst 0
sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 2What im doing wrong ?
-
Hi,
just trying this on Pfsense 2.4.2 and 2.4.1 Release.
Even i install this with ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL
the resulting scheduler type is WF2Q+ …
ipfw sched show
00001: 25.000 kbit/s 0 ms burst 0
sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 1
00002: 5.000 kbit/s 0 ms burst 0
sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
Children flowsets: 2What im doing wrong ?
You should see an error on the command line when you type it in as Unix/Linux is case sensitive. Make sure to use lower case for FQ_Codel:
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel -
Thank you, that worked!
-
I setup fq_codel using floating rules on another system and the same IPv4 traceroute/ICMP problem I mentioned earlier occurs.
Anyone else who uses floating rules to match traffic for fq_codel, do you see IPv4 ICMP traceroute working properly?
-
I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.
You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.
In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-forActually this article does help quite a bit. There was also mention of adjusting some of these tuneables by the OP; I just didn't realize how critical it was until now. Here is the list:
net.inet.ip.dummynet.fqcodel.limit: 10240 net.inet.ip.dummynet.fqcodel.flows: 1024 net.inet.ip.dummynet.fqcodel.quantum: 1514 net.inet.ip.dummynet.fqcodel.interval: 100000 net.inet.ip.dummynet.fqcodel.target: 5000The article you linked suggests the following for a slow connection like mine:
net.inet.ip.dummynet.fqcodel.limit < 1000
net.inet.ip.dummynet.fqcodel.quantum < 300
ECN OFFIs the right way to make these adjustments by adding/modifying them in the GUI (System->Advanced->System Tunables)?
-
Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.
The reason for this is you don't want a single MTU sized packet to trip the drop logic.
-
Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.
The reason for this is you don't want a single MTU sized packet to trip the drop logic.
What is the correct way to set these targets?
-
I assume the "net.inet.ip.dummynet.fqcodel" settings you mentioned in your just prior post.