Playing with fq_codel in 2.4
-
Almost all of my rules are applied LAN-side, because almost all flows are initiated by hosts on the LAN. But if you run a WAN-facing server (e.g. SSH) for which traffic that initiates a flow will come in on the WAN interface, you'd want to assign it to a limiter queue there. At least, that's my understanding of how it works.
-
Would this also be handy, for example, if one would want to assign different limiters to each segment. For example:
WAN - 480/480
LAN1 - 90/90
LAN2 - 200/200
etc.What would happen if the product of the bandwidths of the LANs is more than the WAN? I would assume that fq_codel should do its magic.
Another handy scenario I can think of:
WAN - 300/300
LAN1 - 10/10 per host/IP
LAN2 - 100/100 per host/IP
etc. -
Well in a typical single-WAN scenario, you'd usually only have two limiters . . . one for upload and one for download. You could divide bandwidth among different LAN segments with limiters, but the only way I'd know to do that would impose hard limits on them. For example, say you have 500Mbps downstream and you set a 200Mbps limiter on LAN1 download and a 300Mbps limiter on LAN2 download. Even if LAN2 is using no bandwidth at all, LAN1 is still only going to be allowed to use 200Mbps.
You could instead add child queues to your limiters with different weights. For example, you have your download limiter with 500Mbps, and you could then make two child queues - say Lan1DownQ and Lan2DownQ, with the weight of the former being 40 and the weight of the latter being 60 (the weights are just proportional and can be anything, but conceptually I find it easiest to have them sum to 100 so that you can interpret them as percentages). Then if you assign all LAN1 download traffic to Lan1DownQ and all LAN2 traffic to Lan2DownQ, if both LAN segments are "maxing out" their download, LAN1 should get about 200Mbps and LAN2 should get about 300Mbps. But if LAN2 is idle, LAN1 should get all 500Mbps.
-
@w0w said in Playing with fq_codel in 2.4:
Can you post you configuration, screenshots or what ever you like?
You are not the first person who reported this misbehavior, but I am not sure what is the reason for it.I'd happily revert to that configuration and post whatever you'd like. For the FQ_CODEL, I set it up using the method described in the youtube video, ie, using limiters and then floating rules to catch all the WAN traffic. It's really bizarre behavior, the ipfw sched showed the right pipe throughputs but somehow I was only getting 10% of it. I got the same results through wired cable and using wifi.
-
It makes sense to me to apply the rules on the WAN side, because it means that my OpenVPN connections will also be mixed into the pool of the bandwidth shared around. So someone at home downloading a torrent doesn't impact my ability to remote VPN in to the network and move files/data around.
-
@thenarc said in Playing with fq_codel in 2.4:
Well in a typical single-WAN scenario, you'd usually only have two limiters . . . one for upload and one for download. You could divide bandwidth among different LAN segments with limiters, but the only way I'd know to do that would impose hard limits on them. For example, say you have 500Mbps downstream and you set a 200Mbps limiter on LAN1 download and a 300Mbps limiter on LAN2 download. Even if LAN2 is using no bandwidth at all, LAN1 is still only going to be allowed to use 200Mbps.
You could instead add child queues to your limiters with different weights. For example, you have your download limiter with 500Mbps, and you could then make two child queues - say Lan1DownQ and Lan2DownQ, with the weight of the former being 40 and the weight of the latter being 60 (the weights are just proportional and can be anything, but conceptually I find it easiest to have them sum to 100 so that you can interpret them as percentages). Then if you assign all LAN1 download traffic to Lan1DownQ and all LAN2 traffic to Lan2DownQ, if both LAN segments are "maxing out" their download, LAN1 should get about 200Mbps and LAN2 should get about 300Mbps. But if LAN2 is idle, LAN1 should get all 500Mbps.
Very interesting. What I normally do(since my clients don't get preferential treatment) is the following:
Have one limiter with one queue each for upload, where I set 90% of achievable bandwidth for the limiter. For the queues and the limiter I set a "MASK" which covers all my LAN side networks(I think the term is "supernet"). Then I apply the limiter via a floating rule, with source alias with all the LAN side networks in it. Effectively this leads to all the LANs sharing the total bandwidth configured for each limiter.
I'll try your way when I have some time. Maybe it's more efficient.
-
I can tell you the way I set up my network (simple single WAN single LAN), which may or may not be what you want. I have a 100/10 connection, so I have two limiters: DownloadLimiter and UploadLimiter, with bandwidth limits of 100Mbps and 10Mbps respectively. Neither has a mask set.
DownloadLimiter has two child queues: DownloadQueueDefault - with a weight of 30 - and DownloadQueueHighPriority - with a weight of 70. I have a 32-bit Destination mask set on each, which means that each host will get its own queue as opposed to all download traffic from all LAN hosts being dumped into a single queue. The theory behind that - although I must admit that my understanding at this level is limited - is that the available bandwidth will be shared more equitably among all hosts that way.
UploadLimiter also has two child queues: UploadQueueDefault - with a weight of 30 - and UploadQueueHighPriority - with a weight of 70. I have a 32-bit Source mask set on each, for the same reasons as described above.
The theory here is perhaps best illustrated with an example. Suppose I have four hosts: A, B, C, and D. A and B are assigned to my "Default" upload and download queues and C and D are assigned to my "HighPriority" upload and download queues. For simplicity, let's consider only download, and assume that all four hosts are attempting to download as fast as they can. What should happen, to the best of my knowledge, is that four dynamic download queues are created (because of the mask setting), one per host. The two queues for A and B should get about 30% of my total download, which should be roughly shared between them (i.e. 15% each). Similarly, the two queues for C and D should get about 70% of my total download, which should also be roughly shared between them (i.e. 35% each).
-
So, does anyone know if the issues with floating rules and ICMP are still present with the new kernel in 2.4.4?
-
Seems like people are trying to figure out how share the bandwidth.. which is something I've been trying to achieve to with my "free wifi" :) Basically, I have 120/20 and I run several VLANs off of my Unifi AP. One of these is an open network with free wifi for the casual neighbor. I have two failover PIA VPN connections and send some of my VLANs through it:
VLAN 1 - through WAN, normal router (good for netflix, etc and casual house guests)
VLAN 2 - through PIA (good for privacy browsing, my default on my computer)
VLAN 3 - through PIA - wifi for the people but whatever they do, I don't want it traced back to me :)My idea is to give low priority to my free wifi (VLAN 3) but I have yet to figure out the best way to achieve that. Seems like I need a system that will give me priority but doing with floating rules on WAN, I don't think it will work unless I tag the packets coming out of the VLAN that are NAT'ed. I'm not sure this works. I can't really tag the openvpn client packets either as the openVPN connection is shared. My current approach was:
Create a WAN limiter with 95% of my ISP connections and set the proper queues on my VLAN rule that allows the NAT to happen. Is there a better way? Maybe this would be simpler with a diagram..
-
@dennypage If you mean faulty traceroute when you set floating rules, no, I still see this:
traceroute to news.com (64.30.224.82), 30 hops max, 60 byte packets 1 trogdor.X.com (192.168.0.1) 3.356 ms 3.338 ms 3.317 ms 2 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 5.911 ms 5.913 ms 5.897 ms 3 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 12.221 ms 12.208 ms 12.998 ms 4 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 11.866 ms 11.614 ms 11.632 ms 5 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 11.635 ms 11.608 ms 11.579 ms 6 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 11.576 ms 8.167 ms 8.120 ms 7 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 9.185 ms 9.822 ms 9.739 ms 8 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 133.579 ms 136.458 ms 136.432 ms 9 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 140.571 ms 137.084 ms 137.023 ms 10 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 141.143 ms 141.042 ms 140.487 ms 11 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 145.353 ms 144.733 ms 144.296 ms 12 phx1-rb-gtm3-tron-xw-lb.cnet.com (64.30.224.82) 145.292 ms 144.959 ms 142.029 msI think I will just appy the rules to my LAN interface, not floating rules.
-
@muppet said in Playing with fq_codel in 2.4:
@dennypage If you mean faulty traceroute when you set floating rules, no, I still see this:
Yes, exactly what I was asking. Thanks.
-
Well I upgraded to the 2.4.4 release today and thought I would share a few thoughts and tips on the upgrade process, since I had been using fq_codel in prior versions of pfSense already.
My initial setup had fq_codel enabled via the Shellcmd package, i.e. I enabled the scheduler using the ipfw command upon boot up (if you read back in this thread a little bit you'll see this configuration mentioned). Furthermore, I had two limiters, setup (one for upload and one for download) and 3 queues underneath each (they are weighted queues for three different network subnets). I did not remove any of these settings before upgrading to 2.4.4, hoping that they would maybe just carry over. Unfortunately, the settings did not carry over and after the upgrade to 2.4.4 and reboot, I was left only with two unconfigured limiters and no queues at all. I was able to reconfigure the limiters, but for some reason was unable to add queues (they would simply not show up on I saved and settings were applied). So I decided to just start from scratch: I went ahead and removed the limiters, any remnants of the old queues from my firewall rules, and also deleted the fq_codel setup command from ShellCmd. I then rebooted and after that was able to configure everything just fine - up and down limiters along with 3 queues underneath each of them. I applied the queues back to the firewall rules on the three subnets, and everything is working great like it did before the 2.4.4 upgrade. Also, like the @TheNarc alluded to already, it's not necessary to use a floating rule on the WAN interface, applying queues to the LAN side works fine as well.
If you have similar situation like mine where you already have an existing fq_codel setup in 2.4.3 or prior, and it's not too overly complicated, I would recommend removing it and just reconfiguring everything via the GUI once the upgrade to 2.4.4 is complete.
-
I had a very similar experience to @tman222. One important thing to note, is that if you were using the Shellcmd package in 2.4.3 for fq_codel, it is not sufficient to simply uninstall the package. If you uninstall the Shellcmd package without first deleting any shell commands that you installed with it, those shell commands will persist the package's uninstallation. You must first delete all shell commands that you had installed with the Shellcmd package, and then uninstall the Shellcmd package iteself.
-
@thenarc
Generally you don't need to uninstall it at all if you still using it. But you are right anyway, first delete all unneeded entries. -
Just tried all possible variants, playing with netgate recommended configuration, messed with masks, rules, changed everything wrongly and still getting my normal bandwidth and bufferbloat result, so I do thing that there is some configuration/package conflict which can be cause of those issues users reported above, that's why we need more information about what else is configured on firewall. Since some reported hardware is fully supported, I think the problem is limited to the software.
-
I'm finding no matter what I do, I get the following errors quite often in my dmesg:
config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy!This is straight after a reboot without me editing/configuring anything. They only appear a few times, sometimes once, sometimes twice, or in the example above, three times.
This is my /tmp/rules.limiter
pipe 1 config bw 97Mb codel target 5ms interval 100ms ecn sched 1 config pipe 1 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff codel target 5ms interval 100ms ecn pipe 2 config bw 19Mb droptail sched 2 config pipe 2 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 noecn queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff codel target 5ms interval 100ms noecnI have the pipes configured on my default LAN out rule, because if I configure them on a floating rule, I get the routing loop problem with ICMP as posted above.
-
@w0w said in Playing with fq_codel in 2.4:
Just tried all possible variants, playing with netgate recommended configuration, messed with masks, rules, changed everything wrongly and still getting my normal bandwidth and bufferbloat result, so I do thing that there is some configuration/package conflict which can be cause of those issues users reported above, that's why we need more information about what else is configured on firewall. Since some reported hardware is fully supported, I think the problem is limited to the software.
I hear ya. Wish I had the answer. Colelq shows A+ for bufferbloat. fq_codel I get no better than C and when I run speed tests my gateways show out of service because they are triggering the loss threshold.
-
I just tried changing the roots to droptail and rebooting, but no different:
pipe 1 config bw 97Mb droptail sched 1 config pipe 1 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff codel target 5ms interval 100ms ecn pipe 2 config bw 19Mb droptail sched 2 config pipe 2 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 noecn queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff codel target 5ms interval 100ms noecn <<logs>> ng_pppoe[11]: no matching session ng_pppoe[11]: no matching session ng_pppoe[11]: no matching session config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy!It seems to be working OK though:
[2.4.4-RELEASE][admin@trogdor.muppetz.com]/root: ipfw sched show 00001: 97.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 43 26721 0 0 0 00002: 19.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 29 2695 0 0 0 -
@gsmornot
What packages you have installed? What is WAN connection type? Do you have some other features configured on traffic shaper side? -
@muppet said in Playing with fq_codel in 2.4:
config_aqm Unable to configure flowset, flowset busy!
I've seen those messages also, once I've rebooted this morning and before. I am not sure does it impact anything.
