Playing with fq_codel in 2.4
-
@forbiddenlake said in Playing with fq_codel in 2.4:
@JonH I have the same problem, and after reading 600 more posts in this topic, I believe I have the answer for you.
You're using a traceroute that uses UDP by default, and you're shaping TCP and UDP, and this is a bug in pfSense.
You can work around it by using ICMP for traceroutes, e.g. (disclaimer: I'm using Linux):
alias traceroute='traceroute -I'
Some here have mentioned that you may also be able to work around it by applying the limits on LAN rules, not floating rules, however the alias is good enough for me for now, so I stopped reading at around 600 posts and can't show you what to do there :)
Hi, you have details of this bug? thanks.
-
@chrcoluk I believe this "bug" is being referenced. Certain configuration will cause pfSense to not decrease the TTL when forwarding. Policy routing is used with direction=out limiters, so it's a common cause of the behavior folks are seeing in this thread where traceroute doesn't work. If you're using configuration similar to this, there is a provision for ICMP traceroute but if you are using a traceroute program that is using UDP packets then those packets would use policy routing and the TTL would not decrease on those packets at pfSense.
-
This post is deleted! -
@robnitro said in Playing with fq_codel in 2.4:
What I don't understand is that with no limiters and QOS disciplines not enabled, my traceroutes are still not working. Where else could there be an issue? I don't need qos now with gigabit fiber.
The ISP could be the issue. I know Verizon has issues with traceroutes not showing properly in some areas, essentially showing your router then the destination host in a 2-hop traceroute (or more if you have multiple routers between you and Verizon).
This thread on DSLReports shows it starting back in late 2018, but still noted as happening in August this year in the thread. I'm still seeing the issue though. Has VZ disabled TTL propagation?
-
Hi guys an update from myself.
I did some more messing around with my limiters and changed my main pipe to this.
FQ_PIE target 5ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.1 quantum 300 limit 1000 flows 1024 ECN CapDrop TS Derand
also
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms NoECN
So I ditched droptail.
Now on downstream congestion it performs "way" better. I do still have to provide a sizely overhead for it to not affect latency, but on FQ_CODEL with droptail I needed to supply a massive 50-60% overhead, on this new configuration 2% isnt enough but it seems 12% is. I have yet to try anything between 2% and 12% to see how low I Can get it, but already 12% I consider a massive improvement. :)
Also the masking is set to on src /24 not dest.
-
@w0w said in Playing with fq_codel in 2.4:
both are on the same LAN
I have had the same issues with pf .
FQ_Codel in 2.4.4 doesnt work with floating rules.
Only works via gui as a lan limiter with children and weighted subqueues and even so tcp and udp traffic (udp voip) still experiences spikes under tcp load. ; Added that I have udp voip sub-queues (weighted) under the parent limiter,if not used like this fq_codel and fq_pie (with no interface shaping) its a mess.
On top of this traffic shapers on the interfaces always hinders the floating rule method so I have disabled traffic shping on the interfaces as per the linux method.
What is odd that fq_codel actually works with all ipv4 traffic on debian with all protocols very well,when applied to the wan i/f.
With freebsd for some reason fq_pie only seems to work with udp packets without shaping all ipv4???,whereas fq_codel with altq only works with tcp? as per the original codel implementation.
Hoping smart queuing works soon in the distribution as fq-codel does not perform the same as linux by far.
Anyone else had this issue ?.
-
@m8ee how does your rules look like? I have not had any problems getting fq-CoDel to work in either 2.4.4 or 2.4.5 with limiters and floating rules.
-
Hi all,
I'm a new PFsense user here.
I set up my FQ_Codel in my fresh install as per https://www.youtube.com/watch?v=o8nL81DzTlU
However I get a bunch of flowset errors in my syslog any ideas? I've been reading it's a bug but these messages are from 2017... and I'm unsure if they're still a thing in 2020? I'm currently at work on break and decided to chip in my concern.
Is the video guide maybe outdated? Does anyone have the 2020 version?
Thanks.
-
@Zeny001 try setting queue management algorithm under the queues to Tail drop and see if that helps.
-
Wow. That did it. From C bufferbloat on DSLreports to A+.
Thanks a bunch. For anyone having trouble remember to uncheck ECN since tail drop does not support it
I lost about 300mbps of bandwidth though.
I have a gigabit connection and I'm getting about 600mbps now, was getting 900ish before. I dont really care though, but if anyones got any tips let me know :)
-
@Zeny001 you could try lowering limit by a factor of ten and increase flows by the same and see if that makes a difference.
-
Just wanted to say thanks - I applied the technique in the video and my Bufferbloat and Quality on the DSL reports test went from C/C to A/A+. Hopefully this will resolve some weird network issues I've been having.
-
@uptownVagrant - wanted to thank you for posting these detailed steps on how to setup FQ-CoDel using floating rules.
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815
Originally I had FQ-CoDel setup on my LAN interfaces, but using floating rules the approach is definitely simplified. Everything is working great and as expected. I use Linux so do run into the issue with traceroute (mentioned earlier in this thread) because UDP is used by default. Easily worked around though by using traceroute -I instead (which will use ICMP). Do you have any suggestions how to modify the floating rules so that traceroute using UDP might work properly?
Thanks in advance.
-
This post is deleted! -
Não tive mais problemas depois que fiz essa configuração :
-
@tman222 using traceroute in windows gives the same problem, the only solution so far seems to be making a rule omitting ICMP traffic from FQ-CoDel.
Weird thing is that it works fine with IPfire which i briefly tried some months ago, so it might be something with the FreeBSD implementation of it. -
@tman222 said in Playing with fq_codel in 2.4:
Do you have any suggestions how to modify the floating rules so that traceroute using UDP might work properly?
For me the following works:
Create floating rule with limiters (with match action) not on "WAN out" but on "LAN/VLAN in" (in one floating rule you can select multiple interfaces/vlans) for traffic with source "any" and destination "not-your-local-networks-alias" - that works great for me, and for my easy setup I need only one floating rule for all my VLANS and with "in" traffic you don't need to set gateway.Unfortunately it won't work for traffic with source IP of your pfsense router
-
@uptownVagrant said in Playing with fq_codel in 2.4:
1.) Add quick pass floating rule to handle ICMP traceroute. This rule matches ICMP traceroute packets so that they are not matched by the WAN-Out limiter rule that utilizes policy routing. Policy routing breaks traceroute.
Are you guys actually sure it is this issue? The linked doc page and bug #932 is quite the opposite what we see here. It makes perfect sense that policy routing in certain cases does not decrease TTL making router (pfsense box) invisible in traceroute, but the rest of the hops looks ok.
Our issue is that it shows only one hop which is destination and nothing else. Which arguably doesn't make sense. Have anyone actually look into it or just found not quite similar traceroute issue and call it a day?
With Limiter:
1 <1 ms <1 ms <1 ms <...> [192.168.0.1] 2 * * * Request timed out. 3 11 ms 14 ms 8 ms one.one.one.one [1.1.1.1] 4 12 ms 11 ms 12 ms one.one.one.one [1.1.1.1] 5 12 ms 14 ms 18 ms one.one.one.one [1.1.1.1] 6 13 ms 13 ms 11 ms one.one.one.one [1.1.1.1] 7 13 ms 10 ms 11 ms one.one.one.one [1.1.1.1] 8 55 ms 54 ms 54 ms one.one.one.one [1.1.1.1] 9 49 ms 49 ms 51 ms one.one.one.one [1.1.1.1] 10 37 ms 37 ms 56 ms one.one.one.one [1.1.1.1] 11 55 ms 52 ms 50 ms one.one.one.one [1.1.1.1]
Without Limiter:
1 <1 ms <1 ms <1 ms <...> [192.168.0.1] 2 * * * Request timed out. <removed some hops, but they are properly shown> 8 55 ms 53 ms 50 ms ae-12.r24.amstnl02.nl.bb.gin.ntt.net [129.250.3.81] 9 59 ms 54 ms 53 ms ae-7.r03.amstnl02.nl.bb.gin.ntt.net [129.250.2.103] 10 58 ms 59 ms 60 ms 81.20.65.150 11 56 ms 64 ms 56 ms one.one.one.one [1.1.1.1]
-
I wanted to check to see if anyone else is having issues with the newly released 2.4.5 (released 2020-03-26) using fq_codel. I had a perfectly working limiter setup with 2.4.4_3 and upgraded to 2.4.5 and now It will no longer load up the limiter, having the following in the logs:
- php-fpm[351]: /rc.filter_configure_sync: The command '/sbin/kldload dummynet' returned exit code '1', the output was 'kldload: can't load dummynet: No such file or directory'
- php-fpm[351]: /rc.filter_configure_sync: The command '/sbin/ipfw /tmp/rules.limiter' returned exit code '1', the output was 'Line 2: setsockopt(IP_DUMMYNET_CONFIGURE): Protocol not available'
I completely removed my limiters/queue and floating rule, recreated within the gui and the same occurs. Was the dummynet kernel module not built with the 2.4.5 release or can I manually add it back? I found an old thread that this happened within the builds back in 2010 but nothing else current. At this point limiters are dead for me due to the non existent dummynet kernel module.
uname info:
FreeBSD XXXXXXXXXX 11.3-STABLE FreeBSD 11.3-STABLE #236 21cbb70bbd1(RELENG_2_4_5): Tue Mar 24 15:26:53 EDT 2020 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense amd64Just an FYI I resolved the issue incase anyone else encounters this. Not sure if it is the correct fix. but I utilize ZFS instead of UFS, and it appears the old /boot use to actually be /bootpool/boot but after upgrade to 2.4.5 it has a new /boot under the root filesystem. This new /boot did not contain the /boot/kernel subdirectory and kernel modules so I did the following and rebooted
mkdir /boot/kernel
# cp -p /bootpool/boot/kernel/* /boot/kernel/
# cp -p /boot/loader.conf /boot/loader.conf.orig
# cp /bootpool/boot/loader.conf /boot/loader.confThen rebooted and the system was able to load the dummynet kernel module - resolved.
-
This post is deleted! -
Hi @uptownVagrant - I have a quick question regarding your quick pass floating rule for echo-request and echo-reply from your configuration in post 815:
2.) Add quick pass floating rule to handle ICMP echo-request and echo-reply. This rule matches ping packets so that they are not matched by the limiter rules. See bug 9024 for more info.
Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround
Click Save
I see that this rules also allows the firewall (WAN IP) to respond to pings from the internet. How would you change this rule to avoid that (i.e. only allow outgoing ping but not incoming)? Would it just be as simple as changing the Source from "any" to e.g. "WAN Address"?
Thanks in advance.
-
@tman222 setting direction to "out" and choose your gateway in advanced should give the result you want.
-
@bobbenheim said in Playing with fq_codel in 2.4:
@tman222 setting direction to "out" and choose your gateway in advanced should give the result you want.
Thanks @bobbenheim - that worked! I actually did try setting the direction on the rule to "Out" at first, but then outbound pings from LAN interfaces stopped working. Now I see why - I didn't realize I needed to set the gateway in advanced settings. Thanks again.
-
@gsakes @uptownVagrant etc. I've been looking for some coherent recommendations for zoom, especially, as to how to improve your videoconferencing experience. Also curious as to how just generic fq_codel is treating that for y'all?
Oy, what a long thread this has been!
-
Hi, I have a problem, if after enabling the limiter I start to download a game that saturates the band and in the meantime I perform a ping, the ping increases from 10 ms to 50 ms.
Here are the screenshots of the settings, my bandwidth is 120/20.
Should I change something?spoilerspoiler -
@mikekoke Sorry my English, I play without problems with these settings, stable ping + or - 2ms of variation.
-
@mikekoke My result
-
Hi, thanks for the reply, but i have already tried to enable those two settings and the result does not change.
By doing the test on dslreports with the configuration shown in the screenshots i get three A + but if i download a game and at the same time i ping 8.8.8.8 the latency increases from 10 to 50 ms. -
@mikekoke I did the download test and I didn't have this problem.
-
When I did my tests I ran multiple downloads simultaneously to saturate the band.
-
@mikekoke Mystery.
-
when you say "downloading a game" do you mean, steam? Steam really abuses the network, opening 10 or more full rate flows, and even with cake, it's hard to beat them down.
-
Yes, Steam.
I'll try doing normal downloads.
Would the best settings be Tail Drop, FQ_Codel, Queue length at 10000 and ECN both on download and on upload or only on download? I read that the ECN annoys the upload.
The Queues of both on Tail Drop but without ECN and Queue length?
Sorry for the questions but I have read the discussion and there are many opinions on the matter. -
Does it still give the same issue when you reduce the headline bandwidth you have set in the limiters?
-
Yes, even if the bandwidth is reduced, the ping increases the same.
-
This post is deleted! -
@mikekoke Follow this tutorial that works great!
-
Have you tried changing the quantum to < 536 or 300?, I have had better success at these packet size quantums.
Albeit TCP suffers a bit with a 300 quantum.Also had more success with fq_pie with 300 quantum.
@ dev - fq_pie needs an area in the gui to change the quantum ,pls .
I also run fq_pie lan parents for voip and gaming with greater weights given to the udp in addition to the wan
pipes.I still cant figure out why pfsense wont perform as well with udp packets as a linux router does with QOS/cake on
without quantum changes ?.Edit ;
In testing I have found layer 3 tunnel,gre or openvpn works much better and avoids the tcp overhead,you wont notice as much fluctuation,even with the wan pipes setup as standard for voip and udp games.During full saturation of the pipe the voip and game jitter hardly varies from a no tcp load scenario. This shoudlnt have to be done but works better than cake,pie or fq_codel.Note ;If you try the openvpn method you will need to change the nat and firewall rules for udp to point through the vpn gateway.
-
Assume that I've correctly implemented everything in @uptownVagrant's post #815. It works (splendidly). Thanks for the good instructions, man.
Then, because I have a second physical WAN interface and public IP for my captive portal (working perfectly for several months), let's say I do everything again in parallel - unique limiters, unique queues, and unique rules #3 & 4, setting rule #4's gateway to the captive portal WAN interface.
The LAN still works, but when tested, the captive portal won't do more than 1/4 of the available upload bandwidth.
Anyone feel like helping a brother out with a W.A.G. as to why this might be happening?
-
Thank you so much to everyone in this thread who has helped in testing, and especially to @uptownVagrant for putting in as much time as they have to help the all of us. This has been working well on my network and I'm very much appreciate this massive thread.
I would like to take this one step further and tie this into bandwidth limits for certain VLAN's of mine. Specifically, I'd like to limit bandwidth in both my IoT network and my guest network.
In an ideal scenario, this FQ CoDel rule would cover the entire WAN interface including the Guest and IoT VLAN (which it already does), but then I can set a second child limiter that only limits the bandwidth for specific interfaces. This way there's a single up queue and a single down queue for the entire WAN, but certain VLAN's have an additional restriction on their bandwidth.
What would be the best way to have a single queue as discussed here, but also have bandwidth limits on specific interfaces/VLAN's?