Playing with fq_codel in 2.4
-
i actually just reverted to using a simple TCP/UDP floating rule, setting up the non-floating was a huge pain (I had it working before) but can't remember how.
I'm going to be creating exemptions / rules that exempt specific clients in the future - now that it's working fine (on my primary use case, Windows!)
Here is the exact rule I have (that is, somehow, letting pfSense UDP traceroute work - I am tired of messing with it / don't care). Anything that is missing means it's default.
Action: Pass
Interface: WAN
Direction: Out
Addr Family: IPv4
Protocol: TCP/UDP
Gateway: WAN_DHCP (my gateway selected)
In/Out Pipe: WANUpQ (my name) & WANDownQ (my name)Here are the rules I have:
Firewall Rules > Traffic Shaper > "Limiters""CODEL_QMDown"
Limiter: Enabled
Bandwidth: 320 Mbit
Queue Management Algorithm: CoDel
Scheduler: FQ_CODELSubqueue, "WANDownQ"
Queue algorithm: CoDel"CODEL_QMUp"
Limiter: Enabled
Bandwidth: 340Mbit (yes, I have higher upload than download)
Queue management algorithm: CoDel
Scheduler: FQ_CODELSubqueue, "WANUpQ"
Queue algorithm: CoDel
I know these are basically defaults - I'm not sure why these work? I simply re-created everything, and now everything works great... it's quite odd. I'm curious if there is an issue with pfSense and having the rules edited, or large changes made to the queues (I hit Save/Apply, particularly, after every single change / creation - no going back and editing for me!)
Quite odd. I've used FQ_CoDel in other devices/implementations and have never run into these nuances. It's working great, now, though - but the traceroute/etc was bothering me horribly. With these rules (no floating rule for 'any') it seems to be working great... for... whatever reason.
-
Hi all!
Configured limiters and now my traceroute always shows resolved IP address instead of actual hops.
I`ve read here this is common but just to be sure, is this supposed to be like this or is it a bug?Thanks!
-
Hi guys,
I’m also having an issue with my upload speed... So I have a 400Mbit down / 40 Mbit up connection, and I've followed the guide here to enable the limiters, filling in the 400 and 40 as my down and up speeds.
However when I do speedtests, while my bufferbloat is now gone... I no can no longer reach my maximum speeds.
Down I may get around 360Mbit max which I can live with butup is just pathetic, maybe between 2-8Mbit at most....Is this known? Anything I can do to improve this? I played with the Queue Length value which helped a bit, but in the end didn't really have too much of an impact....
Limiters: 00001: 400.000 Mbit/s 0 ms burst 0 q131073 10000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 40.000 Mbit/s 0 ms burst 0 q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 400.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 00002: 40.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 Queues: q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECNDisabled:
Enabled:
-
You are supposed to set the limiters lower than your actual connection speed. I set mine to 95% of tested(not ISP advertised) speed.
Also please use the guide posted a little while earlier. The one from the hangouts session is outdated and does not cover a bug workaround.
https://forum.netgate.com/post/807490
P.S - I am also on Ziggo NL 240/24 connection.
-
@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.
-
@Veldkornet I don't suppose it's a PPPoE connection? I had a similar problem.
-
Hi how did you resolve pppoe problem?
Thanks! -
@maverick_slo I didn't. I am only able to use the Download shaper for my PPPoE WAN. It works perfectly on my other WAN which is DHCP.
-
@csutcliff Nope, mine is just a DHCPv4 connection
-
@Veldkornet said in Playing with fq_codel in 2.4:
@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.
Hm. That should not be. Did You follow the guide I linked? Also what hardware is the pfsense running on? Limiters add extra CPU usage. Take a look with "htop -d 1" (need to install it first "pkg install htop") and see if You are peaking the CPU while doing the test.
-
@xciter327 ah! I didn't see the guide. I've now re-created everything as per the guide... although it didn't change the results...
I have a PCEngines APU2. I just had a look in "top" while doing the tests. Didn't really see anything climb very high at all. Even then, if it can handle the download, it should be able to manage the upload which is 10% of the download.
If I watch the test, the upload starts strong and climbs to around 20Mbit quickly, for 2 seconds or so, but then drops down to around 4/5 for the remainder of the test where it eventually finishes.
-
Could You perhaps post some picture of your firewall rules and limiter config? I am shaping my Ziggo connection on a Zotac NUC, which theoretically should be less powerful than the APU2. Also make sure You clear the states/reset the firewall when applying the limiters. Keep an eye in the system log for any log messages when You apply the limiters as well.
-
I can post screenshots instead of the below.. but it will take lots of space. I just checked everything though, and except for the speed limits, to me it looks the same as in the post. Might as well just be a copy paste. Looking at the floating rule screenshot though, I see that the WAN-In FQ-CoDel queue is pretty small considering all the tests I was doing... normal?
Also, see how the upload just dies off:
FQ_CODEL_OUT
Name: FQ_CODEL_OUT Bandwidth: 38 Mbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480fq_codel_out_q
Name: fq_codel_out_q Mask: None Queue Management Algorithm: Tail DropFQ_CODEL_IN
Name: FQ_CODEL_IN Bandwidth: 380 Mbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480fq_codel_in_q
Name: fq_codel_in_q Mask: None Queue Management Algorithm: Tail DropFirewall Rules - Floating:
policy routing traceroute workaround
Action: Pass Quick: Tick Apply the action immediately on match. Interface: WAN Direction: out Address Family: IPv4 Protocol: ICMP ICMP subtypes: Traceroute Source: any Destination: any Description: policy routing traceroute workaroundlimiter drop echo-reply under load workaround
Action: Pass Quick: Tick Apply the action immediately on match. Interface: WAN Direction: any Address Family: IPv4 Protocol: ICMP ICMP subtypes: Echo reply, Echo Request Source: any Destination: any Description: limiter drop echo-reply under load workaroundWAN-In FQ-CoDel queue
Action: Match Interface: WAN Direction: in Address Family: IPv4 Protocol: Any Source: any Destination: any Description: WAN-In FQ-CoDel queue Gateway: Default In / Out pipe: fq_codel_in_q / fq_codel_out_qWAN-Out FQ-CoDel queue
Action: Match Interface: WAN Direction: out Address Family: IPv4 Protocol: Any Source: any Destination: any Description: WAN-Out FQ-CoDel queue Gateway: WAN_DHCP In / Out pipe: fq_codel_out_q / fq_codel_in_q -
Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?
-
@xciter327 said in Playing with fq_codel in 2.4:
Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?
Well I have Suricata, no squid. Although I turned Suricata off and it made no difference.
Download maxes the CPU, but upload doesn't seem to do much...
Download:00001: 38.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 2478 112953 488 26508 0 00002: 380.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 33141 49232048 193 287300 12
Upload:00001: 38.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 31 45860 0 0 0 00002: 380.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 16 664 0 0 0
-
Hm, I don't think You will have a great experience with all those thing's You've loaded on that little box. I read somewhere the APU2 is good for ~400Mpbs without Suricata/other heavy software.
I would try disabling all add-on(haProxy, Openvpn, pfblocker, Suricata, snmp, fancy unbound settings, tftp-server etc.) and try vanilla pfsense with just the limiters(via floating rules) and a simple "Allow all" on the LAN side. If your CPU is peaked(like You have on the download test), then better run without limiters.
-
Oh? The CPU load on it is almost non-existant usually. This is the first time I've seen it go so high now with the traffic shaping. Even so, I can understand that the load may be an issue for download and I'm okay with that.
It's the upload speed that's annoying me...... which is not doing much on the CPU side.
-
Well If I look at the picture, the upload is mostly pegged by Surricata. So that would be the first thing I disable. Also note that not all packages work normally with limiters. There used to be issues with Suricata/Squid and limiters, I don't know if it was ever fixed.
-
Okay, I disabled all of the packages, ran the test, upload good.
Then I enabled each one and tested until the upload speed decreased.So it seems I have discovered the culprit... I have an OpenVPN Client setup to PIA, this is what's causing the bad upload speeds. I have an alias with an IP range defined, and basically everything within that alias should go over the VPN. And then I have an interface & gateway which is bound to the OpenVPN client; with a Firewall rule which says anything to that alias should use the VPN gateway.
Is there anything special that I would need to do with my rules in this situation? The traffic itself isn’t going over the VPN.
-
My guess would be to place it at the very top of the floating rule set. I believe pf does "match most specific" unless "quick match" is selected.
