Playing with fq_codel in 2.4
-
@Veldkornet said in Playing with fq_codel in 2.4:
@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.
Hm. That should not be. Did You follow the guide I linked? Also what hardware is the pfsense running on? Limiters add extra CPU usage. Take a look with "htop -d 1" (need to install it first "pkg install htop") and see if You are peaking the CPU while doing the test.
-
@xciter327 ah! I didn't see the guide. I've now re-created everything as per the guide... although it didn't change the results...
I have a PCEngines APU2. I just had a look in "top" while doing the tests. Didn't really see anything climb very high at all. Even then, if it can handle the download, it should be able to manage the upload which is 10% of the download.
If I watch the test, the upload starts strong and climbs to around 20Mbit quickly, for 2 seconds or so, but then drops down to around 4/5 for the remainder of the test where it eventually finishes.
-
Could You perhaps post some picture of your firewall rules and limiter config? I am shaping my Ziggo connection on a Zotac NUC, which theoretically should be less powerful than the APU2. Also make sure You clear the states/reset the firewall when applying the limiters. Keep an eye in the system log for any log messages when You apply the limiters as well.
-
I can post screenshots instead of the below.. but it will take lots of space. I just checked everything though, and except for the speed limits, to me it looks the same as in the post. Might as well just be a copy paste. Looking at the floating rule screenshot though, I see that the WAN-In FQ-CoDel queue is pretty small considering all the tests I was doing... normal?
Also, see how the upload just dies off:
FQ_CODEL_OUT
Name: FQ_CODEL_OUT Bandwidth: 38 Mbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480fq_codel_out_q
Name: fq_codel_out_q Mask: None Queue Management Algorithm: Tail DropFQ_CODEL_IN
Name: FQ_CODEL_IN Bandwidth: 380 Mbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480fq_codel_in_q
Name: fq_codel_in_q Mask: None Queue Management Algorithm: Tail DropFirewall Rules - Floating:
policy routing traceroute workaround
Action: Pass Quick: Tick Apply the action immediately on match. Interface: WAN Direction: out Address Family: IPv4 Protocol: ICMP ICMP subtypes: Traceroute Source: any Destination: any Description: policy routing traceroute workaroundlimiter drop echo-reply under load workaround
Action: Pass Quick: Tick Apply the action immediately on match. Interface: WAN Direction: any Address Family: IPv4 Protocol: ICMP ICMP subtypes: Echo reply, Echo Request Source: any Destination: any Description: limiter drop echo-reply under load workaroundWAN-In FQ-CoDel queue
Action: Match Interface: WAN Direction: in Address Family: IPv4 Protocol: Any Source: any Destination: any Description: WAN-In FQ-CoDel queue Gateway: Default In / Out pipe: fq_codel_in_q / fq_codel_out_qWAN-Out FQ-CoDel queue
Action: Match Interface: WAN Direction: out Address Family: IPv4 Protocol: Any Source: any Destination: any Description: WAN-Out FQ-CoDel queue Gateway: WAN_DHCP In / Out pipe: fq_codel_out_q / fq_codel_in_q -
Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?
-
@xciter327 said in Playing with fq_codel in 2.4:
Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?
Well I have Suricata, no squid. Although I turned Suricata off and it made no difference.
Download maxes the CPU, but upload doesn't seem to do much...
Download:00001: 38.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 2478 112953 488 26508 0 00002: 380.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 33141 49232048 193 287300 12
Upload:00001: 38.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 31 45860 0 0 0 00002: 380.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 16 664 0 0 0
-
Hm, I don't think You will have a great experience with all those thing's You've loaded on that little box. I read somewhere the APU2 is good for ~400Mpbs without Suricata/other heavy software.
I would try disabling all add-on(haProxy, Openvpn, pfblocker, Suricata, snmp, fancy unbound settings, tftp-server etc.) and try vanilla pfsense with just the limiters(via floating rules) and a simple "Allow all" on the LAN side. If your CPU is peaked(like You have on the download test), then better run without limiters.
-
Oh? The CPU load on it is almost non-existant usually. This is the first time I've seen it go so high now with the traffic shaping. Even so, I can understand that the load may be an issue for download and I'm okay with that.
It's the upload speed that's annoying me...... which is not doing much on the CPU side.
-
Well If I look at the picture, the upload is mostly pegged by Surricata. So that would be the first thing I disable. Also note that not all packages work normally with limiters. There used to be issues with Suricata/Squid and limiters, I don't know if it was ever fixed.
-
Okay, I disabled all of the packages, ran the test, upload good.
Then I enabled each one and tested until the upload speed decreased.So it seems I have discovered the culprit... I have an OpenVPN Client setup to PIA, this is what's causing the bad upload speeds. I have an alias with an IP range defined, and basically everything within that alias should go over the VPN. And then I have an interface & gateway which is bound to the OpenVPN client; with a Firewall rule which says anything to that alias should use the VPN gateway.
Is there anything special that I would need to do with my rules in this situation? The traffic itself isn’t going over the VPN.
-
My guess would be to place it at the very top of the floating rule set. I believe pf does "match most specific" unless "quick match" is selected.
-
@xciter327 said in Playing with fq_codel in 2.4:
My guess would be to place it at the very top of the floating rule set. I believe pf does "match most specific" unless "quick match" is selected.
pf is last match wins unless quick is enabled. Quick is enabled by default on interface rules but has to be selected on floating rules
-
Good to hear. I did not know about the default quick on interface rules. That explains some things.
-
I've rebuilt my network a bit, but I am not sure when this was happened — one PC behind pfSense always get the A or A+ rating and the other get the D or even lower grade rating on dslreports, both are on the same LAN. FQ_CODEL configuration is similar to the others in this thread except only that it is applied on the LAN side (LAN to any). When I disable limiters, then both PCs get A or A+ rating. Does anyone have an idea, what is going wrong?
-
What about Dual WAN?
I've used the previous post (https://forum.netgate.com/post/807490), with my best guess modifications for two WANs (duplicated rules for each WAN, and duplicated queues), and received good grades on the DSLReports test. However, I use a dual WAN fallover setup. I noticed, today, when WAN1 went down and PFSense switched to WAN2, all traffic stopped. Removing the floating rules fixed the issue.
The two WANs have different speeds.
Does anyone have this working with dual WAN setup (different WAN speeds) in a fallover configuration? If so, what do your floating rules look like?
@uptownVagrant Do you have any suggestions to modify your steps for dual WAN w/ fallover?
Thanks, in advance, for any suggestions.
-
OK! I have fallen down the rabbit hole of bufferbloat! EEK!
I've followed the Netgate video/slides on creating limiters without much benefit.
I get a B for bufferbloat at DSLReports and A's for Quality and Overall.
I have a 300/25 (advertised) Comcast cable connection. Tried changing the bandwidth down (reducing) , and the queue lengths (+/-/eliminating) w/o benefit. At this point, the DL queue length is set at 2000 and seems to be best so far. UL does not appear to be a problem. Any thing I can do short of getting a life? :) -
@provels without the limiter rules enabled, and your filters reloaded and old states killed, what speeds are you seeing when you run the dslreports.com/speedtest for 60 seconds? It could be that Comcast's burst capacity during short tests may be heavily skewing your results - I seem to remember the default test duration only being 20 seconds. I think you may have to create an account and go into the preferences of the speed test to increase upload and download duration to 60s. There's always the flent project if you're savvy.
Another thought, depending on how close your current limiter config is to the Comcast circuit sustained cap, you may be running into some buffering on your cable modem. What DOCSIS version is your cable modem operating?
I'm assuming you're testing with a computer attached to your router/switch with an Ethernet cable and not using Wi-Fi.
You could try halving your advertised values in your limiters and then increasing from there after each test. Note, if you're using FQ_CoDel then the queue length that you set in the limiter config is not being used. FQ_CoDel manages queue lengths dynamically and works to dequeue packets within the target you set.
Here's a guide that might help although I would suggest leaving target, interval, quantum, limit, and flows set as the pfSense defaults for the initial tests.:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815Best of luck
-
@ohbobva I'm running dual WAN with different speeds, and all you need to do is duplicate your setup. i.e. create four different limiters - upstream WAN1, upstream WAN2, downstream WAN1, downstream WAN2, and then a queue within each of those limiters. Set up four floating rules for WAN1-IPv4, WAN1-IPv6, WAN2-IPv4, WAN2-IPv6 and it all works as advertised. Be careful not to just create two queues within the same limiter as the limit will be the aggregate bandwidth of the two.
-
@Pentangle said in Playing with fq_codel in 2.4:
@ohbobva I'm running dual WAN with different speeds, and all you need to do is duplicate your setup. i.e. create four different limiters - upstream WAN1, upstream WAN2, downstream WAN1, downstream WAN2, and then a queue within each of those limiters. Set up four floating rules for WAN1-IPv4, WAN1-IPv6, WAN2-IPv4, WAN2-IPv6 and it all works as advertised. Be careful not to just create two queues within the same limiter as the limit will be the aggregate bandwidth of the two.
I'm having hard time to follow you, do you mind taking a screenshot of your config ? Thanks in advance, very much appreciated.
-
@uptownVagrant said in Playing with fq_codel in 2.4:
Thanks very much for your quick reply ;)
@provels without the limiter rules enabled, and your filters reloaded and old states killed, what speeds are you seeing when you run the dslreports.com/speedtest for 60 seconds? It could be that Comcast's burst capacity during short tests may be heavily skewing your results - I seem to remember the default test duration only being 20 seconds. I think you may have to create an account and go into the preferences of the speed test to increase upload and download duration to 60s. There's always the flent project if you're savvy.
Did as you suggested, I get results as follows. Looked at flent, but I'm Windows. My upload is no issue, with or without limiters, but the download bloat is normally in the 60-90ms range.
179/24 Mbps 189/23 169/24This result is pretty typical.
This is a typical result using the Ookla Speed Test (client version)
>Another thought, depending on how close your current limiter config is to the Comcast circuit sustained cap, you may be running into some buffering on your cable modem. What DOCSIS version is your cable modem operating?
Modem is a Netgear CM600, DOCSIS 3.0, 24x8, Broadcom
I'm assuming you're testing with a computer attached to your router/switch with an Ethernet cable and not using Wi-Fi.
Yes, testing wired, but my pfSense is a VM running in Hyper-V and I'm testing from the host Desktop session. Maybe I'll wire up my laptop.
You could try halving your advertised values in your limiters and then increasing from there after each test.
I'll start by halving my DL BW and add from there.
Note, if you're using FQ_CoDel then the queue length that you set in the limiter config is not being used. FQ_CoDel manages queue lengths dynamically and works to dequeue packets within the target you set.
Noted and deleted.
Here's a guide that might help although I would suggest leaving target, interval, quantum, limit, and flows set as the pfSense defaults for the initial tests.:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815Best of luck
Thanks for the tips. I'll check out the guide and report back. Thanks again! :)
