QoS / Traffic Shaping / Limiters / FQ_CODEL on 22.05
-
Update: the issue has been fixed! see redmine #13026
I'm hitting the same problem described in redmine #9024: nat + a limiter + fq_codel dropping near all ping traffic under load
Running 22.05.a.20220328.0600 (not using captive portal)
I tried following the steps at https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html as well as the oft-referenced video by LawrenceTech but no matter what, my typically well-behaved 1Gbps FIOS connection becomes basically unusable (100% packet loss after a few seconds of running bufferbloat test), triggering failover to my 4G LTE backup connection which does not have any shaper applied.
Is anyone else having issues? Or is there some new info regarding using limiters in 22.05 / 2.7?
my settings:
- Firewall > Traffic Shaper > Limiters:
- WAN1Down/WAN1DownQ
- bandwidth: 850Mbps
- Queue mgmt algo: Tail Drop
- Scheduler: FQ_CODEL (5/100/1514/10240/1024)
- Queue length: 3000
- ECN: checked
- Firewall > Rules > Floating:
- Action: Pass
- Quick: checked
- Interface: WAN1
- Direction: out
- Family: IPv4
- Protocol: any
- Source: WAN1 address
- Dest: Any
- Gateway: WAN1
- In/Out Pipe: WAN1UpQ / WAN1DownQ
- Firewall > Traffic Shaper > Limiters:
-
Results on 1G/1G FIOS
-
@luckman212 I absolutely have issues that I can't resolve with fatally high ping and packet loss spikes on sustained, multi-stream downloads running CE 2.6.0-RELEASE (as tested using fast.com and maxing out the parallel connections at 30). I'm also running fq_codel limiters but have tried altq too. And even limiting my downstream to 25% (i.e. 100Mbps on a 400Mbps connection) does not help. I know that I didn't have this problem in the past, using the same hardware, and to my knowledge the same limiter configuration. But I can't say I recall whether the problem began with a specific pfSense version update or other change. Now, my downstream was capped at 200Mbps for a long time before it went to 400, but then I limited it to 180 or so and experienced none of these latency and packet loss issues, whereas now even limiting to 100 won't resolve them. Of course, perhaps this just reflects inadequate or poorly configured ISP infrastructure that's beyond my control; I admit that I'm not really certain how to determine that. I'm going to try to get my hands on a cheap consumer-grade router to swap in and see if I get equally poor results with it. I'll report back if I do . . .
-
Just want to add that that I am not using Captive Portal on this system, since it's mentioned often e.g. in Upgrade 2.5.2 to 2.6.0, upgrade success, Limiters not passing
-
@luckman212 try bumping up the queue length to 8000 or so, and don't forget to increase the pipe limit in System Tuneables
-
My current anti-bufferbloat config is a combination of this two guides:
https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
https://isc.sans.edu/forums/diary/Securing+and+Optimizing+Networks+Using+pfSense+Traffic+Shaper+Limiters+to+Combat+Bufferbloat/27102/
It gives me a score A/A+ on https://www.waveform.com/tools/bufferbloat with 400down/15up ISP line.My config:
WAN Upload Limiter
Firewall > Traffic Shaper > Limiters > New Limiter
• Enable > Enable limiter and its children [X]
• Name: WANUpload
• Bandwidth: 95% of your ISP upload speed
• Mask: None
• Description: empty
• Queue Management Algorithm: Tail Drop
• Scheduler: FQ_CODEL
• Queue length: 1000
• ECN: [X]
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 1024WAN Upload Limiter Queue
• Enable > Enable this queue [X]
• Name: WANUploadQ
• Mask: None
• Description: empty
• Queue Management Algorithm: Tail Drop
• Queue Length: empty
• ECN: [ ]WAN Download Limiter
Firewall > Traffic Shaper > Limiters > New Limiter
• Enable > Enable limiter and its children [X]
• Name: WANDownload
• Bandwidth: 95% of your ISP download speed
• Mask: None
• Description: empty
• Queue Management Algorithm: Tail Drop
• Scheduler: FQ_CODEL
• Queue length: 1000
• ECN: [X]
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 1024WAN Download Limiter Queue
• Enable > Enable this queue [X]
• Name: WANDownloadQ
• Mask: None
• Description: empty
• Queue Management Algorithm: Tail Drop
• Queue Length: empty
• ECN: [ ]Floating Rule #1
Firewall > Rules > Floating
• Action: Pass
• Quick: Apply the action immediately on match [X]
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Traceroute
• Source: any
• Destination: any
• Description: Traceroute routing workaround
• Advanced Options > Gateway: WAN_DHCPFloating Rule #2
• Action: Pass
• Quick: Apply the action immediately on match [X]
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: ICMP
• ICMP subtypes: Echo reply, Echo request
• Source: any
• Destination: any
• Description: Limiter drop ping traffic under load workaround
• Advanced Options > Gateway: WAN_DHCPFloating Rule #3
• Action: Pass
• Quick: Apply the action immediately on match [X]
• Interface: WAN
• Direction: out
• Address Family: IPv4
• Protocol: any
• Source: WAN Address
• Destination: any
• Description: WAN CoDel Limiters
• Advanced Options:
Gateway: WAN_DHCP
In Pipe: WANUploadQ
Out Pipe: WANDownloadQ -
@jackyaz To clarify, you're referring to the queue length of the child queue of the limiter, not the queue length of the limiter itself, right? As I understand, the latter is ignored when the scheduler is fq_codel? And what is the pipe limit system tuneable you refer to? I looked but it wasn't obvious to me by name, and I couldn't seem to find it by googling. Thanks for your response!
-
@thenarc I'm referring limiter itself, but I may be wrong, I haven't done any testing against lower limits on my 1000/50 connection.
in the docs from here: https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
Queue Length
Can vary depending on the speed of the link, but 1000 should be a safe default for most high speed WANs (100Mbit/s). For very high speed WANs (e.g. 1Gbit/s+), consider increasing further to 3000-5000.The tunable is referenced on https://docs.netgate.com/pfsense/en/latest/trafficshaper/limiters.html
Tip
In cases where there are several limiters or limiters with large Queue Size values, a System Tunable may need set to increase the value of net.inet.ip.dummynet.pipe_slot_limit above the total number of configured queue lots among all pipes and queues.
-
@jackyaz Thanks for the additional information. I tried these settings on my end and didn't notice any difference, but I'm beginning to believe this is my ISP and not pfSense. I've got a consumer router to try swapping out later today; I just have to wait a bit so as not to knock out other people in the house working from home. I'll provide an update with those test results as soon as I have them.
-
@thenarc Well, I ran some tests with an old Trendnet router I had around and results were the same. It has some rudimentary bandwidth limiting too, and I'm finding that whether I use it or pfSense, I can limit my 400Mbps downstream all the way to 50Mbps or less and I still get catastrophic latency spikes (up to 1s or more) if I run a multi-stream download test (again, I'm just using the fast.com test with the parallel connections maxed out at 30). I'm not really certain how to diagnose further (could it be my cable modem, for example? It's a SB6180, which is not a Puma6 modem) or just crappy ISP configuration? In either case, I'm satisfied based on this testing that it's not pfSense, but I still have no solution :)
-
@emikaadeo Thanks for posting this. I don't know what's wrong, but I've been at this all day today and gotten nowhere.
Can someone explain how the below is even possible?
As a test I set my bandwidth to 25Mbit/s to see if the limiter was even working at all...
# cat /tmp/rules.limiter pipe 1 config bw 25Mb droptail sched 1 config pipe 1 type fq_codel target 5ms interval 100ms quantum 300 limit 10240 flows 1024 ecn queue 1 config pipe 1 droptail pipe 2 config bw 25Mb droptail sched 2 config pipe 2 type fq_codel target 5ms interval 100ms quantum 300 limit 10240 flows 1024 ecn queue 2 config pipe 2 droptail
And yet...
-
Just read through about 9 other threads reporting various breakage with ipfw limiters on 2.6 / 22.0x
Before I lose another day, @jimp or @stephenw10 is it the case that limiters are bugged on the latest builds of pfSense? Specifically for multi-wan setups with gateway groups? It would be nice to know, otherwise if the answer is "no, everything works fine" then I will keep trying or maybe even buy TAC to figure this out because it is driving me nuts.
-
@luckman212 Make sure you have IPv6 disabled on your machine, otherwise the test will not work correct.
-
@luckman212 said in QoS / Traffic Shaping / Limiters / FQ_CODEL on 22.0x:
is it the case that limiters are bugged on the latest builds of pfSense? Specifically for multi-wan setups with gateway groups?
They seem to work fine on my multi-wan setup with gateway groups.
Before I lose another day
Maybe throw in a quick downgrade to 2.4.5-p1 just to be sure?
-
@bob-dig said:
@luckman212 Make sure you have IPv6 disabled on your machine, otherwise the test will not work correct.
Yes at this time I don't have IPv6 enabled at all.
@thiasaef said:
Maybe throw in a quick downgrade to 2.4.5-p1 just to be sure?
I can try that but it's a fair bit of work since my config has changed a lot since 2.5.x/22.x was released, and the configs are not backwards-compatible. So before doing that I'd like to know if I'm barking up the wrong tree here. Since you say it works for you, would you mind sharing how you've got it configured?
-
I'm not sure how helpful this will be, but I've got two separate locations both on 1Gbit/s FiOS circuits running pfSense 22.01 with limiters + FQ-Codel configured. No issues at either site. The instructions I followed for the limiter setup are these originally posted in the large FQ-Codel thread:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/814
The main difference I suppose is that I've only got the one FiOS connection at either location (i.e. no multi-wan or gateway groups configured).
Hope this helps.
-
@luckman212 said in QoS / Traffic Shaping / Limiters / FQ_CODEL on 22.0x:
would you mind sharing how you've got it configured?
Settings:
- Firewall > Traffic Shaper > Limiters:
- WAN1Down/WAN1DownQ
- bandwidth: 265Mbps
- Queue mgmt algo: Tail Drop
- Scheduler: FQ_CODEL (5/100/1514/10240/8192)
- Queue length: empty
- ECN: not checked
- Firewall > Rules > Floating:
- Action: Match
- Quick: unchecked
- Interface: WAN1
- Direction: out
- Family: IPv4
- Protocol: any
- Source: WAN1 address
- Dest: Any
- Gateway: WAN1
- In/Out Pipe: WAN1UpQ / WAN1DownQ
but it also works when I apply your exact settings (except for the different bandwidth).
@luckman212 said in QoS / Traffic Shaping / Limiters / FQ_CODEL on 22.0x:
triggering failover to my 4G LTE backup connection which does not have any shaper applied
As a side note, I also have a shaper on my 4G LTE backup that works wonders in terms of latency under load.
- Firewall > Traffic Shaper > Limiters:
-
@luckman212 I did it exactly like what you already posted in your first post.
-
@thiasaef What version of pfSense are you running there? Do you use gateway groups? What's your System > Routing > default gw IPv4 set to?
-
What version of pfSense are you running there?
2.6.0-RELEASE (amd64)
Do you use gateway groups?
What's your System > Routing > default gw IPv4 set to?
