FQ_Pie no internet
-
The size of traffic going through pie is substantially smaller. Pie is 150bytes and codel is well over 1000bytes.
Before posting for help, I did some searching on the web. I couldn't find anybody that had gotten Pie working. Thanks for the help maybe someone else knows what could be wrong with pie. I will look into it further when version 23.01 is released out of RC.
Network setup:
1 netgate 6100 firewall
4 cisco small business switches
5 ubiquiti uap 6 lite
1 ubiquiti uap 6 lr -
Same here. I waited for 23.01 Stable and still when I send traffic to FQ_Pie limiters using floating rules, it breaks internet.
-
I noticed the same thing.
Fresh install as I wanted to test out FQ_PIE and once I setup this, it stopped passing any traffic to the internet.
I tried numerous things as rebooting / clearing all the rules and nothing worked as I thought I was doing something wrong.
I tested out all the same but changing FQ_PIE to FQ_CODEL for the scheduler and everything worked. I eventually gave up but definitely seems to be broken in 23.01 at least as I ran that with all the system recommended system patches.
-
https://redmine.pfsense.org/issues/13996
If your steps to reproduce differ from those shown you might want to comment there.
Steve
-
Is there any development on this? Issue still seen on 2.7.2
-
@milindhvijay
Still an issue on 23.09.1. -
Mit 24.03-RELEASE funktioniert der FQ-PIE Limiter auch nicht im Gegensatz zu FQ-Codel.
-
Having the same issue on a Chelsio T5 system on 2.7.2.
-
My Prod@Home box is on 24.08-DEVELOPMENT and I got the same issue. But since I got my upgrade from 400/90 Mbit VDSL to 10Gbit Up/Down Fiber a few weeks ago I don't use the limiters.
Having said that I set up an pfSense CE 2.7.2 VM on a Proxmox server and can reproduce it.
Default installation from 2.7.2-CE ISO with limiter created for up- and download. Created with standard values, first the default "Worst-case..." scheduler. Changed to "FQ_CODEL", tested and worked. Then changed doe "FQ_PIE", tested and no internet access.
But in my case it's "only" DNS that doesn't work, I can ping the world just fine.
Is it the same for others, @Martin6734 @christos3105 @Animosity022 @kaj ?
Correction: after disabling and enabling "Enable DNSSEC Support" DNS now works with FQ_PIE but no internet.
Before that I stopped and started Unbound and it didn't work.
Addition: The limiter are on the WAN interface, just to clarify
Addition 2: UDP services do work for me, e.g. NTP and DNS.
As soon as I enable "DNS Query Forwarding" and "Use SSL/TLS for outgoing DNS queries to Forwarding Servers" (using quad9) DNS query stop working. -
@patient0 if you read the "playing with fq codel" forums, around post 720 rasool, the poster, recommends using these combinations of dummynet AQMs and schedulers:
taildrop+fqcodel
codel+wfq
codel+qfq
pie+qfqHe also mentions their code is unfinished. And that codel+fq_codel makes no sense (even though it does work)
No idea how they've changed since updates. I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.
DNS may have to be from ISP or reset repeatedly sometimes. Like, all files. And domains manually set. Same with broadcast domains.
Also, careful with weird vlan switches. I have an "unmanaged pro" switch by tplink which can work with those traffic shapers but sometimes seems a little off
(also my whitebox isn't official). How can an unmanaged switch do vlans 🥸 The netgate book says they can'tThe switch was a gift from an advertisement agency and it is kind of trippy.
-
@HLPPC said in FQ_Pie no internet:
I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.
You tested all the onces you listed or all which are available in pfSense?
But having said that: That of course doesn't invalidate that some user have an issue with FQ_PIE and if you have an idea why only TCP is affected and not UDP (at least in my case) I'm happy to hear it.
-
https://www.theverge.com/23655762/l4s-internet-apple-comcast-latency-speed-bandwidth
The ISP is working on getting these to work with TCP congestion control algorithms and L4S.
I got some beat results setting Windows congestion control to BBR, probably easier to change than rebuilding the pfSense kernel to BBR instead of cubic. Some say BBR can swamp cubic. Others say the router's algorithm doesn't matter unless you are using a proxy. There are a bajillion TCP buffer and tcp timer settings in pfSense to play with that affect the schedulers and shapers.
And the shapers don't let you change the "burst" option, (which you can in Linux fq_codel) because it is supposed to be controlled by the application and how it wants to process UDP and PDU segments. Some are supposed to set their burst rate with these shapers. DNSSEC is tossing a wrench into the fire with its encryption curves and dummynet changing the direction of traffic in CPUs. I found it better to use port 853 and drop all port 53 in and out WAN to keep dns sane while using shapers. 9.9.9.9 who does dnssec for you.
I tested the ones the guy in the forum mentioned who made them. And they all require you to skip firewall rules to get them going, and reboots. The best is by far taildrop+fq_codel. But the dummynet timer eventually starts falling behind without setting my specific kernel's speed to 1100hz. Or giving 1ms-10ms to apps, To make them skip fast-io and go into dummynet. And on top of all that you may need to piggback acknowledgements.
The UDP depends on the TYPE of UDP protocol, there are a lot. And if you feel like mangling them glhf. And UDP can be shoved inside TLS itself. Independent of the DTLS UDP protocol. Sometimes forcing pfSense UDP datagram sizes to 1500 only and disabling fragmentation has helped, and also mangled UDP with fragments anyways. Localhost's mtu being forced to 1500 has also produced results but for only one client at a time for me. I read both suggestions from some college papers on dummynet. Disabling all checksum offloading is necessary too. ifconfig igb -rxcsum -txcsum and managing it for every single interface.
-
I did a further testing (IPv4 only):
And I did it wrong, in the floating rule I set protocol to TCP :/ ... setting it to Any "solves" that, UDP and ICMP are no longer working.
Please ignore the rest of this post.
what protocol seem to work:- UDP
- ICMP
- ARP
what doesn't work:
- TCP
Testing it I run a packet capture on the pfSense on all TCP traffic on the WAN interface. And set the floating rule to enable logging.
The
wgetto a website on port 80 is run from a host behind pfSense.I can see the firewall entry but only a TCP:S nothing more. And the packet captures shows no entry for that host.
-
@patient0 did you try fq-pie+qfq, skipping firewall rules, also you may need to manually look at the configuration file for dummynet. It is probably corrupted.
Backup and restore>download traffic shaper file
And because of that I just manually wipe all firmware occasionally. There are new options evolving letting you set your pipe numbers and associations. Not there in 2.5. Thanks team
-
@HLPPC said in FQ_Pie no internet:
did you try fq-pie+qfq
I assume you're referring to the combination PIE as "Queue Management Algo..." and "Quick Fair Queueing" scheduler? That does indeed work as do a lot of other combinations :).
Naively I assume a scheduler which is a selectable option to work and I'm (and I assume others) just trying to debug the issue.
As mentioned a bit earlier I don't use limiters right now on my prod system since 10Gbit up/down is quite ok without them.
-
@patient0 Yes. I once tried all TCP codel+qfq with source/destination masks and all UDP taildrop+fqcodel. Stuff broke. I guess be careful or something 🫣 child fqcodel queues assigned to hfsc with altq shapers seemed to work alright. No idea how to manage token bucket regulators between the two though.
DSCP definitely makes it more complicated. So does ECN tunables, 1,2,3,4 in/out, in, in/out with interface data, and only in with interface data. And ECN retries. And tcp fast open or syncookies and syncaches, hostcaches.
The best I have ever done is 0 bufferbloat running speed tests with multiple devices using simple tail-drop+fqcodel. And Codel+qfq can kind of do the same. Pie though? Very difficult.
-
@patient0 and all of the traffic shaping works but getting endpoints to say they are ECN capable is kind of difficult too. Should the shapers be transparent? Should devices care about ecn? There are just as many windows settings and linux settings as pfsense settings. Plus windows differentiates between netsh and powershell settings. You may need ECN compatible switches, or some auto edge interface setting if the pfsense is connected to endpoints. Which in my opinion, the auto edge interface setting is dangerous over bridges with an ISP router/switch/all-in-one-wifi, non subnettable thing. AND it may need a crossover cable. Which is just wild. At some point you need more than one firewall. And don't forget to use a vpn that hides your identity. But don't traffic shape the tunnel. Maybe passtos.
-
The last entry for today on that topic, I was confusing enough for one day:
Testing it on FreeBSD 14 using the rules found on the pfSense box, that is /tmp/rules.limiter and /tmp/rules.debug (the line with
dnqueuein it) was not successful either.Rules used to test:
dnctl rules used :
dnctl pipe 1 config bw 88Mb droptail dnctl sched 1 config pipe 1 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand dnctl queue 1 config pipe 1 droptail dnctl pipe 2 config bw 44Mb droptail dnctl sched 2 config pipe 2 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand dnctl queue 2 config pipe 2 droptailNot using any of the parameters after
type fq_piedoesn't change the outcome.pf.conf rule:
pass out log quick on { vtnet0 } route-to ( vtnet0 192.168.1.1 ) inet from any to any ridentifier 1721286853 keep state dnqueue( 2,1 ) label "USER_RULE: Traffic Limiter (out)" label "id:1721286853" label "gw:WANGW"Changing the scheduler type made it work, as on pfSense.
And on pfSense, using the pipes instead of the queues in the floating rule does make it work too.
-
I have updated the redmine issue 13996 on how it is reproducable (for me), including in FreeBSD 14 and later (haven't tried it with earlier version).
-
The developers of opnsense have posted a patch on freebsd regarding the FQ_PIE bug - see: https://reviews.freebsd.org/D46182
Here is the corresponding forum post at opnsense: https://forum.opnsense.org/index.php?topic=41827.0
It looks like opnsense has solved the problem with the fq_pie scheduler.
