FQ_Pie no internet
-
Limiters:
00001: 900.000 Mbit/s 0 ms burst 0
q131073 3000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002: 38.000 Mbit/s 0 ms burst 0
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
sched 65538 type FIFO flags 0x0 0 buckets 0 activeSchedulers:
00001: 900.000 Mbit/s 0 ms burst 0
q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM type PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 NoECN NoCapDrop DRE NoDerand
sched 1 type FQ_PIE flags 0x0 0 buckets 0 active
FQ_PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 ECN NoCapDrop DRE NoDerand
Children flowsets: 1
00002: 38.000 Mbit/s 0 ms burst 0
q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM type PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 NoECN NoCapDrop DRE NoDerand
sched 2 type FQ_PIE flags 0x0 0 buckets 1 active
FQ_PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 ECN NoCapDrop DRE NoDerand
Children flowsets: 2
BKT Prot Source IP/port_ Dest. IP/port Tot_pkt/bytes Pkt/Byte Drp
0 ip 0.0.0.0/0 0.0.0.0/0 1 29 0 0 0Queues:
q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM type PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 NoECN NoCapDrop DRE NoDerand
q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM type PIE target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150ms max_ecnth 0.099 NoECN NoCapDrop DRE NoDerand -
@martin6734
As fare as i can tell i don't see any errors. -
Do you see the traffic in those pipe vary when you are trying to connect through those?
-
@martin6734
What am I checking for? Is there a particular place I should check on the firewall? Do you need to see my firewall rules? Thanks for the help.I’m running version 23.01 on a Netgate 6100.
I followed the instruction listed to setup fq_codel on the Netgate website. Everything is setup exactly as what was instructed.
-
Look at the Diag > Limiters screen when it's configured with CoDel and passing traffic as expected. You'll see traffic going into the pipes as the screens updates.
Now compare that with the Limiters configured using Pie. Do you still see traffic entering the pipes, any variation? -
The size of traffic going through pie is substantially smaller. Pie is 150bytes and codel is well over 1000bytes.
Before posting for help, I did some searching on the web. I couldn't find anybody that had gotten Pie working. Thanks for the help maybe someone else knows what could be wrong with pie. I will look into it further when version 23.01 is released out of RC.
Network setup:
1 netgate 6100 firewall
4 cisco small business switches
5 ubiquiti uap 6 lite
1 ubiquiti uap 6 lr -
Same here. I waited for 23.01 Stable and still when I send traffic to FQ_Pie limiters using floating rules, it breaks internet.
-
I noticed the same thing.
Fresh install as I wanted to test out FQ_PIE and once I setup this, it stopped passing any traffic to the internet.
I tried numerous things as rebooting / clearing all the rules and nothing worked as I thought I was doing something wrong.
I tested out all the same but changing FQ_PIE to FQ_CODEL for the scheduler and everything worked. I eventually gave up but definitely seems to be broken in 23.01 at least as I ran that with all the system recommended system patches.
-
https://redmine.pfsense.org/issues/13996
If your steps to reproduce differ from those shown you might want to comment there.
Steve
-
Is there any development on this? Issue still seen on 2.7.2
-
@milindhvijay
Still an issue on 23.09.1. -
Mit 24.03-RELEASE funktioniert der FQ-PIE Limiter auch nicht im Gegensatz zu FQ-Codel.
-
Having the same issue on a Chelsio T5 system on 2.7.2.
-
My Prod@Home box is on 24.08-DEVELOPMENT and I got the same issue. But since I got my upgrade from 400/90 Mbit VDSL to 10Gbit Up/Down Fiber a few weeks ago I don't use the limiters.
Having said that I set up an pfSense CE 2.7.2 VM on a Proxmox server and can reproduce it.
Default installation from 2.7.2-CE ISO with limiter created for up- and download. Created with standard values, first the default "Worst-case..." scheduler. Changed to "FQ_CODEL", tested and worked. Then changed doe "FQ_PIE", tested and no internet access.
But in my case it's "only" DNS that doesn't work, I can ping the world just fine.
Is it the same for others, @Martin6734 @christos3105 @Animosity022 @kaj ?
Correction: after disabling and enabling "Enable DNSSEC Support" DNS now works with FQ_PIE but no internet.
Before that I stopped and started Unbound and it didn't work.
Addition: The limiter are on the WAN interface, just to clarify
Addition 2: UDP services do work for me, e.g. NTP and DNS.
As soon as I enable "DNS Query Forwarding" and "Use SSL/TLS for outgoing DNS queries to Forwarding Servers" (using quad9) DNS query stop working. -
@patient0 if you read the "playing with fq codel" forums, around post 720 rasool, the poster, recommends using these combinations of dummynet AQMs and schedulers:
taildrop+fqcodel
codel+wfq
codel+qfq
pie+qfqHe also mentions their code is unfinished. And that codel+fq_codel makes no sense (even though it does work)
No idea how they've changed since updates. I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.
DNS may have to be from ISP or reset repeatedly sometimes. Like, all files. And domains manually set. Same with broadcast domains.
Also, careful with weird vlan switches. I have an "unmanaged pro" switch by tplink which can work with those traffic shapers but sometimes seems a little off
(also my whitebox isn't official). How can an unmanaged switch do vlans 🥸 The netgate book says they can'tThe switch was a gift from an advertisement agency and it is kind of trippy.
-
@HLPPC said in FQ_Pie no internet:
I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.
You tested all the onces you listed or all which are available in pfSense?
But having said that: That of course doesn't invalidate that some user have an issue with FQ_PIE and if you have an idea why only TCP is affected and not UDP (at least in my case) I'm happy to hear it.
-
https://www.theverge.com/23655762/l4s-internet-apple-comcast-latency-speed-bandwidth
The ISP is working on getting these to work with TCP congestion control algorithms and L4S.
I got some beat results setting Windows congestion control to BBR, probably easier to change than rebuilding the pfSense kernel to BBR instead of cubic. Some say BBR can swamp cubic. Others say the router's algorithm doesn't matter unless you are using a proxy. There are a bajillion TCP buffer and tcp timer settings in pfSense to play with that affect the schedulers and shapers.
And the shapers don't let you change the "burst" option, (which you can in Linux fq_codel) because it is supposed to be controlled by the application and how it wants to process UDP and PDU segments. Some are supposed to set their burst rate with these shapers. DNSSEC is tossing a wrench into the fire with its encryption curves and dummynet changing the direction of traffic in CPUs. I found it better to use port 853 and drop all port 53 in and out WAN to keep dns sane while using shapers. 9.9.9.9 who does dnssec for you.
I tested the ones the guy in the forum mentioned who made them. And they all require you to skip firewall rules to get them going, and reboots. The best is by far taildrop+fq_codel. But the dummynet timer eventually starts falling behind without setting my specific kernel's speed to 1100hz. Or giving 1ms-10ms to apps, To make them skip fast-io and go into dummynet. And on top of all that you may need to piggback acknowledgements.
The UDP depends on the TYPE of UDP protocol, there are a lot. And if you feel like mangling them glhf. And UDP can be shoved inside TLS itself. Independent of the DTLS UDP protocol. Sometimes forcing pfSense UDP datagram sizes to 1500 only and disabling fragmentation has helped, and also mangled UDP with fragments anyways. Localhost's mtu being forced to 1500 has also produced results but for only one client at a time for me. I read both suggestions from some college papers on dummynet. Disabling all checksum offloading is necessary too. ifconfig igb -rxcsum -txcsum and managing it for every single interface.
-
I did a further testing (IPv4 only):
And I did it wrong, in the floating rule I set protocol to TCP :/ ... setting it to Any "solves" that, UDP and ICMP are no longer working.
Please ignore the rest of this post.
what protocol seem to work:- UDP
- ICMP
- ARP
what doesn't work:
- TCP
Testing it I run a packet capture on the pfSense on all TCP traffic on the WAN interface. And set the floating rule to enable logging.
The
wgetto a website on port 80 is run from a host behind pfSense.I can see the firewall entry but only a TCP:S nothing more. And the packet captures shows no entry for that host.
-
@patient0 did you try fq-pie+qfq, skipping firewall rules, also you may need to manually look at the configuration file for dummynet. It is probably corrupted.
Backup and restore>download traffic shaper file
And because of that I just manually wipe all firmware occasionally. There are new options evolving letting you set your pipe numbers and associations. Not there in 2.5. Thanks team
-
@HLPPC said in FQ_Pie no internet:
did you try fq-pie+qfq
I assume you're referring to the combination PIE as "Queue Management Algo..." and "Quick Fair Queueing" scheduler? That does indeed work as do a lot of other combinations :).
Naively I assume a scheduler which is a selectable option to work and I'm (and I assume others) just trying to debug the issue.
As mentioned a bit earlier I don't use limiters right now on my prod system since 10Gbit up/down is quite ok without them.
