FQ_Pie no internet

Animosity022

I noticed the same thing.

Fresh install as I wanted to test out FQ_PIE and once I setup this, it stopped passing any traffic to the internet.

I tried numerous things as rebooting / clearing all the rules and nothing worked as I thought I was doing something wrong.

I tested out all the same but changing FQ_PIE to FQ_CODEL for the scheduler and everything worked. I eventually gave up but definitely seems to be broken in 23.01 at least as I ran that with all the system recommended system patches.

stephenw10

https://redmine.pfsense.org/issues/13996

If your steps to reproduce differ from those shown you might want to comment there.

Steve

milindhvijay

Is there any development on this? Issue still seen on 2.7.2

Martin6734

@milindhvijay
Still an issue on 23.09.1.

kaj

Mit 24.03-RELEASE funktioniert der FQ-PIE Limiter auch nicht im Gegensatz zu FQ-Codel.

User1337

Having the same issue on a Chelsio T5 system on 2.7.2.

patient0

My Prod@Home box is on 24.08-DEVELOPMENT and I got the same issue. But since I got my upgrade from 400/90 Mbit VDSL to 10Gbit Up/Down Fiber a few weeks ago I don't use the limiters.

Having said that I set up an pfSense CE 2.7.2 VM on a Proxmox server and can reproduce it.

Default installation from 2.7.2-CE ISO with limiter created for up- and download. Created with standard values, first the default "Worst-case..." scheduler. Changed to "FQ_CODEL", tested and worked. Then changed doe "FQ_PIE", tested and no internet access.

But in my case it's "only" DNS that doesn't work, I can ping the world just fine.
Is it the same for others, @Martin6734 @christos3105 @Animosity022 @kaj ?
Correction: after disabling and enabling "Enable DNSSEC Support" DNS now works with FQ_PIE but no internet.
Before that I stopped and started Unbound and it didn't work.
Addition: The limiter are on the WAN interface, just to clarify
Addition 2: UDP services do work for me, e.g. NTP and DNS.
As soon as I enable "DNS Query Forwarding" and "Use SSL/TLS for outgoing DNS queries to Forwarding Servers" (using quad9) DNS query stop working.

HLPPC

@patient0 if you read the "playing with fq codel" forums, around post 720 rasool, the poster, recommends using these combinations of dummynet AQMs and schedulers:

taildrop+fqcodel
codel+wfq
codel+qfq
pie+qfq

He also mentions their code is unfinished. And that codel+fq_codel makes no sense (even though it does work)

No idea how they've changed since updates. I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.

DNS may have to be from ISP or reset repeatedly sometimes. Like, all files. And domains manually set. Same with broadcast domains.

Also, careful with weird vlan switches. I have an "unmanaged pro" switch by tplink which can work with those traffic shapers but sometimes seems a little off (also my whitebox isn't official). How can an unmanaged switch do vlans 🥸 The netgate book says they can't

The switch was a gift from an advertisement agency and it is kind of trippy.

patient0

@HLPPC said in FQ_Pie no internet:

I have tested all the combos with symmetric 1gbps up and down from isp and all of them seemed to work.

You tested all the onces you listed or all which are available in pfSense?

But having said that: That of course doesn't invalidate that some user have an issue with FQ_PIE and if you have an idea why only TCP is affected and not UDP (at least in my case) I'm happy to hear it.

HLPPC

@patient0

https://www.theverge.com/23655762/l4s-internet-apple-comcast-latency-speed-bandwidth

The ISP is working on getting these to work with TCP congestion control algorithms and L4S.

I got some beat results setting Windows congestion control to BBR, probably easier to change than rebuilding the pfSense kernel to BBR instead of cubic. Some say BBR can swamp cubic. Others say the router's algorithm doesn't matter unless you are using a proxy. There are a bajillion TCP buffer and tcp timer settings in pfSense to play with that affect the schedulers and shapers.

And the shapers don't let you change the "burst" option, (which you can in Linux fq_codel) because it is supposed to be controlled by the application and how it wants to process UDP and PDU segments. Some are supposed to set their burst rate with these shapers. DNSSEC is tossing a wrench into the fire with its encryption curves and dummynet changing the direction of traffic in CPUs. I found it better to use port 853 and drop all port 53 in and out WAN to keep dns sane while using shapers. 9.9.9.9 who does dnssec for you.

I tested the ones the guy in the forum mentioned who made them. And they all require you to skip firewall rules to get them going, and reboots. The best is by far taildrop+fq_codel. But the dummynet timer eventually starts falling behind without setting my specific kernel's speed to 1100hz. Or giving 1ms-10ms to apps, To make them skip fast-io and go into dummynet. And on top of all that you may need to piggback acknowledgements.

The UDP depends on the TYPE of UDP protocol, there are a lot. And if you feel like mangling them glhf. And UDP can be shoved inside TLS itself. Independent of the DTLS UDP protocol. Sometimes forcing pfSense UDP datagram sizes to 1500 only and disabling fragmentation has helped, and also mangled UDP with fragments anyways. Localhost's mtu being forced to 1500 has also produced results but for only one client at a time for me. I read both suggestions from some college papers on dummynet. Disabling all checksum offloading is necessary too. ifconfig igb -rxcsum -txcsum and managing it for every single interface.

patient0

I did a further testing (IPv4 only):

And I did it wrong, in the floating rule I set protocol to TCP :/ ... setting it to Any "solves" that, UDP and ICMP are no longer working.

Please ignore the rest of this post.

~~what protocol seem to work:~~

UDP
ICMP
ARP

what doesn't work:

TCP

Testing it I run a packet capture on the pfSense on all TCP traffic on the WAN interface. And set the floating rule to enable logging.

The wget to a website on port 80 is run from a host behind pfSense.

I can see the firewall entry but only a TCP:S nothing more. And the packet captures shows no entry for that host.

Screenshot 2024-07-18 at 11.02.29.png

HLPPC

@patient0 did you try fq-pie+qfq, skipping firewall rules, also you may need to manually look at the configuration file for dummynet. It is probably corrupted.

Backup and restore>download traffic shaper file

And because of that I just manually wipe all firmware occasionally. There are new options evolving letting you set your pipe numbers and associations. Not there in 2.5. Thanks team

patient0

@HLPPC said in FQ_Pie no internet:

did you try fq-pie+qfq

I assume you're referring to the combination PIE as "Queue Management Algo..." and "Quick Fair Queueing" scheduler? That does indeed work as do a lot of other combinations :).

Naively I assume a scheduler which is a selectable option to work and I'm (and I assume others) just trying to debug the issue.

As mentioned a bit earlier I don't use limiters right now on my prod system since 10Gbit up/down is quite ok without them.

HLPPC

@patient0 Yes. I once tried all TCP codel+qfq with source/destination masks and all UDP taildrop+fqcodel. Stuff broke. I guess be careful or something 🫣 child fqcodel queues assigned to hfsc with altq shapers seemed to work alright. No idea how to manage token bucket regulators between the two though.

DSCP definitely makes it more complicated. So does ECN tunables, 1,2,3,4 in/out, in, in/out with interface data, and only in with interface data. And ECN retries. And tcp fast open or syncookies and syncaches, hostcaches.

The best I have ever done is 0 bufferbloat running speed tests with multiple devices using simple tail-drop+fqcodel. And Codel+qfq can kind of do the same. Pie though? Very difficult.

HLPPC

@patient0 and all of the traffic shaping works but getting endpoints to say they are ECN capable is kind of difficult too. Should the shapers be transparent? Should devices care about ecn? There are just as many windows settings and linux settings as pfsense settings. Plus windows differentiates between netsh and powershell settings. You may need ECN compatible switches, or some auto edge interface setting if the pfsense is connected to endpoints. Which in my opinion, the auto edge interface setting is dangerous over bridges with an ISP router/switch/all-in-one-wifi, non subnettable thing. AND it may need a crossover cable. Which is just wild. At some point you need more than one firewall. And don't forget to use a vpn that hides your identity. But don't traffic shape the tunnel. Maybe passtos.

patient0

The last entry for today on that topic, I was confusing enough for one day:

Testing it on FreeBSD 14 using the rules found on the pfSense box, that is /tmp/rules.limiter and /tmp/rules.debug (the line with dnqueue in it) was not successful either.

Rules used to test:

dnctl rules used :

dnctl pipe 1 config  bw 88Mb droptail
dnctl sched 1 config pipe 1 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand
dnctl queue 1 config pipe 1 droptail

dnctl pipe 2 config  bw 44Mb droptail
dnctl sched 2 config pipe 2 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand
dnctl queue 2 config pipe 2 droptail

Not using any of the parameters after type fq_pie doesn't change the outcome.

pf.conf rule:

pass  out log  quick  on {  vtnet0  }   route-to ( vtnet0 192.168.1.1 ) inet from any to any ridentifier 1721286853 keep state  dnqueue( 2,1 ) label "USER_RULE: Traffic Limiter (out)" label "id:1721286853" label "gw:WANGW"

Changing the scheduler type made it work, as on pfSense.

And on pfSense, using the pipes instead of the queues in the floating rule does make it work too.

patient0

I have updated the redmine issue 13996 on how it is reproducable (for me), including in FreeBSD 14 and later (haven't tried it with earlier version).