FQ_Pie no internet

patient0

I did a further testing (IPv4 only):

And I did it wrong, in the floating rule I set protocol to TCP :/ ... setting it to Any "solves" that, UDP and ICMP are no longer working.

Please ignore the rest of this post.

~~what protocol seem to work:~~

UDP
ICMP
ARP

what doesn't work:

TCP

Testing it I run a packet capture on the pfSense on all TCP traffic on the WAN interface. And set the floating rule to enable logging.

The wget to a website on port 80 is run from a host behind pfSense.

I can see the firewall entry but only a TCP:S nothing more. And the packet captures shows no entry for that host.

Screenshot 2024-07-18 at 11.02.29.png

HLPPC

@patient0 did you try fq-pie+qfq, skipping firewall rules, also you may need to manually look at the configuration file for dummynet. It is probably corrupted.

Backup and restore>download traffic shaper file

And because of that I just manually wipe all firmware occasionally. There are new options evolving letting you set your pipe numbers and associations. Not there in 2.5. Thanks team

patient0

@HLPPC said in FQ_Pie no internet:

did you try fq-pie+qfq

I assume you're referring to the combination PIE as "Queue Management Algo..." and "Quick Fair Queueing" scheduler? That does indeed work as do a lot of other combinations :).

Naively I assume a scheduler which is a selectable option to work and I'm (and I assume others) just trying to debug the issue.

As mentioned a bit earlier I don't use limiters right now on my prod system since 10Gbit up/down is quite ok without them.

HLPPC

@patient0 Yes. I once tried all TCP codel+qfq with source/destination masks and all UDP taildrop+fqcodel. Stuff broke. I guess be careful or something 🫣 child fqcodel queues assigned to hfsc with altq shapers seemed to work alright. No idea how to manage token bucket regulators between the two though.

DSCP definitely makes it more complicated. So does ECN tunables, 1,2,3,4 in/out, in, in/out with interface data, and only in with interface data. And ECN retries. And tcp fast open or syncookies and syncaches, hostcaches.

The best I have ever done is 0 bufferbloat running speed tests with multiple devices using simple tail-drop+fqcodel. And Codel+qfq can kind of do the same. Pie though? Very difficult.

HLPPC

@patient0 and all of the traffic shaping works but getting endpoints to say they are ECN capable is kind of difficult too. Should the shapers be transparent? Should devices care about ecn? There are just as many windows settings and linux settings as pfsense settings. Plus windows differentiates between netsh and powershell settings. You may need ECN compatible switches, or some auto edge interface setting if the pfsense is connected to endpoints. Which in my opinion, the auto edge interface setting is dangerous over bridges with an ISP router/switch/all-in-one-wifi, non subnettable thing. AND it may need a crossover cable. Which is just wild. At some point you need more than one firewall. And don't forget to use a vpn that hides your identity. But don't traffic shape the tunnel. Maybe passtos.

patient0

The last entry for today on that topic, I was confusing enough for one day:

Testing it on FreeBSD 14 using the rules found on the pfSense box, that is /tmp/rules.limiter and /tmp/rules.debug (the line with dnqueue in it) was not successful either.

Rules used to test:

dnctl rules used :

dnctl pipe 1 config  bw 88Mb droptail
dnctl sched 1 config pipe 1 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand
dnctl queue 1 config pipe 1 droptail

dnctl pipe 2 config  bw 44Mb droptail
dnctl sched 2 config pipe 2 type fq_pie target 15ms tupdate 15ms alpha 0.125 beta 1.25 max_burst 150 max_ecnth 0.099 quantum 1514 limit 10240 flows 1024 noecn nocapdrop dre noderand
dnctl queue 2 config pipe 2 droptail

Not using any of the parameters after type fq_pie doesn't change the outcome.

pf.conf rule:

pass  out log  quick  on {  vtnet0  }   route-to ( vtnet0 192.168.1.1 ) inet from any to any ridentifier 1721286853 keep state  dnqueue( 2,1 ) label "USER_RULE: Traffic Limiter (out)" label "id:1721286853" label "gw:WANGW"

Changing the scheduler type made it work, as on pfSense.

And on pfSense, using the pipes instead of the queues in the floating rule does make it work too.

patient0

I have updated the redmine issue 13996 on how it is reproducable (for me), including in FreeBSD 14 and later (haven't tried it with earlier version).

kaj

@patient0

The developers of opnsense have posted a patch on freebsd regarding the FQ_PIE bug - see: https://reviews.freebsd.org/D46182

Here is the corresponding forum post at opnsense: https://forum.opnsense.org/index.php?topic=41827.0

It looks like opnsense has solved the problem with the fq_pie scheduler.

stephenw10

@kaj said in FQ_Pie no internet:

https://reviews.freebsd.org/D46182

That will be in 25.03 and 2.8. It's in the 25.03-beta now. Looks like it just missed 24.11.

https://github.com/pfsense/FreeBSD-src/commit/2b7f2890a812eb09dccaa8069483566a3b292338

kaj

@stephenw10 I'm looking forward to the 25.03 version and will test it right away. Thank you for the information