Traceroute omits hops with limiters on 2.4.4

Softener

Fellow networkians,

I have been struggling for days with the apparent bugs of limiters in pfSense 2.4.x but have now found a workaround using CoDel and QFQ to have limiters working again at least. Phew.

However the traceroutes (mtr) looks very weird when limiters are on (using match rules in the floating rules section). This is a typical IPv4 traceroute without limiters:

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. pfsense-lan                       0.0%    12    0.4   0.3   0.1   0.4   0.1
 2. 212.123.255.193                   0.0%    12    0.4   0.5   0.4   0.6   0.1
 3. 62.96.34.45                       0.0%    12    0.9   0.8   0.7   1.0   0.1
 4. 212.74.68.189                     0.0%    12   12.0  11.7  10.1  14.1   1.1
 5. 212.74.68.189                     0.0%    12   10.5  10.8   9.8  12.2   0.8
 6. ???
 7. 108.170.241.129                   0.0%    12    8.7   8.7   8.6   8.8   0.1
 8. 216.239.41.225                    0.0%    12    8.7   8.7   8.7   8.9   0.1
 9. 8.8.8.8                           0.0%    12    8.9   8.7   8.5   8.9   0.1

Once I turn limiters on it looks like this:

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. pfsense-lan                       0.0%   116    0.2   0.2   0.1   0.6   0.1
 2. 8.8.8.8                          56.9%   116    0.5   0.7   0.3   4.8   0.9

First of all I get high packet losses no matter which host I route to. When the bandwidth usage is low pings seem to work okay. But when the limiter kicks in I have around 80-90% ping loss. I hadn't expected that. TCP/UDP traffic seems to flow well and users are not complaining.

However when I do a traceroute to an IPv6 address in an external data center the hops are shown correctly and the loss is not so bad:

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. pfsense-lan                       0.0%    29    0.5   0.3   0.2   0.8   0.1
 2. fd00:212:123:255:193::cafe        0.0%    28    0.8   0.9   0.4   2.9   0.7
 3. 2001:920:0:2::3a0                 0.0%    28    1.3   4.0   0.7  25.4   6.5
 4. 2001:920:c000:0:212:74:91:110     3.6%    28   10.5  11.5  10.1  16.8   1.6
 5. gw6-decix.ffm.netcup.net          0.0%    28   13.7  16.4  13.6  38.9   5.0
 6. jen.workaround.org                3.6%    28   14.5  14.5  13.6  16.5   0.9

Does anyone have an explanation for that? I'd appreciate any hints. Thanks.

…Christoph

kirillkh

Looks like I ran into the same issue. https://forum.netgate.com/topic/139384/weird-issues-with-limiters

uptownVagrant

@softener Take a look at the following guide as it should explain the issue you are witnessing and show how to workaround it - hint floating rule #1.

https://forum.netgate.com/post/807490

uptownVagrant

@softener Also, CoDel is not working on your queues where QFQ is the scheduler. See this: https://forum.netgate.com/topic/137963/codel-does-not-work-on-limiter-queues-in-2-4-4