Traceroute omits hops with limiters on 2.4.4



  • Fellow networkians,

    I have been struggling for days with the apparent bugs of limiters in pfSense 2.4.x but have now found a workaround using CoDel and QFQ to have limiters working again at least. Phew.

    However the traceroutes (mtr) looks very weird when limiters are on (using match rules in the floating rules section). This is a typical IPv4 traceroute without limiters:

                                           Packets               Pings
     Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. pfsense-lan                       0.0%    12    0.4   0.3   0.1   0.4   0.1
     2. 212.123.255.193                   0.0%    12    0.4   0.5   0.4   0.6   0.1
     3. 62.96.34.45                       0.0%    12    0.9   0.8   0.7   1.0   0.1
     4. 212.74.68.189                     0.0%    12   12.0  11.7  10.1  14.1   1.1
     5. 212.74.68.189                     0.0%    12   10.5  10.8   9.8  12.2   0.8
     6. ???
     7. 108.170.241.129                   0.0%    12    8.7   8.7   8.6   8.8   0.1
     8. 216.239.41.225                    0.0%    12    8.7   8.7   8.7   8.9   0.1
     9. 8.8.8.8                           0.0%    12    8.9   8.7   8.5   8.9   0.1
    

    Once I turn limiters on it looks like this:

                                           Packets               Pings
     Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. pfsense-lan                       0.0%   116    0.2   0.2   0.1   0.6   0.1
     2. 8.8.8.8                          56.9%   116    0.5   0.7   0.3   4.8   0.9
    

    First of all I get high packet losses no matter which host I route to. When the bandwidth usage is low pings seem to work okay. But when the limiter kicks in I have around 80-90% ping loss. I hadn't expected that. TCP/UDP traffic seems to flow well and users are not complaining.

    However when I do a traceroute to an IPv6 address in an external data center the hops are shown correctly and the loss is not so bad:

                                           Packets               Pings
     Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. pfsense-lan                       0.0%    29    0.5   0.3   0.2   0.8   0.1
     2. fd00:212:123:255:193::cafe        0.0%    28    0.8   0.9   0.4   2.9   0.7
     3. 2001:920:0:2::3a0                 0.0%    28    1.3   4.0   0.7  25.4   6.5
     4. 2001:920:c000:0:212:74:91:110     3.6%    28   10.5  11.5  10.1  16.8   1.6
     5. gw6-decix.ffm.netcup.net          0.0%    28   13.7  16.4  13.6  38.9   5.0
     6. jen.workaround.org                3.6%    28   14.5  14.5  13.6  16.5   0.9
    

    Does anyone have an explanation for that? I'd appreciate any hints. Thanks.

    …Christoph