Traceroute not working from LAN to any Internet destination



  • Hi,
    I have noticed a strange issue while tracing to Internet destinations from LAN side.
    The output of entire traceroute gets NAT to destination IP . i.e I see only destination IP as all the Intermediate HOPS.

    From WAN interface traceroute works perfectly.

    To verify issue I did packet capture on WAN and LAN interface. Packet Capture clearly shows that WAN is receiving TTL expired from Intermediate HOPS correctly however when it reaches LAN, all the Intermediate HOPs are displayed as just destination IP. This clearly indicates Pfsense is behaving strange. I did multiple tests with ICMP and UDP traceroutes with same results.

    Has anyone faced the issue before? Please guide.

    LAN trace:

    1 * * *
    2 172.217.4.46 8.648 ms 8.025 ms 7.959 ms
    3 172.217.4.46 15.970 ms 14.904 ms 15.991 ms
    4 172.217.4.46 16.988 ms 16.015 ms 17.667 ms
    5 172.217.4.46 16.461 ms 17.790 ms 15.640 ms
    6 172.217.4.46 16.335 ms 18.012 ms 25.032 ms
    7 172.217.4.46 15.972 ms 17.992 ms 22.967 ms
    8 172.217.4.46 42.378 ms 16.748 ms 28.899 ms
    9 172.217.4.46 15.952 ms 17.988 ms 25.003 ms
    10 172.217.4.46 34.956 ms 43.148 ms 71.013 ms
    11 172.217.4.46 40.850 ms 42.575 ms 43.427 ms
    12 172.217.4.46 28.976 ms 27.943 ms 26.994 ms

    WAN trace:

    1 * * *
    2 69.63.242.41 12.863 ms 12.001 ms 12.658 ms
    3 209.148.231.93 16.187 ms 16.024 ms 16.083 ms
    4 209.148.229.229 14.858 ms 16.601 ms 12.398 ms
    5 209.148.235.22 16.547 ms 17.388 ms
    209.148.230.10 14.972 ms
    6 72.14.222.87 14.132 ms 16.429 ms
    72.14.216.54 16.421 ms
    7 108.170.250.231 16.549 ms
    108.170.250.242 15.450 ms
    108.170.250.227 16.954 ms
    8 216.239.42.158 17.973 ms
    216.239.49.82 39.996 ms 47.720 ms
    9 108.170.250.231 15.238 ms
    74.125.244.147 18.021 ms
    108.170.250.227 16.969 ms
    10 108.170.243.174 28.023 ms
    172.253.64.254 26.948 ms 28.039 ms
    11 72.14.239.115 40.012 ms
    72.14.239.123 40.990 ms 40.178 ms
    12 172.217.9.78 27.896 ms 26.768 ms 29.040 ms

    Both above traceroutes are taken from Pfsense



  • I am also having this problem, did you figure it out?



  • @ccb056 Yes; seems like a bug when CODEL is enabled,
    you need a Floating Rule as below:

    Actions: Pass
    Interface: WAN
    Direction: any
    Protocol: ICMP
    ICMP Subtypes: Echo Request and Echo Reply



  • 001.PNG

    I am still getting the traceroute error



  • You need to bring the rule above your Codel Limiter



  • ... and check "Quick" ( Apply the action immediately on match. ) for this rule.

    db535afc-735d-4caa-b15b-88ee00847c70-image.png



  • That worked, thank you!



  • Does creation of that rule not introduce a new security risk by allowing any ICMP traffic inbound from the internet? That does not seem like an acceptable workaround to me.


  • Netgate Administrator

    Well depends how you gauge the risk, responding to pings is not high risk IMO.

    But, yeah, I would not expect to have to allow that. Setting that to OUT only will likely also workaround the problem.

    It's also not set as 'quick' which I would have expected....

    Steve


  • LAYER 8 Moderator

    There are 3-4 ICMP subtypes that should be open per various recommendations and security implications. The last time I checked it was "if one of those 4 get's an attacker inside - you have other problems then ICMP being used."

    A good quick lookup is http://shouldiblockicmp.com/

    For IPv4(legacy) there should be: Type 8/C0 (request), Type 3/C4 (fragreq), Type 11/C0 (timex). This can be reached with ICMP subtype selections: "Echo request", "Destination unreachable" and "Time exceeded" in a IPv4 ICMP rule. We're running that for years in a big datacenter environment - never had problems with it but much more debug capabilities and less problems with other networks upstream.



  • @stephenw10 said in Traceroute not working from LAN to any Internet destination:

    Well depends how you gauge the risk, responding to pings is not high risk IMO.

    But, yeah, I would not expect to have to allow that. Setting that to OUT only will likely also workaround the problem.

    It's also not set as 'quick' which I would have expected....

    Steve

    I just run to the same issue running codel on 2.4.5
    And the fix does the job, using direction out and quick

    What do you mean by >It's also not set as 'quick' which I would have expected.... ?


  • Netgate Administrator

    If you don't set the rule as 'quick' traffic matched by it will go on to be parsed by any other rules. Since it has to respond without matching the CoDel Limiter I expect it to require 'quick' there to prevent it being parsed the Limiter rule below it.

    https://docs.netgate.com/pfsense/en/latest/book/firewall/floating-rules.html#quick

    Steve



  • Yes, of course :)
    It doesn't work without quick. Confirmed



  • Hi, I have the floating rule and quick but Im getting the same problems. Am I missing something in my rules?!

    5f123352-ab82-41cc-89ba-561290f487e1-Screenshot 2020-06-13 09.27.31.png

    traceroute www.ntua.gr
    traceroute to www.ntua.gr (147.102.224.101), 64 hops max, 52 byte packets
     1  XX (X)  4.054 ms  1.243 ms  1.275 ms
     2  www.ntua.gr (147.102.224.101)  8.807 ms  9.811 ms  10.076 ms
     3  www.ntua.gr (147.102.224.101)  15.286 ms  9.946 ms  9.692 ms
     4  * * *
     5  www.ntua.gr (147.102.224.101)  14.548 ms  13.996 ms  13.006 ms
     6  www.ntua.gr (147.102.224.101)  13.220 ms  15.703 ms  13.044 ms
     7  * www.ntua.gr (147.102.224.101)  45.182 ms  43.441 ms
     8  www.ntua.gr (147.102.224.101)  44.675 ms  44.498 ms  47.619 ms
    


  • ca8e612b-ee69-4266-bae3-f8ace3cbcf6a-image.png
    Don't specify an interface
    and try moving it up a bit. The vpn egress seems to match traffic.



  • @netblues the egress is there to stop things going via WAN when the VPN client is down. Shouldnt be first?

    Updated the rules like this. still the same.

    Screenshot 2020-06-13 12.20.53.png

    The other weird thing is that the two codel rules are matching very little compared to what the general "LAN" rule matches on the other tab.. hmmm



  • @daemonix Well, temporarily disable it and see if it matters. Floating rules are powerful but do have side effects.



  • @netblues no fun.. even without it traceroute isnt working.

    What else it might be?



  • It just hit me...
    traceroute on recent linux uses tcp...
    try: traceroute -I ntua.gr


  • Netgate Administrator

    Edited to remove the auto-link.

    Linux uses UDP by default, yeah.



  • @stephenw10 -M method, --module=name
    Use specified method for traceroute operations. Default traditional udp method has name default, icmp (-I) and tcp (-T) have names icmp and tcp respectively.
    So its udp
    at the same time. mtr is using icmp. I don't thing there is any workaround for udp traceroute and fq-codel


Log in to reply