Traceroute Loop On LAN Interface But No Loop On WAN

miki725 · Mar 25, 2020, 1:28 AM

I am seeing strange loop when running traceroute from within LAN however if I run the same traceroute from WAN interface directly in pfSense, it works as expected.

From within LAN:

❯ traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  10.10.20.1  0.148 ms  0.111 ms  0.085 ms
 2  * * *
 3  8.8.8.8  4.941 ms  4.933 ms  4.911 ms
 4  * * *
 5  8.8.8.8  24.779 ms  4.518 ms  4.239 ms
 6  8.8.8.8  4.166 ms  4.599 ms  4.064 ms
 7  * * *
 8  8.8.8.8  4.033 ms  4.653 ms  4.336 ms

but if run the same from the pfsense shell it works as expected:

[2.4.4-RELEASE][admin@10.10.10.1]/root: traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  * * *
 2  100.41.218.210  8.906 ms  7.928 ms  7.935 ms
 3  * * *
 4  140.222.1.81  4.416 ms
    140.222.1.45  4.441 ms  3.439 ms
 5  72.14.208.130  4.586 ms
    72.14.214.38  3.451 ms  4.249 ms
 6  * * *
 7  8.8.8.8  4.977 ms  3.637 ms  3.862 ms

Same happens if I run traceroute from the pfsense UI but change the interface:

There is no crazy FW rules on the LAN interface:

LAN interface has a static IP range 10.10.60.1/24 with DHCP for the connected clients:

WAN is is configured with DHCP.

Outbound NAT mode is Automatic outbound NAT rule generation.

I cant figure where my configuration is incorrect which is causing this behavior. I presume its a loop somewhere. Maybe someone would be able to assist me to debug this further? Ill happily provide any more information.

Thanks in advance

Gertjan · Mar 25, 2020, 9:27 AM

@miki725 said in Traceroute Loop On LAN Interface But No Loop On WAN:

1 * * *

Ask yourself this question : that first hop, is it set up to reply to ICMP packets ?

If it is your own ISP box, then put yourself in front of a mirror, you'll be seeing the admin of that box, and ask him that question.
Better : instruct him to set up the box so it acts upon ICMP packets. ("replies to ping")
Maybe he will tell you he can't, or that that box doesn't have the capability of doing so.

Anyway, keep us informed.

Edit : all the other " * * * *" : you have to connect the admin of those routers and ask them if they can activate the ICMP reply functionality.
They will refuse of course, they have their reasons not to do so.

miki725 · Mar 25, 2020, 1:17 PM

@Gertjan said in Traceroute Loop On LAN Interface But No Loop On WAN:

Ask yourself this question : that first hop, is it set up to reply to ICMP packets ?

I thought that on *posix traceroute uses UDM packets, not ICMP. If I run it with ICMP, I get completely different result - basically no hops whatsoever:

[2.4.4-RELEASE][admin@10.10.10.1]/root: traceroute -n -I  8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 48 byte packets
 1  8.8.8.8  1.584 ms  1.204 ms  2.691 ms

In my case the first hop is a Verizon FIOS fiber termination. I have no control over that. Even if its setup to block ICMP or not decrement TTL, why would traceroute give different results when running inside LAN and directly on WAN? I would expect them to be the same, except inside LAN I would see one additional first hop. That difference I dont understand and cant explain.

Gertjan · Mar 25, 2020, 1:55 PM

@miki725 said in Traceroute Loop On LAN Interface But No Loop On WAN:

different results when running inside LAN and directly on WAN?

From 'inside' pfSense = console or SSH access :

[2.4.5-RC][admin@company-network.net]/root: traceroute -n -I  8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 48 byte packets
 1  192.168.10.1  0.747 ms  0.388 ms  0.493 ms
 2  80.10.232.121  16.704 ms  14.097 ms  9.825 ms
 3  193.253.94.114  20.589 ms  19.496 ms  20.699 ms
 4  193.252.100.25  22.961 ms  22.469 ms  22.471 ms
 5  193.252.160.46  26.084 ms  25.983 ms  25.830 ms
 6  193.252.137.14  25.093 ms *  26.868 ms
 7  193.251.243.103  26.270 ms  25.086 ms  24.838 ms
 8  74.125.147.10  25.960 ms  26.086 ms  25.092 ms
 9  108.170.245.1  27.456 ms  26.870 ms  26.629 ms
10  66.249.95.247  24.775 ms  25.909 ms  25.658 ms
11  8.8.8.8  25.401 ms  25.915 ms  25.400 ms
[2.4.5-RC][admin@company-network.net]/root:

192.168.10.1 = firs hop is my ISP router. I control that router.

One router hop down, on my LAN, from a Windows PC :

C:\Users\Reception-Gauche>tracert 8.8.8.8

Détermination de l'itinéraire vers dns.google [8.8.8.8] avec un maximum de 30 sauts :

  1    <1 ms    <1 ms    <1 ms company-network.net[192.168.1.1]
  2    <1 ms    <1 ms    <1 ms  192.168.10.1
  3    17 ms    18 ms     9 ms  80.10.232.121
  4    19 ms    19 ms    19 ms  193.253.94.114
  5    28 ms    23 ms    26 ms  193.252.100.25
  6    37 ms    37 ms    34 ms  193.252.160.46
  7    26 ms    25 ms    25 ms  193.252.137.14
  8    26 ms    40 ms    29 ms  et-4-1-8-0.pastr3.-.opentransit.net [193.251.243.103]
  9    25 ms    24 ms    26 ms  74.125.147.10
 10    26 ms    26 ms    26 ms  108.170.245.1
 11    26 ms    25 ms    25 ms  66.249.95.247
 12    25 ms    25 ms    25 ms  dns.google [8.8.8.8]

Itinéraire déterminé.

You've met already 192.168.10.1, my ISP router.
Meet my pfsense, the first hop now :

1    <1 ms    <1 ms    <1 ms company-network.net[192.168.1.1]

The last LAN "tracert" has on hop more (12) as the first, which is as excepted.

posix, or not, every admin can do with his router as he sees fit ;)

miki725 · Apr 7, 2020, 2:53 PM

So turns out there is no loop. pfSense rewrites ICMP errors IP addresses. Asking more details about that in https://forum.netgate.com/topic/152252/pfsense-rewrites-source-ip-for-icmp-errors-breaking-traceroute