pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute

[miki725]

So Im trying to debug why my traceroute is broken when running it inside my LAN but running traceroute to outside of LAN:

❯ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
 1  10.10.60.1 (10.10.60.1)  2.680 ms  2.897 ms  3.158 ms
 2  * * *
 3  dns.google (8.8.8.8)  15.396 ms  10.948 ms  7.804 ms
 4  * * *
 5  dns.google (8.8.8.8)  7.802 ms  6.031 ms  4.351 ms
 6  dns.google (8.8.8.8)  8.543 ms  8.123 ms  6.374 ms
 7  * * *
 8  dns.google (8.8.8.8)  14.940 ms  9.293 ms  7.938 ms

Every hop which would return ICMP error I would see my target IP, not the intermediate hop IP. To debug this further I did a packet capture at the same time within my LAN and on WAN interface directly in pfSense:

Left side is packet capture on WAN and right side is packet capture inside LAN. Ive blurred my WAN IP address for security and matched both sides for the corresponding packets to be side-by-side.

You can see that for every ICMP error returned by intermediate hops, pfSense rewrites the source IP to my target destination IP of 8.8.8.8 hence traceroute only shows the target IP within my LAN.

If I run the same traceroute on my WAN interface, I see the same hops (minus the first one) except now I see real IPs of the intermediate hops:

[2.4.5-RELEASE][admin@pfsense]/root: traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  * * *
 2  100.41.218.210  5.323 ms  4.055 ms
    100.41.218.212  7.429 ms
 3  * * *
 4  140.222.227.145  4.247 ms
    140.222.230.215  3.671 ms
    140.222.1.45  4.005 ms
 5  72.14.208.130  4.582 ms  3.582 ms
    72.14.214.38  4.046 ms
 6  * * *
 7  8.8.8.8  3.781 ms  3.720 ms  4.079 ms

Is this a bug? Expected behavior. If so any way I can change it? I cant find any options anywhere to adjust how pfSense handles ICMP NAT.

[Gertjan]

Can't reproduce :

[2.4.5-RELEASE][root@pfsense.front-door.net]/root: traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  192.168.10.1  0.753 ms  0.380 ms  0.478 ms
 2  80.10.232.121  9.467 ms  9.469 ms  10.226 ms
 3  193.253.94.114  20.210 ms  19.501 ms  19.184 ms
 4  193.252.100.25  23.841 ms  22.490 ms  23.433 ms
 5  193.252.160.46  26.457 ms  26.998 ms  26.202 ms
 6  193.252.137.14  25.170 ms  25.860 ms  28.772 ms
 7  193.251.129.185  26.077 ms
    193.251.131.129  25.483 ms
    193.251.151.135  25.699 ms
 8  74.125.50.250  25.696 ms
    193.251.255.104  25.841 ms
    72.14.198.18  26.042 ms
 9  108.170.244.225  21.892 ms *
    108.170.244.161  25.649 ms
10  216.239.48.147  25.803 ms
    8.8.8.8  20.963 ms  21.703 ms

Btw : NATting ICMP ?

[johnpoz]

What is your outbound nat.. All traffic leaving your wan would be natted.. the automatic rules are ANY..

Look in your actual rules.
example here is mine from /tmp/rules.debug

[2.4.5-RELEASE][admin@sg4860.local.lan]/: cat /tmp/rules.debug | grep nat
<snipped>
table <tonatsubnets> { 127.0.0.0/8 ::1/128 192.168.9.0/24 192.168.2.0/24 192.168.200.0/24 192.168.4.0/24 192.168.6.0/24 192.168.7.0/24 192.168.3.0/24 10.0.8.0/24 10.0.200.0/24 }
nat on $WAN inet from <tonatsubnets> to any -> 64.53.x.x/32 port 1024:65535 
<snipped>

If you are using hybrid/manual - you could of just picked tcp/udp etc.. which then no your icmp would not be natted.

Also keep in mind that responses from the hops would be ICMP but if you using linux for example the actual data sent doing the trace would be UDP packets.. Or sure you could even send TCP... But default to udp on linux, and windows would be icmp sent, etc..

[miki725]

@johnpoz said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

What is your outbound nat..

Im using Automatic outbound NAT rule generation. (IPsec passthrough included):

Here are my nat rules I see in /tmp/rules.debug:

# Outbound NAT rules (automatic)

# Subnets to NAT
table <tonatsubnets> { 192.168.1.10/32 127.0.0.0/8 ::1/128 10.10.10.0/24 10.10.5.0/24 10.10.20.0/24 10.10.30.0/24 10.10.40.0/24 10.10.50.0/24 10.10.60.0/24 10.10.70.0/24 10.10.80.0/24 10.10.90.0/24 10.10.100.0/24 10.10.255.1/24 }
nat on $WAN inet from <tonatsubnets> to any port 500 -> x.x.x.x/32  static-port
nat on $WAN inet6 from <tonatsubnets> to any port 500 -> (lagg0.4090)  static-port
nat on $WAN inet from <tonatsubnets> to any -> x.x.x.x/32 port 1024:65535
nat on $WAN inet6 from <tonatsubnets> to any -> (lagg0.4090) port 1024:65535

500 port rule I think is coming from IPSec:

Sorry Im new to NAT in general (thats among the reasons I got pfSense since I want to learn networking better). If my understanding is correct NAT should rewrite the source IP on outbound connections, add entry of the flow table, and finally for inbound connections, change the destination IP address so the packet can be redirected to an IP inside LAN. If so I dont fully understand why source IP is rewritten on the inbound connection.

Also keep in mind that responses from the hops would be ICMP but if you using linux for example the actual data sent doing the trace would be UDP packets.. Or sure you could even send TCP... But default to udp on linux, and windows would be icmp sent, etc..

Thanks! Looks like in this case traceroute is using UDP with ICMP error responses.

[johnpoz]

So yes the nat would send the traffic back into your sender of the traceroute.. So I am just thinking you are confused to how nat works in general... But seeing missing spots in your trace that do not answer is quite normal these days of everyone blocking shit ;)

Are you doing anything odd on your lan side interface for rules or policy routing?

I sure an the hell can not duplicate your issue, and can traceroute from any machine on my network just fine using either icmp, udp or even tcp...

$ tcptraceroute -n www.google.com 443
Selected device ens3, address 192.168.2.12, port 50519 for outgoing packets
Tracing the path to www.google.com (172.217.8.196) on TCP port 443 (https), 30 hops max
 1  192.168.2.253  0.493 ms  0.332 ms  0.324 ms
 2  50.4.135.1  11.620 ms  12.213 ms  11.700 ms
 3  76.73.191.106  10.921 ms  12.476 ms  11.911 ms
 4  76.73.164.142  9.586 ms  18.234 ms  12.953 ms
 5  76.73.164.154  20.807 ms  20.399 ms  20.238 ms
 6  76.73.191.242  14.286 ms  11.709 ms  14.869 ms
 7  143.59.95.224  28.425 ms  12.980 ms  13.262 ms
 8  75.76.35.8  11.605 ms  22.702 ms  10.050 ms
 9  72.14.211.145  11.355 ms  31.802 ms  18.859 ms
10  108.170.243.225  23.928 ms  13.804 ms  14.214 ms
11  72.14.232.187  12.821 ms  12.967 ms  13.243 ms
12  172.217.8.196 [open]  21.723 ms  31.557 ms  17.055 ms

[miki725]

Didnt do anything crazy as far as I know:

Ill keep digging on my end. Thanks for your help.

[Alex Atkin UK]

@miki725 Is it the same problem documented here? https://redmine.pfsense.org/issues/9263

[Gertjan]

You're using traffic (ICMP) shaping ?

[johnpoz]

Sure looks like that is his problem... @Alex-Atkin-UK

[miki725]

@Alex-Atkin-UK

ooooh. I forgot about the limiter:

can confirm that when I disabled floating rule with the queue, traceroute works as expected inside LAN.

Thank a lot for linking to that. Now would be curious why a limiter would have that affect.

[johnpoz]

@miki725 said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

Now would be curious why a limiter would have that affect.

It shouldn't - therefore the bug report ;)

You should of brought up the limiter, when asked doing anything odd on lan side rules (floating would be considered lan side) hehehe.. I will make sure to always mention floating rules going forward..

[miki725]

@johnpoz said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

You should of brought up the limiter, when asked doing anything odd on lan side rules (floating would be considered lan side) hehehe.. I will make sure to always mention floating rules going forward..

My apologies. Forgot about them. Makes perfect sense that they are as legitimate LAN side rule as any other.

Thanks everyone for helping even with limited provided information. Really appreciate everyones time and effort. Stay safe!

[Alex Atkin UK]

@miki725 I remembered it as I had the same problem.

As I recall you have to add a Pass floating rule for the WAN interface at the top of the list for echo reply/requests with Quick ticked, so it bypasses the limiter rule.

[Gertjan]

@Alex-Atkin-UK : Thanks.

That were those for :

( bottom two rules )

[miki725]

Thanks for the suggestion. Didnt even think of bypassing it.

Where is a standard place to add limiter rules? Mine is currently is a floating rule. Even with the ICMP bypass rule with Quick above it, traceroute is still impacted:

I tried removing the Quick from the limiter rule but that does not seem to help. Only thing which helps is disabling the limiter altogether. Im probably not doing something correctly if this works for you.

[Gertjan]

Like me, you added floating rules to do something about buffer bloat.

These are all my floating rules :

You're missing one IPv4 rule : the "direction : IN" rule.
You have the "Out" rule, the one with the gateway (WAN_DHCP).

I also use IPv6 - so that adds 3 more rules.

[bobbenheim]

In your ICMP rule setting direction: out, interface: WAN and change default to your actual gateway in advanced > Gateway
should do the job.

[jamesonp]

This got me today. I can confirm the floating rule for ICMP solves the issue.