pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute

[miki725]

@johnpoz said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

What is your outbound nat..

Im using Automatic outbound NAT rule generation. (IPsec passthrough included):

Here are my nat rules I see in /tmp/rules.debug:

# Outbound NAT rules (automatic)

# Subnets to NAT
table <tonatsubnets> { 192.168.1.10/32 127.0.0.0/8 ::1/128 10.10.10.0/24 10.10.5.0/24 10.10.20.0/24 10.10.30.0/24 10.10.40.0/24 10.10.50.0/24 10.10.60.0/24 10.10.70.0/24 10.10.80.0/24 10.10.90.0/24 10.10.100.0/24 10.10.255.1/24 }
nat on $WAN inet from <tonatsubnets> to any port 500 -> x.x.x.x/32  static-port
nat on $WAN inet6 from <tonatsubnets> to any port 500 -> (lagg0.4090)  static-port
nat on $WAN inet from <tonatsubnets> to any -> x.x.x.x/32 port 1024:65535
nat on $WAN inet6 from <tonatsubnets> to any -> (lagg0.4090) port 1024:65535

500 port rule I think is coming from IPSec:

Sorry Im new to NAT in general (thats among the reasons I got pfSense since I want to learn networking better). If my understanding is correct NAT should rewrite the source IP on outbound connections, add entry of the flow table, and finally for inbound connections, change the destination IP address so the packet can be redirected to an IP inside LAN. If so I dont fully understand why source IP is rewritten on the inbound connection.

Also keep in mind that responses from the hops would be ICMP but if you using linux for example the actual data sent doing the trace would be UDP packets.. Or sure you could even send TCP... But default to udp on linux, and windows would be icmp sent, etc..

Thanks! Looks like in this case traceroute is using UDP with ICMP error responses.

[johnpoz]

So yes the nat would send the traffic back into your sender of the traceroute.. So I am just thinking you are confused to how nat works in general... But seeing missing spots in your trace that do not answer is quite normal these days of everyone blocking shit ;)

Are you doing anything odd on your lan side interface for rules or policy routing?

I sure an the hell can not duplicate your issue, and can traceroute from any machine on my network just fine using either icmp, udp or even tcp...

$ tcptraceroute -n www.google.com 443
Selected device ens3, address 192.168.2.12, port 50519 for outgoing packets
Tracing the path to www.google.com (172.217.8.196) on TCP port 443 (https), 30 hops max
 1  192.168.2.253  0.493 ms  0.332 ms  0.324 ms
 2  50.4.135.1  11.620 ms  12.213 ms  11.700 ms
 3  76.73.191.106  10.921 ms  12.476 ms  11.911 ms
 4  76.73.164.142  9.586 ms  18.234 ms  12.953 ms
 5  76.73.164.154  20.807 ms  20.399 ms  20.238 ms
 6  76.73.191.242  14.286 ms  11.709 ms  14.869 ms
 7  143.59.95.224  28.425 ms  12.980 ms  13.262 ms
 8  75.76.35.8  11.605 ms  22.702 ms  10.050 ms
 9  72.14.211.145  11.355 ms  31.802 ms  18.859 ms
10  108.170.243.225  23.928 ms  13.804 ms  14.214 ms
11  72.14.232.187  12.821 ms  12.967 ms  13.243 ms
12  172.217.8.196 [open]  21.723 ms  31.557 ms  17.055 ms

[miki725]

Didnt do anything crazy as far as I know:

Ill keep digging on my end. Thanks for your help.

[Alex Atkin UK]

@miki725 Is it the same problem documented here? https://redmine.pfsense.org/issues/9263

[Gertjan]

You're using traffic (ICMP) shaping ?

[johnpoz]

Sure looks like that is his problem... @Alex-Atkin-UK

[miki725]

@Alex-Atkin-UK

ooooh. I forgot about the limiter:

can confirm that when I disabled floating rule with the queue, traceroute works as expected inside LAN.

Thank a lot for linking to that. Now would be curious why a limiter would have that affect.

[johnpoz]

@miki725 said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

Now would be curious why a limiter would have that affect.

It shouldn't - therefore the bug report ;)

You should of brought up the limiter, when asked doing anything odd on lan side rules (floating would be considered lan side) hehehe.. I will make sure to always mention floating rules going forward..

[miki725]

@johnpoz said in pfSense Rewrites Source IP for ICMP Errors Breaking Traceroute:

You should of brought up the limiter, when asked doing anything odd on lan side rules (floating would be considered lan side) hehehe.. I will make sure to always mention floating rules going forward..

My apologies. Forgot about them. Makes perfect sense that they are as legitimate LAN side rule as any other.

Thanks everyone for helping even with limited provided information. Really appreciate everyones time and effort. Stay safe!

[Alex Atkin UK]

@miki725 I remembered it as I had the same problem.

As I recall you have to add a Pass floating rule for the WAN interface at the top of the list for echo reply/requests with Quick ticked, so it bypasses the limiter rule.

[Gertjan]

@Alex-Atkin-UK : Thanks.

That were those for :

( bottom two rules )

[miki725]

Thanks for the suggestion. Didnt even think of bypassing it.

Where is a standard place to add limiter rules? Mine is currently is a floating rule. Even with the ICMP bypass rule with Quick above it, traceroute is still impacted:

I tried removing the Quick from the limiter rule but that does not seem to help. Only thing which helps is disabling the limiter altogether. Im probably not doing something correctly if this works for you.

[Gertjan]

Like me, you added floating rules to do something about buffer bloat.

These are all my floating rules :

You're missing one IPv4 rule : the "direction : IN" rule.
You have the "Out" rule, the one with the gateway (WAN_DHCP).

I also use IPv6 - so that adds 3 more rules.

[bobbenheim]

In your ICMP rule setting direction: out, interface: WAN and change default to your actual gateway in advanced > Gateway
should do the job.

[jamesonp]

This got me today. I can confirm the floating rule for ICMP solves the issue.