IPv6 ICMPv6 Type 3 / TTL handling change/bug?

Napsterbater

On 2.3.4 and every version prior and every other platform i can think of, running a trace from a LAN Computer to a Destination on the internet would show pfsenses LAN IP as the first hop or at the appropriate hop, but seems that pfsense is now basing the IP it uses to send a ICMPv6 Type 3 / "Time Expired" on the interface that would have been the exit interface if the TTL hadn't run out.

This has been seen on 2.4.0.r.20170827 and 2.4.0.r.20170831 so far, on 2 different systems/setups/locations.

LAN Computer -pfsenseLAN> pfsense -pfsenseWAN> ISP, etc. = pfsenseWAN IPv6 is seen in the trace.

Internet Looking Glass -pfsenseWAN> pfsense -pfsenseLAN> LAN Computer, etc. = pfsenseLAN IPv6 is seen in the trace.'

Basically it went from using the IP of the interface the packet with TTL = 1 came in, to using the IP of the interface the packet would have went out had it had a TTL = >1.

From Lan system to Internet
Note xxx = same in both places, its my he tunnel prefix.

Tracing route to google.com [2607:f8b0:4006:819::200e]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  ******.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:xxx::2]
  2    29 ms    26 ms    28 ms  ******.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:xxx::1]
  3    21 ms    23 ms    21 ms  ge5-4.core1.ash1.he.net [2001:470:0:90::1]

From Looking Glass to LAN system
Note again xxxx = the same in both hops, as this is part of my /48, I have a /64 on the lan.

racing the route to IPv6 node 2001:470:xxxx:1:7dc5:db6f:aaf8:f195 from 1 to 30 hops

  1    18 ms   18 ms    6 ms 2001:470:0:90::2
  2    23 ms   20 ms   25 ms 2001:470:xxxx:1::3
  3    24 ms   25 ms   74 ms 2001:470:xxxx:1:7dc5:db6f:aaf8:f195

doktornotor

that would have been the exit interface is the TTL hadn't run out.
the interface the packet would have went out had TTL = >1.

Can you try again in plain English?

Napsterbater

When pfsense receives a packet with a TTL of 1, it should respond with a ICMPv6 Type 3 packet with a source IP of the interface the packet came in on. It has always done this prior to 2.4 and every other router i have seen does this.

I.e. If it received the packet on an interface with an IP of fdda:535f:111b:114c::1 the source IP in the ICMPv6 Type 3 packet should be fdda:535f:111b:114c::1.

But what it is doing is if a packet comes in with a TTL = 1 it is sending a ICMPv6 Type 3 packet with the source IP of the interface that the packet would have been routed out had the TTL been more than 1, i.e. its sending the ICMP packet with a source IP of fdda:535f:111b:2000::1 (WAN)

For the example
LAN = fdda:535f:111b:114c::1
WAN = fdda:535f:111b:2000::1

This can be seen clearly in these traces.

From Lan system to Internet

Tracing route to google.com [2607:f8b0:4006:819::200e]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  2001:470:7:yyy::2 <-pfsense "WAN" IP
  2    29 ms    26 ms    28 ms  2001:470:7:yyy::1
  3    21 ms    23 ms    21 ms  2001:470:0:90::1

what it should look like

Tracing route to google.com [2607:f8b0:4006:819::200e]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  2001:470:xxxx:1::3 <-pfsense "LAN" IP
  2    29 ms    26 ms    28 ms  2001:470:7:yyy::1
  3    21 ms    23 ms    21 ms  ge5-4.core1.ash1.he.net [2001:470:0:90::1]

From Looking Glass to LAN system

racing the route to IPv6 node 2001:470:xxxx:1:7dc5:db6f:aaf8:f195 from 1 to 30 hops

  1    18 ms   18 ms    6 ms 2001:470:0:90::2
  2    23 ms   20 ms   25 ms 2001:470:xxxx:1::3 <-pfsense "LAN" IP
  3    24 ms   25 ms   74 ms 2001:470:xxxx:1:7dc5:db6f:aaf8:f195

what it should look like

racing the route to IPv6 node 2001:470:xxxx:1:7dc5:db6f:aaf8:f195 from 1 to 30 hops

  1    18 ms   18 ms    6 ms 2001:470:0:90::2
  2    23 ms   20 ms   25 ms 2001:470:7:yyy::2 <-pfsense "WAN" Ipv6
  3    24 ms   25 ms   74 ms 2001:470:xxxx:1:7dc5:db6f:aaf8:f195

doktornotor

Oh, much better now. 8)

P.S. Yes, confirmed.

bimmerdriver

I'm running the latest 2.4.0 RC.

I tried tracert to a windows 10 host from HE looking glass and also from the same windows 10 host to first hop of HE looking glass. It didn't use the identical route in both directions, but in both cases, the closest hop to the host was the lan address of pfsense. It did not use the wan address of pfsense on the way in or on the way out.

johnpoz

Yeah seeing the same thing - I will update todays RC snap.. Do not get back lan address in the hop, seeing the HE interface as first hop..

Just updated to the most current RC snap..

2.4.0-RC (amd64)
built on Sat Sep 02 16:41:15 CDT 2017
FreeBSD 11.0-RELEASE-p12

The system is on the latest version.
Version information updated at Sun Sep 3 6:57:08 CDT 2017

And confirm this is not correct.. Should see the lan IPv6, then far end of HE tunnel IPv6 address.. What see is my side of the tunnel my outbound IP and then far side of the HE tunnel.

Napsterbater

Created https://redmine.pfsense.org/issues/7839 for this.

jimp

I'm not sure there is anything we can do for that. It's most likely a FreeBSD or pf bug.

Maybe try with pf disabled, see if the same behavior happens. If it does, it's definitely a pf bug. If it works with pf disabled, then it's possible there is some quirk of route-to or reply-to that makes pf do that.

But odds are higher that it's a FreeBSD issue.

Napsterbater

This seems to have been corrected in 2.4.1/FreeBSD 11.1, At Least after testing immediately after reboot from upgrade.

doktornotor

@Napsterbater:

This seems to have been corrected in 2.4.1/FreeBSD 11.1, At Least after testing immediately after reboot from upgrade.

Confirmed.