NPt not working on 6RD tunnel delegated addresses?

dlasher

So I think I found a bug, but want to sanity check it.

Setup
-- Cable IPv6 (native)
-- DSL IPv6 (6rd)
-- NPt

idea: create dummy (vlan) interface, track, pick up /64 via track, use that /64 as oubound NPt

Working perfectly on the native connection
Not working on the 6RD connection

Identical rules for native/6RD connections.

~~Packet capture looks like packets go out, come back, but then aren't forwarded back to the ULA host.~~

EDIT: See below, incorrect conclusion.

~~Example, pinging from ULA host, to google DNS server:~~

11:47:07.025025 AF IPv4 (2), length 72: (tos 0x0, ttl 40, id 21973, offset 0, flags [none], proto IPv6 (41), length 68)
    97.120.xxx.xxx > 205.171.2.64: (hlim 64, next-header ICMPv6 (58) payload length: 8) 2602:61:7891:f500:: > 2001:428::1: [icmp6 sum ok] ICMP6, echo request, seq 1978
11:47:07.058230 AF IPv4 (2), length 72: (tos 0x0, ttl 250, id 34336, offset 0, flags [none], proto IPv6 (41), length 68)
    205.171.2.64 > 97.120.xxx.xxx: (hlim 61, next-header ICMPv6 (58) payload length: 8) 2001:428::1 > 2602:61:7891:f500::: [icmp6 sum ok] ICMP6, echo reply, seq 1978

~~That looks good - sent, replied, should be fine right? However, the ULA host never gets the replies:~~

11:53:10.789161 00:24:e8:xx:xx:xx > 00:1b:21:13:b6:a1, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x427b2, hlim 64, next-header ICMPv6 (58) payload length: 64) fddd::dd20:e19b:9a67:a233:5411 > 2607:f8b0:400a:809::200e: [icmp6 sum ok] ICMP6, echo request, seq 1086
11:53:11.813163 00:24:e8:xx:xx:xx > 00:1b:21:13:b6:a1, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x427b2, hlim 64, next-header ICMPv6 (58) payload length: 64) fddd::dd20:e19b:9a67:a233:5411 > 2607:f8b0:400a:809::200e: [icmp6 sum ok] ICMP6, echo request, seq 1087

~~I think it's because the outbound NPt isn't translating to the right block:~~

binat on $QWEST inet6 from fddd:0:0:dd20::/64 to any -> 2602:61:7891:f502::/64
binat on $QWEST inet6 from any to 2602:61:7891:f502::/64 -> fddd:0:0:dd20::/64

It should be translating to 2602:61:7891:f502::, but it's using 2602:61:7891:f500::: (see first capture)

Something else I'm missing?

dlasher

(Showing it working via native IPv6, same model)

ULA

12:02:03.958015 IP6 fddd::dd20:e19b:9a67:a233:5411 > 2607:f8b0:400e:c02::64: ICMP6, echo request, seq 18, length 64
12:02:03.977828 IP6 2607:f8b0:400e:c02::64 > fddd::dd20:e19b:9a67:a233:5411: ICMP6, echo reply, seq 18, length 64
12:02:04.959713 IP6 fddd::dd20:e19b:9a67:a233:5411 > 2607:f8b0:400e:c02::64: ICMP6, echo request, seq 19, length 64
12:02:04.977801 IP6 2607:f8b0:400e:c02::64 > fddd::dd20:e19b:9a67:a233:5411: ICMP6, echo reply, seq 19, length 64

Native

12:02:48.160809 IP6 2001:558:6025:6d:2026:2d0c:e503:2046 > 2001:558:feed::1: ICMP6, echo request, seq 3834, length 8
12:02:48.168819 IP6 2001:558:feed::1 > 2001:558:6025:6d:2026:2d0c:e503:2046: ICMP6, echo reply, seq 3834, length 8
12:02:48.661833 IP6 2001:558:6025:6d:2026:2d0c:e503:2046 > 2001:558:feed::1: ICMP6, echo request, seq 3835, length 8
12:02:48.670367 IP6 2001:558:feed::1 > 2001:558:6025:6d:2026:2d0c:e503:2046: ICMP6, echo reply, seq 3835, length 8

jimp

That top capture doesn't look at all related to the second part. Not even the same addresses, so it's probably not from your NPt but from something like gateway monitoring.

What shows up in the state table for the 6RD subnet? You should be able to start that ping and check Diagnostics > States and filter on 2602:61:7891:f502.

dlasher

@jimp said in NPt not working on 6RD tunnel delegated addresses?:

That top capture doesn't look at all related to the second part. Not even the same addresses, so it's probably not from your NPt but from something like gateway monitoring.

Good point, grabbing more captures... Nope, that was pinger, you're right.

What shows up in the state table for the 6RD subnet? You should be able to start that ping and check Diagnostics > States and filter on 2602:61:7891:f502.

Nada - good catch.

So looking at the LAN subnet:

Packets	Bytes	
LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22698] -> 2607:f8b0:400e:c02::8a[22698]	NO_TRAFFIC:NO_TRAFFIC	92 / 0	9 KiB / 0 B

But filtering on interface "all" I get no matches for what should be the external IPv6 address. The IPv6 IP changed already, so I rebuilt the NPt rules, but still not working;

Status
up
IPv6 Address
2602:61:78e2:b02::1
Subnet mask IPv6
64
MTU
1500

and

binat on $QWEST inet6 from fddd:0:0:dd20::/64 to any -> 2602:61:78e2:b02::/64
binat on $QWEST inet6 from any to 2602:61:78e2:b02::/64 -> fddd:0:0:dd20::/64

So if the outside IP isn't showing up in Diagnostic->States, where else can I look?

jimp

Does the inside IP address show in states? Or the target?

dlasher

@jimp said in NPt not working on 6RD tunnel delegated addresses?:

Does the inside IP address show in states? Or the target?

pinging ipv6.google.com from the LAN:

LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22723] -> 2607:f8b0:400a:804::200e[22723]	NO_TRAFFIC:NO_TRAFFIC	3.431 K / 0	348 KiB / 0 B	
wan_stf	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22723] -> 2607:f8b0:400a:804::200e[22723]	NO_TRAFFIC:NO_TRAFFIC	3.431 K / 0	348 KiB / 0 B

dlasher

btw, this is what it looks like if I swing to the other gateway, the Native IPv6, which DOES work.

Packets	Bytes	
LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22774] -> 2607:f8b0:400a:804::200e[22774]	NO_TRAFFIC:NO_TRAFFIC	3 / 3	312 B / 312 B	
COMCAST	ipv6-icmp	2601:1c2:4001:33b2:e19b:9a67:a233:5411[22774] (fddd::dd20:e19b:9a67:a233:5411[22774]) -> 2607:f8b0:400a:804::200e[22774]

I'm not sure if wan_stf means wan_6rd, but it looks like wan_stf isn't using the right IP when it's swung over to the 6RD connection.

 QWEST (wan)     -> pppoe0     -> v4/PPPoE: 97.120.X.X/32
                                  v6/6RD: 2602:61:78e2:b00::/24

LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22776] -> 2607:f8b0:400a:804::200e[22776]	NO_TRAFFIC:NO_TRAFFIC	638 / 0	65 KiB / 0 B	
wan_stf	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22776] -> 2607:f8b0:400a:804::200e[22776]	NO_TRAFFIC:NO_TRAFFIC	638 / 0	65 KiB / 0 B

BTW, from the perspective of the PFsense box, the 6RD connection is working. I'm getting delegations, and traceroutes sourcing from the /64's works.

[2.4.3-RELEASE][admin@pfsense001]/root: traceroute6 -s 2602:61:7891:f501::1 2001:4860:4860::8844
traceroute6 to 2001:4860:4860::8844 (2001:4860:4860::8844) from 2602:61:7891:f501::1, 64 hops max, 12 byte packets
 1  *
    2602::205:171:3:250  13.856 ms *
 2  2001:428:0:1:205:171:11:21  14.171 ms  11.433 ms  13.610 ms
 3  2001:428::205:171:3:129  11.604 ms  14.572 ms  13.865 ms
 4  2001:428:6402:10:0:2b:0:2  20.245 ms  19.617 ms  16.102 ms
 5  2001:4860:0:1041::1  14.852 ms
    2001:4860:0:1040::1  12.664 ms  17.040 ms
 6  2001:4860:0:1::1659  15.389 ms
    2001:4860:0:1::165d  15.230 ms
    2001:4860:0:1::17cb  15.811 ms
 7  google-public-dns-b.google.com  11.740 ms  14.457 ms  11.836 ms

2001:428:: = qwest/centurylink

JKnott

What exactly is it you're trying to do? IPv6 supports multiple address ranges on the same LAN. You could have both WAN connections, providing different prefixes. Devices on the LAN will then have both addresses, but the RAs can be configured to give one prefix priority.

dlasher

@jknott said in NPt not working on 6RD tunnel delegated addresses?:

What exactly is it you're trying to do? IPv6 supports multiple address ranges on the same LAN. You could have both WAN connections, providing different prefixes. Devices on the LAN will then have both addresses, but the RAs can be configured to give one prefix priority.

I'm trying to provide survivable IPv6 access to all LAN devices, across (4) sources, without relying on the client to time out from no-longer-working IPv6 IP addresses, when a WAN fails.

My experiments with multiple IPv6 on client, so far, have been painful, but it's possible I haven't configured things as well as they should have been.

NPt, on the other hand, combined with gateway monitoring, seems to accomplish exactly that.. except the single-carrier 6RD challenge experienced above.

dlasher

Any other ideas @jimp ? Still broken..

JKnott

@dlasher said in NPt not working on 6RD tunnel delegated addresses?:

Any other ideas @jimp ? Still broken..

Possibly you're not likely to find what you want because of this:

"I’m trying to provide survivable IPv6 access to all LAN devices, across (4) sources, without relying on the client to time out from no-longer-working IPv6 IP addresses, when a WAN fails."

I don't think there is a way to switch without a time out. Failover doesn't automagically happen. There has to be a detectable failure to trigger it. That trigger could be a keep alive test, such as periodically pinging the gateway. It could be routing protocols determining a route is no longer viable etc. Until the system knows a failure has occurred, there's no way for the switch to occur. About the only thing that's likely to cause an almost immediate switch is total failure of the Ethernet connection.

dlasher

@jknott said in NPt not working on 6RD tunnel delegated addresses?:

I don't think there is a way to switch without a time out. Failover doesn't automagically happen. There has to be a detectable failure to trigger it. That trigger could be a keep alive test, such as periodically pinging the gateway. It could be routing protocols determining a route is no longer viable etc. Until the system knows a failure has occurred, there's no way for the switch to occur. About the only thing that's likely to cause an almost immediate switch is total failure of the Ethernet connection.

It's working perfectly (except the 6RD portion) using gateway monitoring and NPt.

With a gateway group that has

Comcast - Tier 1
HE tunnel - Tier 2

it works. tier 1 as long as it's up, then falls to tier 2.

I'm just trying to figure out why the 6RD NPt doesn't seem to be correctly translating.

LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22776] -> 2607:f8b0:400a:804::200e[22776]	NO_TRAFFIC:NO_TRAFFIC	638 / 0	65 KiB / 0 B	
wan_stf	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22776] -> 2607:f8b0:400a:804::200e[22776]	NO_TRAFFIC:NO_TRAFFIC	638 / 0	65 KiB / 0 B

Notice the source address on the second line. It should be translated, and it's not. It should look something like this:

Packets	Bytes	
LAN20	ipv6-icmp	fddd::dd20:e19b:9a67:a233:5411[22774] -> 2607:f8b0:400a:804::200e[22774]	NO_TRAFFIC:NO_TRAFFIC	3 / 3	312 B / 312 B	
COMCAST	ipv6-icmp	2601:1c2:4001:33b2:e19b:9a67:a233:5411[22774] (fddd::dd20:e19b:9a67:a233:5411[22774]) -> 2607:f8b0:400a:804::200e[22774]

dlasher

Any ideas @jimp ? Still broken. See post right above this one. I can submit an actual bug report if you'd like.

risola316

@dlasher I was able to get this working in release 2.5.4. The problem appears to be because an additional interface is created for the 6rd connection which isn't available to select as an interface in the NPT configuration (in my case opt1_stf for most others probably wan_stf). I was able to get it to work using the following steps but on reboot I'm forced to interact via the command line due to an a warning stating "Warning: Configuration references interfaces that do not exist: opt1_stf" which causes the interface assignment option tool to run.

In Interfaces/Assignments there should be an Interface you can add to a Network Port an"_stf" for me it was called "opt1_stf" and in your case it appears to be called "wan_stf"
Add the interface to the appropriate Network Port but do NOT change any settings or enable the interface
Next in Interfaces/Interface Groups I had to add the newly created Interface (OPT1 in my case) to an Interface group also containing the WAN interface with the 6rd connection (GRP_WAN)
Update your previously created NPT rules to use the GRP_WAN you just created

This allowed me to get IPv6 connectivity while using a ULA address in a multi-wan configuration. In my case I have Cox (DHCPv6) and CenturyLink (PPPoE 6RD)

NOTE: Upon reboot this causes pfsense boot to hang until there is user interaction
This appears to be due to the "_stf" interface not being available earlier in the boot sequence. If you respond no to the question on VLANS and just hit enter when it ask about configuring the WAN booting continues and everything is configured appropriately upon boot. This will NOT work if you're 6rd connection isn't available.

Have you had any other luck with this, I'm thinking about logging a bug as my current solution isn't viable long term.

P.S. This may be fixed in 2.5 as I think it's related to this bug
https://redmine.pfsense.org/issues/7142

I've created the following topic as the above fix doesn't appear to resolve the issue:
https://forum.netgate.com/topic/155053/npt-not-working-on-wan-interface-when-ipv6-is-provided-via-a-6rd-tunnel-bug

dlasher

@risola316 said in NPt not working on 6RD tunnel delegated addresses?:

@dlasher I was able to get this working in release 2.5.4.
<snip>
Have you had any other luck with this, I'm thinking about logging a bug as my current solution isn't viable long term.

Cool workaround, however, like you, I don't find having the box require manual intervention every reboot to be a viable solution.

If you haven't already filed a bug, do so please, I'll add notes to it as well.

viktor_g

Fixed in upstream, see https://redmine.pfsense.org/issues/10757