IPv6 Path MTU Discovery automatically sets MTU to 1280 regardless of interface

[jeremy.duncan]

Cross-posted from the OpnSense forum
Hello, I have OPNsense 21.7.7-amd64 and OPNsense 21.7.8-amd64 - as both behave the same. When doing a path trace on my IPv6 network I see the opnsense firewall sending packet too big messages for 1280 to all hosts on the network (for packets higher than 1280) even though the host is set to 1450, interface is set to 1500, and the RA MTU option is set to 1450. Here's a path trace:

tracepath google.com
1?: [LOCALHOST] 0.040ms pmtu 1450
1: 2001:470:e5bf:1001:cafe:dead:beef:1 0.910ms
1: 2001:470:e5bf:1001:cafe:dead:beef:1 0.533ms
2: 2001:470:e5bf:3000::2 1.178ms
3: 2001:470:e5bf:3000::2 1.229ms pmtu 1280
3: tunnel161881.tunnel.tserv13.ash1.ipv6.he.net 8.896ms
4: 10ge2-2.core1.ash1.he.net 7.659ms
5: pr61.iad07.net.google.com 10.415ms
6: no reply

It's also in packet #9 in the attached PCAP. This behavior is wrong and violates the RFC specs. Anyone have any idea what the issue is?wifi.pcap

[JKnott]

@jeremy-duncan

Well, it's obvious. Hop 3 has a MTU of 1280. I see it's also a tunnel. That's a good clue too. When I used a 6in4 tunnel to get IPv6, it had 1280 MTU. Since that's coming from elsewhere, there's nothing you can do.

BTW, which RFC specs are you referring to? My understanding is path MTU discovery is used to determine maximum packet size and you're seeing it in action. This isn't IPv4 where routers could fragment packets too big for the link.

[johnpoz]

@jeremy-duncan said in IPv6 Path MTU Discovery automatically sets MTU to 1280 regardless of interface:

3: 2001:470:e5bf:3000::2 1.229ms pmtu 1280

What do you have your tunnel mtu set to on HE?

Mine is at 1480.. And that is what I see it change to during tracepath..

[jeremy.duncan]

@johnpoz exactly - mine on HE is set to 1480. Nothing in my network is set to 1280.

[jeremy.duncan]

@jknott @jknott see below. tunnel is set to 1480 - not 1280.

[jeremy.duncan]

OK, I think I figured it out looking at the tunnel interface MTU on the firewall. BY DEFAULT it sets to 1280 unless you set it to match the MTU on the other end of the tunnel - 1480 per HE. When I set to 1480, it no longer sent PMTU for 1280, but for 1480 like it's supposed to. Not at all intuitive...
tracepath google.com
1?: [LOCALHOST] 0.029ms pmtu 1500
1: 2001:470:e073:101::2 0.392ms
1: 2001:470:e073:101::2 0.407ms
2: 2001:470:e073:101::2 0.425ms pmtu 1480
2: tunnel202636.tunnel.tserv13.ash1.ipv6.he.net 29.177ms
3: 10ge2-2.core1.ash1.he.net 13.809ms
4: pr61.iad07.net.google.com 12.468ms

tracepath google.com
1?: [LOCALHOST] 0.033ms pmtu 1400
1: 2001:470:e5bf:1001:cafe:dead:beef:1 8.834ms
1: 2001:470:e5bf:1001:cafe:dead:beef:1 0.516ms
2: 2001:470:e5bf:3000::2 1.576ms
3: tunnel161881.tunnel.tserv13.ash1.ipv6.he.net 7.791ms
4: 10ge2-2.core1.ash1.he.net 7.385ms
5: pr61.iad07.net.google.com 7.862ms