Still weird issues with fragmented IPv6 (DNS) packets 2.4.4-p1

sigi

Hoped on https://redmine.pfsense.org/issues/8934 but still cannot resolve some small flaws. I have native IPv6 and operate a DNS Server behind with DNSSEC enabled. I research this since half a year now, but still no success.
I used tcpdump and external tests like http://icmpcheckv6.popcount.org/ , http://icmpcheck.popcount.org/ and https://dnsflagday.net/ .
The only thing i know for yet, that it happens inside pfSense. When bypassed, the icmpchecks will work. Disable scrubbing and clear DF did also made no change. Dnsflagday fails partly for IPv6:
IPv4: dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok,cookie edns512tcp=ok optlist=ok,expire,cookie,subnet
IPv6: dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=timeout ednsflags=ok docookie=timeout edns512tcp=ok optlist=ok,expire,cookie,subnet
This does not break connectivity but is annoying. No more ideas for now..

lst_hoe

At the moment we inspect what looks like pfsense does not generate IPv6 fragments for locally created "oversized" UDP datagram. We are using pfsense as IPSEC VPN endpoint and since using (native) IPv6 all clients which does not support IKE Fragmentation can not connect. I have taken a dump on the WAN interface and it shows that the packet

Dec 5 14:49:30 10.5.0.3 charon: 05[NET] <con-mobile|76> sending packet: from 2a03:3500:0:a003::100[4500] to 2003:f8:abc0:8401:a05e:95f9:3d24:81c7[4500] (1984 bytes)

simply is missing and the client retry over and over again until timeout. To my knowledge pfsense should create IPv6 UDP fragments for locally created oversized packets, no?

Gertjan

For what it's worth :

My LAN is IPv4/IPv6.
My ISP doesn't know what IPv6 is, I'm using ipv6.he.net as my "ISPv6".
And consider this : I tested on a "Windows XP pro system" (I keep it alive just for the fun).
Both test site :
icmpcheck.popcount.org
and espcially this one :
icmpcheckv6.popcount.org

Say "Ok !". No MTU erros, no fragment errors :
ICMP path MTU packet delivery

✓ All good! ICMP path MTU message was successfully delivered to you.
IP fragmented packet delivery

✓ All good! IP fragments were successfully delivered to your host.

I'm using pfSense 2.4.4p1, the Resolver (activated DNSSC support) and an extra GIF interface for the ipv6.he.net tunnel.

lst_hoe

To my knowledge the packets you are testing are TCP and are not created local on the pfsense box but forwarded. With this we also have no problem. Unfortunately IKE and DNS is UDP and created local on the pfsense box if the resolver or the VPN (Strongswan) is used.
Maybe this one is also related as it is UDP-only too : https://redmine.pfsense.org/issues/7801

johnpoz

Where you can run into problems the path between the devices that are talking over ipv6.. Since intermediate devices can not fragment.. PMTUD is kind of a must between the devices. If that is not working then sure you can have problems.

Only real solution then is to make sure your end devices never send IPv6 larger than the min 1280 mtu...

You would need to test the path between your devices to see if you have a problem to be honest, testing from pfsense to some test site would test your client and path to that site, but doesn't really test the path between pfsense and the client..

The test sites listed by @Gertjan have perl commands and the tcpdump commands you can use to actually validate traffic from pfsense is working correctly to that test site.

lst_hoe

Err, no. As said in my case the packet is created local and does not even leave the desired interface (WAN) as it should. It simply is not there at all. PMTUD only kick in if the MTU on the way is getting lower, not for packets which must already be fragmented at the source. It looks like pfsense/FreeBSD does not like to create fragments outbound.

johnpoz

Why would it need to be fragged out of the gate?

lst_hoe

Dec 5 14:49:30 10.5.0.3 charon: 05[NET] <con-mobile|76> sending packet: from 2a03:3500:0:a003::100[4500] to 2003:f8:abc0:8401:a05e:95f9:3d24:81c7[4500] (1984 bytes)

Strongswan manage the (UDP) IKEv2 connection and if the remote does not support IKEv2 Fragmentation (at application layer) the packet is because of certificates/keysize used too big to send in one IP(v6) piece. With this the only chance to get it out is to fragment at IP layer as UDP has no concept of a stream like TCP does. So the smartes move would be to send out two (or more) fragmented UDP packets with max MTU of 1280 in case of IPv6 to prevent PMTU issues, which is cumbersome for UDP anyway.

JKnott

@johnpoz said in Still weird issues with fragmented IPv6 (DNS) packets 2.4.4-p1:

Why would it need to be fragged out of the gate?

A source can fragment as necessary, if a packet is too large to fit. This might happen with UDP, where the packet contains a lot of data. Of course, routers are not allowed to fragment IPv6 at all and the trend is to using the do not fragment flag on IPv4.

lst_hoe

Strongswan is bound to the WAN interface so there is no routing at this point and pfsense (WAN) is the source. For the OP this might not be true as the DNS case is only relevant for a DNS Server which is not common to operate on the pfsense box. But either way it looks like pfsense or FreeBSD is not able to create outbound fragmented UDP packets, at least not for IPv6.

sigi

@gertjan said in Still weird issues with fragmented IPv6 (DNS) packets 2.4.4-p1:

Say "Ok !". No MTU erros, no fragment errors :

I'm using pfSense 2.4.4p1, the Resolver (activated DNSSC support) and an extra GIF interface for the ipv6.he.net tunnel.

Ok, this should not be a problem, if pfSense is terminating the HE tunnel. Passing is another story.

lst_hoe

So my issue might be related to the OP if FreeBSD does not create/forward any outbound IPv6 UDP packets at all, or it might be two different issues.

lst_hoe

A short test with IPv4 is working as expected. With IPv4 two UDP fragments are created and send out and therefore the tunnel start working, with IPv6 the oversized UDP datagram (around 2000 bytes) is missing.

sigi

At least for DNS/DNSSEC i could solve the issue, because BIND allows to tune maximum udp packet size on application level. With the value 1460 for max-udp-size the tests from http://dnsviz.net/ and https://dnsflagday.net/ works for both IPv4 and IPv6. Still have Issues with SNMPv3 and big packets over IPv6. For me this is a proof, that there is a difference how MSS works on the same OS/Interfaces between IPv6 and IPv4. This could happen on different stages, maybe even on the interface driver, #device re # RealTek 8139C+/8169/8169S/8110S in my case. Will check if Intel Cards make a difference, if time allows.

sigi

@johnpoz said in Still weird issues with fragmented IPv6 (DNS) packets 2.4.4-p1:

Only real solution then is to make sure your end devices never send IPv6 larger than the min 1280 mtu...

This was the "cloudflare-solution" for long time..
https://blog.cloudflare.com/increasing-ipv6-mtu/

maverick_slo

Hello.

Well this seems to be a problem.
domain.com. @85.xx.xx.xx (ns1.domain.com.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok,cookie edns512tcp=ok optlist=ok,expire,cookie,subnet

domain.com. @2001:xxxx:xxx:xxx::92 (ns1.domain.com.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=timeout edns512tcp=ok optlist=ok,expire,cookie,subnet

With HE.NET IPv6 servers all OK.
Also dnssec complains about packet size.

If I understood correctly thhere is nothing I can do on my side to mitigate this issue?

Thanks!

sigi

@maverick_slo The only solution i found was to adjust/reduce max-udp-size on BIND side. (Not MTU!!)

maverick_slo

Did just that:
max-udp-size 1460;

No go, still timeout.
I`m not sure this is pfsense issue.

sigi

@maverick_slo this means not that the same value is ok in your network. Try to go down further....