No IPv6 after upgrade to 23.01

maverickws

Another day without IPv6 on 23.01.

nighthawk1967

@maverickws Can't even reinstall and fix the problem goes directly to 23.01 and no ipv6

forresgeek

Hi all,

Just replying here in case my recent findings help anyone else. I don't think I have (had) the same problem as @maverickws , but it may help.

To re-cap - My ISP (BT Business - UK) allocates me a static \56 on the WAN interface via DHCP. I then use WAN address tracking to allocate a \64 to each subnet/VLAN via RADV.

Prior to 23.01 - things were all working swimmingly - 10/10 on the various ipv6 Test web-sites. After the 23.01 upgrade, this all stopped.

I did notice that I was still receiving IPv6 Addresses + DNS via RADVD. I noticed that IPV6 DNS queries were still working.

Google IPV6 test said that my IPv6 was still working, but any web-site I tried to access would just hang, SSH sessions would just hang etc. (I couldn't even reliably access this forum)

i.e. I was able to make ipv6 "connections" but not usable ones.

I then fell down the rabbit hole of IPV6 MTUs / Router Advertisements / Packet Too Big ICMP, and all of the Path MTU discovery process.

I also found the Packet Tuesday channel on YouTube - very informative. Packet Tuesday

I found the IPvFOO Chrome plugin which helped by showing me how hit-and-miss the IPv6 was - small packets (like DNS) worked, larger streams (like web-pages or SSH sessions) would stall immediately.

I discovered that the PFSense default MTU of 1500 is advertised in the RADVD advertisement packet. I read a load of stuff about MTU's, and decided to have a play with that...

I dropped the MTU down to 1280 (the minimum IPv6 MUST support) - and everything started working again. I was getting 10/10 on all of the ipv6 test sites, and my SSH sessions didn't hang.

I had a play with wireshark, and saw that the "packet too big" ICMP responses I was getting were suggesting MTU of 1492. These "Packet too big" responses were from the local LAN interface of my router to the end-device I was using... I don't know whether the end-devices (laptops etc) actually paid any heed to these packet-too-big responses and started using smaller packets - as my WireShark-Fu is not that good.

I read some more, and saw that google and others use a smaller MTU than my pfSense 1500 default.

To cut a long story short - I dropped my MTU to 1480 - and confirmed that this is being advertised by RADVD AdvLinkMTU 1480 - and IPV6 is working for me again !!!

I'm not sure what the change was in 23.01 that brought this to the fore.

Hope this helps someone.

Lee.

NightlyShark

@maverickws
I know, it worked for years. But, seeing as many an ISP may be using PfSense themselves and update the same day as you (and bring other changes to the table at the same time to minimize deployments) and because the vast majority of users run from CPEs and won't miss IPv6 for an hour anyway, try that anyway.

brmoon

I'm a Verizon FIOS customer for home internet. Like many others, IPv6 stopped working after the upgrade to 23.01. I tried a great many things to get IPv6 working again after the upgrade to 23.01. While I can't possibly recount all the things, I will share the last thing I did that resulted in delegated IPv6 being reassigned to my LAN network.

From the pfSense console I used the 2) Set interface(s) IP address option. Both IPv4 and IPv6 were again set to DHCP. (What I would consider to be no change.)

When the console menu came back up, my IPv6 was present on LAN as I had previous experienced before the upgrade.

Hope this helps someone else.

maverickws

@forresgeek Hi

I tried this: set my MTU to 1480, but unfortunately didn't change the situation.

mhillmann

For what it's worth. We've found a permanent solution for our issue for now (meaning no M+O flags set in RA announcements), testing it for long term validity:
We configure the interface as static with the fixed GUA address next hop gateway extracted from Wireshark capture. Then we change the IPV6 interface from static to dynamic and don't delete the static gateway (in ROUTING tab) to use it as default gateway for the interface. Rtsold() doesn't deliver any gateway for the interface BUT we've got one from the static definition, so we can route IPV6 packtes normally. As the unassigned dynamic gateway doesn't interfere with routing but dhcp6c() renews NA+PD delegations on expiration, we think we've got a stable configuration that will endure over time as long as the defined (fixed) next hop gateway with a GUA address stays valid, as it apperently does.
I acknowledge that this is a VERY nonstandard configuration we get from our ISP, but apparently their whole network is configured this way and their own (mainly Huawei) routers respect this arrangement when set to router mode as configured by default, delivering a correctly working IPV6 network on the internal side, RA-wise, meaning the RA sets the M+O flags as the RFC demands and pfSense can work as intended. The whole issue only becomes relevant if trying to use a different router, needing the ONT to be put into bridge mode.
Nonetheless it would be usefull to acknowledge this use case and see if is possible to force a fixed gateway even with DHCPv6.

maverickws

@mhillmann hi there,

based on your answer I thought of applying this to our setup and see how it would go.

The approach was slightly different, before when working I knew what was the GW address assigned to the IPv6.

So I simply added a new gateway on the routing tab, used that GW address and selected it as gateway.

So this did change something as now I can ping google ipv6 from Diagnostics > Ping

16 bytes from 2a00:1450:4003:806::200e, icmp_seq=0 hlim=59 time=15.372 ms
16 bytes from 2a00:1450:4003:806::200e, icmp_seq=1 hlim=59 time=15.177 ms
16 bytes from 2a00:1450:4003:806::200e, icmp_seq=2 hlim=59 time=15.382 ms
16 bytes from 2a00:1450:4003:806::200e, icmp_seq=3 hlim=59 time=22.012 ms
16 bytes from 2a00:1450:4003:806::200e, icmp_seq=4 hlim=59 time=15.506 ms

--- ipv6.l.google.com ping6 statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 15.328/18.726/30.600/5.971 ms

My issue now is that, for some reason, I don't get IPv6 connectivity on local machines. I get IPv6 address but no route.

On my machine netstat gives me a default v6 gateway which is fe80::abcd:ef01:fe03:bd10 being that this is my pfSense LAN interface local-link address. So I can't really figure out now why I have connectivity on the pfSense LAN interface but not on local hosts.

If I ping that GW my machine has configured:

# ping6 fe80::abcd:ef01:fe03:bd10%en0
PING6(56=40+8+8 bytes) fe80::8af:1f71:bffc:afc1%en0 --> fe80::abcd:ef01:fe03:bd10%en0
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=0 hlim=64 time=1.323 ms
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=1 hlim=64 time=1.381 ms
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=2 hlim=64 time=1.302 ms
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=3 hlim=64 time=1.269 ms
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=4 hlim=64 time=2.170 ms
16 bytes from fe80::abcd:ef01:fe03:bd10%en0, icmp_seq=5 hlim=64 time=2.981 ms
^C
--- fe80::abcd:ef01:fe03:bd10%en0 ping6 statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.269/1.738/2.981/0.638 ms

EDIT: This is a temporary (not totally working still) fix not a solution.
DHCP6 should take care of this without the need of having to find out the GW for the next hop and adding static gateways.

I am appalled how no one from Netgate is getting involved in this. This was a breaking change as MANY ISP's (and I mean many) will have similar setups and hardware.
Sweep it under the rug, don't talk about it, and it's gone, is that it?

@jimp @Derelict

SteveITS

@maverickws Is the mask correct? There was another post recently where the person was saying devices were getting a /128 mask not /64.

IPv6 is working fine here.

maverickws

@steveits I'm attributed a /56 prefix;

WAN configured as DHCP6
LAN, IoT and DOM (all local networks, IoT and DOM are VLANS) configured as tracking WAN interface for IPv6

Each interface gets a /64 prefix delegation size.

This is the same configuration (exactly the same configuration) that has been working for years with IPv6.

The IPv6 addresses are correctly attributed to all the internal interfaces. Actually, on the above test (after creating a static gateway as suggested) pinging ipv6.google.com you can see the pfSense pinging from its IPv6 LAN address.

I would refer to Mr. @mhillmann 's post here: https://forum.netgate.com/post/1088842

As this seems the most on point explanation of what might be happening.

SteveITS

@maverickws If it can be replicated (seems so), I suggest opening an issue at redmine.pfsense.org, with the details and any logs, referencing this thread.

And/or ask the ISP, if they are not following an RFC. Is everyone with this issue using the same ISP?

23.01 skipped from FreeBSD 12 to 14 so it's likely a lot of internal things changed.

Derelict

@maverickws said in No IPv6 after upgrade to 23.01:

DHCP6 should take care of this without the need of having to find out the GW for the next hop and adding static gateways.

IPv6 does not get its default next-hop from DHCP[6] like IPv4 does. It relies on router advertisements. To reiterate, there is nothing built into DHCP6 to propagate a next-hop gateway. It must be in an acceptable router advertisement.

If the router advertisements are non-standard then the problem is at the ISP.

I, personally, find it pretty appalling that an ISP would deploy IPv6 in such a manner, breaking customer routers that adhere to the standards and locking customers into ISP gear that works around the non-standard behavior.

maverickws

@derelict The ISP you're talking about is actually the #1 ISP around here, with the biggest fibre optics infrastructure, most clients, both in the B2C as B2B.

They work with all kinds of OEMs: Cisco, Huawei, Ericsson/Alcatel and others.

The way they implement their solution worked before the update. It still works for all customers who do not use this solution, which will likely be over 95% of the customers. So yeah you can find it appalling, I'd say it isn't them (the ISP) but the OEM's yet many customers around the world will face these issues, but never enough to force them.

If I plug a Cisco where the pfSense is now, will IPv6 work? [YES, it does].

Now, what do you think it will happen? An infinitely small percentage of the customers will make the OEM's and the ISP's to change, or people will just move to something that works?

I believe you know the answer. It's not even a matter of personal opinion, it's a very simple business decision.

nighthawk1967

@maverickws This ipv6 problem started a few updates ago and before that everything worked for years now its just problems .

mhillmann

@maverickws
If you only define a fixed gateway and don't use DHCPv6 for prefix delegation too, the ISP's GW probably will answer pings from pfSense, but as it has no associaton between the delegated internal prefix and the routing address of the pfSense appliance it refuses to route anything else coming from the public address of your router. This is the whole point of keeping the DHCPV6 configuration (even if it doesn't work correctly): IA-NA to IA-PD binding. The routing infrastructure of your ISP must know which is YOUR local GUA that has the delegated prefix behind itself for the packets to get there...
This gets even worse if the particular ISP does not keep the delegated prefix fixed (as in our case) changing it randomly when our router issues a DHCP REFRESH message. To resolve this latter issue we use ULA + NPt for our internal network, with the added benefit of easy IPv6 gateway failover (we use several ISP's on the same router) without mandatory internal renumbering of hosts as the prefix changes. Such a configuration mimicks NAT of IPv4 but only changing the first 64 bits of your local host to the public delegated prefix on the pfSense router.

jimp

https://redmine.pfsense.org/issues/14072

maverickws

@jimp I really appreciate the report, thank you. You guys address the issue description way better than me.

mhillmann

@jimp We have another twist to the already acknowledged regression: our ISP refuses to delegate a prefix shorter than a /64, but grants as many(?) /64 prefix delegations as we request as long as the IA-PD identifier is unique. This gives roughly the same result as requesting a shorter PD and splitting it up on the track interface, but regrettably there is no way to inform unique IAID´s from the IA-PD request to individual track interfaces in order to create an association between the granted PD and the local interfaces onto which these PD´s should be assigned. The implied logic in pfSense is that we only get one PD of a certain length and associate it to the routing interface via its name, not the IAID. It might be useful to consider this use case too, as I have read somewhere that even ATT has this use policy.
The following should be possible and not restricted/blocked by the pfSense configuration (0 and 1 are the IAID's):
interface ix2 {
send ia-na 0;
send ia-pd 0;
send ia-pd 1;
script "/var/etc/dhcp6c_opt13_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
prefix-interface ix2.666 {
sla-id 0;
sla-len 0;
};
};
id-assoc pd 1 {
prefix-interface ix2.667 {
sla-id 0;
sla-len 0;
};
};

jimp

That is completely unrelated to the problem here and wouldn't have changed in 23.01. It belongs in a separate thread.

mhillmann

@jimp sorry, my mistake. I´ve started a new thread already.