Comcast IPv6 working on Linux clients, but not Windows clients
-
Suspecting an issue with ISC DHCP server, I tried to switch to KEA through System / Advanced / Networking .
This immediately resulted in an unbootable system, without any recovery option offered. Admin UI not accessible from LAN. Only able to login locally thanks to the system being on a KVM switch.
KEA looks like it really shat the bed. Maybe it can't handle the 350 IPv4 DHCP reservations. It's a really awful state for the system to be in.
Fortunately, I was able to recover the previous working config by copying it manually with cp /cf/conf/backup/config-XXX.xml to /cp/conf/config.xml .
Still not sure what it would take to get KEA actually working, sigh.
Edit: I tried deleting all the staticmap entries. Still no go. My system just won't boot with KEA.
-
That's no kea doing anything wrong.
And it's not a IPv6 issue, but an IPv4 issue.
It might be a left over from a GUI issue that wrote something 'not allowed' to the config.
After the admin entered info (of course) that was plain wrong - and not flagged by the GUI.
tells the story.
What I see :
During the boot process - because initiated during /etc/rc.bootup stage
The /etc/hosts file is created (even windows has one, every system has this file - go have a look and you know what it is, why it is so universally known) - and this is where kea_earlydnsreg_mappings is called, as, imho, this is the place where static DHCPv4 leases are copied to the host file, as these never change, and can be seen as static host file entries.
My take : one of your DHCPv4 static leases entries - go have a look at the config file, you'll find the one that has an "illegal" format ....
I suspect that one of your :<dhcpd> <lan> ..... <staticmap> // one of these has an error .......has some info that is wrong.
and this is the reason /etc/utils/inc at line 3999 fails ...
and because the fail happens during system wide init (boot), pfSense breaks the boot process.
Quick and dirty solution : edit the config.xml with a good editor - not msword, use notepad++
Ditch all the
.....
<staticmap>
</staticmap>
.......
save, put the file in place, and done.You'll loose your DHCP4 static entries, but these can be put in place the same way you've removed them, when you've found the entry that contains the syntax error.
it's xml, the format is pretty straight forward. -
@Gertjan
As I said in my last message, I tried deleting all the <staticmap> entries from the <dhcpd> section. And KEA DHCP still would not boot.What I did to finally get it working.
- use admin GUI to disable LAN interface
- reboot to console
- change <dhcpdbackend> from isc to kea
- from console, manually reset IP subnet on LAN, to re-enable DHCP
- at that point, pfSense booted successfully with KEA
- I then downloaded the DHCP server section, and pasted back all the <staticmap> entries exactly as they were before
- KEA DHCP still worked fine, even after reboot
- I also had to manually enable KEA on LAN for DHCPv6, and re-enable RA / stateless mode
My guess is that it had to be something else in the <dhcpd> or <dhcpdv6> sections from the ISC config that KEA did not like - but it was not the <staticmap> entries.
At this point, my system is working again, with KEA. All my devices have their expected IPv4 addresses, from existing reservations.
test-ipv6 is still working fine on all the Unix hosts I checked.
But it is not working on any of the Windows hosts as I type this.The one improvement I see is that the "DHCPv6 leases" screen now shows names in the "Hostname" column, which was previously empty when using ISC.
"Flash" and "Higgs" are both Windows PCs. Flash has an Intel X550-T2 NIC (only one port in use). Higgs has an Aquantia AQN-107 NIC.
The DUID for both systems contains the 12 hex digits of the Aquantia NIC from "Higgs". I'm only showing the last 6 here.
An extremely large number of devices is missing from the DHCPv6 list. I have 7 WiiM Pro Plus, 2 Narwal robots, 2 Raspberry Pi controlling my dual HVAC systems, etc.
Anyway, it is 3am now, and I'm going to leave the network in this state for now.
-
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
But it is not working on any of the Windows hosts as I type this
On these windows devices,
ipconfig /renew6doesn't do it's job ?
IPv6 is disabled on these devices ?When I packet capture DHCPv6 requests, I can see the DHCPv6 request (from the windows device) and proposal (from pfSense).
-
@Gertjan ipconfig /renew6 gives the Windows hosts IPv6 addresses that look fine. The problem is that they are not usable. Ping fails with "General failure" with any IPv6 address I try.
>ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure.There is likely some kind of routing issue. But it only affects the Windows hosts - and only temporarily. On a Linux host, it always works :
madbrain@office-pi4:~ $ ping 2001:4860:4860::8888 PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes 64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=116 time=18.4 ms -
Two different leases have a very similar DUID - one that ends with the 12 hex digits of the MAC of one Windows machine, the currently working one. That is extremely strange. They are using different brand NICs, too.
I figured this one out. The Aquantia NIC had been moved between those two systems. Windows generated the DUID based on the MAC of that NIC, and stored it in the registry. I tried deleting it from both systems. Windows regenerated a new DUID that still contained the same MAC. The DUID generation algorithm must be relying on another store to find that MAC. I can't be bothered to try to fix it. Now that the hostnames are showing in the DHCPv6 leases with KEA, I am able to identify the hosts for each lease, without needing to check the MAC embedded in the DUID. The DUIDs are still unique, so it is not causing actual issues, just confusion.
-
@Gertjan So, clearly, the problem is the default route for ::/0 .
Windows clients should be getting this route from the RA. But somehow, they are not.I spent a lot of time with chatgpt trying to figure this out, changing a variety of settings on the pfSense and Windows sides, including packet captures for ICMPv6 - to no avail.
The only solution that was ultimately found was to add the route manually, with a PowerShell script like this :
PS C:\> type .\addroute.ps1 # SCRIPT TO FIX PF SENSE ROUTING ISSUE PERMANENTLY # Run this script as Administrator whenever IPv6 connectivity is lost. # 1. Configuration Variables # Interface Alias is "Ethernet" based on Get-NetAdapter output. $TargetInterface = "Ethernet" # This is the stable Link-Local address of your pfSense LAN interface. $PfSenseLinkLocal = "fe80::a236:9fff:fefa:8a31" $Destination = "::/0" # 2. Remove Existing Route (Ensures we don't duplicate or conflict) Write-Host "Attempting to remove old default route to $PfSenseLinkLocal..." Remove-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -Confirm:$false -ErrorAction SilentlyContinue # 3. Add New Persistent Route Write-Host "Adding new persistent static route via PowerShell..." # New-NetRoute is persistent by default when used in this manner. New-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -RouteMetric 10 Write-Host "SUCCESS: IPv6 default route is now active and persistent."For good measure, here is the deletion script.
PS C:\> type .\delroute.ps1 # SCRIPT TO DELETE THE PERSISTENT STATIC IPV6 DEFAULT ROUTE # 1. Configuration Variables $TargetInterface = "Ethernet" $PfSenseLinkLocal = "fe80::a236:9fff:fefa:8a31" $Destination = "::/0" # 2. Remove the Existing Route Write-Host "Attempting to delete the persistent static default route..." Remove-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -Confirm:$false -ErrorAction SilentlyContinue # 3. Verification Write-Host "Route deletion process complete. Checking routing table..." Get-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -ErrorAction SilentlyContinue if ($LASTEXITCODE -ne 0) { Write-Host "SUCCESS: Route ::/0 pointing to pfSense has been successfully removed." } else { Write-Host "WARNING: Route may still be present. Please check manually."Test :
PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), PS C:\> .\addroute.ps1 Attempting to remove old default route to fe80::a236:9fff:fefa:8a31... Adding new persistent static route via PowerShell... ifIndex DestinationPrefix NextHop RouteMetric ifMetric PolicyStore ------- ----------------- ------- ----------- -------- ----------- 8 ::/0 fe80::a236:9fff:fefa:8a31 10 15 ActiveStore 8 ::/0 fe80::a236:9fff:fefa:8a31 10 Persiste... SUCCESS: IPv6 default route is now active and persistent. PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: Request timed out. Reply from 2001:4860:4860::8888: time=14ms Reply from 2001:4860:4860::8888: time=10ms Reply from 2001:4860:4860::8888: time=8ms Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 3, Lost = 1 (25% loss), Approximate round trip times in milli-seconds: Minimum = 8ms, Maximum = 14ms, Average = 10ms PS C:\> .\delroute.ps1 Attempting to delete the persistent static default route... Route deletion process complete. Checking routing table... WARNING: Route may still be present. Please check manually. PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), PS C:\>This works, but is very brittle, since the scripts hardcode the name of the LAN interface as "Ethernet". It will be different on every Windows system or VM. If I install Hyper-V on my host, the interface name changes also. Maybe the script can be made smarter. But the point is that it really shouldn't be necessary to set the route manually. Something is just not right with the way pfSense interacts with Windows IPv6 clients.
I'm inclined to think that this is a pfSense configuration issue, since these Windows clients all work fine with IPv6 when I use the Comcast XB8 as router instead of pfSense.
I'm at a loss as to what to do to fix the routing in the pfSense config, though.
-
Actually, the "addroute" script only works on one single Windows system. It does not help the other one at all.
I spent a ton of time today trying to figure out the root cause.
I went back to my old SSD all the way to a 23.09 boot environment, and an old config with a /24 only, back when I had far fewer devices. I did not plug-in the Wifi APs so as not to exceed 255 devices. Everything worked fine with the /24 . All Windows clients had IPv6 access at all times.
I believe I may be running into a bug in pfSense here with my network size.
-
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
Everything worked fine with the /24
That's 'IPv4'.
Your routing issue is with IPv6.I don't understand why you would need scripts to put a IPv6 gateway in place.
DHCPv6 - RA actually - should take care of this. It works and is easy to test : buy a new PC (go always for Windows Pro ^^) and hook it up : it works.@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
he Aquantia NIC had been moved
The DUID of an interface can be based on the MAC - but more info is added. For exemple : a random time stamp, the position of the moon, etc.
So when you take a NIC from one device to another, the DUID of a device using that NIC will be different as the previous device.When I add a new IPv6 cable device, I let it get a DHCPv6 lease out of the DHCPv6 pool.
I copied the DUID I saw in the IPv6 lease list, and assign a static DHCP lease with this DUID.
Done.
It's the same procedure as IPv4 static leases, with the difference DUID is used, not a MAC.Btw : as changing /24 to something else can have nice hidden consequences (I stay away from it, like VLANs) the same could be said for what is /64 for IPv6. Double check ?
-
@Gertjan I set up a brand new Windows 11 Enterprise VM earlier today just to test, with its own MAC, bridged. It did not work. IPv6 routing still broken.
Buying a new PC will not help. -
@Gertjan and yes I know very well that the /24 is for IPv4. I'm just reporting what I observed - when I reverted to the old configuration with /24 on IPv4, and fewer than 100 clients connected, the IPv6 routing on the Windows hosts got fixed. It doesn't make any sense why that would be, but there you have it.
Whereas when I use the /22 on IPv4, and all 350 clients are connected, the IPv6 routing breaks on the Windows hosts.
As I said, makes no sense which is why I'm thinking there is a bug somewhere, either in pfSense or Windows.
-
@Gertjan And I do not use VLANs either. All my switches are unmanaged.
While experimenting today with various configuration, I ran into massive problems with IPv4 too. Sometimes the hosts wouldn't get IPv4 at all, only IPv6. This was across all hosts, though, not just Windows hosts
At present, another oddity is that when I ping the same public IPv4 host, say 1.1.1.1 or 8.8.8.8, from 2 Windows machines on my LAN at once, only the first machine gets the ICMP packets back. Second box only gets ttimeouts. If I ctrl-c on the first box, the second box starts getting ICMP replies back. This may be a separate problem from the IPv6 issue, but is sure is extremely puzzling.
No issue if 2 LAN hosts simultaneously ping the same public host.
-
@Gertjan As to why I went to a /22 earlier this year, it is for smart lighting. I replaced about 220 LED bulbs with Wi-Fi bulbs. They are powered at all times in order to be controllable remotely by Wi-Fi. The 97 switches on the wall are Z-wave. They are used as scene controllers only- the relay is always on.
I could temporarily turn off the relay on enough z-wave switches to bring the total number of IP devices down. For example, my home theater has almost 40 light bulbs on just 3 switches, so that would be a quick reduction. With a few more switches/ rooms turned off, I could bring down the device count below 255.
I can of course also turn off the WiFi SSID, which will remove about 300 devices at once, but then I won't have any way to turn lights on or off. -
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
I set up a brand new Windows 11 Enterprise VM earlier today just to test, with its own MAC, bridged. It did not work. IPv6 routing still broken.
Buying a new PC will not help.Ok, then the IPv6 problem is on the pfSense side.
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
As to why I went to a /22 earlier this year
Yeah, but again, imho, IPv4 works.
Your issue is IPv6 related. -
Yes, the problem with IPv6 is definitely on the pfSense side.
My entire network - all 350 devices - works perfectly fine if I remove pfSense and set the XB8 to router mode, with a 10.0.0.1/16 . No IPv4 or IPv6 issues anywhere, including Windows systems.
But the XB8 is very limited as router - no local DNS, cannot handle 350 DHCP reservations, no Wireguard inbound VPN. These are the reasons I'm using pfSense. IPv6 not working is a problem, though.
-
@Gertjan said :
Yeah, but again, imho, IPv4 works.
Not fully. The IPv4 ping issue I ran into is still there, even with a very basic network with pfSense using factory defaults. I suspect a NAT problem. Again, this problem does not exist when using the Comcast XB8 as router, bypassing pfSense.
I filed a report in the NAT forum :
https://forum.netgate.com/topic/199273/2-windows-hosts-cannot-receive-echo-ping-from-the-same-ipv4-internet-host-simultaneously
That issue might not be related to the IPv6 problem I have been running into in this thread, though.
-
@Gertjan said in Comcast IPv6 working on Linux clients, but not Windows clients:
.
So, I have been pulling my hair today trying to figure out what triggers the issue, making so many config changes to pfSense, rebooting, and putting some devices offline temporarily.
I have not come to a conclusion as to the root cause, still, but I did isolate something that triggers the problem.
-
I went back to my original pfSense configuration, before I started this thread, with IPv6 not working on Windows clients. That config includes a 192.168.100.1/22 on the IPv4 side, as well as 350 DHCP reservations.
-
I disabled all SSIDs on my Ubiquiti Wifi APs (9 x U6-LR and 1 x NanoHD). Doing so had the effect of disconnecting about 300 Wifi clients from the LAN/WLAN.
-
I put 2 Windows systems to sleep, and woke them back up, to reset the network settings.
-
Immediately after waking them up, full IPv6 connectivity was restored on both Windows systems, including proper routing.
-
If I re-enable the Wifi SSIDs, and suspend & wake both Windows systems again, the IPv6 gets broken in both of them upon wakeup.
I'm still not really sure what could be causing this issue. One theory is that there is some kind of IPv6 maximum client limit is being hit. I am guessing this is the case, because I only see a maximum of 16 leases on the "DHCPv6 leases" screen, even with all 350 clients connected. While it is probably that not all 350 clients fully support IPv6, I would still expect there to be many more than 16 DHCPv6 leases.
The other theory would be that some client(s) are interfering with the RA packets, somehow, confusing the Windows boxes, and interferring their IPv6 routes. Not sure how likely that is. I have not taken the step of running Wireshark in promiscuous mode to find out.
-
-
@madbrain
Did you try just Unmanaged RA?
with disabled dhcpv6 server? All my devices including android and windows are working just fine. -
@w0w Thanks. I had no tried it yet. I just did. pfSense asked me to disable DHCPv6, which I did. I cleared all the DHCPv6 leases, also. Put my Windows clients to sleep, and woke them up. Unfortunately, same behavior - they get an IPv6 address, but routing is broken. Android & Linux clients are still fine.
Edit: turned off the Wifi IOT SSID again, and Windows clients are back to working in IPv6 mode, even with RA in Unmanaged mode.
I have been writing some scripts with the Unifi API to block specific clients based on MAC prefix. I'm down to 48 unblocked clients on the IOT SSID now. They are still causing IPv6 to break on the wired Windows clients ... Got a few brands of devices left to block. Hopefully it is a specific device causing the issue, and not a device limit I'm hitting.
-
So, it wasn't until I got down to 0 unblocked IOT clients that the problem resolved. Meaning, the problem wasn't caused by a specific client.
I went to check the IOT SSID setting in the Unifi controller. It had something called "Proxy ARP" enabled. I disabled it. Miraculously, all problems with IPv6 on the wired Windows hosts went away. This is really crazy.
