Comcast IPv6 working on Linux clients, but not Windows clients
-
@Gertjan I already have DHCPv4 reservations for all 350 IP devices I own, along with custom hostnames. The pace of adding and removing devices in my home is much more rapid - usually a couple each month.
For DHCPv6, I see only a grand total of 16 leases. It just can't be that only 16 out of the 350 support IPv6. It is difficult to identify which DUID corresponds to which piece of hardware. It looks like the MAC is embedded into the DUID. The DUID is variable-length, though. I found the entry for my currently working desktop PC. It has the MAC of the Aquantia NIC as the last 12 hex digits of the DUID. The IPv6 address shown corresponds to the 'IPv6 address" from ipconfig on that system.
For the other non-working Windows system, I can't find any corresponding entry under leases.
I'm going to take a look at the remainder of the DHCPv6 leases, since there are so few of them, and try to figure out what they are.
-
@Gertjan said in Comcast IPv6 working on Linux clients, but not Windows clients:
edit :
I've switched to kea last year, and never came back. Some 'minor' issues existed back then, but nothing mission critical that broke.
These days, with "25.07.1", is good enough for me.be ware that I can't compare kea with ISC.
You might also be dealing with old ISC IPV6 bugs (that won't get fixed anymore).
I deal with the new bugs - that are discussed here on the forum - and will get fixed ;)
That said, IPv6, when I was using ISC, was working well enough for me. With kea it also works well (now).Thanks. I just checked my version, and it says it's on 24.11 . It may be because of a backup I restored. I don't think restoring a backup would roll back the firmware, though. I could have sworn I had already upgraded to 25.07.1 .
I went to System / Update / System update just now, and tried to select the current stable version, 25.07.1 . But it says "Another instance of pfSense-upgrade is running. Please try again in a few moments".
That's probably because "Boot environment" is shown as "broken-20250127220801". Looks like there is another one called default_20251006131313 . I don't want to switch back to it, since it seems like it's proceeding to upgrade the current boot env. sigh.
Edit: that default_20251006131313 environment shows "Boot verification failed". I cannot switch to it. It goes back to 24.11 as a result. It complains during boot that the current configuration was created with a newer version, also.
It looks like my pfSense SSD may be seriously hosed. Good thing I have a hotswap SATA drive bay and a few spare SSDs. Time to figure out how to flash another one, sigh.
-
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
But it says "Another instance of pfSense-upgrade is running. Please try again in a few moments".
and if for some reason you (pfSense) have issues reaching the Netgate upgrade servers, it will stay in that mode forever. Thus locking you out for any updates (and pfSense package updates).
Short cut : as always : go to the place where you have power : console or SSH.
Option 8 and kill all 'pkg' processes.
Back up one level, and use option 13: update from console.You see issues know : you have a 99,99 % change that issue is already documented in the pfSense upgrade documentation or here ion the forum.
Better cut : export you config.xml.
Get the installer and burn it to a USB drive with etcher (see pfSense doc).
Install your device with the USB drive.
Import, if needed, the config.
Done.
Now you have a clean system, with all the latest stuff, latest ZFS layout and partitions, everything.@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
Time to figure out how to flash another one, sigh
That might be valid for the big desktop OSes or servers with loads of data, programs etc.
Not valid for pfSense : there is one (1 !) config file to keep.
Re installing is waaaay faster. -
@Gertjan said
Time to figure out how to flash another one, sigh
That might be valid for the big desktop OSes or servers with loads of data, programs etc.
Not valid for pfSense : there is one (1 !) config file to keep.
Re installing is waaaay faster.Thanks. I have nightly backups of my config.xml . I agree reinstalling is much faster.
I meant that I needed to figure out how to flash the USB stick for the reinstall. Which I'm doing now. But Etcher just failed after the verification, sigh.
-
-
@Gertjan Thanks. I was able to recover the system. Etcher somehow reported an error, but the media was good. I tried 2 different media.
pfSense is up and running again, this time with 25.07.1 .The problem of IPv6 inconsistency is still there. I took a look at the DHCPv6 leases again, and found something very interesting :
Two different leases have a very similar DUID - one that ends with the 12 hex digits of the MAC of one Windows machine, the currently working one. That is extremely strange. They are using different brand NICs, too.
-
Suspecting an issue with ISC DHCP server, I tried to switch to KEA through System / Advanced / Networking .
This immediately resulted in an unbootable system, without any recovery option offered. Admin UI not accessible from LAN. Only able to login locally thanks to the system being on a KVM switch.
KEA looks like it really shat the bed. Maybe it can't handle the 350 IPv4 DHCP reservations. It's a really awful state for the system to be in.
Fortunately, I was able to recover the previous working config by copying it manually with cp /cf/conf/backup/config-XXX.xml to /cp/conf/config.xml .
Still not sure what it would take to get KEA actually working, sigh.
Edit: I tried deleting all the staticmap entries. Still no go. My system just won't boot with KEA.
-
That's no kea doing anything wrong.
And it's not a IPv6 issue, but an IPv4 issue.
It might be a left over from a GUI issue that wrote something 'not allowed' to the config.
After the admin entered info (of course) that was plain wrong - and not flagged by the GUI.
tells the story.
What I see :
During the boot process - because initiated during /etc/rc.bootup stage
The /etc/hosts file is created (even windows has one, every system has this file - go have a look and you know what it is, why it is so universally known) - and this is where kea_earlydnsreg_mappings is called, as, imho, this is the place where static DHCPv4 leases are copied to the host file, as these never change, and can be seen as static host file entries.
My take : one of your DHCPv4 static leases entries - go have a look at the config file, you'll find the one that has an "illegal" format ....
I suspect that one of your :<dhcpd> <lan> ..... <staticmap> // one of these has an error .......has some info that is wrong.
and this is the reason /etc/utils/inc at line 3999 fails ...
and because the fail happens during system wide init (boot), pfSense breaks the boot process.
Quick and dirty solution : edit the config.xml with a good editor - not msword, use notepad++
Ditch all the
.....
<staticmap>
</staticmap>
.......
save, put the file in place, and done.You'll loose your DHCP4 static entries, but these can be put in place the same way you've removed them, when you've found the entry that contains the syntax error.
it's xml, the format is pretty straight forward. -
@Gertjan
As I said in my last message, I tried deleting all the <staticmap> entries from the <dhcpd> section. And KEA DHCP still would not boot.What I did to finally get it working.
- use admin GUI to disable LAN interface
- reboot to console
- change <dhcpdbackend> from isc to kea
- from console, manually reset IP subnet on LAN, to re-enable DHCP
- at that point, pfSense booted successfully with KEA
- I then downloaded the DHCP server section, and pasted back all the <staticmap> entries exactly as they were before
- KEA DHCP still worked fine, even after reboot
- I also had to manually enable KEA on LAN for DHCPv6, and re-enable RA / stateless mode
My guess is that it had to be something else in the <dhcpd> or <dhcpdv6> sections from the ISC config that KEA did not like - but it was not the <staticmap> entries.
At this point, my system is working again, with KEA. All my devices have their expected IPv4 addresses, from existing reservations.
test-ipv6 is still working fine on all the Unix hosts I checked.
But it is not working on any of the Windows hosts as I type this.The one improvement I see is that the "DHCPv6 leases" screen now shows names in the "Hostname" column, which was previously empty when using ISC.
"Flash" and "Higgs" are both Windows PCs. Flash has an Intel X550-T2 NIC (only one port in use). Higgs has an Aquantia AQN-107 NIC.
The DUID for both systems contains the 12 hex digits of the Aquantia NIC from "Higgs". I'm only showing the last 6 here.
An extremely large number of devices is missing from the DHCPv6 list. I have 7 WiiM Pro Plus, 2 Narwal robots, 2 Raspberry Pi controlling my dual HVAC systems, etc.
Anyway, it is 3am now, and I'm going to leave the network in this state for now.
-
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
But it is not working on any of the Windows hosts as I type this
On these windows devices,
ipconfig /renew6doesn't do it's job ?
IPv6 is disabled on these devices ?When I packet capture DHCPv6 requests, I can see the DHCPv6 request (from the windows device) and proposal (from pfSense).
-
@Gertjan ipconfig /renew6 gives the Windows hosts IPv6 addresses that look fine. The problem is that they are not usable. Ping fails with "General failure" with any IPv6 address I try.
>ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure.There is likely some kind of routing issue. But it only affects the Windows hosts - and only temporarily. On a Linux host, it always works :
madbrain@office-pi4:~ $ ping 2001:4860:4860::8888 PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes 64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=116 time=18.4 ms -
Two different leases have a very similar DUID - one that ends with the 12 hex digits of the MAC of one Windows machine, the currently working one. That is extremely strange. They are using different brand NICs, too.
I figured this one out. The Aquantia NIC had been moved between those two systems. Windows generated the DUID based on the MAC of that NIC, and stored it in the registry. I tried deleting it from both systems. Windows regenerated a new DUID that still contained the same MAC. The DUID generation algorithm must be relying on another store to find that MAC. I can't be bothered to try to fix it. Now that the hostnames are showing in the DHCPv6 leases with KEA, I am able to identify the hosts for each lease, without needing to check the MAC embedded in the DUID. The DUIDs are still unique, so it is not causing actual issues, just confusion.
-
@Gertjan So, clearly, the problem is the default route for ::/0 .
Windows clients should be getting this route from the RA. But somehow, they are not.I spent a lot of time with chatgpt trying to figure this out, changing a variety of settings on the pfSense and Windows sides, including packet captures for ICMPv6 - to no avail.
The only solution that was ultimately found was to add the route manually, with a PowerShell script like this :
PS C:\> type .\addroute.ps1 # SCRIPT TO FIX PF SENSE ROUTING ISSUE PERMANENTLY # Run this script as Administrator whenever IPv6 connectivity is lost. # 1. Configuration Variables # Interface Alias is "Ethernet" based on Get-NetAdapter output. $TargetInterface = "Ethernet" # This is the stable Link-Local address of your pfSense LAN interface. $PfSenseLinkLocal = "fe80::a236:9fff:fefa:8a31" $Destination = "::/0" # 2. Remove Existing Route (Ensures we don't duplicate or conflict) Write-Host "Attempting to remove old default route to $PfSenseLinkLocal..." Remove-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -Confirm:$false -ErrorAction SilentlyContinue # 3. Add New Persistent Route Write-Host "Adding new persistent static route via PowerShell..." # New-NetRoute is persistent by default when used in this manner. New-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -RouteMetric 10 Write-Host "SUCCESS: IPv6 default route is now active and persistent."For good measure, here is the deletion script.
PS C:\> type .\delroute.ps1 # SCRIPT TO DELETE THE PERSISTENT STATIC IPV6 DEFAULT ROUTE # 1. Configuration Variables $TargetInterface = "Ethernet" $PfSenseLinkLocal = "fe80::a236:9fff:fefa:8a31" $Destination = "::/0" # 2. Remove the Existing Route Write-Host "Attempting to delete the persistent static default route..." Remove-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -NextHop $PfSenseLinkLocal -Confirm:$false -ErrorAction SilentlyContinue # 3. Verification Write-Host "Route deletion process complete. Checking routing table..." Get-NetRoute -InterfaceAlias $TargetInterface -DestinationPrefix $Destination -ErrorAction SilentlyContinue if ($LASTEXITCODE -ne 0) { Write-Host "SUCCESS: Route ::/0 pointing to pfSense has been successfully removed." } else { Write-Host "WARNING: Route may still be present. Please check manually."Test :
PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), PS C:\> .\addroute.ps1 Attempting to remove old default route to fe80::a236:9fff:fefa:8a31... Adding new persistent static route via PowerShell... ifIndex DestinationPrefix NextHop RouteMetric ifMetric PolicyStore ------- ----------------- ------- ----------- -------- ----------- 8 ::/0 fe80::a236:9fff:fefa:8a31 10 15 ActiveStore 8 ::/0 fe80::a236:9fff:fefa:8a31 10 Persiste... SUCCESS: IPv6 default route is now active and persistent. PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: Request timed out. Reply from 2001:4860:4860::8888: time=14ms Reply from 2001:4860:4860::8888: time=10ms Reply from 2001:4860:4860::8888: time=8ms Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 3, Lost = 1 (25% loss), Approximate round trip times in milli-seconds: Minimum = 8ms, Maximum = 14ms, Average = 10ms PS C:\> .\delroute.ps1 Attempting to delete the persistent static default route... Route deletion process complete. Checking routing table... WARNING: Route may still be present. Please check manually. PS C:\> ping 2001:4860:4860::8888 Pinging 2001:4860:4860::8888 with 32 bytes of data: PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. PING: transmit failed. General failure. Ping statistics for 2001:4860:4860::8888: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), PS C:\>This works, but is very brittle, since the scripts hardcode the name of the LAN interface as "Ethernet". It will be different on every Windows system or VM. If I install Hyper-V on my host, the interface name changes also. Maybe the script can be made smarter. But the point is that it really shouldn't be necessary to set the route manually. Something is just not right with the way pfSense interacts with Windows IPv6 clients.
I'm inclined to think that this is a pfSense configuration issue, since these Windows clients all work fine with IPv6 when I use the Comcast XB8 as router instead of pfSense.
I'm at a loss as to what to do to fix the routing in the pfSense config, though.
-
Actually, the "addroute" script only works on one single Windows system. It does not help the other one at all.
I spent a ton of time today trying to figure out the root cause.
I went back to my old SSD all the way to a 23.09 boot environment, and an old config with a /24 only, back when I had far fewer devices. I did not plug-in the Wifi APs so as not to exceed 255 devices. Everything worked fine with the /24 . All Windows clients had IPv6 access at all times.
I believe I may be running into a bug in pfSense here with my network size.
-
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
Everything worked fine with the /24
That's 'IPv4'.
Your routing issue is with IPv6.I don't understand why you would need scripts to put a IPv6 gateway in place.
DHCPv6 - RA actually - should take care of this. It works and is easy to test : buy a new PC (go always for Windows Pro ^^) and hook it up : it works.@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
he Aquantia NIC had been moved
The DUID of an interface can be based on the MAC - but more info is added. For exemple : a random time stamp, the position of the moon, etc.
So when you take a NIC from one device to another, the DUID of a device using that NIC will be different as the previous device.When I add a new IPv6 cable device, I let it get a DHCPv6 lease out of the DHCPv6 pool.
I copied the DUID I saw in the IPv6 lease list, and assign a static DHCP lease with this DUID.
Done.
It's the same procedure as IPv4 static leases, with the difference DUID is used, not a MAC.Btw : as changing /24 to something else can have nice hidden consequences (I stay away from it, like VLANs) the same could be said for what is /64 for IPv6. Double check ?
-
@Gertjan I set up a brand new Windows 11 Enterprise VM earlier today just to test, with its own MAC, bridged. It did not work. IPv6 routing still broken.
Buying a new PC will not help. -
@Gertjan and yes I know very well that the /24 is for IPv4. I'm just reporting what I observed - when I reverted to the old configuration with /24 on IPv4, and fewer than 100 clients connected, the IPv6 routing on the Windows hosts got fixed. It doesn't make any sense why that would be, but there you have it.
Whereas when I use the /22 on IPv4, and all 350 clients are connected, the IPv6 routing breaks on the Windows hosts.
As I said, makes no sense which is why I'm thinking there is a bug somewhere, either in pfSense or Windows.
-
@Gertjan And I do not use VLANs either. All my switches are unmanaged.
While experimenting today with various configuration, I ran into massive problems with IPv4 too. Sometimes the hosts wouldn't get IPv4 at all, only IPv6. This was across all hosts, though, not just Windows hosts
At present, another oddity is that when I ping the same public IPv4 host, say 1.1.1.1 or 8.8.8.8, from 2 Windows machines on my LAN at once, only the first machine gets the ICMP packets back. Second box only gets ttimeouts. If I ctrl-c on the first box, the second box starts getting ICMP replies back. This may be a separate problem from the IPv6 issue, but is sure is extremely puzzling.
No issue if 2 LAN hosts simultaneously ping the same public host.
-
@Gertjan As to why I went to a /22 earlier this year, it is for smart lighting. I replaced about 220 LED bulbs with Wi-Fi bulbs. They are powered at all times in order to be controllable remotely by Wi-Fi. The 97 switches on the wall are Z-wave. They are used as scene controllers only- the relay is always on.
I could temporarily turn off the relay on enough z-wave switches to bring the total number of IP devices down. For example, my home theater has almost 40 light bulbs on just 3 switches, so that would be a quick reduction. With a few more switches/ rooms turned off, I could bring down the device count below 255.
I can of course also turn off the WiFi SSID, which will remove about 300 devices at once, but then I won't have any way to turn lights on or off. -
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
I set up a brand new Windows 11 Enterprise VM earlier today just to test, with its own MAC, bridged. It did not work. IPv6 routing still broken.
Buying a new PC will not help.Ok, then the IPv6 problem is on the pfSense side.
@madbrain said in Comcast IPv6 working on Linux clients, but not Windows clients:
As to why I went to a /22 earlier this year
Yeah, but again, imho, IPv4 works.
Your issue is IPv6 related.
