pfSense fails to restore IPV6 after WAN side service interruption
-
@jwj SB6121, Arris chip. pfSense 2.4.4-RELEASE-p3
-
I don't think it's a Cablemodem issue, at least not one that couldn't be mitigated in software. Re-initializing the WAN interface fixes the issue.
-
Yeah, I agree. Just seeing what is common and what is different. I'm also on 2.4.4-RELEASE-p3. pfblockerng-devel and nut. Those should be of no relevance. I'm not doing anything special with my ipv6 prefix, defaults across the board.
My WAN is using the igb driver. My hardware is a supermicro 5018D-FN4T, which is massive overkill for my needs, but I had it leftover from another use. That makes the interface a Intel I350-AM2 according to the spec-sheet.
I haven't really put much into isolating the issue because my service is also very reliable. I see this once every two or three months.
We'll see if anyone else has something to add.
-
First noted the issue in version 2.1.
-
I suspect it may have something to do with the way DOCSIS handles service outages. A cablemodem will not drop the link to the router during service outages (RF side failures), instead it keeps the link up and fires up an internal DHCP server that hands out 192.168.100.x addresses with 30 second leases. You'll see a 192.168.100.10 IPV4 WAN address during such outages, Upon service restoration the modems internal DHCP server is shut down and further DHCP request are passed on to the ISP resulting in the WAN being reassigned a proper network address within 30 seconds.
None of this applies to IPV6 and radvd seems to permanently give up with the initial service loss. I need to run a few tests to see if physically disconnecting and reconnecting the modem to router link behaves the same as a cable side service outage.
-
Disconnecting, then reconnecting restores IPV6 connectivity correctly. Cable (RF) side outages do not! Could it link down/up events that don't occur with cable outages be the cause?
-
@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:
A cablemodem will not drop the link to the router during service outages (RF side failures), instead it keeps the link up and fires up an internal DHCP server that hands out 192.168.100.x addresses with 30 second leases.
If that happens, then the router will not know about the internal DHCP server and continue with the regular address. Unless something happens, such as dropping the link, the router will continue with the assigned address, until the lease expires. Perhaps someone should do some packet capture, to see what happens when the RF side is disconnected and reconnected.
-
@JKnott Correct, that is the reason I configure DHCP to hand out short leases. pfSense gets no indication that the route is down when an RF side failure occurs. Every single Cable modem in the world behaves this way... its mandated by the DOCSIS spec. What were they thinking!!!
This issue has existed since the introduction of IPV6 support in pfSense (version 2.1 or thereabouts). Which leads me to conclude that very few are trying to support native IPV6 over Cablemodem...
-
Same here whenever I reboot the modem. I just re-save the WAN settings to reconnect. If I remember... :) I've also setup the WAN interface to ignore leases offered by the cable modem.
-
Odd thing is, IPV6 quickly becomes aware something is wrong and drops V6 addresses of all interfaces. You'd think DHCPV6 would keep retrying in that situation, and kick off the whole V6 delegation thing with radvd upon establishing a new V6 WAN address?
-
@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:
This issue has existed since the introduction of IPV6 support in pfSense (version 2.1 or thereabouts). Which leads me to conclude that very few are trying to support native IPV6 over Cablemodem...
What is it that's causing IPv6 to fail? As for LAN lease, that shouldn't fail for a very long time and there should be RAs on the WAN side that indicate the connection is up. Has anyone done a packet capture to see what happens when it fails?
-
@jcyr your post train is the most useful insight I have read on the Internet on a long, long time. I have observed the same behaviours (Rogers, technologically identical to Comcast) on my end, but haven't documented or researched it as far as you did.
Thanks.
-
Ok, know I'm confused! Did mess up the capture by forgetting to update the capture count from the default 100, and won't be able to try this again without incurring substantial grief. Thing is it worked perfectly!!!
- Pulled the coax from the modem.
- Both V4 and V6 gateways go offline.
- WAN link receives 192.168.100.10 IP address from modem's internal DHCP
- Wait about 2 minutes for the modem to realize the signal isn't coming back, and starts scanning.
- Modem re-reisters with ISP.
- pfSense gets valid WAN network V4 address upon expiry of the 30 second 192.168.100.10 lease.
- new V6 address assigned to WAN.
- delegated V6 addresses assigned to LANs.
Exactly how you'd expect it to happen!
Why would the service interruptions I've seen lately due to road work be any different? Fiber cuts to the CMTS providing the the RF modulation, without RF signal loss on the cable? Not sure I could easily reproduce that.
-
Adding, the the V6 addresses are dropped from all interfaces, around the time the 192.168.100.10 is installed on the WAN interface.
-
@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:
Adding, the the V6 addresses are dropped from all interfaces, around the time the 192.168.100.10 is installed on the WAN interface.
Then something is telling pfSense the modem is down. Your post says the WAN port gets a 192 address from the internal DHCP server. How does that happen, if pfSense didn't request an address? Normally, a device holds onto an address until the lease expires. The RF side failing can't tell pfSense it's down, unless something, perhaps the Ethernet link dropping, tells pfSense. For example, my DHCP lease, from my ISP is 72 hours. So, my firewall "owns" that address until the end of the lease and nothing, short of dropping the Ethernet link will change that. Also, the lease is normally renewed well before it expires, so the RF side would have to be down at least 20 - 30 hours, before pfSense loses the lease.
-
@JKnott I'm pretty sure it's the modem... nevermind that they are Puma chipsets, as soon as they see an error on the uplink (as recorded in the DOCSIS status page of the modem) they fall back into 192.168.100.0/24 with DHCP mode. That shouldn't be happening in "bridged mode" as opposed to "gateway mode", but... well, that's what we have to work with.
An "error" on the uplink could be anything from coax being disconnected, to CMTS issues (no ranging response, etc.) to a signal level issue.
Short of a "work around", I don't think pfSense can really do anything about this...
-
@JKnott I hand out short leases, about 30 minutes, but you're right. I'm not sure what triggers DHCP to request a new lease so quickly.
-
This brings up the question, is the OPs modem in bridge or gateway mode? If I have time later, I'll have to run Wireshark between my modem and pfSense to see what happens. However, I have a Hitron modem. It's in bridge mode.
If it's a problem with certain make modems, then it's not likely a pfSense problem.
-
@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:
@JKnott I hand out short leases, about 30 minutes, but you're right. I'm not sure what triggers DHCP to request a new lease so quickly.
There's something else to test. Place a switch between the modem and pfSense and then kill the RF connection to the modem. If pfSense does not respond, then it's the Ethernet being dropped that causes the address change. Since pfSense can now no longer see that drop, then it won't run the DHCP process.
-
@JKnott said in pfSense fails to restore IPV6 after WAN side service interruption:
This brings up the question, is the OPs modem in bridge o
Bridge, of course.
-
My setup:
ISP - Comcast
Modem - Netgear CM600
Bridge mode
Non-Puma chipset (Broadcom)
Does not do LAN DHCP when ISP drops AFAICT
pfSense is direct connect to the modem (but as a VM)
IPv4 comes back fine, have to bounce the IPv6.
(Just stirrin' the pot! :) -
@JKnott Ok, tried that, interesting result.
With the switch inline, pfSense never notices that the modem's gone away, other than apinger and NTP complaining of loss of connectivity. Reconnecting the coax lets the modem re-register, and connectivity is restored on the very same V4 and V6 addresses.
Without the inline switch I managed to reproduce the original problem. Pulling the coax immediately cause the IP address to switch to 192.168.100.10, so I guess there is a short Ethernet link interruption. However, V6 connectivity was not restored. Restarting the radvd service re-established V6 connectivity.
-
@jcyr interesting, I will be putting a dummy switch between the modem and router, that may indeed be the simplest work around.
It's rather odd that there is an interruption at the ethernet layer when the coax is pulled - you'd think that the switch ports at the back of the modem are exactly that, just a switch.
I don't think it has anything to do with the Puma chipsets (which are notoriously terrible at maintaining decent first hop latency), but it may have something to do with firmware. Given that @provels reproduced the issue with a Broadcom chipset, I'm going to assume there is a gremlin in Comcast's firmware that causes a momentary ethernet interruption. Again, Rogers exactly mirrors Comcast's technology - identical hardware, identical firmware.
At this point I believe we ruled out that the issue lies with pfSense, it is widely reproducible, and I doubt any of the cable providers will consider it an issue as any non-standard use of their garbage "residential gateways" are explicitly unsupported.
tl;dr
Mainstream DOCSIS residential gateways are never truly bridged even operating in bridge mode. This worked fine prior to dual-stack going mainstream, although I do have to praise Comcast (and Rogers) for their otherwise superb IPv6 support. (/56 per gateway is pretty cool) -
@ljr said in pfSense fails to restore IPV6 after WAN side service interruption:
I will be putting a dummy switch between the modem and router, that may indeed be the simplest work around.
That will only work if Comcast assigns the same V4 and V6 addresses to you upon re-registration. I know this to be mostly true for Comcast, but not guaranteed.
-
@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:
That will only work if Comcast assigns the same V4 and V6 addresses to you upon re-registration. I know this to be mostly true for Comcast, but not guaranteed.
I don't recall my v4 address ever changing, unless I change the MAC address on my WAN interface... same for v6, as long as you'd have the same DUID you should receive the same prefix...
I suppose one could write a script to restart the interface periodically until the 192.168.100.0/24 address clears.
-
pfSense has a gateway monitoring function. Perhaps that might help. However, I haven't tried it, as I don't have a need to. What is the intended function of gateway monitoring?
-
@ljr said in pfSense fails to restore IPV6 after WAN side service interruption:
I suppose one could write a script to restart the interface periodically until the 192.168.100.0/24 address clears.
Or start with RTFM: https://docs.netgate.com/pfsense/en/latest/book/interfaces/ipv4-wan-types.html#dhcp and try to reject the lease.
-
@Grimson I'll certainly try that. But that's IPV4 stuff and never had a problem with V4.
-
@Grimson said in pfSense fails to restore IPV6 after WAN side service interruption:
Or start with RTFM: https://docs.netgate.com/pfsense/en/latest/book/interfaces/ipv4-wan-types.html#dhcp and try to reject the lease.
Well, in retrospect it was amateur to suggest hacking a script. But, shit, that does the trick.
Sorry.
-
@Grimson I have been rejecting leases from my cable modem, 192.168.100.1, for some time now. Doesn't appear to have an impact on this issue(s).
Rebooted pfsense late last night and the WAN did not get an ipv6 address. Waited a good long time. A second reboot set it right. I then disconnected the cable, waited a few minutes, and reconnected. It worked.
Too many variables, I need to set aside some time and work this through methodically.
-
@jwj said in pfSense fails to restore IPV6 after WAN side service interruption:
@Grimson I have been rejecting leases from my cable modem, 192.168.100.1, for some time now. Doesn't appear to have an impact on this issue(s).
Rebooted pfsense late last night and the WAN did not get an ipv6 address. Waited a good long time. A second reboot set it right. I then disconnected the cable, waited a few minutes, and reconnected. It worked.
Too many variables, I need to set aside some time and work this through methodically.
Indeed, there appears to be more than one path to failure.
-
@Grimson Well that didn't work! Had another service interruption today and ignoring leases from the modem's DHCP prevented even IPV4 from recovering when the modem registered upon resumption of service. Had to bounce the WAN link to restore LAN side connectivity.
-
I see this as well, IPv6 will not come up until apply button is pressed in the WAN interface or a second reboot (sometimes)
This is with Arris CM8200B and 2.4.5-RELEASE-p1
-