pfSense fails to restore IPV6 after WAN side service interruption

jcyr

This has been a long standing pfSense issue that has not been much of a problem given Comcast's reliability, but lately due to road work in the area it's come to the forefront.

The pfSense WAN is to a DOCSIS cablemodem served by Comcast which supports native IPV6. After a WAN side service interruption IPV6 connectivity isn't restored, only IPV4. All WAN and LAN side interfaces lose their IPV6 addresses.

A pfSense reboot, or releasing and restoring the WAN connection via the Status/Interfaces panel.

Is this how it's supposed to work?

jcyr

@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:

A pfSense reboot, or releasing and restoring the WAN connection via the Status/Interfaces panel.

That should have said: A pfSense reboot, or releasing and restoring the WAN connection via the Status/Interfaces panel restores IPv6 connectivity.

I see similar behavior. I'm with Spectrum (formerly Time Warner) using an ARRIS SB6190 cable modem.

What cable modem are you using?

jcyr

@jwj SB6121, Arris chip. pfSense 2.4.4-RELEASE-p3

jcyr

I don't think it's a Cablemodem issue, at least not one that couldn't be mitigated in software. Re-initializing the WAN interface fixes the issue.

Yeah, I agree. Just seeing what is common and what is different. I'm also on 2.4.4-RELEASE-p3. pfblockerng-devel and nut. Those should be of no relevance. I'm not doing anything special with my ipv6 prefix, defaults across the board.

My WAN is using the igb driver. My hardware is a supermicro 5018D-FN4T, which is massive overkill for my needs, but I had it leftover from another use. That makes the interface a Intel I350-AM2 according to the spec-sheet.

I haven't really put much into isolating the issue because my service is also very reliable. I see this once every two or three months.

We'll see if anyone else has something to add.

jcyr

First noted the issue in version 2.1.

jcyr

I suspect it may have something to do with the way DOCSIS handles service outages. A cablemodem will not drop the link to the router during service outages (RF side failures), instead it keeps the link up and fires up an internal DHCP server that hands out 192.168.100.x addresses with 30 second leases. You'll see a 192.168.100.10 IPV4 WAN address during such outages, Upon service restoration the modems internal DHCP server is shut down and further DHCP request are passed on to the ISP resulting in the WAN being reassigned a proper network address within 30 seconds.

None of this applies to IPV6 and radvd seems to permanently give up with the initial service loss. I need to run a few tests to see if physically disconnecting and reconnecting the modem to router link behaves the same as a cable side service outage.

jcyr

Disconnecting, then reconnecting restores IPV6 connectivity correctly. Cable (RF) side outages do not! Could it link down/up events that don't occur with cable outages be the cause?

JKnott

@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:

A cablemodem will not drop the link to the router during service outages (RF side failures), instead it keeps the link up and fires up an internal DHCP server that hands out 192.168.100.x addresses with 30 second leases.

If that happens, then the router will not know about the internal DHCP server and continue with the regular address. Unless something happens, such as dropping the link, the router will continue with the assigned address, until the lease expires. Perhaps someone should do some packet capture, to see what happens when the RF side is disconnected and reconnected.

jcyr

@JKnott Correct, that is the reason I configure DHCP to hand out short leases. pfSense gets no indication that the route is down when an RF side failure occurs. Every single Cable modem in the world behaves this way... its mandated by the DOCSIS spec. What were they thinking!!!

This issue has existed since the introduction of IPV6 support in pfSense (version 2.1 or thereabouts). Which leads me to conclude that very few are trying to support native IPV6 over Cablemodem...

provels

Same here whenever I reboot the modem. I just re-save the WAN settings to reconnect. If I remember... :) I've also setup the WAN interface to ignore leases offered by the cable modem.

jcyr

Odd thing is, IPV6 quickly becomes aware something is wrong and drops V6 addresses of all interfaces. You'd think DHCPV6 would keep retrying in that situation, and kick off the whole V6 delegation thing with radvd upon establishing a new V6 WAN address?

JKnott

@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:

This issue has existed since the introduction of IPV6 support in pfSense (version 2.1 or thereabouts). Which leads me to conclude that very few are trying to support native IPV6 over Cablemodem...

What is it that's causing IPv6 to fail? As for LAN lease, that shouldn't fail for a very long time and there should be RAs on the WAN side that indicate the connection is up. Has anyone done a packet capture to see what happens when it fails?

ljr

@jcyr your post train is the most useful insight I have read on the Internet on a long, long time. I have observed the same behaviours (Rogers, technologically identical to Comcast) on my end, but haven't documented or researched it as far as you did.

Thanks.

jcyr

Ok, know I'm confused! Did mess up the capture by forgetting to update the capture count from the default 100, and won't be able to try this again without incurring substantial grief. Thing is it worked perfectly!!!

Pulled the coax from the modem.
Both V4 and V6 gateways go offline.
WAN link receives 192.168.100.10 IP address from modem's internal DHCP
Wait about 2 minutes for the modem to realize the signal isn't coming back, and starts scanning.
Modem re-reisters with ISP.
pfSense gets valid WAN network V4 address upon expiry of the 30 second 192.168.100.10 lease.
new V6 address assigned to WAN.
delegated V6 addresses assigned to LANs.

Exactly how you'd expect it to happen!

Why would the service interruptions I've seen lately due to road work be any different? Fiber cuts to the CMTS providing the the RF modulation, without RF signal loss on the cable? Not sure I could easily reproduce that.

jcyr

Adding, the the V6 addresses are dropped from all interfaces, around the time the 192.168.100.10 is installed on the WAN interface.

JKnott

@jcyr said in pfSense fails to restore IPV6 after WAN side service interruption:

Adding, the the V6 addresses are dropped from all interfaces, around the time the 192.168.100.10 is installed on the WAN interface.

Then something is telling pfSense the modem is down. Your post says the WAN port gets a 192 address from the internal DHCP server. How does that happen, if pfSense didn't request an address? Normally, a device holds onto an address until the lease expires. The RF side failing can't tell pfSense it's down, unless something, perhaps the Ethernet link dropping, tells pfSense. For example, my DHCP lease, from my ISP is 72 hours. So, my firewall "owns" that address until the end of the lease and nothing, short of dropping the Ethernet link will change that. Also, the lease is normally renewed well before it expires, so the RF side would have to be down at least 20 - 30 hours, before pfSense loses the lease.

ljr

@JKnott I'm pretty sure it's the modem... nevermind that they are Puma chipsets, as soon as they see an error on the uplink (as recorded in the DOCSIS status page of the modem) they fall back into 192.168.100.0/24 with DHCP mode. That shouldn't be happening in "bridged mode" as opposed to "gateway mode", but... well, that's what we have to work with.

An "error" on the uplink could be anything from coax being disconnected, to CMTS issues (no ranging response, etc.) to a signal level issue.

Short of a "work around", I don't think pfSense can really do anything about this...

jcyr

@JKnott I hand out short leases, about 30 minutes, but you're right. I'm not sure what triggers DHCP to request a new lease so quickly.