Dhcp6c IA_NA address renewal strangeness

[quantumx]

Some time in the last couple of months, a testing pfSense box, started losing IPv6 connectivity.  This box connects to a native IPv6 network, acquiring and providing connectivity using DHCPv6 services.  LAN connectivity is set to TRACK the WAN interface

Network DHCP structure (basic):
DHCP Server (ISC) – DHCP Relay (pfsense beta) -- Client (pfsense beta)

Either dhcp6c or another process on the Client pfsense seems to be removing the public address from the WAN interface, while leaving the LAN address/prefix intact :o

To speed up the debugging process, the lease times for both addresses and prefixes on the DHCP server were shortened to just under 3 minutes.  It was found that although dhcp6c was showing both WAN address and LAN prefix as being renewed successfully, the WAN IP address never survives beyond 1 lease term.  It first enters the 'deprecated' state and soon afterwards disappears from the interface.  The LAN address/prefix always remains intact.

Details attached include:

-dhcp server relevant software version / config lines
-dhcp relay configuration
-relevant chunks of config file of client device (interfaces)
-dhcp6c debug log (launched manually from shell)
-system log (from client webui)
-dhcp log (from client webui)
-ifconfig output of interfaces, added inline to dhcp6c debug logging

Additional tidbits:

-If absolutely required, packet traces can be provided, but seem redundant given that dhcp6c reports the lease as renewed.
-Several other platforms are using the Server/Relay infrastructure without issue.
-If the relay is removed from the environment, the Client WAN address persists, but beyond the first lease term shows 'deprecated' which still would seem to indicate a problem.

All thoughts and suggestions welcome.

Thanks for reading  ;D
dhcp6c-debug.txt

[quantumx]

After further testing:

2.2.x Works (2.2.1 & 2.2.6 tried) - Address stays attached to WAN interface

2.3beta(latest) Fails - Address is removed from WAN interface after ~lease time

Attached is an almost vanilla config that exhibits the problem .

config-pfSense.localdomain-20160218170038.xml.txt

[cmb]

That's a pretty basic setup to have stopped working. There are a slew of people, myself included, running DHCP6 WANs with no issues upon renewal.

Am I misinterpreting the timestamps, or is your lease time something like 4 minutes? So it's renewing after ~2 minutes. There has to be something unusual there, and the timing is the only thing that stands out.

[quantumx]

The server lease time was originally configured for 1 day, but for debugging purposes, it seemed more reasonable to use a shorter interval - easier to track the sequence of events.  Extending the time simply seems to extend the failure interval.

It is my understanding that it is reasonable for a DHCP client to begin attempting renewal after approximately 50% of the lease time has elapsed - maybe this is wrong?

As of build: Sat Feb 27 13:27:14 CST 2016

the problem persists.  What additional information can I provide?

[cmb]

That's correct for that short of a lease time. That wouldn't be a good real-world scenario, but if it's just shortening the time it takes to hit a problem, that's understandable.

I turned lease length down to 2 minutes on a test setup, so it's renewing every minute. There is some kind of race condition there where it logs something like:

rc.newwanipv6: Failed to update WAN[wan] IPv6, restarting

which I can hit from time to time with the lease renewing every minute. You had that once in your log as well. I updated that log message to be more clear. It just means when rc.newwanipv6 ran, the interface it was called on had no IPv6 address. It didn't actually restart anything.

That doesn't seem to be related to the problem at hand either though. Even in that circumstance, dhcp6c still puts the address back on the interface.

@quantumx:

What additional information can I provide?

Don't have an answer for that just yet, still looking.

[quantumx]

Might this patch be relevant?

https://svnweb.freebsd.org/ports/head/net/dhcp6/files/patch-dhcp6c.c?revision=405216&view=markup

[cmb]

@quantumx:

Might this patch be relevant?

https://svnweb.freebsd.org/ports/head/net/dhcp6/files/patch-dhcp6c.c?revision=405216&view=markup

I was wondering that, given that's the only relevant change I see between 2.2.x and 2.3. But it sounds like it should actually fix problems like you're describing.

You on 64 bit?

I built a 64 bit dhcp6 package from prior to those changes, which should be equivalent to what was in 2.2.x.
http://files.atx.pfsense.org/cmb/dhcp6-20080615_3.txz

If you're on 32 bit I can build that instead.

To install that:

fetch http://files.atx.pfsense.org/cmb/dhcp6-20080615_3.txz
pkg delete dhcp6
pkg install dhcp6-20080615_3.txz

Then try again.

[quantumx]

This package resolves the issue  :D

[cmb]

@quantumx:

This package resolves the issue  :D

Good. Now to figure out what's broken in that changeset. My best guess is the sacreate check.

This package has all those patches minus the sacreate one originally from here.

http://files.atx.pfsense.org/cmb/dhcp6-20080615_4.txz

Please give that one a try and let me know.

I'm not sure what changed but the test system I dropped to 120s lease time started dropping its WAN-side IP too. It stopped happening with the _3 pkg version. It hasn't happened yet with that _4 pkg version.

[cmb]

That definitely seems to have fixed the issue. I went ahead and reverted that in our ports to get that change out to a wider audience. If you just upgrade a stock 2.3 now, you'll get the _7 dhcp6 package version which has the same change as what I posted earlier.

[quantumx]

Confirming _4 package works as well.