rtsold not running, IPv6 WAN (DHCP) keeps losing connectivity

luckman212

I'm troubleshooting an issue where every day like clockwork when my DHCP6 WAN connection resets (4G connection) my gateway goes down and I lose IPV6 connectivity.

Going into Interfaces, changing nothing, and then clicking Save brings everything back online. As does resetting the dhcp6c client with reset_dhcp6client_process() or pkill -HUP -F /var/run/dhcp6c.pid.

So it's something with DHCP6 or that my system just isn't handling the RA (Router Advertisement) packets being sent to it properly. I have my WAN DHCP6 Client Configuration set as:

DHCPv6 Prefix Delegation size: 64
Send IPv6 prefix hint: checked
Do not wait for a RA: unchecked

I ran tcpdump -vvvv -ni ix2 icmp6 and 'ip6[40] = 134' to confirm and see that the RAs are being sent and contain the correct delegated subnets etc.

Can anyone tell me the sequence of how interfaces.inc, rtsold and dhcp6c are supposed to work together? I understand interfaces.inc creates the scripts that in turn trigger rtsold to run and that in turn runs dhcp6c, correct?. On my system pgrep -lf rtsold indicates there is NO rtsold running, which may be the core of the problem.

I do have these files present:

/var/run/rtsold_ix2.pid
/var/run/rtsold.dump
/var/etc/rtsold_ix2_script.sh

but ps and pgrep reveal there is no process 4254...

cat /var/run/rtsold_ix2.pid
4254
pgrep -lF /var/run/rtsold_ix2.pid
pgrep: Cannot get process list (kvm_getprocs: No such process)

also /var/run/rtsold.dump is empty.

x-post: /r/PFSENSE
related: forum thread

Bob.Dig

@luckman212 That is also the case when another router is in front of pfSense and is doing the IPv6. If the connection resets, IPv4 will come back but not IPv6. You have to manually cycle WAN for it to come back. Would like to see some progress in this regard, some sort of watchdog that will intervene.

luckman212

Why should I need to manually cycle the WAN? Something is definitely off here.

pfSense receives the new RA from the upstream router, and even assigns the new prefix/IP to the interface (confirmed with ifconfig). The problem is, dpinger is not restarted, the old IPs (which are now marked deprecated in ifconfig) are not removed, and the "default route" for lack of a better word is not updated.

At that point, if I manually delete the dead IPs with ifconfig inet6 xxxx delete then things start working again. This should happen automatically when RAs are received which reconfigure the interface...

luckman212

@jimp I saw some back and forth between you and David Myers in redmine #12947 That seems like exactly the issue I'm facing here. I am also using a 4G device that changes IP/PD without a linkdown/linkup event to trigger rc.newwanipv6

Other than a nasty cronjob that runs every minute to check for this condition, do you have any ideas on what needs to be done to solve it?

Bob.Dig

@luckman212 said in rtsold not running, IPv6 WAN (DHCP) keeps losing connectivity:

Why should I need to manually cycle the WAN? Something is definitely off here.

At least that is what I was doing to "fix" it. If less is necessary the better. Here is what it looked like for me.
I now do PPPoE in pfSense and everything is working fine.

mazarian

Following! I have the same problem with Spectrum in SoCal.

luckman212

It appears we are out of luck on having devd fire events for IP address changes. There is a commit here but that seems like it might not even be in FreeBSD13 (much less 12.3...). Maybe we can get this backported to 12.x somehow? Seems like it would be immensely useful in these cases.

My poor man's solution is parsing the output of ifconfig on all DHCP6/SLAAC interfaces with a cronjob and firing an event if a change is detected. But this places unnecessary load on the system and also has a potential for up to 1 minute of downtime before the change is picked up. Still, it's the best I can come up with for now until devd can trigger on address changes. I have a PR for this that I will submit later today.

luckman212

I just pushed an update to my PR #4595 that attempts to mitigate this issue. Testers wanted!