WAN_DHCP6 pending / unknown and dhcpv6 server not working

bimmerdriver · Mar 3, 2022, 5:22 PM

I run pfSense on a Hyper-V server. I maintain several pfSense versions on this server. Up to mid-January, I was running the 2.7.0 development snapshot without any problem (at least not that I noticed). I switched over to the 2.6.0 RC and stayed on it until 2.6.0 was released. A couple of days ago, I switched back to 2.7.0 and updated it to the latest version.

It appears that some time between mid-January and now, there was a change in dhcpv6. The WAN_DHCP6 gateway is showing pending / unknown status. There is no process running. Also, dhcpv6 is not working. IPv6 is working. My ISP requires the "do not wait for RA" setting.

Here are the dhcp* processes that are running:

_dhcp   10940   0.0  0.1  11560  2804  -  Ss   18:18      0:00.05 dhclient: hn0 (dhclient)
root    11484   0.0  0.1  11184  2636  -  Is   18:18      0:00.01 /usr/local/sbin/dhcp6c -n -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid hn0
root    37424   0.0  0.1  11400  2840  -  Ss   18:19      0:00.71 /usr/sbin/syslogd -O rfc3164 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
dhcpd   79026   0.0  0.6  22776 11748  -  Ss   18:19      0:04.22 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid hn1

There are only 4 processes. Normally, there should be 6 processes.

There are no messages in the log for WAN_DHCP6, only for WAN_DHCP.

Is anyone else experiencing this?

bimmerdriver · Mar 4, 2022, 9:50 PM

I noticed there are several issues in Redmine related to IPv6 gateways and DHCPv6. Is it possible that someone made a change in this code to fix one of these issues that had the consequence of causing a regression? My experience over the years is that this code is notoriously brittle and there have been quite a few cases of unintended consequences of people trying to fix one issue causing another issue.

If someone would like to get on my system, I would be happy to make it available.

In the interim, if logs are needed, let me know which ones.

bimmerdriver · Mar 11, 2022, 3:20 PM

I kept updating to the latest daily snapshots and the problem did not go away, so I reverted to a checkpoint from 2.6.0-DEVELOPMENT and updated to 2.7.0-DEVELOPMENT. The problem is the same. The only thing I haven't tried is a clean installation of 2.7.0-DEVELOPMENT, which I will try to do this weekend. This seems to be a solid problem.

bimmerdriver · Mar 14, 2022, 8:50 PM

I did a clean reinstallation of 2.7.0-DEVELOPMENT and using the saved configuration. I'm still having the same problem.

IPv6 is working, so it's getting a prefix.

The IPv6 gateway is not showing an address either on the dashboard or in status / gateways. The interface is not showing up in status / interfaces The gateway has a link-local address.

DHCPv6 is not working, so the client is not getting a lease.

I noticed there is this bug, which sounds suspiciously related:

Bug #11764
IPv6 link local gateway default status not indicated in GUI

The file /tmp/hn0_defaultgwv6 is missing. I manually created it with the address of the gateway, but it didn't seem to have any effect. Restarting dpinger and dhcp still did not result in either of them running.

That may explain the gateway problem, but could it also cause the DHCPv6 problem?

bimmerdriver · Mar 22, 2022, 2:05 AM

Am I the only one experiencing this? I'd really like to get the bottom of it, but I'm out of ideas.

bimmerdriver · Mar 23, 2022, 4:38 AM

I posted on dslreports.com/forum/telus and this has been duplicated by another user. So it seems to be something specific to the Telus router, which in my case is Nokia.

bimmerdriver · Mar 27, 2022, 1:22 AM

Anyone? This problem isn't just cosmetic. It seems to affect updating, in addition to DHCPv6 and the gateway status.

My system is available for someone to take a look at or I can post whatever logs or other information is required.

Thanks in advance.

jimp · Apr 3, 2022, 2:04 AM

There have been a few changes around gateway handling for IPv6 (such as the one you linked) but I checked in my lab and every one I looked at is OK. I have a mix of CE and plus, and some on current snapshots and prior releases. Some are DHCPv6, some are static.

Do you have any special settings on the WAN interface for DHCPv6? Anything in the log from dhcp6c or similar? It would on the DHCP tab of the system logs.

bimmerdriver · Apr 3, 2022, 2:04 AM

@jimp Thank you for your reply. I'm not using any special settings, although I am probably in the minority of users that depend on "Do not wait for a RA". I'm also using "Request only an IPv6 prefix", with a /56. One other thing that may be of interest is that all of the flags are unset in the RA message from the ISP router. I'm wondering if this is causing there to be no /tmp/hn0_defaultgwv6 file for the IPv6 gateway.

jimp · Apr 4, 2022, 1:36 PM

It is entirely possible it's related to the "do not wait for RA" option. There are not many people who need it, and if the setup doesn't need it, it's not really an accurate test to just check the box and see what happens. There is some code that is duplicated when that option is set and it's possible a change was made to the code path without that option that didn't get mirrored inside that block.

Does it log anything noteworthy in the DHCP log or the system log?

bimmerdriver · Apr 4, 2022, 3:55 PM

@jimp Agreed, it's not a valid test to check "Do not wait for a RA" when it's not required. I will take a look at the log of my 2.6.0 system and compare it with the log of the 2.7.0 system. I'm wondering why the file /tmp/hn0_defaultgwv6 is not being created. Does the system use rtsold? If so, I wonder if the M and O flags not being set in the RA are causing a problem. If rtsold is being invoked with scripts to handle M and O, they will not run.

bimmerdriver · Apr 17, 2022, 5:51 PM

@jimp There are notable differences in running processes.

Below are the dhcp processes running on 2.6.0:

login-to-view

Lines 1, 6 and 7 are not running on 2.7.0. Also, the dpinger process for the IPv6 gateway is not running. It dies with exiting on signal 15.

Does that help? Let me know if you need any additional information. If you would like to come on my system to take a look, that's easy to arrange.

I'm suspicious about the missing file /tmp/hn0_defaultgwv6. As I mentioned before, the ISP router does not set the M or O flags in the RA message. Could this be causing rtsol to not work properly?

bimmerdriver · May 3, 2022, 5:24 PM

@jimp said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:

It is entirely possible it's related to the "do not wait for RA" option. There are not many people who need it, and if the setup doesn't need it, it's not really an accurate test to just check the box and see what happens. There is some code that is duplicated when that option is set and it's possible a change was made to the code path without that option that didn't get mirrored inside that block.

Does it log anything noteworthy in the DHCP log or the system log?

I originally thought this might be related to the "do not wait for RA" option, but if that was the problem, IPv6 would not be working at all. The ISP router does not send an RA in response to an RS until after the DHCP solicit has been done. Since IPv6 is working, it must have been done. So there must be some other problem, which I think is related to the missing /tmp/hn0_defaultgwv6 file for the IPv6 gateway.

I can make my hyper-v system available for you or someone else if you want to take a look. I can also send you the packet capture from wireshark.

bimmerdriver · May 19, 2022, 7:53 PM

@jimp I can see that you're busy with so many developments going on. If there is anything that I can do to help to identify the problem, let me know. If you or someone else from the development team wants to take a look at my system, I just need to know when.

nedyah700 · May 23, 2022, 5:45 PM

@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.

I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.

nedyah700 · May 24, 2022, 5:49 PM

@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:

@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.

I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.

@bimmerdriver and @jimp I would also add that my Gateway / Monitor IP is an fe80, link-local, address. I swear I remember having this issue a year or so ago and it was resolved with an update.

bimmerdriver · May 24, 2022, 9:13 PM

@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:

@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:

@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.

I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.

@bimmerdriver and @jimp I would also add that my Gateway / Monitor IP is an fe80, link-local, address. I swear I remember having this issue a year or so ago and it was resolved with an update.

In my situation, it has always been the case the gateway address is a link-local address. That does not cause a problem on 2.6.0 and it didn't cause a problem on 2.7.0 up to the point where this regression was introduced. As I pointed out above, I think this problem is related to the missing file /tmp/hn0_defaultgwv6.

bimmerdriver · May 31, 2022, 10:09 PM

I created a new bug in redmine, https://redmine.pfsense.org/issues/13238.

jimp · Jun 1, 2022, 1:23 PM

I think I've spotted the problem here. In #6880 the scripts were changed around a bit and the withoutra path isn't getting the script header that should be setting the router file(s).

Try the attached patch and see if it helps for you.

13238.diff

jimp · Jun 2, 2022, 1:47 PM

The fix on #13238 fixed the gateway with "do not wait for RA"

The DHCPv6 daemon not running was a regression from https://redmine.pfsense.org/issues/12527 and it wasn't related to "do not wait for RA".

The attached patch here should fix the DHCPv6 daemon problem. It worked for me in my lab on several systems.

814.diff

Fixes for both are going to be in 22.05.