WAN_DHCP6 pending / unknown and dhcpv6 server not working
-
I run pfSense on a Hyper-V server. I maintain several pfSense versions on this server. Up to mid-January, I was running the 2.7.0 development snapshot without any problem (at least not that I noticed). I switched over to the 2.6.0 RC and stayed on it until 2.6.0 was released. A couple of days ago, I switched back to 2.7.0 and updated it to the latest version.
It appears that some time between mid-January and now, there was a change in dhcpv6. The WAN_DHCP6 gateway is showing pending / unknown status. There is no process running. Also, dhcpv6 is not working. IPv6 is working. My ISP requires the "do not wait for RA" setting.
Here are the dhcp* processes that are running:
_dhcp 10940 0.0 0.1 11560 2804 - Ss 18:18 0:00.05 dhclient: hn0 (dhclient) root 11484 0.0 0.1 11184 2636 - Is 18:18 0:00.01 /usr/local/sbin/dhcp6c -n -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid hn0 root 37424 0.0 0.1 11400 2840 - Ss 18:19 0:00.71 /usr/sbin/syslogd -O rfc3164 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf dhcpd 79026 0.0 0.6 22776 11748 - Ss 18:19 0:04.22 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid hn1There are only 4 processes. Normally, there should be 6 processes.
There are no messages in the log for WAN_DHCP6, only for WAN_DHCP.
Is anyone else experiencing this?
-
I noticed there are several issues in Redmine related to IPv6 gateways and DHCPv6. Is it possible that someone made a change in this code to fix one of these issues that had the consequence of causing a regression? My experience over the years is that this code is notoriously brittle and there have been quite a few cases of unintended consequences of people trying to fix one issue causing another issue.
If someone would like to get on my system, I would be happy to make it available.
In the interim, if logs are needed, let me know which ones.
-
I kept updating to the latest daily snapshots and the problem did not go away, so I reverted to a checkpoint from 2.6.0-DEVELOPMENT and updated to 2.7.0-DEVELOPMENT. The problem is the same. The only thing I haven't tried is a clean installation of 2.7.0-DEVELOPMENT, which I will try to do this weekend. This seems to be a solid problem.
-
I did a clean reinstallation of 2.7.0-DEVELOPMENT and using the saved configuration. I'm still having the same problem.
IPv6 is working, so it's getting a prefix.
The IPv6 gateway is not showing an address either on the dashboard or in status / gateways. The interface is not showing up in status / interfaces The gateway has a link-local address.
DHCPv6 is not working, so the client is not getting a lease.
I noticed there is this bug, which sounds suspiciously related:
Bug #11764
IPv6 link local gateway default status not indicated in GUIThe file /tmp/hn0_defaultgwv6 is missing. I manually created it with the address of the gateway, but it didn't seem to have any effect. Restarting dpinger and dhcp still did not result in either of them running.
That may explain the gateway problem, but could it also cause the DHCPv6 problem?
-
Am I the only one experiencing this? I'd really like to get the bottom of it, but I'm out of ideas.
-
I posted on dslreports.com/forum/telus and this has been duplicated by another user. So it seems to be something specific to the Telus router, which in my case is Nokia.
-
Anyone? This problem isn't just cosmetic. It seems to affect updating, in addition to DHCPv6 and the gateway status.
My system is available for someone to take a look at or I can post whatever logs or other information is required.
Thanks in advance.
-
There have been a few changes around gateway handling for IPv6 (such as the one you linked) but I checked in my lab and every one I looked at is OK. I have a mix of CE and plus, and some on current snapshots and prior releases. Some are DHCPv6, some are static.
Do you have any special settings on the WAN interface for DHCPv6? Anything in the log from dhcp6c or similar? It would on the DHCP tab of the system logs.
-
@jimp Thank you for your reply. I'm not using any special settings, although I am probably in the minority of users that depend on "Do not wait for a RA". I'm also using "Request only an IPv6 prefix", with a /56. One other thing that may be of interest is that all of the flags are unset in the RA message from the ISP router. I'm wondering if this is causing there to be no /tmp/hn0_defaultgwv6 file for the IPv6 gateway.
-
It is entirely possible it's related to the "do not wait for RA" option. There are not many people who need it, and if the setup doesn't need it, it's not really an accurate test to just check the box and see what happens. There is some code that is duplicated when that option is set and it's possible a change was made to the code path without that option that didn't get mirrored inside that block.
Does it log anything noteworthy in the DHCP log or the system log?
-
@jimp Agreed, it's not a valid test to check "Do not wait for a RA" when it's not required. I will take a look at the log of my 2.6.0 system and compare it with the log of the 2.7.0 system. I'm wondering why the file /tmp/hn0_defaultgwv6 is not being created. Does the system use rtsold? If so, I wonder if the M and O flags not being set in the RA are causing a problem. If rtsold is being invoked with scripts to handle M and O, they will not run.
-
@jimp There are notable differences in running processes.
Below are the dhcp processes running on 2.6.0:
Lines 1, 6 and 7 are not running on 2.7.0. Also, the dpinger process for the IPv6 gateway is not running. It dies with exiting on signal 15.
Does that help? Let me know if you need any additional information. If you would like to come on my system to take a look, that's easy to arrange.
I'm suspicious about the missing file /tmp/hn0_defaultgwv6. As I mentioned before, the ISP router does not set the M or O flags in the RA message. Could this be causing rtsol to not work properly?
-
@jimp said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:
It is entirely possible it's related to the "do not wait for RA" option. There are not many people who need it, and if the setup doesn't need it, it's not really an accurate test to just check the box and see what happens. There is some code that is duplicated when that option is set and it's possible a change was made to the code path without that option that didn't get mirrored inside that block.
Does it log anything noteworthy in the DHCP log or the system log?
I originally thought this might be related to the "do not wait for RA" option, but if that was the problem, IPv6 would not be working at all. The ISP router does not send an RA in response to an RS until after the DHCP solicit has been done. Since IPv6 is working, it must have been done. So there must be some other problem, which I think is related to the missing /tmp/hn0_defaultgwv6 file for the IPv6 gateway.
I can make my hyper-v system available for you or someone else if you want to take a look. I can also send you the packet capture from wireshark.
-
@jimp I can see that you're busy with so many developments going on. If there is anything that I can do to help to identify the problem, let me know. If you or someone else from the development team wants to take a look at my system, I just need to know when.
-
@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.
I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.
-
@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:
@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.
I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.
@bimmerdriver and @jimp I would also add that my Gateway / Monitor IP is an fe80, link-local, address. I swear I remember having this issue a year or so ago and it was resolved with an update.
-
@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:
@nedyah700 said in WAN_DHCP6 pending / unknown and dhcpv6 server not working:
@bimmerdriver I tried a past development release of 22.05 and had the exact same issue you are experiencing. Traffic flows without any issue but the gateway shows pending. I rolled back when I noticed the issue and haven't tried since.
I also have "Do not wait for a RA" and "Request only an IPv6 prefix" selected.
@bimmerdriver and @jimp I would also add that my Gateway / Monitor IP is an fe80, link-local, address. I swear I remember having this issue a year or so ago and it was resolved with an update.
In my situation, it has always been the case the gateway address is a link-local address. That does not cause a problem on 2.6.0 and it didn't cause a problem on 2.7.0 up to the point where this regression was introduced. As I pointed out above, I think this problem is related to the missing file /tmp/hn0_defaultgwv6.
-
I created a new bug in redmine, https://redmine.pfsense.org/issues/13238.
-
I think I've spotted the problem here. In #6880 the scripts were changed around a bit and the withoutra path isn't getting the script header that should be setting the router file(s).
Try the attached patch and see if it helps for you.
-
The fix on #13238 fixed the gateway with "do not wait for RA"
The DHCPv6 daemon not running was a regression from https://redmine.pfsense.org/issues/12527 and it wasn't related to "do not wait for RA".
The attached patch here should fix the DHCPv6 daemon problem. It worked for me in my lab on several systems.
Fixes for both are going to be in 22.05.
