DHCP v6 - no route to host

wallabybob

After updating to 2.0-RC1-IPv6 (i386)
built on Sun Mar 20 02:20:38 EDT 2011 I noticed in /var/log/system.log:

Mar 21 06:29:42 pfsense php: : The command ' bridge0 inet6 fe80::/64 eui64' returned exit code '127', the output was ''

and

Mar 21 06:41:53 pfsense dhcpd: send_packet6: No route to host
Mar 21 06:41:53 pfsense dhcpd: dhcpv6: send_packet6() sent -1 of 84 bytes

so I guess the eui64 option doesn't work on bridges. (I have a wired LAN and wireless LAN bridged together and DHCP server on the bridge.)

databeestje

That means I need to write a php function to perform that calculation. Sometime this century then.

Attempted fix committed.

rjairam

How is the MAC address of a bridge calculated anyway? Mine seems to change on every boot. Is it some random value?

wallabybob

@rjairam:

How is the MAC address of a bridge calculated anyway? Mine seems to change on every boot. Is it some random value?

From the FreeBSD bridge man page:

The if_bridge interface randomly chooses a link (MAC) address in the range reserved for locally administered addresses when it is created. This address is guaranteed to be unique only across all if_bridge interfaces on the local machine. Thus you can theoretically have two bridges on the different machines with the same link addresses. The address can be changed by assigning the desired link address using ifconfig(8 ).

rjairam

Ah.

There is a utility called ipv6calc that can be used to generate EUI 64 link local addresses.

http://www.tunnelbroker.net/forums/index.php?topic=751.0

In reality I've been able to get away with any random nonsense and it seems to work. It may not be right but so is assigning a link local to a bridge.

databeestje

Calculating the eui64 from the existing hardware address isn't too bad with a bad of calculus and string replacement. I just have not tried reproducing it yet.

MrKoen

@wallabybob:

I was feeling brave and enabled DHCP6 on my pfSense. After some futzing around with firewall rules and client software I finally got a DHCP Solicit to pfSense dhcpd which wasn't able to process the solicit.

I still can not get the DHCPv6 service to work on my pfSense box. Can't figure out why. All it lists in the logs is:

May 2 11:23:38 dhcpd: Sending on Socket/14/em0/2001:470:xxx:51::/64
May 2 11:23:38 dhcpd: Listening on Socket/14/em0/2001:470:xxx:51::/64
May 2 11:23:38 dhcpd: Bound to *:547
May 2 11:23:38 dhcpd: Wrote 0 leases to leases file.
May 2 11:23:38 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
May 2 11:23:38 dhcpd: All rights reserved.
May 2 11:23:38 dhcpd: Copyright 2004-2010 Internet Systems Consortium.
May 2 11:23:38 dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1

DHCPv4 works fine, however DHCPv6 requests never arrive at or get logged by pfSense. Since you mention in your post you had to play with firewall rules to get it to work, I was wondering what exactly did you set in your rules? Possibly that's the issue here as well.

MrKoen

One step further.. it was indeed a firewall rule that blocked the DHCPv6 requests. I have an alias which specifies the IPv4 and IPv6 ranges on my local network which are allowed to send traffic through pfSense. From the logs posted here I noticed the DHCPv6 requests come from fe80::/64 addresses. So I added that to my alias specifying my local network addresses and the DHCP logs started to report the DHCPv6 requests and give out leases.

The problem I'm facing now is that my client does get an IPv6 lease from pfSense, but it doesn't request the gateway and it assumes the DHCPv6 address to have a CIDR of 128. So it can not connect to the outside world and can not reach other IPv6 hosts within the same /64 address space.

MrKoen

I think I have found the problem.

It is caused by this change about a week ago:
https://rcs.pfsense.org/projects/pfsense/repos/pfSense-smos/commits/753bd64dffb8352ea273c1bf1f00aa099ec38634

The raflags for the rtadvd router discovery were changed from a numeric format to a letter in the /etc/inc/services.inc file. The translation for the "assist" mode wasn't correct though. The 192 number has been replaced by an O while this should be a M and an O. This will cause my Windows 7 client to discover the pfSense router first and get the assignment from it to request an IPv6 address via a DHCP broadcast resulting in both a gateway and an IPv6 lease. Currently since it only gives the O assignment, it only uses DHCPv6 to discover DNS services on the network, but not retrieve a DHCPv6 address. This causes it to generate its own address based on the address space the gateway is in.

Documentary supporting this thought: http://blogs.technet.com/b/teamdhcp/archive/2009/03/03/dhcpv6-understanding-of-address-configuration-in-automatic-mode-and-installation-of-dhcpv6-server.aspx from which I qoute:

Both M and O Flags are Set to 1. DHCPv6 is used for both addresses and other configuration settings. This combination is known as DHCPv6 stateful, in which DHCPv6 is assigning stateful addresses to IPv6 hosts.

And actually the /etc/inc/services.inc file also contains a comment supporting this:

/* raflags, other o, managed=64 m, stateful=128, both=192 */

@Databeestje, could you have a look at this and change it in the code if you confirm my suspicion to be correct?

I changed my DHCPv6 operating mode to Managed /services_dhcpv6.php to see what that would do and now my pfSense installation seems to have gotten corrupt. I'm getting loads of the following messages in my logs:

dhcpd: /var/db/dhcpd6.leases line 834: corrupt lease file; expecting ia_na contents, got ' '

When starting the dhcpd deamon it actually kills the whole webinterface and will eventually result in the following message in my browser:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 24 bytes) in /etc/inc/util.inc on line 905

When I run killall dhcpd through the console, the webinterface comes back up. Even trying to set it back to operating mode Assisted will not get it working anymore. Also the mentioned file /var/db/dhcpd6.leases file does not exist so I have no idea why it tries to read line 834 from it.

Does anyone have a clue how I can fix this again?

MrKoen

I got the dhcpd deamon back up again. I simply went to Diagnostics -> Edit file -> /cf/conf/config.xml -> Load and replaced the <lan>contents within <dhcpdv6>with an empty set of nodes copied from a NIC on which DHCPv6 was not enabled. Now the dhcpd deamon starts again. However, when I try to enable the DHCPv6 service again in any operation mode, the problem returns. So there is a serious bug in there somewhere which once triggered will eliminate the use of the DHCPv6 service.</dhcpdv6></lan>

databeestje

Now I know how to reproduce the out of memory issue. I'll have a look later this evening.

MrKoen

I saw you have adjusted the raflags accordingly. Just gitsynced and ran some tests and it works like a charm now! Only thing I can't get to work yet is the DHCPv6 reservations. Reservations are simply ignored and an address from the address pool is provided. Is this a thing you're still working on or should it work and do I need to play with it some more to try to get it to work?

MrKoen

Actually what I can not do anymore with the latest version is let pfSense send out the advertisements through rtadvd and let my Windows DHCPv6 hand out the IPv6 addresses as it used to do (since I have the IPv6 reservations working on my Windows server). For it to work I need pfSense to advertise itself using the MO flags, but when I configure pfSense in Assisted mode, it automatically also enables its own DHCPv6 service thus causing a conflict on the network with pfSense which is also giving out addresses. Ideally it should be that I can always choose an operating mode on the DHCPv6 page in pfSense, but only when I check the box to enable the DHCPv6 server, it will also enable the DHCPv6 service. Sounds rather logical too if you think about it. Is this a thing you agree on Databeestje and can realize? I can change the Javascript on the page to allow this if you want.

MrKoen

Databeestje, I saw you have followed up on my proposal already and checked in some changes regarding this matter. Great work! I have found a few issues with it so far:

1. When I keep the mode at advertised and disable the DHCPv6 service, it still provides DHCPv6 addresses
2. The DHCP service reports an error on startup in the system log:

php: /services_dhcpv6.php: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em0_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.1.1-P1 Copyright 2004-2010 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 23: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ /etc/dhcpdv6.conf line 28: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ Configuration file errors encountered – exiting If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If yo

3. When entering a DUID at the DHCPv6 reservation page, it does not get past the validator which keeps stating that it is not a valid DUID while it is. I also do not understand the remark posted under the field:

Enter a DUID Identifier in the following format:
"DUID-LLT - ETH – TIME --- ---- address ----"
"xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"

My DUID matches the xx:xx format shown, but is not accepted. What is meant with the line above the xx:xx?

MrKoen

I see new checkins have been done which fix the DUID problem. I can indeed add a DUID reservation now. However, the whole DHCPv6 system doesn't work anymore with the latest gitsync. I'm getting this error in the system log:

php: /services_dhcpv6.php: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em0 em0_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.1.1-P1 Copyright 2004-2010 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 23: expecting string or hexadecimal data. host-identifier option dhcp6.client-id 00-01-00-01-12-0D-0D-6C-00-1E-8C-4A-56-F ^ /etc/dhcpdv6.conf line 36: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ /etc/dhcpdv6.conf line 41: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ Configuration file errors encountered – exiting If you did not get this software from ftp.isc.org, pleas

I noticed it complaining about my DUID reservation. I have added it as Windows reports it: in the xx-xx-xx format instead of the required xx:xx:xx format. You might want to add a little regular expression JavaScript validation on the DUID field to make sure its added in the xx:xx:xx format. Or better yet, add a piece of JavaScript to the onblur of the textfield that will replace the '-' with ':' so it's still Windows copy/paste friendly :)

I just changed the DUID to the correct xx:xx:xx format. I'm still getting an error in the system logs and am still not getting a DHCPv6 lease. Also still nothing in the DHCP logs regarding IPv6 requests or responses. The error I'm getting in the system logs is:

php: /services_dhcpv6.php: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em0 em0_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.1.1-P1 Copyright 2004-2010 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 36: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ /etc/dhcpdv6.conf line 41: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ Configuration file errors encountered – exiting If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. I

It seems there is still something wrong with my reservation so I removed the reservation and restarted the dhcp deamon. Now I'm getting in the situation again where the dhcpd service kills the webinterface of pfSense. Only this time, doing a killall dhcpd on the console did not bring it back up. So next I tried rebooting the pfSense server. Luckily this brings it back up again. I did notice an error during the boot though. Something with a foreach loop in some interfaces file. I tried rebooting again and recording the bootup into an AVI to capture the error, but it did not occur anymore.

After the reboot, still no response from the DHCPv6 service and still this error with all DUID reservations now being removed:

php: : The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid em0 em0_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.1.1-P1 Copyright 2004-2010 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 31: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ /etc/dhcpdv6.conf line 36: expecting string or hexadecimal data. host-identifier option dhcp6.client-id ; ^ Configuration file errors encountered – exiting If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If you intend to requ

The router advertisement flags are still handed out correctly using assisted mode, DHCPv4 addresses are still provided, but no response at all from the DHCPv6 service anymore. May it have something to do with the firwall rules you have cooked into the software? The behavior is the same as when I had my firewall rules set incorrectly.

When I set it to unmanaged mode on the router advertisements, my clients discover the gateway and will auto-generate an IPV6 address to use. So that is okay.

MrKoen

More findings. Checking the in the error reported config file at /var/dhcpd/etc/dhcpdv6.conf I noticed the error is referring to the following content of this file:

}host s_opt1_0 {
host-identifier option dhcp6.client-id ;
fixed-address6 2001:470:-removed-:52::21;
option host-name DC4;
}
host s_opt1_1 {
host-identifier option dhcp6.client-id ;
fixed-address6 2001:470:-removed-:52::55;
option host-name KoenLaptop;
}

Turns out on a different interface I still had two reservations. I removed these, restarted the dhcp deamon and no errors in the system logs anymore and it did start to provide DHCPv6 addresses again from the pool!

Next I tried adding a DUID reservation again. I did an ipconfig /renew6 on my Windows 7 x64 client and voila: my reserved IPv6 address was returned. Databeestje, you da man! :D

MrKoen

One more question.. I notice that when getting an DHCPv6 address from pfSense (either a static reservation or pool address), my Windows 7 x64 client will add a route to the routing table like this:

2001:470:-removed-:51::1111/128

Meaning.. all data between hosts on my local 2001:470:-removed-:51::/64 subnet will be routed to the default IPv6 gateway being pfSense. This brings extra unnecessary load on pfSense and causes the pfSense logs to be filled with entries regarding my local IPv6 traffic. Is it possible to have the clients designate a /64 route to the assigned IPv6 address?

Another question.. what is the IPv6 Prefix Pool field for in a DCHPv6 reservation?

Last thing.. don't forget to put updating /status_dhcpv6_leases.php on the todo list. It does show the leases, but not the IPv6 addresses, still shows a column MAC address instead of DUID and does not allow for pool leases to be translated to a reservation.

MrKoen

I was playing with the IPv6 Prefix Pool field and apparently it can kill whole pfSense. I now know why I didn't see the error during boot on the interfaces.inc file anymore. It only appears when having this field set and if so, it will kill the WAN interface completely. I caught the error. Check the attached screenshot. The error points to this section in the interfaces.inc file:

/* Setup the prefix delegation */
foreach($IfList as $pdinterface => $friendly) {
if(is_numeric($interface['ia-pd'])) {
$realif = get_real_interface($friendly);
$dhcp6cconf .= " prefix-interface {$realif} {\n";
$dhcp6cconf .= " sla-id {$interface['ia-pd']};\n";
$dhcp6cconf .= " sla-len {$config['interface'][$interface]['dhcp-ia-pd-len']};\n";
$dhcp6cconf .= " };\n";
}
}

I removed this block completely from the interfaces.inc file, rebooted pfSense and now I have a working pfSense installation again. So there is a bug in this block that kills the WAN interface.

ErrorInInterfaces.png_thumb

MrKoen

I also just found out that the DHCPv6 service is still very unstable. After a couple of successful DHCPv6 requests, it will stop responding to DHCPv6 requests. Restarting the dhcpd service will get it going again for a few requests. No errors logged though, so probably very difficult to trace to the cause.

databeestje

The dhcp support is still very much a work in progress. The prefix pool setting for the dhcp client static mapping is for dhcp-pd support. This is what preactically the world will use for deploying IPv6 to end users.

You can save the prefix pool but is not used anywhere yet. That will come later. I need to BBQ first.