IPv6 changes in 2.2.5
-
Any thoughts on this? I had a look at the code for rc.newwanipv6 but the route change command must be called by an external library as it doesn't seem to be there.
This is unrelated to the issue we were discussing, but it is worth thinking through.
The code in question is in the system_resolvconf_generate() function in /etc/inc/system.inc.
I'm wondering if a piece of code is getting confused by the PPPoE connection as the routing table has fe80::xxx:xxxx:xxxx:bc00%pppoe0 as the default route gateway not just fe80::xxx:xxxx:xxxx:bc00.
As fe80::/10 addresses are link local, they only have meaning in a single scope. %pppoe0 is the correct syntax to indicate that scope.
It looks as if the IPv6 nameservers your ISP is returning are not reachable via the IPv6 gateway returned via IPv6CP (which will have a link local address). Can you ping6 the addresses?
-
It looks as if the IPv6 nameservers your ISP is returning are not reachable via the IPv6 gateway returned via IPv6CP (which will have a link local address). Can you ping6 the addresses?
Yes I can ping the DNS servers. I figured out what it was though and it wasn't related to the 2.2.5 fixes specifically. I had those DNS servers manually entered in the System - General Setup - DNS Servers list and somehow had forced them to use the WAN_DHCP6 gateway as the gateway for the traffic to them. Once I removed that the errors are gone from the logs.
The only issue I have now is that for the last 20 hours or so access via IPV6 to only the pfSense websites (www, blog, packages, doc, etc) are all very slow and timeout frequently. This are the only IPV6 sites that I am having trouble with. Everything was working fine after the upgrade to 2.2.5 so I don't think its specifically related to that. Given no one else seems to be having issues I'm not sure what is going on. I no longer have any errors in the system logs and have verified no traffic to the pfSense IPs is being blocked. This occurs from inside the network but also affects the pfSense GUI as well. If I check "Prefer IPv4" in System - Advanced Settings or connect from a machine with only IPv4 then everything is fine.
-
The only issue I have now is that for the last 20 hours or so access via IPV6 to only the pfSense websites (www, blog, packages, doc, etc) are all very slow and timeout frequently.
Yeah it's been completely broken for about a day now. Certainly not related to upgrade.
-
Yeah it's been completely broken for about a day now. Certainly not related to upgrade.
Thanks! I'll stop trying to troubleshoot. :D
-
I too am having problems with IPv6 in 2.2.5. My previously rock solid IPv6 connection is now disconnecting after only a few hours.
Different from others in this thread, I'm not on PPPoE. I just have a normal Comcast connection.
In that case, the new code I contributed to 2.2.5 cannot be to blame, as it is PPPoE specific.
You could usefully try:
ps -auwwx | grep -E -e '(dhcp6c|rtsold)'
clog /var/log/dhcpd.log | grep dhcp6c | tail -n 40As explained above, the first command displays details of any running dhcp6c and rtsold processes, whilst the second one displays the last 40 lines of available dhcp6c related messages.
You should have one dhcp6c process for every interface that uses DHCP6 and/or DHCP-PD. Likely, this will only be your WAN interface.
I have seen dhcp6c disappear once on my WAN interface in 2.2.5, apparently dying silently, so I have no idea whether that was a one-off or not.
-
…
There is a nasty 'immediately after boot' issue with IPv6 PPPoE that I never got round to characterising and reporting, which is not caused by the new code in 2.2.5 but might interact negatively with it. On my system at least, with the PPPoE parent interface on a vlan, the MAC address of the PPPoE interface is random for the first connection after boot, resulting in a random link local address for the PPPoE interface. Once the system is booted, this settles down to a predictable MAC address based on the system hardware and PPPoE works reliably. It is possible that the new code in 2.2.5 allows the random link local address to be used for IPV6CP, but the predictable link local address for dhcp6c, which will likely result in strange IPv6 brokenness.The workround for this issue is to Disconnect the PPPoE interface in Status -> Interfaces, then Connect it again. All will then work properly until the system is next booted.
...I "suffer" from this phenomenon too…
Could you tell if [System: Advanced: System Tunables](net.inet6.ip6.use_tempaddr OR net.inet6.ip6.prefer_tempaddr) is/should be related and should be a fix ?
I tested and changing the value to '0' has no effect.
-
@hda:
On my system at least, with the PPPoE parent interface on a vlan, the MAC address of the PPPoE interface is random for the first connection after boot, resulting in a random link local address for the PPPoE interface. Once the system is booted, this settles down to a predictable MAC address based on the system hardware and PPPoE works reliably.
I "suffer" from this phenomenon too…
Is your PPPoE parent interface a vlan or a physical interface?
@hda:
Could you tell if [System: Advanced: System Tunables](net.inet6.ip6.use_tempaddr OR net.inet6.ip6.prefer_tempaddr) is/should be related and should be a fix ?
I tested and changing the value to '0' has no effect.
Those two sysctls control IPv6 Privacy Extensions (see RFC 4941 for details). I'm pretty certain they are not involved here, especially as their default value under FreeBSD is 0. If these sysctls were set to 1, the link local address would always be generated using privacy extensions.
I instrumented up interface_ppps_configure() and have verified that the MAC addresses of the parent interface (the VLAN) and the parent of the parent interface (the physical interface) are set as I expect and the ngctl msg <parent interface="">: setautosrc 1 call made by interface_ppps_configure() has succeeded. I've also verified the two sysctls you mention are set to 0, as I expect.
I added a six second delay in interface_ppps_configure() just before the call to invoke mpd5 if the system is booting, but that didn't correct the behaviour.
I have a suspicion that the IPv6CP code in mpd5 is responsible for this problem. The current code arguably goes against the spirit of section 4.1 of RFC 5072, which notes (emphasis added):
The non-zero value of the tentative interface identifier SHOULD be chosen such that the value is unique to the link and, preferably, consistently reproducible across initializations of the IPV6CP finite state machine (administrative Close and reOpen, reboots, etc.). The rationale for preferring a consistently reproducible unique interface identifier to a completely random interface identifier is to provide stability to global scope addresses (see Appendix A) that can be formed from the interface identifier.
On boot, /etc/rc.bootup calls interfaces_configure() in /etc/inc/interfaces.inc. This walks through the configured interfaces, initialising them sequentially. It may well be that the WAN interface is configured before any other interfaces on the machine are up, which is significant as CreateInterfaceID() in ipv6cp.c calls GetEther(NULL, &hwaddr) in an attempt to discover a hardware address to base the interface identifier on before falling back to a random interface identifier.
If you look at the definition of GetEther() in util.c, it ignores interfaces that only have point-to-point or loopback addresses even if they have a MAC address (i.e. an EUI-48). This failure to use an available EUI-48 violates section 4.1 of RFC 5072 as the RFC requires the use of any EUI-64 or EUI-48 on the machine before falling back to other sources of uniqueness and then a random interface identifier. CreateInterfaceID() should use MAC addresses of interfaces that have no addresses at all.
I suspect the fix will require changing to CreateInterfaceID() to follow the RFC more closely, basing the interface ID on the first valid value from:
-
the MAC address(es) of any interface used as part of the PPP link (if any - PPPoE is not necessarily in use)
-
the MAC address(es) of any other interface on the machine
-
GetEther() (as now - though this is arguably redundant as all MAC addresses should be considered by the first two steps)
-
randomness (as now)</parent>
-
-
Is your PPPoE parent interface a vlan or a physical interface?
Yes, a physical interface in my case.
…
I suspect the fix will require changing to CreateInterfaceID() to follow the RFC more closely,...OK, I will continue my procedure, ever since 2.2.x, to disconnect/connect once every time after a reboot/boot of pfSense (until your recommendation will lead to code change).
Doing so, my IPv6 ISP lease will be continued every hour and not terminated after 2 hours (remarkably, the IPv4 has no problems with hourly re-lease).Thank you for the clear insight and references in the case.
-
Thank you for the clear insight and references in the case.
David_W, many many thanks from myself too for the advice you have shared here and in the Zen forums. I think I have pfSense working with IPv6 and Zen, although I need to go through everything to be one thousand percent sure I certainly would not have got this far without your taking the time to discuss your setup.
-
@hda:
I suspect the fix will require changing to CreateInterfaceID() to follow the RFC more closely,…
OK, I will continue my procedure, ever since 2.2.x, to disconnect/connect once every time after a reboot/boot of pfSense (until your recommendation will lead to code change).
Doing so, my IPv6 ISP lease will be continued every hour and not terminated after 2 hours (remarkably, the IPv4 has no problems with hourly re-lease).I've pretty much proved my diagnosis that the problem is my earlier supposition about CreateInterfaceID() by proving you can work round the problem by assigning a bogon IPv4 address to the PPPoE parent interface temporarily at boot.
Install the Shellcmd package and create an new entry:
| Command | sh -c 'ifconfig igb0 inet 192.0.2.248/31 alias > /dev/null 2>&1 ; sleep 120 ; ifconfig igb0 inet 192.0.2.248/31 -alias > /dev/null 2>&1' >/dev/null 2>/dev/null & |
| Shellcmd type | earlyshellcmd |
| Description | Temporarily assign a bogon (RFC 5737) IPv4 address to an interface to ensure sane IPv6CP interface identifier allocation immediately after boot - https://forum.pfsense.org/index.php?topic=101967.0 |You will need to change igb0 (twice) to to the interface that is normally used to set the interface identifier. You should be able to recognise this interface from its MAC address.
When you reboot, the IPv6 WAN address should be the same on the initial connection as on subsequent connections.
When I have the time, I will open a redmine bug about this issue.
-
…
When you reboot, the IPv6 WAN address should be the same on the initial connection as on subsequent connections.Yes, Confirmed 1st step after reboot. Great temporary solution, thanks David.
Will report again after 1 and 2 hr uptime.(Alix-box on 2.2.6)
-
Well, maybe half a solution :)
No IPv6 lease renewal after 1 hr and not after the 2hr limit, then connection is gone as usual in such case (for IPv6 only).Then:
Do Status-Interfaces PPPoE-Disconnect then Diagnostic-Command Prompt(ps ax | grep dhcp6c) :: still the dhcp6c PID there ! (bad)
Do Diagnostic-Command Prompt(kill -9 $PID) :: OKDo Status-Interfaces PPPoE-Connect :: Diagnostic-Command Prompt(ps ax | grep dhcp6c) - new PID dhcp6c, but address change from WAN-MAC to LAN-MAC !?
Check back after 1 & 2 hr if lease renewal is reported in System Logs.
-
1 hr, no lease renewal. Wait for definite 2hr…. No and lost IPv6 as expected.
-
@hda:
Well, maybe half a solution :)
No IPv6 lease renewal after 1 hr and not after the 2hr limit, then connection is gone as usual in such case (for IPv6 only).Then:
Do Status-Interfaces PPPoE-Disconnect then Diagnostic-Command Prompt(ps ax | grep dhcp6c) :: still the dhcp6c PID there ! (bad)
Do Diagnostic-Command Prompt(kill -9 $PID) :: OKDo Status-Interfaces PPPoE-Connect :: Diagnostic-Command Prompt(ps ax | grep dhcp6c) - new PID dhcp6c, but address change from WAN-MAC to LAN-MAC !?
Check back after 1 & 2 hr if lease renewal is reported in System Logs.
The temporary work-round I posted earlier today only fixes the random IPv6CP interface identifier after boot. That problem is now clearly characterised. Leave the temporary work-round in place pending a more permanent solution. I'm thinking of implementing this work-round in interfaces_configure() as the next stage, because reimplementing mpd5's CreateInterfaceID() to follow section 4.1 of RFC 5072 more closely is a longer term project.
The ongoing symptoms you are describing relate to the issue we were discussing earlier in the thread. I suspect I've already identified the fix:
If I was to change /etc/inc/interfaces.inc, I would suggest changing interface_configure() so that it does not call interface_dhcpv6_configure() for a PPPoE interface, allowing ppp-ipv6 to call interface_dhcpv6_configure() unconditionally once mpd5 signals IPv6 link up.
Edit to add: On further reflection, I'd make this change for all PPP interfaces, not just PPPoE.
I tried hard not to change /etc/inc/interfaces.inc, because my local systems run with the RFC 4638 patch, which changes /etc/inc/interfaces.inc in various places. I hope that the pull request to merge the RFC 4638 patch into pfSense 2.3 will be looked at soon.
I suspect that dhcp6c is somehow starting twice on your system following initial boot-up, presumably from a race condition. I know someone e-mailed me some debugging output a while back, and I can't remember whether it was you. In any event, I wonder if you would be kind enough to reboot and send me the output of the debugging commands I gave earlier in the thread (PM preferred):
clog /var/log/ppp.log | grep -A 1 -E -e 'IPV6CP: LayerUp' | tail -n 2
ifconfig pppoe0 inet6 | grep -E -e '( fe80::|nd6)'
ps -auwwx | grep -E -e '(dhcp6c|rtsold)'
clog /var/log/dhcpd.log | grep dhcp6c | tail -n 40Your interface appears from your screen shots to be pppoe1 - you will need to make the appropriate substitution.
If the expected lease renewal doesn't happen or you note there is more than one dhcp6c process running on the pppoe1 interface, try:
/usr/local/sbin/ppp-ipv6 pppoe0 down ; pkill -xf '^.*dhcp6c.*pppoe0$' ; sleep 2 ; /usr/local/sbin/ppp-ipv6 pppoe0 upAgain, if the problem is with pppoe1, make the three substitutions.
-
…
I suspect that dhcp6c is somehow starting twice on your system following initial boot-up, presumably from a race condition. I know someone e-mailed me some debugging output a while back, and I can't remember whether it was you. In any event, I wonder if you would be kind enough to reboot and send me the output of the debugging commands I gave earlier in the thread (PM preferred):
...I can confirm this all and have sent you the data. Thanks sofar. :)
EDIT:
/usr/local/sbin/ppp-ipv6 pppoe0 down ; pkill -xf '^.*dhcp6c.*pppoe0$' ; sleep 2 ; /usr/local/sbin/ppp-ipv6 pppoe0 up
This does the jobs on console shell as root, not with GUI command prompt as admin. Did it immediately after a reboot, keeps the addresses same (..:b371), makes an entry in system.log for rc.newwanipv6 and yeah after 1hr there is a renewal of the lease as I expect and know to work out OK until the next (re)boot. Thanks again David !
-
@hda:
/usr/local/sbin/ppp-ipv6 pppoe0 down ; pkill -xf '^.*dhcp6c.*pppoe0$' ; sleep 2 ; /usr/local/sbin/ppp-ipv6 pppoe0 up
This does the jobs on console shell as root, not with GUI command prompt as admin. Did it immediately after a reboot, keeps the addresses same (..:b371), makes an entry in system.log for rc.newwanipv6 and yeah after 1hr there is a renewal of the lease as I expect and know to work out OK until the next (re)boot. Thanks again David !
A fairly trivial patch to /etc/inc/interfaces.inc will hopefully fix both problems discussed in this thread.
Install System Patches (if you haven't done so already) and create a patch as follows:
| Field | Contents |
| Description | PPP IPv6 fixes |
| URL/Commit ID | https://dl.dropboxusercontent.com/u/107909287/pfSense%202.2%20patches/2.2.6-RELEASE-ppp-ipv6.patch |
| Patch Contents | (leave blank} |
| Path Strip Count | 1 |
| Base Directory | / |
| Ignore Whitespace | (checked) |
| Auto Apply | (unchecked) |Press the Save button, then, if necessary, press 'Fetch' next to the patch. At this point, the option to 'Apply' should appear, so press 'Apply'.
If you have the RFC 4638 patch applied, you should install this patch before that patch. If the RFC 4638 patch is already installed, revert the RFC 4638 patch, apply this one, then apply the RFC 4638 patch again.
Once you've installed this patch, do a Disconnect and Reconnect on the affected interface in Status -> Interfaces. Test that IPv6 works correctly.
If IPv6 works correctly, set the shellcmd entry from my earlier work round to 'disabled' (rather than 'earlyshellcmd'), as this patch incorporates a version of that work round (the temporary bogon IPv4 address is set on the first configured parent interface of a PPPoE connection and removed once the PPP connection is up). Before you reboot, take a note of the IPv6 link local address for the PPPoE interface, so that you can check after rebooting that the IPv6 addressing is the same as on subsequent connections. Please do not disconnect and reconnect for several hours, as the key test for this patch is that IPv6 keeps working correctly without any manual intervention. If IPv6 addressing is not as you wish on the first connection, you can re-enable the shellcmd, reboot and try again.
If IPv6 does not work correctly with this patch installed, revert the patch and do another Disconnect / Reconnect to resume normal operation.
I'd be grateful if you can update this thread on whether this patch works and whether it is sufficient to ensure IPv6 addressing is the same on the first connect after reboot as on subsequent connections. I'm hoping to get these issues to the point where I can submit pull requests soon, as they seem to be affecting quite a few people.
Anyone else seeing either of these issues in 2.2.5 or 2.2.6 is welcome to try this patch.
-
…
I'd be grateful if you can update this thread on whether this patch works and whether it is sufficient to ensure IPv6 addressing is the same on the first connect after reboot as on subsequent connections...Applied your patch OK.
Tested Status:Interfaces Disconnect(~10sec)/Connect(~25sec) ADSL OK.
IPv6 still good, [both IPv6(linklocal, address) changed from the WAN-fe80::EUI-MAC (initial boot) to the LAN-fe80::EUI-MAC]
Disabled the shellcommand OK.
Rebooted the ALIX-on-2.2.6 (i386/32) OK.
Looked Status:Interfaces WAN-PPPoE addresses OK.
Tested browser IPv6 from a LAN host OK.
Inspected System.Log for the 1hr lease renewal OK.It works !
Sofar Sogood for me :)Then:
Tested Status:Interfaces Disconnect(~10sec)/Connect(~20sec) ADSL OK.
IPv6 still good, but both IPv6(linklocal, address) changed from the WAN-fe80::…b371 (WAN initial boot) to the LAN-fe80::...b370 (LAN).Thanks great solution.
-
@hda:
I'd be grateful if you can update this thread on whether this patch works and whether it is sufficient to ensure IPv6 addressing is the same on the first connect after reboot as on subsequent connections…
Applied your patch OK.
Tested Status:Interfaces Disconnect(~10sec)/Connect(~25sec) ADSL OK.
IPv6 still good, [both IPv6(linklocal, address) changed from the WAN-fe80::EUI-MAC (initial boot) to the LAN-fe80::EUI-MAC]
Disabled the shellcommand OK.
Rebooted the ALIX-on-2.2.6 (i386/32) OK.
Looked Status:Interfaces WAN-PPPoE addresses OK.
Tested browser IPv6 from a LAN host OK.
Inspected System.Log for the 1hr lease renewal OK.It works !
Sofar Sogood for me :)Then:
Tested Status:Interfaces Disconnect(~10sec)/Connect(~20sec) ADSL OK.
IPv6 still good, but both IPv6(linklocal, address) changed from the WAN-fe80::…b371 (WAN initial boot) to the LAN-fe80::...b370 (LAN).That sounds pretty encouraging.
I was hoping that the interface identifier on boot would be the same as on reconnection (in your case ending :b371), but the approach I used to selecting an interface to assign a temporary bogon IPv4 addresses in the version of the patch you tried is clearly wrong now I think about it.
When I get the time, I'll change the patch to assign a temporary IPv4 address to each interface that will have an IPv4 address when pfSense has booted up, start the PPP connection, then remove those temporary addresses when the PPP connection has started. This should result in the same interface identifier on boot as on subsequent reconnections, also it will work with non-PPPoE connections.
-
Running PPPoE from clean boot now for 24hrs OK without interrupts (no need for Disconnect/Connect). Patch by David_W recommended.
-
Hi David,
Is this fix going to get included for 2.3? I have the same issue and am running 2.3 right now and could help you test.
Thanks,
Robbert
-
@rrijkse:
Is this fix going to get included for 2.3? I have the same issue and am running 2.3 right now and could help you test.
My intention is to rework the patch as described a few posts ago, so that the interface ID immediately after reboot is the same as for subsequent reconnections. Once that is done, I'll rebase the patch on 2.3, post a link here and submit pull requests for RELENG_2_2 (2.2.x) and master (2.3).
/etc/inc/interfaces.inc is essentially identical in 2.2.x and 2.3, other than PPTP support having been removed from 2.3.
I will try to get this done as soon as possible, but I'm rather busy at the moment. M_Devil has suggested that the 2.2.6 patch posted earlier in this thread will apply OK on 2.3. Ordinarily applying a patch from one branch will not work on another, but it seems this will work in the interim.
-
Thanks, there is no rush, like I said I have a 2.3 install ready so let me know when you have a pull request ready for 2.3 and I'll try it out. I have had IPv6 disabled for a while since it is not stable enough and I'd rather apply the fix when its fully ready since I'm running in somewhat production right now.
Robbert
-
@hda:
Applied your patch OK.
Tested Status:Interfaces Disconnect(~10sec)/Connect(~25sec) ADSL OK.
IPv6 still good, [both IPv6(linklocal, address) changed from the WAN-fe80::EUI-MAC (initial boot) to the LAN-fe80::EUI-MAC]
Disabled the shellcommand OK.
Rebooted the ALIX-on-2.2.6 (i386/32) OK.
Looked Status:Interfaces WAN-PPPoE addresses OK.
Tested browser IPv6 from a LAN host OK.
Inspected System.Log for the 1hr lease renewal OK.It works !
Sofar Sogood for me :)Then:
Tested Status:Interfaces Disconnect(~10sec)/Connect(~20sec) ADSL OK.
IPv6 still good, but both IPv6(linklocal, address) changed from the WAN-fe80::…b371 (WAN initial boot) to the LAN-fe80::...b370 (LAN).That sounds pretty encouraging.
I was hoping that the interface identifier on boot would be the same as on reconnection (in your case ending :b371), but the approach I used to selecting an interface to assign a temporary bogon IPv4 addresses in the version of the patch you tried is clearly wrong now I think about it.
When I get the time, I'll change the patch to assign a temporary IPv4 address to each interface that will have an IPv4 address when pfSense has booted up, start the PPP connection, then remove those temporary addresses when the PPP connection has started. This should result in the same interface identifier on boot as on subsequent reconnections, also it will work with non-PPPoE connections.
I've now reworked the patch as outlined above. If you (and anyone else using this patch) can revert the patch, refetch it (the location is unchanged), apply it and test it again, I'd be grateful. In particular, I'd be grateful for confirmation that the IPv6 address of your PPP interface(s) is the same immediately after boot as on subsequent connections (Disconnect then Connect in Status->Interfaces).
If this version of the patch solves the issue with the previous version, I'll sort out a pull request in the hope of these fixes being merged into pfSense in due course.
-
…
In particular, I'd be grateful for confirmation that the IPv6 address of your PPP interface(s) is the same immediately after boot as on subsequent connections (Disconnect then Connect in Status->Interfaces)...pfSense 2.2.6 on DSL, no rfc4638 patch.
After reboot, checked Status-Interfaces:
it picks the Alix default LAN(fe80..b370)/nic vr0, not the WAN(..b_371)/vr1 (or OPT1(..b372)/vr2),
for Link-Local & Address, but it seems logical to select for lowest iface device.
Tested browser for IPv6 and both LAN's iface vr0 & vr2 OK !After ~20min. Disconnect/Connect, checked Status-Interfaces:
it still has the 1st LAN-MAC(fe80..b370)=vr0.
Tested browser for IPv6 and both iface vr0 & vr2 OK !After 1hr inspect Status-System Log, the lease-renewal: OK !
Tested browser for IPv6 and both iface vr0 & vr2 OK !It works :) Thank You again.
-
Those wanting a 2.3 version of the latest patch should install System Patches (if you haven't done so already) and create a patch as follows:
| Field | Contents |
| Description | PPP IPv6 fixes |
| URL/Commit ID | https://github.com/pfsense/pfsense/compare/master…davidjwood:ppp-ipv6.diff |
| Patch Contents | (leave blank} |
| Path Strip Count | 2 |
| Base Directory | / |
| Ignore Whitespace | (checked) |
| Auto Apply | (unchecked) |Press the Save button, then, if necessary, press 'Fetch' next to the patch. At this point, the option to 'Apply' should appear, so press 'Apply'.
If this solves the problems covered in this thread, I will submit this as a pull request.
-
M_Devil has commented that the 2.3 version of the patch works for them.
I think, finally, we've solved these two issues. I'll endeavour to get bug reports and pull requests sorted out as soon as time allows.
-
tl;dr The patches you have been testing so far do not deal properly with the full range of scenarios for dhcp6c. I need as many people as possible to test revised patches (for 2.2.5/2.2.6 and for 2.3) using the instructions at the foot of the post. If these patches prove successful, I'll submit pull requests.
Issue 1: Multiple dhcp6c processes on one interface issue, leading to broken IPv6 connectivity
Analysis
I now believe I understand the root cause of multiple dhcp6c processes running on the same interface, which typically leads to broken IPv6 connectivity on that interface.
interface_dhcp6_configure() is called unconditionally by interface_configure().
The normal scenario for DHCPv6 and/or DHCP-PD on PPPoE is to set "Use IPv4 connectivity as parent interface", which tells pfSense to run dhcp6c on the PPP interface rather than the parent interface of the PPP connection. When the interface gets an IPv4 address, /etc/rc.newwanip runs, which calls interface_dhcp6_configure() when "Use IPv4 connectivity as parent interface" is enabled. This created a possible race condition that could lead to two dhcp6c processes running on the same interface.
The ppp-ipv6 helper script (new in 2.2.5) calls interface_dhcp6_configure() when the IPv6CP layer starts on a PPP connection, but only when acceptance of router advertisements were disabled on the PPP interface. Acceptance of router advertisements were being enabled globally on all new interfaces as interface_dhcp6_configure() set the net.inet6.ip6.accept_rtadv sysctl to 1 for reasons that are unclear from the git logs. As sysctl -d notes, this sysctl controls the default for new interfaces, so there is no need for interface_dhcp6_configure() to set this sysctl to 1 merely to enable acceptance of router advertisements on an interface as ifconfig <interface>inet6 accept_rtadv is sufficient. Setting this sysctl to 1 meant that a newly created PPP interface would be created with acceptance of router advertisements switched on, so ppp-ipv6 would only call interface_dhcp6_configure() on an interface once IPv6CP had gone down and come back up at least once.
Fixing the problem
The patches you have all been testing up until now prevented the race by preventing the call to interface_dhcp6_configure() in interface_configure() when the interface is a PPP type interface. This logic is inadequate to deal with the full breadth of potential scenarios, though it happened to work correctly when "Use IPv4 connectivity as parent interface" is set on a PPP type interface (as is normal for PPP type interfaces).
I believe L2TP and PPTP connections should be treated the same way as PPPoE connections, as L2TP and PPTP typically use PPP as the transport layer. When I say "PPP type interfaces", I mean L2TP, PPTP, PPPoE and PPP interfaces.
You can have "Use IPv4 connectivity as parent interface" unset on a PPP connection, but doing so has no effect, as there are no parent network interfaces. That complicates the logic somewhat. With that in mind, I believe the correct rules are:
-
for all PPP type interfaces with "Use IPv4 connectivity as parent interface" as well as all PPP connections, interface_dhcp6_configure() should only be called when IPv6 is up, which means the ppp-ipv6 helper should be making this call.
-
for all connections with "Use IPv4 connectivity as parent interface" set, other than PPP type interfaces, interface_dhcp6_configure() should be called when an IPv4 address is obtained, which means this call should be made by /etc/rc.newwanip.
-
for all remaining connections (all connections with "Use IPv4 connectivity as parent interface" not set, but not including any PPP connections), interface_dhcp6_configure() should be called by interface_configure().
I believe I have implemented this logic correctly in the patches at the foot of this post.
Outstanding issues requiring further work
/etc/rc.newwanip makes no attempt to handle a second call for the same interface correctly. If, for example, a DHCP renewal results in a changed IPv4 address, which is a fairly common experience on some cable providers, interface_dhcp6_configure() will be called again. This may well result in two dhcp6c processes running on the same IPv6 interface, likely resulting in an IPv6 outage on that interface. This leads on to…
interface_dhcp6_configure() really needs reworking to handle subsequent calls for the same interface in a sane manner. I think the correct solution is to introduce an additional parameter, force_restart, which defaults to false. If force_restart is false, a second call to interface_dhcp6_configure() should exit without changing anything - this will probably require a volatile global array that stores which interfaces interface_dhcp6_configure() has been called on in order to prevent race conditions. If force_restart is true, rtsold and/or dhcp6c should be stopped on the interface, then interface_dhcp6_configure() restarted.
If this rework takes place, it would make sense to create a new function interface_dhcp6_stop() to stop rtsold and/or dhcp6c, which would allow some refactoring and simplification of the ppp-ipv6 script.
A further enhancement would be to add some logic that deals with scenarios where delegated prefix(es) change, either in a force_restart scenario or as dhcp6c continues to operate. In either case, track interface assignments are no longer valid but, so far as I recall, pfSense makes no attempt to handle this scenario.
IPv6 is a complex beast indeed!
Issue 2: Random interface identifiers when interfaces connect at boot time
Analysis
At boot time, the interface identifier is observed to be random, rather than a predictable value based on the MAC address of one of the interfaces in the machine. This not only results in a random link local address on the first connection after boot, but the interface identifier part of a SLAAC address is also the same random value. This means pfSense does not respond on its normal IPv6 address until a Disconnect and Connect is carried out using Status -> Interfaces. Clearly, this is a problem if you use pfSense's IPv6 address as a VPN endpoint.
The root cause is as I explained earlier in the thread. The IPv6CP code in mpd5 is responsible for this problem. The current code arguably goes against the spirit of section 4.1 of RFC 5072, which notes (emphasis added):
The non-zero value of the tentative interface identifier SHOULD be chosen such that the value is unique to the link and, preferably, consistently reproducible across initializations of the IPV6CP finite state machine (administrative Close and reOpen, reboots, etc.). The rationale for preferring a consistently reproducible unique interface identifier to a completely random interface identifier is to provide stability to global scope addresses (see Appendix A) that can be formed from the interface identifier.
On boot, /etc/rc.bootup calls interfaces_configure() in /etc/inc/interfaces.inc. This walks through the configured interfaces, initialising them sequentially. It may well be that the WAN interface is configured before any other interfaces on the machine are up, which is significant as CreateInterfaceID() in ipv6cp.c calls GetEther(NULL, &hwaddr) in an attempt to discover a hardware address to base the interface identifier on before falling back to a random interface identifier.
If you look at the definition of GetEther() in util.c, it ignores interfaces that only have point-to-point or loopback addresses even if they have a MAC address (i.e. an EUI-48). This failure to use an available EUI-48 violates section 4.1 of RFC 5072 as the RFC requires the use of any EUI-64 or EUI-48 on the machine before falling back to other sources of uniqueness and then a random interface identifier. CreateInterfaceID() should use MAC addresses of interfaces that have no addresses at all.
I suspect the fix will require changing to CreateInterfaceID() to follow the RFC more closely, basing the interface ID on the first valid value from:
-
the MAC address(es) of any interface used as part of the PPP link (if any - PPPoE is not necessarily in use)
-
the MAC address(es) of any other interface on the machine
-
GetEther() (as now - though this is arguably redundant as all MAC addresses should be considered by the first two steps)
-
randomness (as now)
Fixing the problem
An mpd5 bug needs opening about this issue, in the hope that CreateInterfaceID() will eventually be changed as described in the previous section. A redmine bug also needs opening to track this issue, not least as a record that the work round should be reverted if CreateInterfaceID() is changed to address this issue.
Until this issue is fixed in mpd5, it is possible to work round this issue by assigning temporary bogon IPv4 addresses to all interfaces that will eventually have an IPv4 or IPv6 address when a PPP type interface is initialised during bootup. Once mpd5 is running, the temporary addresses can be removed.
I am content this work round is behaving correctly - it solves the problem on my system and on hda's system.
Patches to test
IMPORTANT: After following the instructions to apply the patch for your pfSense version, you must either reboot or execute sysctl net.inet6.ip6.accept_rtadv=0 from a shell command prompt (e.g. via Diagnostics -> Command Prompt).
2.2.6 (or, possibly, 2.2.5)
Revert the RFC 4638 patch, if you have that installed.
Revert any previous patch from this thread.
Install the System Patches package (if you haven't done so already) and create a patch as follows. If you are updating from an older version of the patch, the URL has changed, so you need to use this new URL and blank the Patch Contents box.
| Field | Contents |
| Description | PPP IPv6 fixes |
| URL/Commit ID | https://github.com/pfsense/pfsense/compare/RELENG_2_2…davidjwood:RELENG_2_2-ppp-ipv6-new.diff |
| Patch Contents | (leave blank) |
| Path Strip Count | 1 |
| Base Directory | / |
| Ignore Whitespace | (checked) |
| Auto Apply | (unchecked) |Press the Save button, then, if necessary, press 'Fetch' next to the patch. At this point, the option to 'Apply' should appear, so press 'Apply'.
Apply the RFC 4638 patch, if you are using that.
Execute the sysctl command at the top of this section or reboot.
2.3
N.B. RFC 4638 functionality has now been merged into pfSense 2.3. If you are using the RFC 4638 patch, I suggest you update to the latest builds and remove the RFC 4638 patch from System Patches.
If you have the previous patch installed, revert it, re-fetch and apply, then execute the sysctl command at the top of this section or reboot.
If you don't have the previous patch installed, install System Patches (if you haven't done so already) and create a patch as follows:
| Field | Contents |
| Description | PPP IPv6 fixes |
| URL/Commit ID | https://github.com/pfsense/pfsense/compare/master…davidjwood:ppp-ipv6.diff |
| Patch Contents | (leave blank) |
| Path Strip Count | 2 |
| Base Directory | / |
| Ignore Whitespace | (checked) |
| Auto Apply | (unchecked) - or (checked) if you don't want to have to reapply the patch every time you update |Press the Save button, then, if necessary, press 'Fetch' next to the patch. At this point, the option to 'Apply' should appear, so press 'Apply'.
Execute the sysctl command at the top of this section or reboot.
Testing the patch
As might be apparent from the explanation of issue 1 above, this patch involves deeper changes to the code than I had previously thought necessary. This means it needs testing on a broad range of interfaces using SLAAC, DHCPv6 and/or DHCP-PD.
PPPoE connections and other PPP type connections (PPP to a modem or cellular device, L2TP, PPTP) using SLAAC, DHCPv6 and/or DHCP-PD
To confirm issue 1 is fixed, IPv6 connectivity should work immediately after boot, and should stay working across several dhcp6c renewal cycles. IPv6 connectivity should be re-established correctly after a Disconnect and Connect in Status -> Interfaces (or the interface being disabled, Apply Changes, enabled, Apply Changes) and should stay working across several dhcp6c renewal cycles.
To confirm issue 2 is fixed, the link local IPv6 address and any global IPv6 interface address should be the same immediately after boot as on a subsequent connection (e.g. Disconnect and Connect in Status -> Interfaces).
Non-PPP type connections using SLAAC, DHCPv6 and/or DHCP-PD (e.g. cable modem with native IPv6)
To confirm issue 1 is fixed, IPv6 connectivity should work immediately after boot, and should stay working across several dhcp6c renewal cycles. IPv6 connectivity should be re-established correctly after a Disconnect and Connect in Status -> Interfaces (or the interface being disabled, Apply Changes, enabled, Apply Changes) and should stay working across several dhcp6c renewal cycles.
Issue 2 does not apply to these connection types.</interface>
-
-
Thanks for the very detailed explanation David, I have been running the original 2.3 patch since last night and it has so far been far more stable than without it.
I will update to the new patch tonight and let you know how it goes.
Thanks,
Robbert
-
Testing the patch RELENG_2_2…davidjwood:RELENG_2_2-ppp-ipv6-new.diff
Alix2D13/i386 with 2.2.6 /no rfc4638 patch. PPPoE/DHCP6c (PD) & Advanced for IPv6 fixed /48, with Static & RA and DHCPv6-Server & RA for LAN's configged. (so no Track Interface).
REBOOT:
(non-fatal) crash report: PHP Parse error: syntax error, unexpected end of file in /etc/rc.newwanip on line 255.
Link-Local & Address == fe80::b370/vr0 :: OK.
tested browser IPv6 vr0 & vr2 OK.DISCONNECT/CONNECT:
~10min : Link-Local & Address == fe80::b370/vr0 :: OK.
(non-fatal) crash report: PHP Parse error: syntax error, unexpected end of file in /etc/rc.newwanip on line 255.
tested browser IPv6 vr0 & vr2 OK.INTERFACES-WAN ((CHANGE) || SAVE):
Link-Local & Address == fe80::b370/vr0 :: OK.
(non-fatal) crash report: PHP Parse error: syntax error, unexpected end of file in /etc/rc.newwanip on line 255.
tested browser IPv6 vr0 & vr2 OK.LEASE-RENEWAL:
1hr renewal no entry in syslog…
ultimate 2hr renewal... Nope. IPv6 discontinuedNot good enough for me so I revert back to previous patch for the time being :D
-
Hi David,
Thanks a lot for your work on this issue and on the RFC 4638 patch.
I've installed the patch on a 2.2.6 machine with a dual stack connection provided by my ISP through a PPPoE connection. My ISP (RDS-RCS) is delegating a dynamic /64.
Current settings on WAN interface:
-Use IPv4 connectivity as parent interface - check
-Request only an IPv6 prefix - unckeck
-DHCPv6 Prefix Delegation size - 64
-Send IPv6 prefix hint - check
LAN interface is set to track interface.Now, without the patch i had a hard time acquiring an IPV6 address and delegated prefix (it involved multiple reboots or a combination of re-saving general setup settings and/or re-saving WAN configuration).
After i applied the patch the system acquired the IPV6 address and a delegated prefix without a hitch after every reboot. I'm testing remote atm so i cannot disconnect/connect the wan interface right now. The only problem is that after rebooting, pfsense is trowing out a crash report complaining about "syntax error, unexpected end of file in /etc/rc.newwanip on line 255"
Here are some SS of Status:Interfaces (first without patch , 2nd after applying the patch and rebooting 3rd after another reboot)…
Later edit:
The link local IPV6 address of the parent interface for the pppoe connection (em0) is identical between connect/disconnect attempts (based on mac address of em0). The pppoe link local changes every time.
![with_ patch_2nd_boot.png](/public/imported_attachments/1/with_ patch_2nd_boot.png)
![with_ patch_2nd_boot.png_thumb](/public/imported_attachments/1/with_ patch_2nd_boot.png_thumb) -
Hi David,
I just made a post inquiring about a not-getting-an-IPv6-on-pppoe problem before I found this thread.
My setup is described here: https://forum.pfsense.org/index.php?topic=104895.0I've tried applying your patch to see if I could get ipv6 on WAN, but it doesn't appear to work for my problem.
If there's anything I can do to help, I'd be happy to.
Screenshot from first boot after applying the patch attached.
-
@hda:
(non-fatal) crash report: PHP Parse error: syntax error, unexpected end of file in /etc/rc.newwanip on line 255.
Bother. Such are the perils of rebasing patches from master (2.3) to RELENG_2_2 (2.2), as updated coding style has been applied to all code in 2.3.
The fix is trivial - I've pushed it to the git repository, so if you revert, re-fetch and reapply the patch using the instructions in the earlier post, everything should work correctly now.
-
I've tried applying your patch to see if I could get ipv6 on WAN, but it doesn't appear to work for my problem.
You have to wait until @M_Devil chimes in. He has a 500/500 working (on Track Interface) I understood.
Did you try my config in: https://forum.pfsense.org/index.php?topic=103990.msg579876#msg579876 on the vLAN in pfSense
or are you ?And you know the "permanent" /48 you have, in order to config static LAN(s) ;)
-
@hda:
I've tried applying your patch to see if I could get ipv6 on WAN, but it doesn't appear to work for my problem.
You have to wait until M_Devil chimes in. He has a 500/500 I understood.
Did you try my config in: https://forum.pfsense.org/index.php?topic=103990.msg579876#msg579876 on the vLAN in pfSense
or are you ?And you know the "permanent" /48 you have, in order to config static LAN(s) ;)
I had not set dhcp6 advanced options. I've just tried it, and it does not appear to make a difference. IPv6 still comes up on LAN reliably, and local machines get assigned IP's as well, but no IPv6 route and no IP for the WAN.
Note: my WAN is set up as PPPOE(igb0,igb0_vlan6). IP connectivity is on vlan6, IPTV is on vlan4 (which I haven't gotten to work just yet)
![Screen Shot 2016-01-06 at 20.40.17.png](/public/imported_attachments/1/Screen Shot 2016-01-06 at 20.40.17.png)
![Screen Shot 2016-01-06 at 20.40.17.png_thumb](/public/imported_attachments/1/Screen Shot 2016-01-06 at 20.40.17.png_thumb) -
I just made a post inquiring about a not-getting-an-IPv6-on-pppoe problem before I found this thread.
My setup is described here: https://forum.pfsense.org/index.php?topic=104895.0I've tried applying your patch to see if I could get ipv6 on WAN, but it doesn't appear to work for my problem.
If there's anything I can do to help, I'd be happy to.
Screenshot from first boot after applying the patch attached.I've just replied in your other thread.
I urge you to give MTU 1500 operation a go using RFC 4638, as I described in that thread - it should work with your setup, assuming bhyve and your NIC is capable of using jumbo frames on your WAN interface.
-
I just made a post inquiring about a not-getting-an-IPv6-on-pppoe problem before I found this thread.
My setup is described here: https://forum.pfsense.org/index.php?topic=104895.0I've tried applying your patch to see if I could get ipv6 on WAN, but it doesn't appear to work for my problem.
If there's anything I can do to help, I'd be happy to.
Screenshot from first boot after applying the patch attached.I've just replied in your other thread.
I urge you to give MTU 1500 operation a go using RFC 4638, as I described in that thread - it should work with your setup, assuming bhyve and your NIC is capable of using jumbo frames on your WAN interface.
MTU 1500 works as advertised. But I'm not getting an IPv6 on WAN (yet)
-
-
Steps
revert previous patch, delete it
update to newest built:
2.3-BETA (amd64)
built on Wed Jan 06 07:53:18 CST 2016
FreeBSD 10.2-STABLEcreate newest patch (from your description), fetch it
After fetch, test it, but…
Apply = clean
Reverted = not clean/usr/bin/patch --directory=/ -f -p2 -i /var/patches/568d8e2e969e1.patch --check --reverse --ignore-whitespace Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |diff --git a/src/etc/inc/interfaces.inc b/src/etc/inc/interfaces.inc |index 49fd2ca..73879ce 100644 |--- a/src/etc/inc/interfaces.inc |+++ b/src/etc/inc/interfaces.inc -------------------------- Patching file etc/inc/interfaces.inc using Plan A... Hunk #1 failed at 1998. Hunk #2 failed at 2020. Hunk #3 failed at 3349. Hunk #4 failed at 3958. 4 out of 4 hunks failed while patching etc/inc/interfaces.inc Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -------------------------- |diff --git a/src/etc/rc.newwanip b/src/etc/rc.newwanip |index 45cef96..5bc6a61 100755 |--- a/src/etc/rc.newwanip |+++ b/src/etc/rc.newwanip -------------------------- Patching file etc/rc.newwanip using Plan A... Hunk #1 failed at 170. 1 out of 1 hunks failed while patching etc/rc.newwanip Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -------------------------- |diff --git a/src/usr/local/sbin/ppp-ipv6 b/src/usr/local/sbin/ppp-ipv6 |index aa0536c..4d47de2 100755 |--- a/src/usr/local/sbin/ppp-ipv6 |+++ b/src/usr/local/sbin/ppp-ipv6 -------------------------- Patching file usr/local/sbin/ppp-ipv6 using Plan A... Hunk #1 failed at 23. Hunk #2 failed at 63. 2 out of 2 hunks failed while patching usr/local/sbin/ppp-ipv6 done
Apply patch
rebootTest 1
after reboot no IPv6 traffic
Multiple Interface disconnect-connects, still no IPv6Test 2
Same after several reboots:- IPv6 Link Local
- IPv6 Address
- Gateway IPv6
Leaving me in a non operationel IPv6 state now.
If you like, I can do more testsEdit: If I find the time, I will reinstall 2.3.b and recover the configuration to make sure the problem has nothing to do with all the updates I performed.
-
After uninstalling patch, again updated to newest (2.3.b 7-jan 2:26) version of pfSense refetch, apply en reboot IPv6 is working again. Also after several reboots
Don't have a clue why it did't work te last attempt, maybe I made an configuration mistake (It was late on the evening…)
-
After uninstalling patch, again updated to newest (2.3.b 7-jan 2:26) version of pfSense refetch, apply en reboot IPv6 is working again. Also after several reboots
Don't have a clue why it did't work te last attempt, maybe I made an configuration mistake (It was late on the evening…)
It's good news that things are now stable with your IPv6 connectivity.
It would be useful if you'd try the patch again. It hasn't changed since you last tried it.
Please also give outline details of the connection - especially the settings for "IPv4 Configuration Type", "IPv6 Configuration Type" and "Use IPv4 connectivity as parent interface" (in the "DHCP6 Client Configuration" section).
I hope to reach a point of having confidence in this patch so that I can submit a pull request. I'm confident in the work-round used for Issue 2 (a proper fix needs changes in mpd5). Though I now believe I understand Issue 1 well, I would like some additional user experience of the fix for Issue 1 before submitting a pull request because of the complexity of the underlying issues.