NPt not working on WAN interface when IPv6 is provided via a 6RD tunnel. Bug?
-
Hi all,
A little background on where I'm eventually trying to get to.
I'm a home user working to get MultiWAN IPv6 with failover up and running with the use of ULA addresses. I have COX cable (1G/40M) who provides IPv6 via DHCPv6 and CenturyLink DSL (80M/40M) who provides IPv6 connectivity via a 6rd Tunnel. Cox has a data cap associated with their plan whereas my CenturyLink plan has no data cap.
I want to use ULA addresses for the purpose of eventually routing traffic for specific computers on my network to use Cox (faster but data cap) and have all other devices use CenturyLink, with failover for all devices if one of the connections fails.I have been able to get MultiWAN IPv6 using a single ULA prefix with failover working but believe I've stumbled across a bug with NPTv6 for 6rd interfaces in the process.
My Initial Setup was as follows but with this configuration I could not ping IPv6 addresses from my LAN when CenturyLink (6rd) was the default ISP. However, I can ping if I would pull my CenturyLink connection and the failover to Cox occurs then IPv6 is correctly routed over my LAN. It should also be noted that I can ping IPv6 addresses in this setup from the shell (even with WAN1 disconnected) :
PfSense 2.4.5-RELEASE-p1
Interfaces:
- WAN0 - CenturyLink (Static IP)
- IPv4 - PPPoE
- IPv6 - 6rd Tunnel
- 6RD Prefix - 2602::/24
- 6RD Border relay - 205.171.2.64
- WAN1 - Cox
- IPv4 - DHCPv6
- IPv6 - DHCPv6
- LAN
- IPv4 Configuration Type - Static IPv4
- IPv4 Address - 192.168.X.X/24
- IPv6 Configuration Type - Static IPv6
- IPv4 Address - fd83:XXXX:XXXX:XXXX::1/64 (was randomly generated)
- IPv4 Configuration Type - Static IPv4
Gateways:
- WANGWGR (Default gateway IPv4 )
- WAN_PPPOE (CenturyLink - Tier 1)
- WAN_DHCP (Cox - Tier 2)
- WANGWGR_IPv6 (Default gateway IPv6 )
- WAN_6RD (CenturyLink v6)
- WAN_DHCP6 (Cox v6)
DHCPv6: Configured for the full range of /64 ULA prefix
NPt:
- Interface - WAN0
- External: 2602:XXXX:XXXX:XXXX::/64
- Internal: fd83:XXXX:XXXX:XXXX::1/64
- Interface - WAN1
- External: 2600:XXXX:XXXX:XXXX::/64
- Internal: fd83:XXXX:XXXX:XXXX::1/64
I was able to get things working (i.e. 6rd IPv6 traffic routed over my LAN with NPt and ULA addresses) by doing the following:
- In Interfaces/Assignments there was Interface I could add to Network Port "opt1_stf" (The CenturyLink 6rd tunnel interface). This might be "wan_stf" for most people.
a) It should be noted that the adding of this interface shouldn't be allowed as upon reboot this will cause the boot sequence to hang due to the error "Warning: Configuration references interfaces that do not exist" because the 6rd interface isn't up yet. This issue is being resolved in 2.5.0 under bug but this fix will prevent my workaround: https://redmine.pfsense.org/issues/10626#change-46806 - I added the interface to the appropriate Network Port and did NOT change any settings or enable the interface
- Next in Interfaces/Interface Groups I had to add the newly created Interface (OPT1 in my case) to an Interface group also containing the WAN0 interface with the 6rd connection which I called (GRP_WAN)
- I then updated my previously created NPt rule to use the GRP_WAN instead of WAN0
This allowed me to get IPv6 connectivity while using a ULA address in a multi-wan configuration.
What led me to try the above steps was that when looking in Diagnostics/States and filtering on v6 I saw what looked like appropriate NPt entries for the WAN1 (Cox connection) but nothing analogous for WAN0 (CenturyLink 6rd). I did however see "opt1_stf" in the interface list with IPv6 source/destination that I would have expected to see listed for the WAN0 interface.
Based on the above to me the problem appears to be that NPT rules that should get bound the the virtual 6rd interface when an NPT rule is configured for the physical WAN interface do not get applied. I thought this might be resolved by the bug fix outlined in this bug report: https://redmine.pfsense.org/issues/7142
But when I updated my "etc/inc/filter.inc" file to match the change outlined in the commit for the above bug report and reverted my workaround for the issue I was still unable to get IPv6 traffic routed to my LAN via the 6rd interface.
Is there something I'm missing or should this be logged as a bug?
- WAN0 - CenturyLink (Static IP)
-
did you change this too?
https://redmine.pfsense.org/projects/pfsense/repository/revisions/bae04c3730d58f59a71ec3801e0e144f6452b7a6/diff
and this of course:
https://github.com/pfsense/pfsense/commit/bae04c3730d58f59a71ec3801e0e144f6452b7a6 -
@Alitai Yes, I did update the filter.inc file and then rebooted and then reboot but still had the same issue once I reverted the changes for my workaround. I did mention this in my initial post but I should clarify the file was updated in "/etc/inc/filter.inc" I pasted the wrong file location (included /src/) in my original post.
Both of the links you posted appear to be for fix referenced in bug #7142 was there another change you also meant to link to?
-
I'm wondering if this is due to a similar bug to what was corrected in #7142. In /etc/inc/filter.inc/filter_nat_rules_generate() I see where the NPt filtering rules get applied but it only appears to occur for the main interface. For 6rd and 6to4 interfaces is there additional logic needed to correctly apply the NPt filters rules to the virtual interfaces?
-
Fixed in https://redmine.pfsense.org/issues/7142
You need to apply Patch ID bae04c3730d58f59a71ec3801e0e144f6452b7a6 -
@viktor_g Thanks for your reply. I did stumble across bug #7142 in my initial attempt to resolve this problem and I have replaced my filter.inc file with the one linked to in the patch ID you provided. However, after applying this update I'm still not seeing the NPt rules being appropriately applied to the _stf interface. I did verify that the changes were present in the file and tried both a filter reload and a reboot of the system to ensure they were in effect. I did notice there is another place in filter.inc where the NPt rules are generated (/etc/inc/filter.inc/filter_nat_rules_generate()). Does something similar to the fix in #7142 need to be done in that function as well? Is it possible there are also other patches I need ( I did see there were some other 6rd bug fixes targeted for the 2.5 release)?
I'm happy to provide any additional information/logs if that would be helpful. Any additional thoughts/suggestions are appreciated. -
@risola316 Did you apply Patch bae04c3730d58f59a71ec3801e0e144f6452b7a6 ?
I have successfully tested the firewall rules, NPt and 1:1 NAT after applying this patch.
Please show the/tmp/rules.debugandpfctl -s rulesafter applying this patch and system reboot. -
@viktor_g I do believe that I applied the appropriate patch. Please see below for the relevant section of the patched file.
I wasn't able to paste the info into my post so I've included it in the following links.
/tmp/rules.debug
https://pastebin.com/wptxDd4d
pfctl -s rules
https://pastebin.com/9rJa52QP
Thank you again for your help.
-
@risola316 That seems correct:
WAN0_CNT = "{ pppoe0 opt1_stf }" ... binat on $WAN0_CNT inet6 from fd83:8359:2e69:54d5::/64 to any -> 2602:d8:a1aa:bd00::/64 binat on $WAN0_CNT inet6 from any to 2602:d8:a1aa:bd00::/64 -> fd83:8359:2e69:54d5::/64But it seems that you forget fw pass rules 'from outside to 2602:d8:a1aa:bd00::/64'
Also please show the output of
pfctl -s nat -
@viktor_g said in NPt not working on WAN interface when IPv6 is provided via a 6RD tunnel. Bug?:
pfctl -s nat
I'm not sure what might cause the fw pass rules for outside not to be set. I'm using the default ruleset that was configured for the 6rd gateway/firewall. These are also the same I have configured for my other WAN (DHCPv6) which I am able to get connectivity to from the LAN.
Here is the output of the requested command
#anchor(pfctl -s nat)[2.4.5-RELEASE][admin@pfSense.localdomain]/root: pfctl -s nat no nat proto carp all nat-anchor "natearly/*" all nat-anchor "natrules/*" all nat on vmx2 inet proto tcp from 192.168.13.0/24 to 192.168.100.0/24 -> (vmx2) port 1024:65535 round-robin nat on vmx1 inet proto tcp from 192.168.13.0/24 to 192.168.0.0/24 -> 192.168.0.13 port 1024:65535 nat on vmx2 inet from <tonatsubnets> to any port = isakmp -> (vmx2) round-robin static-port nat on vmx2 inet6 from <tonatsubnets> to any port = isakmp -> (vmx2) round-robin static-port nat on vmx2 inet from <tonatsubnets> to any -> (vmx2) port 1024:65535 round-robin nat on vmx2 inet6 from <tonatsubnets> to any -> (vmx2) port 1024:65535 round-robin nat on pppoe0 inet from <tonatsubnets> to any port = isakmp -> 216.161.170.189 static-port nat on opt1_stf inet from <tonatsubnets> to any port = isakmp -> 216.161.170.189 static-port nat on pppoe0 inet6 from <tonatsubnets> to any port = isakmp -> 2602:d8:a1aa:bd00:: static-port nat on opt1_stf inet6 from <tonatsubnets> to any port = isakmp -> 2602:d8:a1aa:bd00:: static-port nat on pppoe0 inet from <tonatsubnets> to any -> 216.161.170.189 port 1024:65535 nat on opt1_stf inet from <tonatsubnets> to any -> 216.161.170.189 port 1024:65535 nat on pppoe0 inet6 from <tonatsubnets> to any -> 2602:d8:a1aa:bd00:: port 1024:65535 nat on opt1_stf inet6 from <tonatsubnets> to any -> 2602:d8:a1aa:bd00:: port 1024:65535 no rdr proto carp all rdr-anchor "tftp-proxy/*" all rdr-anchor "miniupnpd" all binat on vmx2 inet6 from fd83:8359:2e69:54d5::/64 to any -> 2600:8800:4700:3820::/64 binat on vmx2 inet6 from any to 2600:8800:4700:3820::/64 -> fd83:8359:2e69:54d5::/64 binat on pppoe0 inet6 from fd83:8359:2e69:54d5::/64 to any -> 2602:d8:a1aa:bd00::/64 binat on pppoe0 inet6 from any to 2602:d8:a1aa:bd00::/64 -> fd83:8359:2e69:54d5::/64 [2.4.5-RELEASE][admin@pfSense.localdomain]/root:One other thing I am surprised to see, is "opt1_stf" showing up in Diagnostics/States as I thought that should reflect WAN0_CNT
This opt1_stf disappears when I do the workaround described in my original post.
-
@viktor_g I'm not sure if this will help but I figured I would post the items your requested for the case where the workaround as described above is applied. After the workaround is applied I do have IPv6 connectivity from my LAN, but this workaround poses issues as I should not be able to create an interface assignment for the _stf interface.
Post Workaround: pfctl -s rules
https://pastebin.com/0smufdep
Post Workaround: pfctl -s rules
https://pastebin.com/iq6zyBqS
Post Workaround: pfctl -s nat
Maybe this issue is related to the combination of IPV6 via 6rd over a pppoe connection. Looking at the binat rules in the case that doesn't work the only difference I can see is that the NPt rules are being applied to the pppoe interface instead of to the opt1_stf interface. In the binat instruction that works the GRP_WAN_CNT is used which consists of (WAN0_CNT, opt1_stf).
[2.4.5-RELEASE][admin@pfSense.localdomain]/root: pfctl -s nat no nat proto carp all nat-anchor "natearly/*" all nat-anchor "natrules/*" all nat on vmx2 inet proto tcp from 192.168.13.0/24 to 192.168.100.0/24 -> (vmx2) port 1024:65535 round-robin nat on vmx1 inet proto tcp from 192.168.13.0/24 to 192.168.0.0/24 -> 192.168.0.13 port 1024:65535 nat on vmx2 inet from <tonatsubnets> to any port = isakmp -> (vmx2) round-robin static-port nat on vmx2 inet6 from <tonatsubnets> to any port = isakmp -> (vmx2) round-robin static-port nat on vmx2 inet from <tonatsubnets> to any -> (vmx2) port 1024:65535 round-robin nat on vmx2 inet6 from <tonatsubnets> to any -> (vmx2) port 1024:65535 round-robin nat on pppoe0 inet from <tonatsubnets> to any port = isakmp -> 216.161.170.189 static-port nat on opt1_stf inet from <tonatsubnets> to any port = isakmp -> 216.161.170.189 static-port nat on pppoe0 inet6 from <tonatsubnets> to any port = isakmp -> 2602:d8:a1aa:bd00:: static-port nat on opt1_stf inet6 from <tonatsubnets> to any port = isakmp -> 2602:d8:a1aa:bd00:: static-port nat on pppoe0 inet from <tonatsubnets> to any -> 216.161.170.189 port 1024:65535 nat on opt1_stf inet from <tonatsubnets> to any -> 216.161.170.189 port 1024:65535 nat on pppoe0 inet6 from <tonatsubnets> to any -> 2602:d8:a1aa:bd00:: port 1024:65535 nat on opt1_stf inet6 from <tonatsubnets> to any -> 2602:d8:a1aa:bd00:: port 1024:65535 no rdr proto carp all rdr-anchor "tftp-proxy/*" all rdr-anchor "miniupnpd" all binat on vmx2 inet6 from fd83:8359:2e69:54d5::/64 to any -> 2600:8800:4700:3820::/64 binat on vmx2 inet6 from any to 2600:8800:4700:3820::/64 -> fd83:8359:2e69:54d5::/64 binat on GRP_WAN_CNT inet6 from fd83:8359:2e69:54d5::/64 to any -> 2602:d8:a1aa:bd00::/64 binat on GRP_WAN_CNT inet6 from any to 2602:d8:a1aa:bd00::/64 -> fd83:8359:2e69:54d5::/64 [2.4.5-RELEASE][admin@pfSense.localdomain]/root:States showing NPt rules applied on _stf interface (OPT6)
Thanks
-
@viktor_g I added some additional log messages to the filter Reload Status and it appears that floating rules are only generated for the WAN interface. They are not generated for the OPT1 interface even thought it is configured as a WAN.
Filter Reload Status
Initializing Creating aliases Creating gateway group item... Generating Limiter rules Generating NAT rules - Test Creating 1:1 rules... 1) The interface: wan was found for NPt rules. 2) The interface: wan is being setup for NPt rules. 3) The interface: WAN1_COX is having it's NPt rules saved. 1) The interface: opt1 was found for NPt rules. 2) The interface: opt1 is being setup for NPt rules. 3) The interface: WAN0_CNT is having it's NPt rules saved. Creating advanced outbound rule Cox Modem Management Access (.100.1) Creating advanced outbound rule CenturyLink Modem Management Access (.0.1) Creating outbound NAT rules Creating automatic outbound rules Setting up TFTP helper Generating filter rules Creating default rules Pre-caching ICMPv6 Allow Rules... Creating filter rule ICMPv6 Allow Rules ... Creating filter rules ICMPv6 Allow Rules ... 1) Creating floating filter rules for interface wan ... 2) array_key_exists for interface wan ... 3a) isset(Array['if']) for interface wan ... Setting up pass/block rules Setting up pass/block rules ICMPv6 Allow Rules Creating rule ICMPv6 Allow Rules Pre-caching Default allow LAN to any rule... Creating filter rule Default allow LAN to any rule ... Creating filter rules Default allow LAN to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN to any rule Creating rule Default allow LAN to any rule Pre-caching Default allow LAN IPv6 to any rule... Creating filter rule Default allow LAN IPv6 to any rule ... Creating filter rules Default allow LAN IPv6 to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN IPv6 to any rule Creating rule Default allow LAN IPv6 to any rule Creating IPsec rules... Creating uPNP rules... Generating ALTQ queues Loading filter rules Setting up logging information Setting up SCRUB information Processing down interface states Running plugins DoneAdditional Logging: filter.inc
filter.inc_filter_nat_rules_generate
/* Add binat rules for Network Prefix translation */ if (is_array($config['nat']['npt'])) { foreach ($config['nat']['npt'] as $rule) { if (isset($rule['disabled'])) { continue; } if (!$rule['interface']) { $natif = "wan"; } else { $natif = $rule['interface']; } update_filter_reload_status(sprintf(gettext("1) The interface: %s was found for NPt rules."), $natif)); if (!isset($FilterIflist[$natif])) { continue; } update_filter_reload_status(sprintf(gettext("2) The interface: %s is being setup for NPt rules."), $natif)); $srcaddr = filter_generate_address($rule, 'source'); $dstaddr = filter_generate_address($rule, 'destination'); $srcaddr = trim($srcaddr); $dstaddr = trim($dstaddr); $natif = $FilterIflist[$natif]['descr']; /* Do not form an invalid NPt rule. * See https://redmine.pfsense.org/issues/8575 */ if (!(is_subnetv6($srcaddr) || is_ipaddrv6($srcaddr)) || !(is_subnetv6($dstaddr) || is_ipaddrv6($dstaddr))) { continue; } update_filter_reload_status(sprintf(gettext("3) The interface: %s is having it's NPt rules saved."), $natif)); $natrules .= "binat on \${$natif} inet6 from {$srcaddr} to any -> {$dstaddr}\n"; $natrules .= "binat on \${$natif} inet6 from any to {$dstaddr} -> {$srcaddr}\n"; } }filter.inc_filter_generate_user_rule
/* Check to see if the interface is in our list */ if (isset($rule['floating'])) { if (isset($rule['interface']) && $rule['interface'] <> "") { $interfaces = explode(",", $rule['interface']); $ifliste = ""; foreach ($interfaces as $iface) { update_filter_reload_status(sprintf(gettext("1) Creating floating filter rules for interface %s ..."), $iface)); if (array_key_exists($iface, $FilterIflist)) { update_filter_reload_status(sprintf(gettext("2) array_key_exists for interface %s ..."), $iface)); if (isset($FilterIflist[$iface]['if'])) { update_filter_reload_status(sprintf(gettext("3a) isset($FilterIflist[$iface]['if']) for interface %s ..."), $iface)); $ifliste .= " " . $FilterIflist[$iface]['if'] . " "; if (($FilterIflist[$iface]['type6'] == '6rd') || ($FilterIflist[$iface]['type6'] == '6to4')) { update_filter_reload_status(sprintf(gettext("3) 6rd/6to4 for interface %s ..."), $iface)); $ifliste .= " " . $FilterIflist[$iface]['ifv6'] . " "; } } elseif (isset($FilterIflist[$iface][0]['if'])) { update_filter_reload_status(sprintf(gettext("3b) isset($FilterIflist[$iface][0]['if']) for interface %s ..."), $iface)); $ifliste .= " " . $FilterIflist[$iface][0]['if'] . " "; } } }===
-
@risola316 Thanks for info
Please check https://redmine.pfsense.org/issues/10757#note-3
(apply it after #7142) -
@viktor_g Thanks so much for your help. After applying the second patch things are working as expected. Thanks again.