PPPoE IPv6 + Hurricane Electrics IPv6 - Assymmetric routing / no ping from External to pppoe assigned IPv6 Adddress possibile, despite that, everything is working...?
-
Hi,
I just encountered a strange Problem accordning interface assigned routing, resulting in non working assymetric routing.
My setup consists of a couple of internal vlans, both IPv6 + IPv4.
All internal VLANs get a IPv6 Network by my Hurricane Electrics Tunnel, since they are static an nearly as fast as my providers IPv6.The default Gatetway for the IPv6 Internet is the Hurricane electrics Tunnel.
So, I get only one IP Address from my provider (Deutsche Telekom) on my pppoe Interface - in the routing
context I see the default gw is the fe80::103:103:3f9a:fa33....Link Local Address... okay, that might work....
Despite that, with netstat I see there is also a interface route to pppoe[2.4.4-RELEASE][root@router]/root: netstat -n -6 -r | grep ppp
2003:0:0:ad64::/64 link#26 U pppoe0
2620:fe::9 pppoe0 UHS pppoe0
fe80::%pppoe0/64 link#26 U pppoe0
fe80::329c:23ff:fe21:whatever%pppoe0 link#26 UHS lo0So far so good.
The funny thing is, I use the IPv6 assigned IP Address on the pfsense for connecting to two ipsec endpoints in the internet which works fine - despite the fact it seems to problematic regarding pmtu.
After further investigations, the reason for this behaviour is, that icmpv6 replies are sent out the wrong interface for that address-range.[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi pppoe0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 10:12:38.779234 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 17, length 64 10:12:39.803005 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 18, length 64 10:12:40.823277 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 19, length 64 10:12:41.847030 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 20, length 64 10:12:42.871295 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 21, length 64 ^C 5 packets captured 656 packets received by filter 0 packets dropped by kernelIf I Initiate a (outgoing) ping on the pppoe Ipv6 Interface, everything is working fine:
[2.4.4-RELEASE][root@rotorouter]/root: tcpdump -nnfi pppoe0 icmp6 and host 2620:fe::9 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 10:16:24.688594 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6558, length 8 10:16:24.703858 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6558, length 8 10:16:25.195955 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6559, length 8 10:16:25.210875 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6559, length 8 10:16:25.699812 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6560, length 8 10:16:25.714615 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6560, length 8 10:16:26.210344 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6561, length 8 10:16:26.225103 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6561, length 8 10:16:26.734244 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6562, length 8 10:16:26.749107 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6562, length 8 ^C 10 packets captured 331 packets received by filter 0 packets dropped by kernelOutgoing pings via the WAN Interfae work fine and are answered on the same Network Interface.
But Incoming Pings are reply'ed on the wrong interface, the leave the firewall on the hurricane electrics tunnel, not on the interface they are received in the first place:
[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi gif0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9 tcpdump: WARNING: foreign (-f) flag used but: gif0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes 10:18:31.035483 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 361, length 64 10:18:32.055503 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 362, length 64 10:18:33.079271 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 363, length 64 10:18:34.103536 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 364, length 64 10:18:35.127287 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 365, length 64 10:18:36.155275 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 366, length 64 ^C 6 packets captured 144 packets received by filter 0 packets dropped by kernelIs this a known - whilst buggy? - behaviour or do I miss anything in my configuration? Also I thing I remember it was working fine a couple of years ago... ( I use this setup since about 2013 or so)
Normally, despite of any "default route" every packed should be answered on the same interface it is received in the first place, or am I wrong?
However, what could I do to gain such interface based default routes again?
Thanks a lot!
Cheers,
4920441
-
Do you have IPv6 enabled on the PPPoE interface, if yes why? Disable it there, you don't need it with a he.net tunnel.
-
@4920441-0 said in PPPoE IPv6 + Hurricane Electrics IPv6 - Assymmetric routing / no ping from External to pppoe assigned IPv6 Adddress possibile, despite that, everything is working...?:
The funny thing is, I use the IPv6 assigned IP Address on the pfsense for connecting to two ipsec endpoints in the internet which works fine - despite the fact it seems to problematic regarding pmtu.
First, that has nothing to do with this Problem, because yes off course If I had only one IPv6 default route the problem would exist, because there is only one route to go - yes of course I have enabled IPv6 on PPPoE else there would not be an IPv6 Address nor could it work in any way I have described...
Since when shouldn't pfsense be able to have more than one default route in the first place?
Second, I use the native IPv6 - as mentioned above - ('The funny thing is, I use the IPv6 assigned IP Address on the pfsense for connecting to two ipsec endpoints in the internet which works fine - despite the fact it seems to problematic regarding pmtu.') for two Ipsec connections, since the latency is only about 8 ms compared to appx 18 ms its much nicer this way.
The routing itself, if it is initated by pfsense itself works without problems, but the problem lies in the incoming queue somewhere since received icmp6 packets are sent out to the wrong interface.
Only using one interface is not a solution....
Cheers,
4920441
-
@4920441-0 said in PPPoE IPv6 + Hurricane Electrics IPv6 - Assymmetric routing / no ping from External to pppoe assigned IPv6 Adddress possibile, despite that, everything is working...?:
Since when shouldn't pfsense be able to have more than one default route in the first place?
Read up on what a default route is, you can only have one for IPv4 and one for IPv6. Use policy routing if you need more than one gateway. -
Sure - regarding I would like to route packets from an internal network to another network.
Since this is a interface route (Packet comes from somewhere in pppoe Interface) the reply has to be sent out through the interface it came in (pppoe).Your suggestion were true, if it would not work initiating a connection via the IPv6 part directly on the firewall or through a connected IPv6 Subnet - but that Part is absolutely working fine!
Only 'non-state' incoming packets are sent out via the standard default route and that should not be the case since they should bound to an interface route not to a default and not to a policy based route.
Its as easy as that, broken down to IPv4 - its nearly the same:
If you have two interfaces
192.168.1.1
172.16.1.1And you ping the interface 192.168.1.1 - the packet is received by 192.168.1.1 and replied by 192.168.1.1 and NOT by 172.16.1.1 - even if it would be the set default route.
That is not working right now in IPv6 context as mentioned.
Cheers
4920441
P.S.: If I set IPv6 'default gw' to 'none' - and make some policy based rules as you mentioned - as I had not tried that before - everything else is working finde, but the same Problem comes up... If I have one rule allowd to from HE tunnel which is used for outgoing , the other policy rule regarding incomiing connections on the pppoe interface are routed through the appropriate gateway back the same happens! Every incoming icmp packet on the pppoe Interface is routet via the he tunnel - despite all policy based routes.
It only works if I set the pppoe IPv6 gateway as default gateway, then incoming packets a also sent out where there are received.... but then my policy based routes through the hurricane tunnel don't work any more. -
@4920441-0 said in PPPoE IPv6 + Hurricane Electrics IPv6 - Assymmetric routing / no ping from External to pppoe assigned IPv6 Adddress possibile, despite that, everything is working...?:
It only works if I set the pppoe IPv6 gateway as default gateway, then incoming packets a also sent out where there are received.... but then my policy based routes through the hurricane tunnel don't work any more.
Then you need to fix your rules for policy routing, show screenshots if you can't do it yourself.
-
Ok ok.... I clean up all my policy based rules once again, and when I am down to only a couple of rules and it's still not working I'll come back to you.
Cheers,
4920440
-
So, lastly, I got the following:
I completely disabled all default ipv6 gateways.
And two pairs of 'floating' rules, with the 'Apply the action immediately on match' Flag, so they should only be the first rules which match.
The first two are for the WAN IPv6 Address (this is the address which is tried to be pinged from an external Ipv6 Network)
One for Incoming one for outgoing traffic - just to be sure.
The second two are for the LAN IPv6 Subnet (which is still Hurricane Electrics), the same one for in one for out.And the symptons are exactly the same! HE still works in any direction but the WAN IPv6 Address receives ICMP Req on its interface but pfsense sends it out on the Hurricane Electric Interface - WITHOUT any default route acitvated and with the policy based route that every Packet which is received or send on the pppoe Interface should get the pppoe IPv6 Gateway as next hop.
Even worse: the floating rule does not catch any traffic at all, despite it should.
Did I mention that the floating rule for Hurricane Elecrics works? Maybe it has something to do with a faulty implementation of the DHCPv6 over pppoe implementation?
-
I bet if I switch to the dev branch, suddenly everything is working fine....
-
@4920441-0 And why should that be? Besides if you were reading the announcements about the changes in the dev-version you wouldn't talk about that.
-
Simple: because its a pppoe related bug.
I got another IPv6 network on another interface propagated via SLAAC and it is also routet, this one is pingable without any problems.
Only pppoe Connections via DHCP over pppoe behave strange...Cheers,
-
I have been working on a similar setup. Dual WAN IPv4+IPv6. I get native IPv4 from my ISP. For IPv6 I have been using Hurricane Electric for at least a decade. Recently, I stumbled upon a tunnel service that does both IPv4 and IPv6. This makes it possible to rather easily move services, yet keeping IPs the same, both IPv4 and IPv6.
But that's more of a backstory. I have been researching quite the same problem you describe. Packets that are generated on the router (e.g. ICMP TTL Exceeded when doing a traceroute) should be sent back through the same interface they entered, but for IPv6, this doesn't work.
It seems that in FreeBSD, the backing operating system for pfSense, this is simply not implemented for IPv6. There is code in review for this, but it may take some more time before that reaches FreeBSD itself, and consequently pfSense.
Hope this helps.
