Puzzled by connection tracking in IPv6



  • Short version, I have IPv6 working fine on the home LAN and can access google and friends via IPv6.  I can also ping6 internet addresses from the LAN.

    I also run a webserver at 175.111.102.128 which is natted to 10.28.1.2 and that works fine.

    The webserver has a manually set IPv6 address of [2400:6900:ffff:1::1:2] and that does not work.
    I can try a ping6 from the internet and I get this:

    on the WAN interface of pfsense I see this:
    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(18): tcpdump -nn -i pppoe0 host 2400:6900:ffff:1::1:2
    17:27:55.869223 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
    17:27:56.867144 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
    17:27:57.866348 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64

    LAN interface of pfsense box
    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(19): tcpdump -nn -i re0 host 2400:6900:ffff:1::1:2 and icmp6
    17:06:28.602516 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 44, length 64
    17:06:28.602643 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 44, length 64

    And on the internal host I see this:
    caffeine:/var/log# tcpdump -nn -i eth0 icmp6
    17:29:15.871401 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 81, length 64
    17:29:15.871430 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 81, length 64
    17:29:16.870856 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 82, length 64
    17:29:16.870887 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 82, length 64

    So the ping6 request gets to the target box, it replies, but the pfsense box is not allowing the return traffic out to the WAN

    Other details:

    • 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 is my work desktop

    • The same thing happens with http traffic to the webserver - the SYN packet hits the webserver then the SYN-ACK packet enters pfsense LAN and never exits the WAN interface.

    • There are no deny messages or any relevant messages in the firewall logs or syslog.

    • My ISP does not have functional IPv6 reverse DNS yet, however I can't see that having an effect.

    • I shouldn't have to run separate WAN rules and matching LAN rules to let the IPv6 traffic back out?

    The puzzle is that I have IPv6 working fine for connections sourced from the LAN, I can admin the box via v6 from the LAN.

    So why can't I route v6 in from the WAN to the LAN?



  • I can confirm from my LAN I can read this web forum with IPv6, so something's right. 
    Anyone got any ideas?



  • Hi Criggie

    It is a little difficult for me to say something specific.

    Please do the following and tell me you experiences:

    1. Try http://test-ipv6.com/
      Do you get 10/10 or does some of the tests fail?

    2. If you e.g. use Linux try from a host outside your LAN to get into your pfSense network (through the WAN of course :-) ), but by using these commands
      ping6 (icmpv6 echoes)
      rltraceroute6 (UDP)
      tcptraceroute6 (TCP)
      tracert6 (IMCPv6 Echo)
      , which you can get by installing ndisc6 (in Ubuntu at least). (There are some other useful tools like ndisc6 and rdisc6 within the ndisc6-package.)

    Please post the outcome when calling these commands with the trace of which routers it reaches.

    1. You might have to specify some rules on you WAN side to allow ipv6 TCP/UDP as well as ICMP, but I am not able to tell from your posts. Likewise you need some ipv6 rules in your LAN e.g. a default allow all ipv6 (e.g. any transport layer protocol instead of only tcp/udp) to get out of your LAN.


  • @al:

    1. Try http://test-ipv6.com/
      Do you get 10/10 or does some of the tests fail?

    2. If you e.g. use Linux try from a host outside your LAN to get into your pfSense network (through the WAN of course :-) ), but by using these commands
      ping6 (icmpv6 echoes)
      rltraceroute6 (UDP)
      tcptraceroute6 (TCP)
      tracert6 (IMCPv6 Echo)
      Please post the outcome when calling these commands with the trace of which routers it reaches.

    3. You might have to specify some rules on you WAN side to allow ipv6 TCP/UDP as well as ICMP, but I am not able to tell from your posts. Likewise you need some ipv6 rules in your LAN e.g. a default allow all ipv6 (e.g. any transport layer protocol instead of only tcp/udp) to get out of your LAN.

    1) I get a full 10/10 for IPv6 readyness from my desktop

    2) I have access to a test host outside my LAN which is fully IPv6, at  2400:6900:ffff:fffe::3, call it "ksp"

    ksp pings my webserver at 2400:6900:ffff:1::1:2 which is criggie.org.nz, and ping requests come in my pppoe interface

    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(7): tcpdump -nn -i pppoe0 icmp6
    11:53:13.123553 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
    11:53:14.123328 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
    11:53:15.123327 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64

    Ping requests go out my LAN on vr0 and replies come in vr0

    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(3): tcpdump -nn -i re0 icmp6
    11:53:13.123708 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
    11:53:13.123860 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 1, length 64
    11:53:14.123403 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
    11:53:14.123507 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 2, length 64
    11:53:15.123390 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64
    11:53:15.123517 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 3, length 64

    And at the internal server I see the request and reply being sent
    caffeine:~# tcpdump -i eth0 icmp6 -nn
    11:53:13.119289 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
    11:53:13.119323 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 1, length 64
    11:53:14.118949 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
    11:53:14.118991 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 2, length 64
    11:53:15.118971 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64
    11:53:15.119004 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 3, length 64

    Here's a  mtr -6 2400:6900:ffff:fffe::3    from the webserver to the test box
                                            Packets              Pings
    Host                                  Loss%  Snt  Last  Avg  Best  Wrst StDev
    1. pfsense.criggie.org.nz              0.0%    15    0.4  0.2  0.1  0.4  0.0
    2. ? ? ?
    3. 2400:6900:ffff:fffe::3              0.0%    14    5.4  5.4  5.0  5.7  0.0

    Here's the reverse    mtr -6 2400:6900:ffff:1::1:2
                                          Packets              Pings
    Host                              Loss%  Snt  Last  Avg  Best  Wrst StDev
    1. 2400:6900:ffff:fffe::1          0.0%    18    0.7  0.8  0.7  1.7  0.3
    2. ? ? ?

    I can ping the ksp test box perfectly from the webserver
    caffeine:~# ping6 2400:6900:ffff:fffe::3
    PING 2400:6900:ffff:fffe::3(2400:6900:ffff:fffe::3) 56 data bytes
    64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=1 ttl=62 time=6.65 ms
    64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=2 ttl=62 time=5.85 ms
    64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=3 ttl=62 time=5.67 ms
    ^C
    –- 2400:6900:ffff:fffe::3 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 5.676/6.061/6.657/0.436 ms

    3) I've checked my syslog server and the local logs on the pfsense firewall and there are no denies for ICMP6.
    Firewall rules are a big fat "allow any icmp6 from any to any" on both the LAN and WAN tabs.

    This could be a clue…. when I start pinging from ksp to the webserver, I see this
    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(13): tcpdump -nn -i re0_vlan10 icmp6
    tcpdump: WARNING: re0_vlan10: no IPv4 address assigned
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on re0_vlan10, link-type EN10MB (Ethernet), capture size 96 bytes
    12:04:57.056920 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
    12:04:58.056673 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
    12:04:59.056704 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
    12:05:01.054929 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32

    Background, I am on VDSL and the session is delivered to the pfsense box tagged with VLAN 10, over which I have to connect via PPPoE.

    That IP address of  fe80::c664:13ff:fe9e:bf80  is on my ISP's cisco router and is my default gateway, but via %pppoe0

    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(18): ping6 fe80::c664:13ff:fe9e:bf80%pppoe0
    PING6(56=40+8+8 bytes) fe80::230:1bff:fe82:bcac%pppoe0 –> fe80::c664:13ff:fe9e:bf80%pppoe0
    16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=0 hlim=64 time=5.500 ms
    16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=1 hlim=64 time=5.200 ms
    ^C
    --- fe80::c664:13ff:fe9e:bf80%pppoe0 ping6 statistics ---
    2 packets transmitted, 2 packets received, 0.0% packet loss
    round-trip min/avg/max/std-dev = 5.200/5.350/5.500/0.150 ms

    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(19): ping6 fe80::c664:13ff:fe9e:bf80%re0_vlan10
    PING6(56=40+8+8 bytes) fe80::230:1bff:fe82:bcac%re0_vlan10 –> fe80::c664:13ff:fe9e:bf80%re0_vlan10
    ^C
    --- fe80::c664:13ff:fe9e:bf80%re0_vlan10 ping6 statistics ---
    5 packets transmitted, 0 packets received, 100.0% packet loss

    It does NOT respond on interface re0_vlan10, only on the pppoe0 interface.

    So here's my routing table for v6 traffic

    [2.1-RELEASE][root@pfsense.criggie.org.nz]/root(20): netstat -arn -6
    Internet6:
    Destination                      Gateway                      Flags      Netif Expire
    default                          fe80::c664:13ff:fe9e:bf80%pppoe0 UGS      pppoe0
    ::1                              ::1                          UH          lo0
    2400:6900:ffff:1::/64            link#1                        U          re0
    2400:6900:ffff:1::1:1            link#1                        UHS        lo0
    2400:6900:ffff:1::90:0/120        link#9                        U      re0_vlan
    2400:6900:ffff:1::90:1            link#9                        UHS        lo0
    2400:6900:ffff:1:230:1bff:fe82:bcac link#10                      UHS        lo0
    fe80::%re0/64                    link#1                        U          re0
    fe80::230:1bff:fe82:bcac%re0      link#1                        UHS        lo0
    fe80::%lo0/64                    link#4                        U          lo0
    fe80::1%lo0                      link#4                        UHS        lo0
    fe80::%re0_vlan10/64              link#6                        U      re0_vlan
    fe80::230:1bff:fe82:bcac%re0_vlan10 link#6                        UHS        lo0
    fe80::%re0_vlan30/64              link#7                        U      re0_vlan
    fe80::230:1bff:fe82:bcac%re0_vlan30 link#7                        UHS        lo0
    fe80::%re0_vlan40/64              link#8                        U      re0_vlan
    fe80::230:1bff:fe82:bcac%re0_vlan40 link#8                        UHS        lo0
    fe80::%re0_vlan90/64              link#9                        U      re0_vlan
    fe80::230:1bff:fe82:bcac%re0_vlan90 link#9                        UHS        lo0
    fe80::%pppoe0/64                  link#10                      U        pppoe0
    fe80::230:1bff:fe82:bcac%pppoe0  link#10                      UHS        lo0
    ff01::%re0/32                    fe80::230:1bff:fe82:bcac%re0  U          re0
    ff01::%lo0/32                    ::1                          U          lo0
    ff01::%re0_vlan10/32              fe80::230:1bff:fe82:bcac%re0_vlan10 U      re0_vlan
    ff01::%re0_vlan30/32              fe80::230:1bff:fe82:bcac%re0_vlan30 U      re0_vlan
    ff01::%re0_vlan40/32              fe80::230:1bff:fe82:bcac%re0_vlan40 U      re0_vlan
    ff01::%re0_vlan90/32              fe80::230:1bff:fe82:bcac%re0_vlan90 U      re0_vlan
    ff01::%pppoe0/32                  2400:6900:ffff:1:230:1bff:fe82:bcac U        pppoe0
    ff02::%re0/32                    fe80::230:1bff:fe82:bcac%re0  U          re0
    ff02::%lo0/32                    ::1                          U          lo0
    ff02::%re0_vlan10/32              fe80::230:1bff:fe82:bcac%re0_vlan10 U      re0_vlan
    ff02::%re0_vlan30/32              fe80::230:1bff:fe82:bcac%re0_vlan30 U      re0_vlan
    ff02::%re0_vlan40/32              fe80::230:1bff:fe82:bcac%re0_vlan40 U      re0_vlan
    ff02::%re0_vlan90/32              fe80::230:1bff:fe82:bcac%re0_vlan90 U      re0_vlan
    ff02::%pppoe0/32                  2400:6900:ffff:1:230:1bff:fe82:bcac U        pppoe0

    Note - my IPv6 doesn't work by default unless I manually run this command:

    route change -inet6 default fe80::c664:13ff:fe9e:bf80%pppoe0

    So it feels to me like IPv6 reply traffic from the LAN going out the WAN doesn't pay attention to my system routing table.
    Whereas traffic sourced in the LAN takes another path.

    Or am I completely wrong?



  • @al:

    Please post the outcome when calling these commands with the trace of which routers it reaches.

    Oh yes - I should have added that I get exactly the same symptoms when using http on port 80 and smtp on 25, packets go in OK, but reply traffic from internal host enters but doesn't leave the pfsense firewall.

    Its not ICMP specific, its IPv6 specific but only with reply traffic from LAN to inet.

    Anything going the other way is fine.



  • Hi Criggie :-)

    I have been busy and your problem is a bit complex for me.

    First of all
    I have not worked with VLANs so I am definitely not on solid ground, but something comes to my mind.
    You might have a problem with that your VLAN re0_vlan/link#9 runs with a /120 prefix since a normal ipv6 network with hosts ought to be run with a /64 prefix to make stateless autoconfiguration work. However the VLAN might only be seen by network equipment such as pfSense, switches and so on and therefore maybe not a problem?!
    So you might consider to give your VLANs one common unique /64 prefixed network or a unique /120 prefixed network (that is not a subnet of another network also managed by pfSense, but read on)
    2400:6900:ffff:1::90:0/120        link#9                        U      re0_vlan
    2400:6900:ffff:1::90:1            link#9                        UHS        lo0
    And when I say one common unique network I do not mean the networks which the hosts of those VLANs use. I mean the network used to setup and connect the VLANs with each other.

    Second is that it seems strange that you have to specify a default gateway (I am unaware if that is something one has to do when dealing with VLANs) however normally you would have to specify that in the menu: Interfaces - [Name of your interface].

    Third I think you might get into trouble having a subnet 2400:6900:ffff:1::90:0/120 within your 2400:6900:ffff:1::/64 network. It might might mess things up when routing to/from your hosts or sub-routers if you had such - e.g. home routers. But I guess the problem in this case is that pfSense does not know how to route, because of the /120 network is within the /64 network. But I might be wrong.
    I guess it would be much better if you can use one /64 network for each VLAN if the VLANs are supposed to be unique networks, but at least make sure that pfSense does not use a subnetwork (/120) within a /64 network which one of the VLANs use. However my knowledge is non-existent regarding VLANs. But it is my very best suggestion to why it does not work!
    So the problem seems to be centralized around the subnet 2400:6900:ffff:1::90:0/120 within your 2400:6900:ffff:1::/64 network and the common gateway:
    ff02::%pppoe0/32                  2400:6900:ffff:1:230:1bff:fe82:bcac U        pppoe0
    ff01::%pppoe0/32                  2400:6900:ffff:1:230:1bff:fe82:bcac U        pppoe0
    default                          fe80::c664:13ff:fe9e:bf80%pppoe0 UGS      pppoe0

    Please also remember to give your hosts the right prefix if you change prefixes in your network (else you will just end out in a situation where routing does not seem to work). To keep it simple use /64 prefixes whenever you can when dealing with networks having hosts.

    You might want to read the last part of this post http://forums.dlink.com/index.php?topic=57422.msg225194#msg225194
    regarding:

    Receiving an IPv6 block of size greater than /64 (greater = shorter prefix length, e.g. /56) via DHCP-PD will not mean, that the CPE would deploy it to a single attached LAN. Rather, as /64 is the standard size of any IPv6 network (RFC4291, Chapter 2.5.1:  "For all unicast addresses, except those that start with the binary value 000, Interface IDs are required to be 64 bits long and to be constructed in Modified EUI-64 format". Exceptions: /127, see RFC6164, or statically configured IPv6 addresses with embeddes IPv4 addresses resulting from IP4v/IPv6 address translation, see RFC6052) and hence SLAAC (RFC4862) only works with /64 networks, the CPE will subdivide the delegated block (e. g. a /56) into several /64, namely one per directly attached LAN segment (in most cases meaning only one) and might use the rest of the unused address space for deployment of smaller subranges (e.g. /60) to "second hierarchy" routers that connect to additional lan segments.

    In contrast to DHCPv4 statefull DHCPv6 will only deploy /128 addresses without any prefix length (subnet mask in IPv4) and default gateway information. Hence in addition to DHCPv6 you always need a router interface connected to a LAN segment with DHCPv6 clients that sends router advertisments ("RA", RFC4861). From these RA the DHCPv6 clients should learn the default gateway (always the link local address of the router the RA was send from) and at least one IPv6 prefix being "onlink" (the one that starts with the first 64 bits of the address received from the DHCPv6 server) and usually having a prefix length of 64. Ideally within this RA the A-flag for this prefix should be set to 0 to prevent the clients to autoconfigure a second IPv6 address in addition to the one received via stateful DHCPv6.

    In case you need to make some static routes you can do that in pfSense in the menu. System - Routing - Routes.
    You define the network (global routing prefix + subnet ID) and from that you choose the subnet prefix - preferably /64. E.g. see http://www.roesen.org/files/ipv6_cheat_sheet.pdf
    Then you might select an existing gateway or in case you have a home router (sub-router) create a new gateway by specifying an interface (e.g. LAN) where you want the route to be made, a name and a gateway IP that e.g. your sub-router should use on its WAN-side.

    Remember static routes should never be made/added on addresses of interfaces that are part of the pfSense fw/router when you define a route to a (sub-)network! A static route should only be made to a network whenever another router manages that network.

    Also you might want to consider using (from the menu) System - Advanced - Firewall/Nat - Static route filtering (Bypass firewall rules for traffic on the same interface). But it might not be the case if you use VLANs. Again I do not know how to configure VLANs.
    Read more about that in the (old) pfSense book. I really hope that I soon can buy the new pfSense 2.x book when it is released. :)

    As I said earlier on VLANs are not my expertise at all. Maybe you will also need to check up on e.g. link-local and interface-local network references in your routes list. But I guess pfSense made a lot of these by itself when you created the VLAN so it should not be needed I guess. I am merely speculating.



  • @al:

    I have been busy and your problem is a bit complex for me.

    Its complex for me too :)    On the positive side, version  pfSense-Full-Update-2.1.1-PRERELEASE-amd64-20140221-1118.tgz  fixes the default gateway routing for outbound IPv6 traffic, its now sent to the ISP's v6 address on the pppoe0 interface, not the re0_vlan10 interface.

    My inbound problem is unchanged, but I feel its related.  The inbound  ping6 www.criggie.org.nz  works inwards, through firewall, hits internal box, and goes back to pfsense.  But pfsense starts doing Neighbour Solicitation out re0_vlan10 again, which is wrong.

    Your suggestions are all good and I tried them, including disabling the internal guest VLAN completely but that made no difference.

    I think this redmine bug is going to be related:    https://redmine.pfsense.org/issues/3357



  • @Criggie:

    I think this redmine bug is going to be related:    https://redmine.pfsense.org/issues/3357

    Turns out that bug was not related.  Here's the current bug report for this  https://redmine.pfsense.org/issues/3544



  • @Criggie:

    @Criggie:

    I think this redmine bug is going to be related:    https://redmine.pfsense.org/issues/3357

    Turns out that bug was not related.  Here's the current bug report for this  https://redmine.pfsense.org/issues/3544

    SOLVED!  I don't know when the option appeared but under interface -> WAN there is now an option

    Use IPv4 connectivity as parent interface   [X]	Request a IPv6 prefix/information through the IPv4 connectivity link 
    

    So now I have full IPv6 native connectivity both inwards and outwards.