Puzzled by connection tracking in IPv6

Criggie

Short version, I have IPv6 working fine on the home LAN and can access google and friends via IPv6. I can also ping6 internet addresses from the LAN.

I also run a webserver at 175.111.102.128 which is natted to 10.28.1.2 and that works fine.

The webserver has a manually set IPv6 address of [2400:6900:ffff:1::1:2] and that does not work.
I can try a ping6 from the internet and I get this:

on the WAN interface of pfsense I see this:
[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(18): tcpdump -nn -i pppoe0 host 2400:6900:ffff:1::1:2
17:27:55.869223 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
17:27:56.867144 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
17:27:57.866348 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64
…

LAN interface of pfsense box
[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(19): tcpdump -nn -i re0 host 2400:6900:ffff:1::1:2 and icmp6
17:06:28.602516 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 44, length 64
17:06:28.602643 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 44, length 64
…

And on the internal host I see this:
caffeine:/var/log# tcpdump -nn -i eth0 icmp6
17:29:15.871401 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 81, length 64
17:29:15.871430 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 81, length 64
17:29:16.870856 IP6 2400:6900:ffff:fe00:225:b3ff:fe0a:b422 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 82, length 64
17:29:16.870887 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fe00:225:b3ff:fe0a:b422: ICMP6, echo reply, seq 82, length 64
…

So the ping6 request gets to the target box, it replies, but the pfsense box is not allowing the return traffic out to the WAN

Other details:

2400:6900:ffff:fe00:225:b3ff:fe0a:b422 is my work desktop
The same thing happens with http traffic to the webserver - the SYN packet hits the webserver then the SYN-ACK packet enters pfsense LAN and never exits the WAN interface.
There are no deny messages or any relevant messages in the firewall logs or syslog.
My ISP does not have functional IPv6 reverse DNS yet, however I can't see that having an effect.
I shouldn't have to run separate WAN rules and matching LAN rules to let the IPv6 traffic back out?

The puzzle is that I have IPv6 working fine for connections sourced from the LAN, I can admin the box via v6 from the LAN.

So why can't I route v6 in from the WAN to the LAN?

Criggie

I can confirm from my LAN I can read this web forum with IPv6, so something's right.
Anyone got any ideas?

al

Hi Criggie

It is a little difficult for me to say something specific.

Please do the following and tell me you experiences:

Try http://test-ipv6.com/
Do you get 10/10 or does some of the tests fail?
If you e.g. use Linux try from a host outside your LAN to get into your pfSense network (through the WAN of course :-) ), but by using these commands
ping6 (icmpv6 echoes)
rltraceroute6 (UDP)
tcptraceroute6 (TCP)
tracert6 (IMCPv6 Echo)
, which you can get by installing ndisc6 (in Ubuntu at least). (There are some other useful tools like ndisc6 and rdisc6 within the ndisc6-package.)

Please post the outcome when calling these commands with the trace of which routers it reaches.

You might have to specify some rules on you WAN side to allow ipv6 TCP/UDP as well as ICMP, but I am not able to tell from your posts. Likewise you need some ipv6 rules in your LAN e.g. a default allow all ipv6 (e.g. any transport layer protocol instead of only tcp/udp) to get out of your LAN.

Criggie

@al:

Try http://test-ipv6.com/
Do you get 10/10 or does some of the tests fail?

If you e.g. use Linux try from a host outside your LAN to get into your pfSense network (through the WAN of course :-) ), but by using these commands
ping6 (icmpv6 echoes)
rltraceroute6 (UDP)
tcptraceroute6 (TCP)
tracert6 (IMCPv6 Echo)
Please post the outcome when calling these commands with the trace of which routers it reaches.

You might have to specify some rules on you WAN side to allow ipv6 TCP/UDP as well as ICMP, but I am not able to tell from your posts. Likewise you need some ipv6 rules in your LAN e.g. a default allow all ipv6 (e.g. any transport layer protocol instead of only tcp/udp) to get out of your LAN.

1) I get a full 10/10 for IPv6 readyness from my desktop

2) I have access to a test host outside my LAN which is fully IPv6, at 2400:6900:ffff:fffe::3, call it "ksp"

ksp pings my webserver at 2400:6900:ffff:1::1:2 which is criggie.org.nz, and ping requests come in my pppoe interface

[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(7): tcpdump -nn -i pppoe0 icmp6
11:53:13.123553 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
11:53:14.123328 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
11:53:15.123327 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64

Ping requests go out my LAN on vr0 and replies come in vr0

[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(3): tcpdump -nn -i re0 icmp6
11:53:13.123708 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
11:53:13.123860 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 1, length 64
11:53:14.123403 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
11:53:14.123507 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 2, length 64
11:53:15.123390 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64
11:53:15.123517 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 3, length 64

And at the internal server I see the request and reply being sent
caffeine:~# tcpdump -i eth0 icmp6 -nn
11:53:13.119289 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 1, length 64
11:53:13.119323 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 1, length 64
11:53:14.118949 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 2, length 64
11:53:14.118991 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 2, length 64
11:53:15.118971 IP6 2400:6900:ffff:fffe::3 > 2400:6900:ffff:1::1:2: ICMP6, echo request, seq 3, length 64
11:53:15.119004 IP6 2400:6900:ffff:1::1:2 > 2400:6900:ffff:fffe::3: ICMP6, echo reply, seq 3, length 64

Here's a mtr -6 2400:6900:ffff:fffe::3 from the webserver to the test box
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. pfsense.criggie.org.nz 0.0% 15 0.4 0.2 0.1 0.4 0.0
2. ? ? ?
3. 2400:6900:ffff:fffe::3 0.0% 14 5.4 5.4 5.0 5.7 0.0

Here's the reverse mtr -6 2400:6900:ffff:1::1:2
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2400:6900:ffff:fffe::1 0.0% 18 0.7 0.8 0.7 1.7 0.3
2. ? ? ?

I can ping the ksp test box perfectly from the webserver
caffeine:~# ping6 2400:6900:ffff:fffe::3
PING 2400:6900:ffff:fffe::3(2400:6900:ffff:fffe::3) 56 data bytes
64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=1 ttl=62 time=6.65 ms
64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=2 ttl=62 time=5.85 ms
64 bytes from 2400:6900:ffff:fffe::3: icmp_seq=3 ttl=62 time=5.67 ms
^C
–- 2400:6900:ffff:fffe::3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 5.676/6.061/6.657/0.436 ms

3) I've checked my syslog server and the local logs on the pfsense firewall and there are no denies for ICMP6.
Firewall rules are a big fat "allow any icmp6 from any to any" on both the LAN and WAN tabs.

This could be a clue…. when I start pinging from ksp to the webserver, I see this
[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(13): tcpdump -nn -i re0_vlan10 icmp6
tcpdump: WARNING: re0_vlan10: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0_vlan10, link-type EN10MB (Ethernet), capture size 96 bytes
12:04:57.056920 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
12:04:58.056673 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
12:04:59.056704 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32
12:05:01.054929 IP6 fe80::230:1bff:fe82:bcac > ff02::1:ff9e:bf80: ICMP6, neighbor solicitation, who has fe80::c664:13ff:fe9e:bf80, length 32

Background, I am on VDSL and the session is delivered to the pfsense box tagged with VLAN 10, over which I have to connect via PPPoE.

That IP address of fe80::c664:13ff:fe9e:bf80 is on my ISP's cisco router and is my default gateway, but via %pppoe0

[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(18): ping6 fe80::c664:13ff:fe9e:bf80%pppoe0
PING6(56=40+8+8 bytes) fe80::230:1bff:fe82:bcac%pppoe0 –> fe80::c664:13ff:fe9e:bf80%pppoe0
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=0 hlim=64 time=5.500 ms
16 bytes from fe80::c664:13ff:fe9e:bf80%pppoe0, icmp_seq=1 hlim=64 time=5.200 ms
^C
--- fe80::c664:13ff:fe9e:bf80%pppoe0 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 5.200/5.350/5.500/0.150 ms

[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(19): ping6 fe80::c664:13ff:fe9e:bf80%re0_vlan10
PING6(56=40+8+8 bytes) fe80::230:1bff:fe82:bcac%re0_vlan10 –> fe80::c664:13ff:fe9e:bf80%re0_vlan10
^C
--- fe80::c664:13ff:fe9e:bf80%re0_vlan10 ping6 statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

It does NOT respond on interface re0_vlan10, only on the pppoe0 interface.

So here's my routing table for v6 traffic

[2.1-RELEASE][root@pfsense.criggie.org.nz]/root(20): netstat -arn -6
Internet6:
Destination Gateway Flags Netif Expire
default fe80::c664:13ff:fe9e:bf80%pppoe0 UGS pppoe0
::1 ::1 UH lo0
2400:6900:ffff:1::/64 link#1 U re0
2400:6900:ffff:1::1:1 link#1 UHS lo0
2400:6900:ffff:1::90:0/120 link#9 U re0_vlan
2400:6900:ffff:1::90:1 link#9 UHS lo0
2400:6900:ffff:1:230:1bff:fe82:bcac link#10 UHS lo0
fe80::%re0/64 link#1 U re0
fe80::230:1bff:fe82:bcac%re0 link#1 UHS lo0
fe80::%lo0/64 link#4 U lo0
fe80::1%lo0 link#4 UHS lo0
fe80::%re0_vlan10/64 link#6 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan10 link#6 UHS lo0
fe80::%re0_vlan30/64 link#7 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan30 link#7 UHS lo0
fe80::%re0_vlan40/64 link#8 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan40 link#8 UHS lo0
fe80::%re0_vlan90/64 link#9 U re0_vlan
fe80::230:1bff:fe82:bcac%re0_vlan90 link#9 UHS lo0
fe80::%pppoe0/64 link#10 U pppoe0
fe80::230:1bff:fe82:bcac%pppoe0 link#10 UHS lo0
ff01::%re0/32 fe80::230:1bff:fe82:bcac%re0 U re0
ff01::%lo0/32 ::1 U lo0
ff01::%re0_vlan10/32 fe80::230:1bff:fe82:bcac%re0_vlan10 U re0_vlan
ff01::%re0_vlan30/32 fe80::230:1bff:fe82:bcac%re0_vlan30 U re0_vlan
ff01::%re0_vlan40/32 fe80::230:1bff:fe82:bcac%re0_vlan40 U re0_vlan
ff01::%re0_vlan90/32 fe80::230:1bff:fe82:bcac%re0_vlan90 U re0_vlan
ff01::%pppoe0/32 2400:6900:ffff:1:230:1bff:fe82:bcac U pppoe0
ff02::%re0/32 fe80::230:1bff:fe82:bcac%re0 U re0
ff02::%lo0/32 ::1 U lo0
ff02::%re0_vlan10/32 fe80::230:1bff:fe82:bcac%re0_vlan10 U re0_vlan
ff02::%re0_vlan30/32 fe80::230:1bff:fe82:bcac%re0_vlan30 U re0_vlan
ff02::%re0_vlan40/32 fe80::230:1bff:fe82:bcac%re0_vlan40 U re0_vlan
ff02::%re0_vlan90/32 fe80::230:1bff:fe82:bcac%re0_vlan90 U re0_vlan
ff02::%pppoe0/32 2400:6900:ffff:1:230:1bff:fe82:bcac U pppoe0

Note - my IPv6 doesn't work by default unless I manually run this command:

route change -inet6 default fe80::c664:13ff:fe9e:bf80%pppoe0

So it feels to me like IPv6 reply traffic from the LAN going out the WAN doesn't pay attention to my system routing table.
Whereas traffic sourced in the LAN takes another path.

Or am I completely wrong?

Criggie

@al:

Please post the outcome when calling these commands with the trace of which routers it reaches.

Oh yes - I should have added that I get exactly the same symptoms when using http on port 80 and smtp on 25, packets go in OK, but reply traffic from internal host enters but doesn't leave the pfsense firewall.

Its not ICMP specific, its IPv6 specific but only with reply traffic from LAN to inet.

Anything going the other way is fine.

al

Hi Criggie :-)

I have been busy and your problem is a bit complex for me.

First of all
I have not worked with VLANs so I am definitely not on solid ground, but something comes to my mind.
You might have a problem with that your VLAN re0_vlan/link#9 runs with a /120 prefix since a normal ipv6 network with hosts ought to be run with a /64 prefix to make stateless autoconfiguration work. However the VLAN might only be seen by network equipment such as pfSense, switches and so on and therefore maybe not a problem?!
So you might consider to give your VLANs one common unique /64 prefixed network or a unique /120 prefixed network (that is not a subnet of another network also managed by pfSense, but read on)
2400:6900:ffff:1::90:0/120 link#9 U re0_vlan
2400:6900:ffff:1::90:1 link#9 UHS lo0
And when I say one common unique network I do not mean the networks which the hosts of those VLANs use. I mean the network used to setup and connect the VLANs with each other.

Second is that it seems strange that you have to specify a default gateway (I am unaware if that is something one has to do when dealing with VLANs) however normally you would have to specify that in the menu: Interfaces - [Name of your interface].

Third I think you might get into trouble having a subnet 2400:6900:ffff:1::90:0/120 within your 2400:6900:ffff:1::/64 network. It might might mess things up when routing to/from your hosts or sub-routers if you had such - e.g. home routers. But I guess the problem in this case is that pfSense does not know how to route, because of the /120 network is within the /64 network. But I might be wrong.
I guess it would be much better if you can use one /64 network for each VLAN if the VLANs are supposed to be unique networks, but at least make sure that pfSense does not use a subnetwork (/120) within a /64 network which one of the VLANs use. However my knowledge is non-existent regarding VLANs. But it is my very best suggestion to why it does not work!
So the problem seems to be centralized around the subnet 2400:6900:ffff:1::90:0/120 within your 2400:6900:ffff:1::/64 network and the common gateway:
ff02::%pppoe0/32 2400:6900:ffff:1:230:1bff:fe82:bcac U pppoe0
ff01::%pppoe0/32 2400:6900:ffff:1:230:1bff:fe82:bcac U pppoe0
default fe80::c664:13ff:fe9e:bf80%pppoe0 UGS pppoe0

Please also remember to give your hosts the right prefix if you change prefixes in your network (else you will just end out in a situation where routing does not seem to work). To keep it simple use /64 prefixes whenever you can when dealing with networks having hosts.

You might want to read the last part of this post http://forums.dlink.com/index.php?topic=57422.msg225194#msg225194
regarding:

Receiving an IPv6 block of size greater than /64 (greater = shorter prefix length, e.g. /56) via DHCP-PD will not mean, that the CPE would deploy it to a single attached LAN. Rather, as /64 is the standard size of any IPv6 network (RFC4291, Chapter 2.5.1: "For all unicast addresses, except those that start with the binary value 000, Interface IDs are required to be 64 bits long and to be constructed in Modified EUI-64 format". Exceptions: /127, see RFC6164, or statically configured IPv6 addresses with embeddes IPv4 addresses resulting from IP4v/IPv6 address translation, see RFC6052) and hence SLAAC (RFC4862) only works with /64 networks, the CPE will subdivide the delegated block (e. g. a /56) into several /64, namely one per directly attached LAN segment (in most cases meaning only one) and might use the rest of the unused address space for deployment of smaller subranges (e.g. /60) to "second hierarchy" routers that connect to additional lan segments.

In contrast to DHCPv4 statefull DHCPv6 will only deploy /128 addresses without any prefix length (subnet mask in IPv4) and default gateway information. Hence in addition to DHCPv6 you always need a router interface connected to a LAN segment with DHCPv6 clients that sends router advertisments ("RA", RFC4861). From these RA the DHCPv6 clients should learn the default gateway (always the link local address of the router the RA was send from) and at least one IPv6 prefix being "onlink" (the one that starts with the first 64 bits of the address received from the DHCPv6 server) and usually having a prefix length of 64. Ideally within this RA the A-flag for this prefix should be set to 0 to prevent the clients to autoconfigure a second IPv6 address in addition to the one received via stateful DHCPv6.

In case you need to make some static routes you can do that in pfSense in the menu. System - Routing - Routes.
You define the network (global routing prefix + subnet ID) and from that you choose the subnet prefix - preferably /64. E.g. see http://www.roesen.org/files/ipv6_cheat_sheet.pdf
Then you might select an existing gateway or in case you have a home router (sub-router) create a new gateway by specifying an interface (e.g. LAN) where you want the route to be made, a name and a gateway IP that e.g. your sub-router should use on its WAN-side.

Remember static routes should never be made/added on addresses of interfaces that are part of the pfSense fw/router when you define a route to a (sub-)network! A static route should only be made to a network whenever another router manages that network.

Also you might want to consider using (from the menu) System - Advanced - Firewall/Nat - Static route filtering (Bypass firewall rules for traffic on the same interface). But it might not be the case if you use VLANs. Again I do not know how to configure VLANs.
Read more about that in the (old) pfSense book. I really hope that I soon can buy the new pfSense 2.x book when it is released. :)

As I said earlier on VLANs are not my expertise at all. Maybe you will also need to check up on e.g. link-local and interface-local network references in your routes list. But I guess pfSense made a lot of these by itself when you created the VLAN so it should not be needed I guess. I am merely speculating.

Criggie

@al:

I have been busy and your problem is a bit complex for me.

Its complex for me too :) On the positive side, version pfSense-Full-Update-2.1.1-PRERELEASE-amd64-20140221-1118.tgz fixes the default gateway routing for outbound IPv6 traffic, its now sent to the ISP's v6 address on the pppoe0 interface, not the re0_vlan10 interface.

My inbound problem is unchanged, but I feel its related. The inbound ping6 www.criggie.org.nz works inwards, through firewall, hits internal box, and goes back to pfsense. But pfsense starts doing Neighbour Solicitation out re0_vlan10 again, which is wrong.

Your suggestions are all good and I tried them, including disabling the internal guest VLAN completely but that made no difference.

I think this redmine bug is going to be related: https://redmine.pfsense.org/issues/3357

Criggie

@Criggie:

I think this redmine bug is going to be related: https://redmine.pfsense.org/issues/3357

Turns out that bug was not related. Here's the current bug report for this https://redmine.pfsense.org/issues/3544

Criggie

@Criggie:

@Criggie:

I think this redmine bug is going to be related: https://redmine.pfsense.org/issues/3357

Turns out that bug was not related. Here's the current bug report for this https://redmine.pfsense.org/issues/3544

SOLVED! I don't know when the option appeared but under interface -> WAN there is now an option

Use IPv4 connectivity as parent interface   [X]	Request a IPv6 prefix/information through the IPv4 connectivity link

So now I have full IPv6 native connectivity both inwards and outwards.