IPv6 ping craziness



  • Ok, this issue has me (he.net ipv6 sage), my ISP, and those in ##pfsense ive spoken to utterly stumped.

    I will attempt to describe this as best I can without screaming!

    And before I go any further, understand that there are firewall rules in place currently to allow ALL ipv6 traffic from any source, to any destination, via any protocol, so this is not a rules issue.

    Network setup:

    Pfsense:
    WAN: IPv6 SLAAC, which is assigned 2a02:13a0:ad02:21:21c:25ff:fe4d:f77d  pppoe1 on sge0
    LAN: IPv6 Static: 2a02:13a0:a006:1::/64  em0

    Pfsense v6 routing table:

    IPv6
    Destination	Gateway	Flags	Refs	Use	Mtu	Netif	Expire
    default	fe80::f2f7:55ff:fe0c:5700%pppoe1	UGS	0	0	1492	pppoe1	 
    ::1	::1	UH	0	0	16384	lo0	 
    2001:4860:4860::8844	fe80::f2f7:55ff:fe0c:5700	UGHS	0	692	1492	pppoe1	 
    2001:4860:4860::8888	fe80::f2f7:55ff:fe0c:5700	UGHS	0	2061	1492	pppoe1	 
    2a02:13a0:a006:1::	link#5	UHS	0	0	16384	lo0	=>
    2a02:13a0:a006:1::/64	link#5	U	0	1681	1500	em0	 
    2a02:13a0:a006:2::	link#12	UHS	0	0	16384	lo0	=>
    2a02:13a0:a006:2::/64	link#12	U	0	842	1500	re0_vlan90	 
    2a02:13a0:ad02:21::/0	link#18	U	0	35	1492	pppoe1	 
    2a02:13a0:ad02:21:21c:25ff:fe4d:f77d	link#18	UHS	0	3	16384	lo0	 
    fe80::%sge0/64	link#4	U	0	0	1500	sge0	 
    fe80::21c:25ff:fe4d:f77d%sge0	link#4	UHS	0	0	16384	lo0	 
    fe80::%em0/64	link#5	U	0	125	1500	em0	 
    fe80::202:a5ff:fe4f:2a81%em0	link#5	UHS	0	0	16384	lo0	 
    fe80::%em1/64	link#6	U	0	0	1500	em1	 
    fe80::202:a5ff:fe4f:2a80%em1	link#6	UHS	0	0	16384	lo0	 
    fe80::%re0/64	link#7	U	0	0	1500	re0	 
    fe80::4a02:2aff:fe07:35d5%re0	link#7	UHS	0	0	16384	lo0	 
    fe80::%lo0/64	link#10	U	0	0	16384	lo0	 
    fe80::1%lo0	link#10	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan90/64	link#12	U	0	962	1500	re0_vlan90	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan90	link#12	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan1/64	link#13	U	0	0	1500	re0_vlan1	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan1	link#13	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan99/64	link#14	U	0	0	1500	re0_vlan99	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan99	link#14	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan64/64	link#15	U	0	0	1500	re0_vlan64	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan64	link#15	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan20/64	link#16	U	0	0	1500	re0_vlan20	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan20	link#16	UHS	0	0	16384	lo0	 
    fe80::%re0_vlan30/64	link#17	U	0	0	1500	re0_vlan30	 
    fe80::21c:25ff:fe4d:f77d%re0_vlan30	link#17	UHS	0	0	16384	lo0	 
    fe80::%pppoe1/64	link#18	U	0	0	1492	pppoe1	 
    fe80::21c:25ff:fe4d:f77d%pppoe1	link#18	UHS	0	0	16384	lo0	 
    fe80::548c:995a:9b35:e530%pppoe1	link#18	UHS	0	0	16384	lo0	 
    ff01::%sge0/32	fe80::21c:25ff:fe4d:f77d%sge0	U	0	0	1500	sge0	 
    ff01::%em0/32	fe80::202:a5ff:fe4f:2a81%em0	U	0	0	1500	em0	 
    ff01::%em1/32	fe80::202:a5ff:fe4f:2a80%em1	U	0	0	1500	em1	 
    ff01::%re0/32	fe80::4a02:2aff:fe07:35d5%re0	U	0	0	1500	re0	 
    ff01::%lo0/32	::1	U	0	0	16384	lo0	 
    ff01::%re0_vlan90/32	fe80::21c:25ff:fe4d:f77d%re0_vlan90	U	0	0	1500	re0_vlan90	 
    ff01::%re0_vlan1/32	fe80::21c:25ff:fe4d:f77d%re0_vlan1	U	0	0	1500	re0_vlan1	 
    ff01::%re0_vlan99/32	fe80::21c:25ff:fe4d:f77d%re0_vlan99	U	0	0	1500	re0_vlan99	 
    ff01::%re0_vlan64/32	fe80::21c:25ff:fe4d:f77d%re0_vlan64	U	0	0	1500	re0_vlan64	 
    ff01::%re0_vlan20/32	fe80::21c:25ff:fe4d:f77d%re0_vlan20	U	0	0	1500	re0_vlan20	 
    ff01::%re0_vlan30/32	fe80::21c:25ff:fe4d:f77d%re0_vlan30	U	0	0	1500	re0_vlan30	 
    ff01::%pppoe1/32	fe80::21c:25ff:fe4d:f77d%pppoe1	U	0	0	1492	pppoe1	 
    ff02::%sge0/32	fe80::21c:25ff:fe4d:f77d%sge0	U	0	0	1500	sge0	 
    ff02::%em0/32	fe80::202:a5ff:fe4f:2a81%em0	U	0	0	1500	em0	 
    ff02::%em1/32	fe80::202:a5ff:fe4f:2a80%em1	U	0	0	1500	em1	 
    ff02::%re0/32	fe80::4a02:2aff:fe07:35d5%re0	U	0	0	1500	re0	 
    ff02::%lo0/32	::1	U	0	0	16384	lo0	 
    ff02::%re0_vlan90/32	fe80::21c:25ff:fe4d:f77d%re0_vlan90	U	0	0	1500	re0_vlan90	 
    ff02::%re0_vlan1/32	fe80::21c:25ff:fe4d:f77d%re0_vlan1	U	0	0	1500	re0_vlan1	 
    ff02::%re0_vlan99/32	fe80::21c:25ff:fe4d:f77d%re0_vlan99	U	0	0	1500	re0_vlan99	 
    ff02::%re0_vlan64/32	fe80::21c:25ff:fe4d:f77d%re0_vlan64	U	0	0	1500	re0_vlan64	 
    ff02::%re0_vlan20/32	fe80::21c:25ff:fe4d:f77d%re0_vlan20	U	0	0	1500	re0_vlan20	 
    ff02::%re0_vlan30/32	fe80::21c:25ff:fe4d:f77d%re0_vlan30	U	0	0	1500	re0_vlan30	 
    ff02::%pppoe1/32	fe80::21c:25ff:fe4d:f77d%pppoe1	U	0	0	1492	pppoe1	 
    

    So first of all outbound ipv6, this works fine.

    Now the problems come when tryign to connect to this network from outside, but NOT on the initial connection.
    Selected lines from the 60+ tcpdumps i have done today….

    First an outgoing ping,

    
    C:\Users\James>ping 2001:1b40:5000:22::123
    
    Pinging 2001:1b40:5000:22::123 with 32 bytes of data:
    Reply from 2001:1b40:5000:22::123: time=10ms
    Reply from 2001:1b40:5000:22::123: time=9ms
    Reply from 2001:1b40:5000:22::123: time=9ms
    Reply from 2001:1b40:5000:22::123: time=9ms
    
    Ping statistics for 2001:1b40:5000:22::123:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 9ms, Maximum = 10ms, Average = 9ms
    
    

    And the attempt packet captured from the pfsense LAN:

    17:46:10.349597 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 314, length 40
    17:46:10.358630 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 314, length 40
    17:46:11.352816 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 315, length 40
    17:46:11.361746 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 315, length 40
    17:46:12.355661 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 316, length 40
    17:46:12.365088 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 316, length 40
    17:46:13.361255 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 317, length 40
    17:46:13.370471 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 317, length 40
    
    

    And the same thing tcpdump'ed from the WAN interface:

    17:47:06.735520 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 318, length 40
    17:47:06.744655 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 318, length 40
    17:47:07.738672 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 319, length 40
    17:47:07.747773 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 319, length 40
    17:47:08.741265 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 320, length 40
    17:47:08.750426 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 320, length 40
    17:47:09.744745 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo request, seq 321, length 40
    17:47:09.753764 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo reply, seq 321, length 40
    
    

    Ok, so far so good, but now the random issue begins when i try to ping my internal ip from the outside.

    
    james@observium:~$ ping6 2a02:13a0:a006:1::dead:beef:cafe
    PING 2a02:13a0:a006:1::dead:beef:cafe(2a02:13a0:a006:1:0:dead:beef:cafe) 56 data bytes
    ^C
    --- 2a02:13a0:a006:1::dead:beef:cafe ping statistics ---
    21 packets transmitted, 0 received, 100% packet loss, time 20159ms
    
    

    This is the capture from the pfsense WAN:

    17:48:39.990853 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 1, length 64
    17:48:40.998513 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 2, length 64
    17:48:42.006188 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 3, length 64
    17:48:43.014294 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 4, length 64
    17:48:44.022198 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 5, length 64
    17:48:45.030307 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 6, length 64
    17:48:46.038432 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 7, length 64
    
    

    So request is coming in just fine, but now for the wierd part
    packet capture from the pfsense LAN:

    17:49:36.790236 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 1, length 64
    17:49:36.790403 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 1, length 64
    17:49:37.798329 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 2, length 64
    17:49:37.798494 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 2, length 64
    17:49:38.806216 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 3, length 64
    17:49:38.807339 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 3, length 64
    17:49:39.814559 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 4, length 64
    17:49:39.814677 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 4, length 64
    17:49:39.858532 IP6 2a02:13a0:a006:1:: > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, destination unreachable, unreachable address 2001:1b40:5000:22::123, length 112
    17:49:40.822219 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 5, length 64
    17:49:40.823268 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 5, length 64
    17:49:41.829894 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 6, length 64
    17:49:41.830610 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 6, length 64
    17:49:42.838462 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 7, length 64
    17:49:42.839326 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 7, length 64
    17:49:43.057770 IP6 2a02:13a0:a006:1:: > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, destination unreachable, unreachable address 2001:1b40:5000:22::123, length 112
    17:49:43.846133 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 8, length 64
    17:49:43.846295 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 8, length 64
    17:49:44.854030 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 9, length 64
    17:49:44.854540 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 9, length 64
    17:49:45.862159 IP6 2001:1b40:5000:22::123 > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, echo request, seq 10, length 64
    17:49:45.862353 IP6 2a02:13a0:a006:1:0:dead:beef:cafe > 2001:1b40:5000:22::123: ICMP6, echo reply, seq 10, length 64
    17:49:46.912023 IP6 2a02:13a0:a006:1:: > 2a02:13a0:a006:1:0:dead:beef:cafe: ICMP6, destination unreachable, unreachable address 2001:1b40:5000:22::123, length 112
    
    

    So what this shows me is that the ping request came in, was sent out of the LAN, the local PC responded to it, and only then every few times does the pfsense say destination unreachable.
    Given that I can simultaneously ping the host outbound, while inbound ping responses are told 'destination unreachable' makes absolutely no sense to me.

    Can anyone shed any light on this?

    Update:
    ifconfig and pfctl -sr as requested by someone on IRC: https://gist.github.com/tandyuk/b8d97d127f2e20f9624f



  • Just to note that all the above was done on pfsense 2.1.5, and the router has now been upgraded to 2.2.4 with exactly the same issues present.



  • Ok turns out this is an actual bug!

    https://redmine.pfsense.org/issues/5258

    If you suffer from this, System -> Advanced > Firewall/NAT and Disable reply-to rules (tick the box).
    Not sure what multi WAN ipv6 users can do to fix it.


Log in to reply