IPv6 with HE Tunnel: ping works, but TCP fails to establish

xurubin · Aug 15, 2016, 9:18 PM

I have set up a HE IPv6 tunnel using the official guide (https://doc.pfsense.org/index.php/Using_IPv6_with_a_Tunnel_Broker). I'm able to ping any ipv6 addresses from pfsense, and gif0's IPv6 address is also pingable from another host. The problem is, I cannot initiate TCP connection from pfsense, nor can another host connect to the pfsense box via TCP.

I did a tcpdump on gif0 and vtnet1 (my WAN interface). It looks like when initiating a TCP connection from pfsense for example by "curl -v -6 http://ipv6.google.com", during the three-way handshake, the third ACK packet is sent out (visible on both dumps, see attached screenshots dump-gif0.png and dump-vtnet1.png) but somehow not received by the remote host (a retransmission of SYN/ACK packet is received after around 1 second).

When receiving a TCP connection directed at gif0 (by doing "nc -l -6 8080" on my pfsense box, and performing "curl -6 http://my_ipv6_address:8080" from another IPv6 capable host), it's slightly different: on gif0, there is the initial SYN received followed outgoing SYN/ACK, but the third ACK is not received so SYN/ACK is re-transmitted (screenshot dump-listening-gif0.png). However the pcap on vtnet1 does contain the incoming third ACK (screenshot dump-listening-vtnet1.png), suggesting some packet filtering issue. In both cases there is no indication of dropped packets related to the Ipv6 addresses in question in pfsense's system log/firewall.

I have set the firewall rule to very permissive ones (screenshots rule-ipv6.png, rule-lan.png, rule-ipv6.png): in the LAN tab, allowing IPv6 from LAN subnet; in the GIF interface tab, allowing IPv6 on any source or destination. The pfsense box's WAN interface sits behind my ISP's modem/router as a DMZ, but I doubt this matters as IPv6 ping over the HE tunnel works both ways.

The output of pfctl -sr is attached as pfctl.txt

Any help is appreciated. This problem has bugged me for days :(

pfctl.txt

bimmerdriver · Aug 15, 2016, 6:57 PM

@xurubin:

I have set up a HE IPv6 tunnel using the official guide. I'm able to ping any ipv6 addresses from pfsense, and gif0's IPv6 address is also pingable from another host. The problem is, I cannot initiate TCP connection from pfsense, nor can another host connect to the pfsense box via TCP.

[deleted]

Any help is appreciated. This problem has bugged me for days :(

I'm using pfsense with an HE tunnel. It has been working perfectly for months. The only thing I notice is that speedtest on ipv4 is faster than through the tunnel, which is not unexpected.

I'm not clear what your problem is. Do you literally mean connect to pfsense or do you really mean connect to a host on the lan?

If go to test-ipv6.com and ipv6-test.com, what results do you get? Do either of them show any issues? (You have to add pfsense firewall rules to pass icmp echo-request, as well as icmpv6 echo-request on the client.) Except for the lack of a host name, both of those websites show no problems.

xurubin · Aug 15, 2016, 7:13 PM

I meant connecting to any outside IPv6 sites via TCP from pfsense does not work (for example curl -6 http://ipv6.google.com just timeout). The exact symptom is TCP three-way handshake fails to complete, which I found out via tcpdump as in my initial post.

johnpoz · Aug 15, 2016, 8:51 PM

I run HE tunnel on pfsense 2.3.2 and have no issues at all.. Have both /64 and multiple segments from my /48 running..


[2.3.2-RELEASE][root@pfSense.local.lan]/tmp: curl -6 http://ipv6.google.com
<title>Google</title>

official guide where? Can you actually post up your config and your firewall rules (screenshots are so much better than asci art) so we can see what you might have done wrong. Your saying like pfsense itself can not do curl to some ipv6 site like you posted..

But you are saying you get back syn,ack from your syn or you don't get anything back at all?

xurubin · Aug 15, 2016, 9:20 PM

I uploaded screenshots for the firewall rules as well as wireshark captures in the initial post. Hope they make sense.

grandrivers · Aug 15, 2016, 11:01 PM

try setting gateway for your ipv6 lan rule

xurubin · Aug 16, 2016, 9:42 AM

Thanks grandrivers. Tried to set gateway on both/either LAN/HENETV6 ipv6 rules, but still no luck.

johnpoz · Aug 16, 2016, 11:51 AM

what is the point of those rules on your lan below the any any for tcp to 46.101 ?? Anything below your any any rule never going to be seen..

I see no hits on your ipv6 rule in your lan?

Looks like your getting dupes of the syn,ack - so they are never seeing your ack.. Did you contact HE?

Can you post up config of your gif, and your he interface.. Your not getting any sort of IPv6 from your isp are you on your wan? Your he tunnel should be listed as default gateway in routes section you should not setup any specific gateways on any rules.

Your testing this all from pfsense, what happens when you test from client on your lan?

andrewknots · Aug 16, 2016, 2:09 PM

I have had the same problem since I moved. Last night I downloaded and re-flashed the very latest pfSense release just in case…

I attach 4 tcpdump run on the endpoints of attempts to run ssh by way of an HE tunnel terminated on my pfSense firewall. SYN and Reset packets get through but the conversation gets no further

It may or may not be relevant that I'm connecting to an HE server in London 216.66.80.26, and it appears that the missing packets are disappearing before being delivered to my pfSense down the HE tunnel

SSH_to_Internet_Source.txt
SSH_to_Internet_Dest.txt
SSH_from_Internet_Source.txt
SSH_from_Internet_Dest.txt

yon · Aug 16, 2016, 3:16 PM

me too, my he.net ipv6 not work now. I had use he.net ipv6 with pfsense years. but this pf 2.3.1-2.3.2 version not normal work.

bimmerdriver · Aug 16, 2016, 6:25 PM

My system is working perfectly. I had to reboot the modem a few days ago and it came back up on its own. I'm currently running 2.3.1-RELEASE-p5.

xurubin · Aug 16, 2016, 11:56 PM

After much Googling, I have now a suspicion that this has something to do with my ISP (UK Sky Fiber) or the router it provides (SR102): http://helpforum.sky.com/t5/Fibre/Hurricane-electric-IPv6-Tunnel-Broker-on-BSkyB/td-p/2366610. That would probably explain the missing ACK when connecting to remote sites, if the router/ISP is dropping it. However, it's not apparent why the incoming connection behaves strangely: the WAN port does receive the tunneled TCP handshake packets, but pfsense is not passing them through to gif0. Unless maybe the packet is mangled somehow by the ISP causing pfsense to consider it invalid and drop it?

I'm attaching pcap files on the WAN & gif0 interfaces for the incoming connection case, which are what the screenshots in #1 are based on.

dump-listening-vtnet1.pcap
dump-listening-gif0.pcap

andrewknots · Aug 17, 2016, 2:10 PM

After discussing with Alex of HE I dropped the MSS way down to no avail.

I then switched to HE's other London Tunnel server, again no joy

I can ping6 and traceroute6 (ie ICMP and UDP), and syn packets get through. However using the pfSense packet Capture on the tunnel I only get the inbound SYN packet, followed by the target host sending syn-ack packets back to the initiator, which does receive them, and sends an ack which does not appear in the tunnel packet capture.

I get no further traffic from the initiator of the connection INBOUND from HE

Inbound-tunnel-traffic.txt

Paint · Aug 17, 2016, 4:30 PM

@andrewknots:

After discussing with Alex of HE I dropped the MSS way down to no avail.

I then switched to HE's other London Tunnel server, again no joy

I can ping6 and traceroute6 (ie ICMP and UDP), and syn packets get through. However using the pfSense packet Capture on the tunnel I only get the inbound SYN packet, followed by the target host sending syn-ack packets back to the initiator, which does receive them, and sends an ack which does not appear in the tunnel packet capture.

I get no further traffic from the initiator of the connection INBOUND from HE

weird… maybe it is your ISP? I have no problems with pfSense and he.net tunnels in NYC

Gertjan · Sep 24, 2016, 6:14 AM

More weird, because : The 'tunnel' to London, UK 216.66.80.26 is nothing less or more as a TCPv4 stream, which contains the IPv6 connection.

Your ISP is filtering these kind of packets ?
Did you check your modem / device that coverts your local Ethernet to the cable / phone (ADSL) / whatever ?

andrewknots · Sep 24, 2016, 8:22 AM

My ISP decided they could supply IPv6 directly (Zen Internet) so I've now dropped the HE tunnel in favour of native support. Sorry if that means I'm not helping others to resolve the issue, but for me it is a satisfactory outcome.

Thanks everyone for the help and support

xurubin · Jan 14, 2018, 9:38 AM

Some updates:

I recently switched to a new ISP (BT Infinity) so decided to give this another go. Unfortunately the exact same ACK dropping issue still happens with BT's Smart Hub (Home Hub 6A). This time I come across this post https://ttlexpired.co.uk/2016/02/12/ipv6-tunnel-and-failing-tcp-sessions/ describing a very similar issue from an engineer working for SKY and he concluded this is a bug with Broadcom SoC's "flow cache" mechanism, and by disabling flow cache the issue can be mitigated. Both my old SKY router SR102 and BT's new hub use Broadcom's SoC, so I have a strong suspicion that this is indeed the root cause.

I'm no longer with SKY so can't experiment with it, but for anyone stumbling across this post via Google, you might be able to play with it by compiling your own SR102 firmware from SKY's GPL tarball and try to disable flow cache. Unfortunately BT has yet to release the source code for its Smart Hub, so I'm still stuck.