HE Tunnel Problem
-
I'm using an HE tunnel for ipv6. I've never had any problems with it over a long period of time. Lately, I can't access mail.yahoo.com or speedtest.xfinity.com. If I disable ipv6, I can access both sites. tracert -4 and -6 both work. In order to eliminate a problem with pfsense, I upgraded from 2.3.1_5 to 2.3.2_1. That made no difference. I also have a pfsense 2.4 development system that's connected using native ipv6. The problem doesn't exist on that system so it appears to be a tunnel-related issue.
The problem is independent of which PC and which browser. It also occurs on mobile phones. Both ipv6-test.com and test-ipv6.com report no issues.
I posted on the HE forum and the response was the following:
If traces work but HTTP(S) doesn't, that isn't a routing issue, that sounds more like PMTUD. Try tuning your MTU on both sides of the tunnel.
I'm not sure how to investigate this or how to tune MTU. Any suggestions?
-
What is your WAN type?
You can adjust the MTU for your tunnel in your he.net settings by logging into tunnelbroker.net, click the tunnel and then click "Advanced". This is especially important if your WAN is PPPoE or has similar encapsulation that naturally reduces the MTU.
I'd login and lower it a bit, 1452 or lower.
You shouldn't have to do anything on the pfSense end, but you could set an MSS value on your assigned HE.net interface if you wanted.
-
I ran the ICSI / UC Berkeley Netalyzer. The DNS portion of the result is attached below. For no reason in particular, pfsense is configured to use the forwarder. Considering the result, should I try switching to the resolver to see if the results change?
-
What is your WAN type?
You can adjust the MTU for your tunnel in your he.net settings by logging into tunnelbroker.net, click the tunnel and then click "Advanced". This is especially important if your WAN is PPPoE or has similar encapsulation that naturally reduces the MTU.
I'd login and lower it a bit, 1452 or lower.
You shouldn't have to do anything on the pfSense end, but you could set an MSS value on your assigned HE.net interface if you wanted.
Your post arrived right as I posted. The WAN is PPPoE, DHCP. I ran Netalyzer again and here is what it reported for MTU:
The path between your network and our system supports an MTU of at least 2037 bytes, and the path between our system and your network has an MTU of 1500 bytes.
My other system (the one that doesn't have this problem) is set up to use the resolver, not the forwarder.
-
When I ran the test, ipv6 was disabled on the client. I enabled it and ran the test again. The results are attached. It looks like there are a few problems. It's strange because up to now, I haven't had any such problems.
 -
I changed the dns settings for the system that's not working. It's now configured to use the resolver and all dns related settings are set to default values, exactly the same as the 2.4 system. That didn't make any difference. Netalyzer is reporting this problem:
One popular name has a moderate anomaly: we are unable to find a reverse name associated with the IP address provided by your ISP's DNS server, although we expected to find a name. This is most likely due to a slow responding DNS server. If you rerun Netalyzr and see this condition remain, it could be due to a misconfiguration on the part of the domain owner, deliberate blocking using DNS, or your DNS server could be misconfigured or enabling a Man-in-the-Middle attack.
I ran netalyzer on my other system that doesn't have the problem accessing mail.yahoo.com. It reports the same thing, so perhaps that's not the cause of the problem.
Also, I reconfigured MTU down to 1280 per advice from the HE forum. It made no difference.
What's also interesting is that netalyzer is reporting fragmentation problems on the 2.4 system:
Your system can send fragmented traffic, but can not receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1496 bytes. The path between our system and your network has an MTU of 1500 bytes. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.Is this normal?
-
Changing MTU didn't have any effect on the problem, so I'm setting up another 2.4 system with a tunnel. If the 2.4 system with native ipv6 works and the 2.4 system with the tunnel doesn't work that will narrow the problem down.
I tried to setup the tunnel on the 2.4 system similarly to the tunnel on the 2.3.2_1 system, but it's not working. I compared all of the status pages and settings, but I haven't found anything that could be the cause.
The tunnel appears to be operational. It's online and there is some traffic going in both directions, probably from dpinger. DHCP4 / ipv4 are working fine. DHCP6 is working. I can ping ipv6 addresses of the pfsense LAN interface from the pc client, as well as both ends of the tunnel, but I can't ping any foreign hosts such as google.com using ipv6. The name seems to resolve, but the pings all get dropped. Similarly for tracert. Nothing is being logged in the firewall, so I wonder if the problem is routing.
Any suggestions? I can post screen captures.
-
I compared Diagnostics / Routes of the working and not-working systems. The working system has a default ipv6 route that points at the server end of the route. There is no equivalent route on the not-working system. I didn't set up the working system but I don't think any routes had to be manually created. In System / Routing / Gateways, the tunnel gateway is set to be the default ipv6 route on both systems. Is it possible that this is the problem? If so, how do I fix it?
-
If you assigned the tunnel interface and set it all properly, and set the gateway as default, it should handle that on its own:
https://doc.pfsense.org/index.php/Using_IPv6_with_a_Tunnel_Broker
There is a problem on 2.4 with setting gateways in some cases but that wouldn't affect your 2.3.x box
-
If you assigned the tunnel interface and set it all properly, and set the gateway as default, it should handle that on its own:
https://doc.pfsense.org/index.php/Using_IPv6_with_a_Tunnel_Broker
There is a problem on 2.4 with setting gateways in some cases but that wouldn't affect your 2.3.x box
Thank you for the reply. I looked at that procedure. The tunnel on the 2.3.2_1 system is the one that's working. I'm having problems getting it working on 2.4.
-
In the first post of the thread you said the one that wasn't working was 2.3.2_1, or did I misread that?
If it fails on 2.4 with a tunnel, then you're probably hitting this:
https://redmine.pfsense.org/issues/6828Any static interface with a gateway will not have a default route due to that, until we fix it. 2.4 isn't meant to be used in production or for more than lab testing at the moment.
-
In the first post of the thread you said the one that wasn't working was 2.3.2_1, or did I misread that?
If it fails on 2.4 with a tunnel, then you're probably hitting this:
https://redmine.pfsense.org/issues/6828Any static interface with a gateway will not have a default route due to that, until we fix it. 2.4 isn't meant to be used in production or for more than lab testing at the moment.
Sorry if my original post wasn't clear. My "production" system is running 2.3 (currently 2.3.2_1) and it uses an HE tunnel for ipv6. This system has been working fine for months. I also have a test system which is running 2.4 and it is providing native ipv6. (I have to use 2.4 because the "dhcp6 before RA" fix isn't in the release version yet.) Recently I found that it's not possible to access mail.yahoo.com from the 2.3 system (on any host), but it was working on the test system. I tried tuning MTU to see if it would help, but it didn't make any difference. I'm trying to set up a 2.4 system using a tunnel to confirm that the problem is being caused by the tunnel, not pfsense.
The bug report sounds like exactly the problem.
-
I ran the ICSI / UC Berkeley Netalyzer. The DNS portion of the result is attached below. For no reason in particular, pfsense is configured to use the forwarder. Considering the result, should I try switching to the resolver to see if the results change?
When I was using a tunnel, it was set to the minimum IPv6 MTU of 1280. That should work with anything. Once that's verified to work, you can try larger MTUs.
-
I ran the ICSI / UC Berkeley Netalyzer. The DNS portion of the result is attached below. For no reason in particular, pfsense is configured to use the forwarder. Considering the result, should I try switching to the resolver to see if the results change?
When I was using a tunnel, it was set to the minimum IPv6 MTU of 1280. That should work with anything. Once that's verified to work, you can try larger MTUs.
As I said above, I tried lowering MTU all the way down to 1280 and it made no difference. (I was changing the setting of the tunnel and the opt1 interface.) Currently, the tunnel is set to the default MTU, which is 1480, and the opt1 interface is left blank. You can see below that one of the routes has an MTU of 1280. Not sure where that came from.
IPv6 Routes
Destination, Gateway, Flags, Use, Mtu, Netif
default, 2001:xxx:c:211::1, UGS, 45566, 1500, gif0
::1, link#4, UH, 0, 16384, lo0
2001:xxx:c:211::1, link#7, UH, 318447, 1280, gif0
2001:xxx:c:211::2, link#7, UHS, 0, 16384, lo0
2001:xxx:d:211::/64, link#5, U, 150524, 1500, hn0
2001:xxx:d:211::1, link#5, UHS, 1, 16384, lo0Also, when I used the iea-software.com mtupath.exe utility, it showed 1500 to google.com and 1480 to mail.yahoo.com using ipv6, so I'm not convinced mtu has anything to do with this issue.
-
^^^^
I read your later comments after I posted that. Sorry. -
-
Does anyone know why the bold route has an MTU of 1280? Where is it defined? The destination of that route is the server end of the tunnel. OPT1 is currently configured with the default MTU.
IPv6 Routes
Destination, Gateway, Flags, Use, Mtu, Netif
default, 2001:xxx:c:211::1, UGS, 45566, 1500, gif0
::1, link#4, UH, 0, 16384, lo0
2001:xxx:c:211::1, link#7, UH, 318447, 1280, gif0
2001:xxx:c:211::2, link#7, UHS, 0, 16384, lo0
2001:xxx:d:211::/64, link#5, U, 150524, 1500, hn0
2001:xxx:d:211::1, link#5, UHS, 1, 16384, lo0I did some testing using ping -6. I found that the largest size for mail.yahoo.com is 1432 and the largest size for google.com is 1452. Does that imply the MSS is 1432 and 1452, respectively? I ran mtupath and it reported 1452 MSS for both, suggesting 1500 "estimated MTU". Not sure why there's a discrepancy. What should I add to the MSS to get the MTU?
-
I did some additional testing and I'd appreciate some feedback. I can access mail.yahoo.com on my test pfsense system, running 2.4 with native ipv6. I can also access it using a host through a mullvad vpn (dual-stack) terminating in the USA. I cannot access it using my "production" pfsense system, running 2.3.2_1 and with an HE tunnel.
I noticed when I looked at the route table that the route to the server end of the tunnel had mtu of 1280 so I set the mtu of opt1 and the tunnel to 1280. It did not fix the problem. I still can't load mail.yahoo.com. Is there a way to change the MTU? I'm not clear how it got set to 1280 in the first place, because the MTU of opt1 was originally left at the default.
I also did a ping test to mail.yahoo.com and google.com using the 1280 mtu setting. I can ping -6 both of them up to 1232 bytes.
I've attached the ipv6 routes, opt1 configuration, tracert -6 mail.yahoo.com and the results of the dns and ipv6 sections of the netalyzer test. What's interesting is that now netalyzr is now saying the bottleneck is within the HE network, at 2001:470:0:9d::2. Does that indicate a problem in the HE network? I don't even know if MTU is the cause of this problem or whether it's dns (or something else).
-
I am also having this exact same issue on 2.3.2 (about to upgrade to 2.3.2_1).
I can't get to mail.yahoo.com, kb.vmware.com, my.vmware.com.. and pandora won't stream (though I'm not 100% sure that's related)
[2.3.2-RELEASE][root@router]/root: ping6 -s 1405 -d my.vmware.com
PING6(1453=40+8+1405 bytes) 2001:470:1f10:xxxx::2 –> 2a02:e980:39::13
^C
--- 5alxq.x.incapdns.net ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[2.3.2-RELEASE][root@router]/root: ping6 -s 1404 -d my.vmware.com
PING6(1452=40+8+1404 bytes) 2001:470:1f10:xxxx::2 –> 2a02:e980:39::13
1412 bytes from 2a02:e980:39::13, icmp_seq=0 hlim=61 time=25.633 ms
1412 bytes from 2a02:e980:39::13, icmp_seq=1 hlim=61 time=26.127 ms
^C
--- 5alxq.x.incapdns.net ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 25.633/25.880/26.127/0.247 ms
[2.3.2-RELEASE][root@router]/root:I just changed my MTU on my IPv6 Tunnel interface to 1280 and I am able to get to the vmware sites now, but still not mail.yahoo.com.
What is causing the requirement for the MTU to be set so incredibly low? I know it's data -> IPv6 -> GRE -> IPv4, so there's a bunch of headers on there, but I've never seen a link that low. I'm wondering if there's anything I can do to fix this.
-
I am also having this exact same issue on 2.3.2 (about to upgrade to 2.3.2_1).
I can't get to mail.yahoo.com, kb.vmware.com, my.vmware.com.. and pandora won't stream (though I'm not 100% sure that's related)
[2.3.2-RELEASE][root@router]/root: ping6 -s 1405 -d my.vmware.com
PING6(1453=40+8+1405 bytes) 2001:470:1f10:xxxx::2 –> 2a02:e980:39::13
^C
--- 5alxq.x.incapdns.net ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
[2.3.2-RELEASE][root@router]/root: ping6 -s 1404 -d my.vmware.com
PING6(1452=40+8+1404 bytes) 2001:470:1f10:xxxx::2 –> 2a02:e980:39::13
1412 bytes from 2a02:e980:39::13, icmp_seq=0 hlim=61 time=25.633 ms
1412 bytes from 2a02:e980:39::13, icmp_seq=1 hlim=61 time=26.127 ms
^C
--- 5alxq.x.incapdns.net ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 25.633/25.880/26.127/0.247 ms
[2.3.2-RELEASE][root@router]/root:I just changed my MTU on my IPv6 Tunnel interface to 1280 and I am able to get to the vmware sites now, but still not mail.yahoo.com.
What is causing the requirement for the MTU to be set so incredibly low? I know it's data -> IPv6 -> GRE -> IPv4, so there's a bunch of headers on there, but I've never seen a link that low. I'm wondering if there's anything I can do to fix this.
Sorry to hear you are also experiencing this problem, although at least it's confirmation that something is wrong. After testing with ping and mtupath, I'm not convinced this is an MTU issue. Both methods of testing indicate the MTU is 1480. I manually configured the routes to 1480, since pfsense sets them to 1280. It made no difference.
However, I read the article "Evaluating IPv4 and IPv6 Packet Fragmentation" from RIPE on path fragmentation: https://labs.ripe.net/Members/gih/evaluating-ipv4-and-ipv6-packet-fragmentation and the behaviour really does seem like a result of path fragmentation. Considering that this problem arose with no changes to pfsense or the tunnelbroker service (according to Hurricane Electric) and it stayed after I upgraded to 2.3.2_1 and replaced the tunnel, maybe something changed in yahoo's network.
