Unstable HE.NET tunnel with MTU > 1280
-
Hello,
I have an IPv6 connection to HE.NET. For users' complaints, identified as IPv6-related, I made a comprehensive test with http://test-ipv6.com and noticed my pfSense (x64 edition, release v.2.4.4 - p3) fails a fair amount of tests for many of the targets (i.e. mirrors), especially those beyond North America and West Europe.
Failed tests typically include "IPv6 big packet test" and "IPv6 big DNS query" test. Errors are classified as "slow" (> 5s) or "time-out" (> 15s).
It can pass all tests at (almost) all targets only if I set MTU = 1280 and MSS = 1260 at the HE.NET interface.
Looking at the route tables, I see a possible reason. The HE.NET interface was setup with big MTUs, say, 1480, 1472, 1400, 1350 bytes etc (and the "MSS" was set accordingly as MTU - 20 bytes) but the underlying GIF interface, say gif0, stays at 1280 bytes. Maybe this has caused excessive fragmentation and increased probability of fragment loss in peak times.
Is there a way to change the MTU for the underlying GIF interface?
1280-byte MTU is sub-optimal. For my Internet connection, which is PPPoE with MTU 1492 bytes, the ideal MTU would be 1472 bytes.
(I've search Forum and Redmine and found some relevant topics/issues. But all are Closed/Resolved. Maybe my observation is a new problem.)
-
I'm using he.net like you for IPv6.
@dusan said in Unstable HE.NET tunnel with MTU > 1280:
Is there a way to change the MTU for the underlying GIF interface?
On the "HE.NET" interface, I guess :
-
Actually your HENETV6 is the HE.NET interface, not the underlying GIF interface. It's similar to WAN interface: you set its MTU whatever you want, but the underlying interface (say, pppoe0) stays at 1492 bytes.
Attached figure shows that the GIF interface actually uses MTU of 1280 bytes.
-
I use HE for my tunnel.. And get 10/10, that big packet test
Test IPv6 large packet
ok (0.190s) using ipv6Tested via some mirrors that are outside NA, and not seeing any issues all 10/10
-
@johnpoz Maybe you have good connection to HE, so fragmentation is not a problem for you.
-
Maybe sure.. What is your closest pop? They have lots of them - which one are you using.. Did you check its status maybe its having issues?
https://tunnelbroker.net/status.php
Did you check with HE, they are very helpful!!
-
@johnpoz I check status every time I test. I checked with both HE status page and pfSense's Gateway status page. The PoP is online and gives normal latency at the test time. And as noted, at MTU 1280 bytes, it passed all tests.
I tested with 4 ISPs and 2 best PoPs (see below) and obtained the same result.
My HE connection is not very good. Around one [short] outage every few days. Two nearest PoPs are Singapore and Hong Kong, with latency of 184 ms and 187 ms, respectively. For comparison, Tokyo (JP) 220 ms, Phoenix (AZ, US) 235 ms, Amsterdam (NL) 240 ms. All latencies are measured at the moment. In peak time, normally latency jumps up by 10 - 50 ms more.
EDIT: the aforementioned latencies are measured for one ISP. The other 3 ISPs have much better latencies (30 - 50 ms) but much worse throughput (an order of magnitude). No matter which ISP I test, I get the same result.
-
Sounds like its your isp's issue and not HE.. HE has some of the most peering of any company..
https://he.net/peering.html
I would suggest you work with your ISPs to peer with HE, if your isp is not going to support IPv6.. Which I take it none of them do if your using HE?
Not sure what you think pfsense can do to help here? The min required mtu for IPv6 is 1280.. If your sending larger packets than that - then yeah you can all kinds of problems. If not supported end to end..
What do you get when you go here
http://ipv6-test.com/pmtud/You should get back
Testing packet size: 1500
Result: no PMTUD problems detectedIt will ramp up testing different packet sizes.. Maybe your issue is with PMTUD.. Which is really a requirement with IPv6..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
https://redmine.pfsense.org/issues/6868
-
@johnpoz said in Unstable HE.NET tunnel with MTU > 1280:
You should get back
Testing packet size: 1500
Result: no PMTUD problems detectedThat what I have. "no PMTUD problems detected"
But this to :
Nice catch.
Rare to see that the GUI gives more info as (basic) command line interface commands.I applied what is proposed in https://redmine.pfsense.org/issues/6868 but that didn't change the "1280".
-
@johnpoz Exactly the contrary. Since local ISPs all offer IPv6, HE is not a friend to them. Akamai is. Local ISPs aren't trying to develop international connections. They try to force international content devilvery providers "moving" clouds into their so-called data centers in the country. It's cheaper to buy a congressman (hope I'm using the right term) than to pay a peering agreement or a share in regional or intercontinental optical cables.
Unfortunately I can't use IPv6 offered by local ISPs due to numerous unacceptable terms of use. Dynamic address, small address range, countless inter-AS routing problems, inability to operate ISP-agnostic DNS servers, ISP-locked CPE routers, btw full of bugs and errors, unwilling to answer customer's questions, incompetence of the sale and help desk in IPv6-specific issues. On every single incident, every ISP advises me "disable IPv6 and shut up".
Regarding PMTUD, I did see something like this
Testing packet size: 1500 Result: probable PMTUD problem detected beyond ~1500 bytesBut that occurred only a few times out of ~300 times I ran the PMTUD test during ~2h of peak time today (around 3 p.m. local time). I got the "no problems" result, exactly like yours, in 99% of the times.
So, it's not real PMTUD problem, I mean one caused by a mis-configured router/firewall on the path. It was caused by random packet loss, possibly amplified by fragmentation in the tunnel. By the way, for the purpose of testing, I've set the HE tunnel back to 1472-byte MTU.
pfSense can help me, I think, to eliminate or to reduce the silent fragmentation in the data link layer (i.e. IPv4 layer of the GIF tunnel).
@jimp Thank you for pointing out the relevant issue and the patch. Unfortunately, like Gertjan, it didn't work for me. The interface (or route?) MTU remains 1280 bytes. I tried deleting and re-creating the tunnel. I tried also rebooting the pfSense machine. No luck.
-
i have pppoe and my tunnel is set this way
10/10 everywhere
http://test-ipv6.hkg.vr.org/
http://test-ipv6.epic.network/
http://test-ipv6.sin.vr.org/ -
As noted, so long as you have zero-loss connection, fragmentation is not an issue. For example, my HE.NET connection is a corporate one and out of the peak times (around 3 p.m. work days) it passes all tests for any MTU <= 1472 and MSS <= MTU - 20. Problems arise only under heavy load.
By the way, the harder targets (to me) are those in Eastern Europe, Latin America, and Africa.
