Unstable HE.NET tunnel with MTU > 1280

johnpoz

I use HE for my tunnel.. And get 10/10, that big packet test

Test IPv6 large packet
ok (0.190s) using ipv6

Tested via some mirrors that are outside NA, and not seeing any issues all 10/10

dusan

@johnpoz Maybe you have good connection to HE, so fragmentation is not a problem for you.

johnpoz

Maybe sure.. What is your closest pop? They have lots of them - which one are you using.. Did you check its status maybe its having issues?

https://tunnelbroker.net/status.php

Did you check with HE, they are very helpful!!

dusan

@johnpoz I check status every time I test. I checked with both HE status page and pfSense's Gateway status page. The PoP is online and gives normal latency at the test time. And as noted, at MTU 1280 bytes, it passed all tests.

I tested with 4 ISPs and 2 best PoPs (see below) and obtained the same result.

My HE connection is not very good. Around one [short] outage every few days. Two nearest PoPs are Singapore and Hong Kong, with latency of 184 ms and 187 ms, respectively. For comparison, Tokyo (JP) 220 ms, Phoenix (AZ, US) 235 ms, Amsterdam (NL) 240 ms. All latencies are measured at the moment. In peak time, normally latency jumps up by 10 - 50 ms more.

EDIT: the aforementioned latencies are measured for one ISP. The other 3 ISPs have much better latencies (30 - 50 ms) but much worse throughput (an order of magnitude). No matter which ISP I test, I get the same result.

johnpoz

Sounds like its your isp's issue and not HE.. HE has some of the most peering of any company..

https://he.net/peering.html

I would suggest you work with your ISPs to peer with HE, if your isp is not going to support IPv6.. Which I take it none of them do if your using HE?

Not sure what you think pfsense can do to help here? The min required mtu for IPv6 is 1280.. If your sending larger packets than that - then yeah you can all kinds of problems. If not supported end to end..

What do you get when you go here
http://ipv6-test.com/pmtud/

You should get back
Testing packet size: 1500
Result: no PMTUD problems detected

It will ramp up testing different packet sizes.. Maybe your issue is with PMTUD.. Which is really a requirement with IPv6..

jimp

https://redmine.pfsense.org/issues/6868

Gertjan

@johnpoz said in Unstable HE.NET tunnel with MTU > 1280:

You should get back
Testing packet size: 1500
Result: no PMTUD problems detected

That what I have. "no PMTUD problems detected"

But this to :

Nice catch.
Rare to see that the GUI gives more info as (basic) command line interface commands.

I applied what is proposed in https://redmine.pfsense.org/issues/6868 but that didn't change the "1280".

dusan

@johnpoz Exactly the contrary. Since local ISPs all offer IPv6, HE is not a friend to them. Akamai is. Local ISPs aren't trying to develop international connections. They try to force international content devilvery providers "moving" clouds into their so-called data centers in the country. It's cheaper to buy a congressman (hope I'm using the right term) than to pay a peering agreement or a share in regional or intercontinental optical cables.

Unfortunately I can't use IPv6 offered by local ISPs due to numerous unacceptable terms of use. Dynamic address, small address range, countless inter-AS routing problems, inability to operate ISP-agnostic DNS servers, ISP-locked CPE routers, btw full of bugs and errors, unwilling to answer customer's questions, incompetence of the sale and help desk in IPv6-specific issues. On every single incident, every ISP advises me "disable IPv6 and shut up".

Regarding PMTUD, I did see something like this

Testing packet size: 1500
Result: probable PMTUD problem detected beyond ~1500 bytes

But that occurred only a few times out of ~300 times I ran the PMTUD test during ~2h of peak time today (around 3 p.m. local time). I got the "no problems" result, exactly like yours, in 99% of the times.

So, it's not real PMTUD problem, I mean one caused by a mis-configured router/firewall on the path. It was caused by random packet loss, possibly amplified by fragmentation in the tunnel. By the way, for the purpose of testing, I've set the HE tunnel back to 1472-byte MTU.

pfSense can help me, I think, to eliminate or to reduce the silent fragmentation in the data link layer (i.e. IPv4 layer of the GIF tunnel).

@jimp Thank you for pointing out the relevant issue and the patch. Unfortunately, like Gertjan, it didn't work for me. The interface (or route?) MTU remains 1280 bytes. I tried deleting and re-creating the tunnel. I tried also rebooting the pfSense machine. No luck.

kiokoman

i have pppoe and my tunnel is set this way

10/10 everywhere
http://test-ipv6.hkg.vr.org/
http://test-ipv6.epic.network/
http://test-ipv6.sin.vr.org/

dusan

As noted, so long as you have zero-loss connection, fragmentation is not an issue. For example, my HE.NET connection is a corporate one and out of the peak times (around 3 p.m. work days) it passes all tests for any MTU <= 1472 and MSS <= MTU - 20. Problems arise only under heavy load.

By the way, the harder targets (to me) are those in Eastern Europe, Latin America, and Africa.