MSS has to be set manually for IPv6 to work correctly with PPPoE
-
TL;DR - set MSS to 1472 and disable TCPmssFix to make IPv6 work properly.
Forgive me for lack of knowledge of proper terminology as I am very new to IPv6 but after spending three days trying to configure it on pfSense I have finally discovered there seems to be some sort of bug within the software.
I am running CE 2.8.1 in Proxmox and the NIC is a Mellanox CX321B with interfaces passed to pfSense in software as there is no hardware passthrough possible.
My ISP is IDNet in the UK; it is an FTTP connection over PPPoE and they offer an IPv6 /48 block (including an address for the firewall itself) via DHCPv6 Prefix Delegation. My firewall WAN port is connected through a media converter to the fibre ONT.
I have WAN configured with the following settings:
IPv4 type: PPPoE
IPv6 type: DHCP6
Use IPv4 connectivity as parent interface
DHCPv6 Prefix Delegation size 48
Send IPv6 prefix hint
Then just PPPoE username and password since that is all is required for the connection.
In the interface status I can see pfSense has set the WAN MTU to 1492 which I believe is correct for PPPoE. The LAN MTU is 1500.For LAN interface I have IPv6 type set to Track Interface with WAN selected as the interface to be tracked, and using prefix ID 1.
For Router Advertisement I have Router Mode set to Unmanaged and I DHCPv6 Server is not enabled.
What this results in:
Firewall WAN interface gets assigned a routable IPv6 address and can ping IPv6 hosts on the internet
Firewall LAN interface gets assigned a routable IPv6 address
IPv6-capable devices on LAN get assigned a unique IPv6 address, and some (e.g. Windows also get a "Temporary IPv6 address" which I believe is some sort of privacy featureThe problem:
IPv6 usability is sporadic; pinging IPv6 hosts from LAN clients works intermittently or not at all; IPv6 tests on the internet sometimes work and say IPv6 is OK, sometimes work and say IPv6 is not OK, and sometimes do not load at all. Some IPv6 websites work (e.g. ipv6.google.com) sometimes but some do not load at all and the connection times out. If I disable IPv6 I am able to load those same websites fine over IPv4.So after a lot of back and forth with ChatGPT and trying several things with no luck I finally managed to figure out that in order to make IPv6 actually work properly and reliably, you need to set the following settings:
WAN interface MSS must be 1472
In PPPoE advanced settings, the "TCPmssFix" setting must be DISABLED (box ticked)After this, IPv6 works completely on all LAN clients, all websites load correctly over IPv6, all online IPv6 tests work and show 10/10 for IPv6 connectivity, and running a continuous ping to a remote IPv6 host results in zero packetloss even after hours.
-
@jpns I too discovered TCPmssfix doesnt work quite right, when I was investigating why fragment issues I had for a long time were mysteriously fixed with if_pppoe.
The good news is TCPmssfix does the same as manually clamping MSS now (scrub rules) when if_pppoe is enabled, on old PPPoE it used some kind of mechanism in the PPPoE software directly.
My preference however even with if_ppoe is to disable TCPmssfix and just manually configure the clamp I want.
I ended up using 1220 for broad compatibility, and what was originally agreed between network vendors as a default when IPv6 launched.
MTU is at 1500 for me (I have baby jumbo frames support ISP side and on ONT).
-
@chrcoluk how do you enable
if_pppoe? -
@jpns system->advanced->networking, scroll to bottom, toggle it and reboot. Needs a recent version of pfSense.
-
@chrcoluk I don't see it in CE 2.8.1. Is it only in the beta version?
-
said in MSS has to be set manually for IPv6 to work correctly with PPPoE:
@chrcoluk I don't see it in CE 2.8.1. Is it only in the beta version?
Never mind - the GUI option suddenly appeared (very weird) and I enabled it but it hasn't fixed the issue. I still have to disable tcpmssfix and set my MSS to a low value or IPv6 just does not work properly. I actually discovered earlier that 1452 is too high, and sometimes it still wouldn't work 100% of the time, so I have now lowered it to 1440.
-
@jpns Was TCPmssfix setting it to 1452?
Overhead for IPv6 is 60 bytes, so absolute maximum on a 1492 MTU would be 1432.
Also if you include timestamps IPv4 is better at 1440.
Unlike MTU this doesnt need to be a high as possible mentality, it doesnt start high and work its way down, any modern stack will work fine providing you give it a MSS thats compatible with your setup.
I just looked a the the 2.8.x filter.inc code and default behaviour if using if_pppoe and leaving MSS box unpopulated, it will apply a 40 byte overhead for IPv4 and 60 byte overhead for IPV6 via SCRUB rules.
Some info here, referenced in the code.
https://redmine.pfsense.org/issues/11409
-
@chrcoluk I'm not sure what TCPmassfix was setting it to - how do you find out?
It seems like I have more underlying issues:
Pinging ipv6.l.google.com [2a00:1450:4009:c0f::65] with 32 bytes of data: Destination host unreachable. Destination host unreachable. Destination host unreachable. Destination host unreachable. Destination host unreachable. Reply from 2a00:1450:4009:c0f::65: time=2356ms Reply from 2a00:1450:4009:c0f::65: time=11ms Request timed out. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out.I have now gone to if_pppoe with TCPmssfix enabled and no MSS manually set. IPv6 seems to be working fine now, but I will see if it lasts.
-
said in MSS has to be set manually for IPv6 to work correctly with PPPoE:
@chrcoluk I'm not sure what TCPmassfix was setting it to - how do you find out?
It seems like I have more underlying issues:
Pinging ipv6.l.google.com [2a00:1450:4009:c0f::65] with 32 bytes of data: Destination host unreachable. Destination host unreachable. Destination host unreachable. Destination host unreachable. Destination host unreachable. Reply from 2a00:1450:4009:c0f::65: time=2356ms Reply from 2a00:1450:4009:c0f::65: time=11ms Request timed out. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out.I have now gone to if_pppoe with TCPmssfix enabled and no MSS manually set. IPv6 seems to be working fine now, but I will see if it lasts.
Yep, here we go. After about an hour of it working perfectly, it just stops working for no reason:
Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=16ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Destination host unreachable. Request timed out. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Request timed out. Destination host unreachable. Destination host unreachable. Destination host unreachable. Request timed out. Request timed out.and if I reset the PPPoE connection:
Request timed out. Request timed out. Destination host unreachable. Request timed out. Request timed out. Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=15ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=16ms Reply from 2a00:1450:4009:c0f::65: time=13ms Reply from 2a00:1450:4009:c0f::65: time=14ms Reply from 2a00:1450:4009:c0f::65: time=12ms Reply from 2a00:1450:4009:c0f::65: time=13msIPv4 stays working perfectly all the time throughout this. Anyone know what could be wrong?
-
@jpns you seem to have underlying ipv6 problems, possibly at the ISP level, MSS will have no impact on 32byte UDP/ICMP pings.
It is a mechanism for large TCP packets.
I think the only way to see what legacy TCPMSSfix is doing is to inspect packet headers, its one reason I dont like using it. However on if_pppoe, it uses scrub rules, so this command should tell you what it is configuring max-mss to.
'grep max-mss /tmp/rules.debug'
But this wont be causing problems on your pings, something else is going on there.
-
@chrcoluk said in MSS has to be set manually for IPv6 to work correctly with PPPoE:
@jpns you seem to have underlying ipv6 problems, possibly at the ISP level, MSS will have no impact on 32byte UDP/ICMP pings.
It is a mechanism for large TCP packets.
I think the only way to see what legacy TCPMSSfix is doing is to inspect packet headers, its one reason I dont like using it. However on if_pppoe, it uses scrub rules, so this command should tell you what it is configuring max-mss to.
'grep max-mss /tmp/rules.debug'
But this wont be causing problems on your pings, something else is going on there.
I think you are right. From further testing it appears that this specific client has issues with IPv6. Other clients are working just fine.
-
Can I hijack this thread to ask about something similar?
My ISP has a very bad implementation of RFC4638 "baby jumbo frames". It properly supports MTU 1500 for IPv4, but hasn't bothered to make the same work over IPv6 where my MTU seems to be 1492.
I was wondering how can I make this work with 1500 for IPv4 and 1492 for IPv6. At one point you could send MTU info over RAs, but it seems that option has been taken out from recent versions of pfSense+.
Any other ideas? I know I could just use 1492 for both stacks and call it a day, but not having to fragment every big packets that goes outside my network does sound like a pretty good deal, even though it's just for IPv4.
-
@IonutIT MTU is a per interface value, so you have the same MTU for both stacks.
Just to confirm, you are saying when you set MTU to 1500, IPv6 can not deliver unfragmented 1500 byte packets?
If your ISP has set MTU too low to support a proper baby jumbo frames configuration, then I think you only have two options.
Run the 1492 MTU, or use single IPv4 stack only with 1500 MTU.
If you think you wont ever use large UDP packets, then you could just rely on MSS fix for IPv6 to at least ensure TCP is ok. But if you do this make sure you disable use of QUIC on your network. I wouldnt go down this route.
-
@chrcoluk said in MSS has to be set manually for IPv6 to work correctly with PPPoE:
Just to confirm, you are saying when you set MTU to 1500, IPv6 can not deliver unfragmented 1500 byte packets?
Yup. I can send 1500 bytes over IPv4 no issue, but can't send more than 1492 bytes over IPv6. Tried contacting my ISP, just got a Level 1 boilerplate answer "It's a feature that will be implemented at some point".