IPv6 PPPoE MSS incorrect

BM118

Hi All,

I am wondering if anyone has had any issues similar and if I am doing something wrong, or if this is just a bug.

With IPv6 enabled and with the systems preferring IPv6 over IPv4, I lose access to some websites, with most of them providing a "took too long to respond" error page. Google and test-ipv6.com work, but forum.netgate.com or ipv6-test.com don't and Wireshark shows the three-way TCP and TLS Client Hello, but we never see the Server Hello, which to me indicates an MTU/MSS or Fragmentation issue somewhere. My findings are below. IPv4 works perfectly.

The client starts the connection with the SYN requesting an MSS of 1440 as I would expect; 1500 MTU - 20 TCP - 40 IPv6 = 1440 MSS. As the packet goes through PFSense and out PPPoE, it does not change the MSS as expected, and therefore the far end agrees on 1440 as well, unless the far end is configured lower. My understanding is it should instead be; 1500 MTU - 8 PPPoE - 20 TCP - 40 IPv6 = 1432 MSS as it passes through the PPPoE.
The exception to this is Google and test-ipv6.com who is limiting their side, i.e. 1360, which is why they work.

Looking at the logs and traces I have, it appears that PFsense is incorrectly applying the IPv4 calculations to IPv6 packets; 1500 MTU - 8 PPPoE - 20 TCP - 20 IPv4 = 1452 MSS. Therefore IPv6 packets have the IPv4 40-byte subtraction instead of the required IPv6 60-byte subtraction when calculating MSS and will, therefore, use the 1440 MSS as offered by the client as it is lower.

If I set the MSS on the PPPoE interface manually to 1472, therefore, accommodating the additional 20 bytes of IPv6 header manually, then I see the MSS is adjusted as expected, and the far end negotiates down to 1432 and everything works. Leaving the MTU blank correctly calculates a 1492 MTU for the PPPoE interface.
My only problem with this is that IPv4 is also using the 1432 MSS as well and while this is Ok in the grand scheme, it is not ideal for the perfectionist within me.

My setup is 2 x PFsense virtuals, 2.4.5-Release, one on VMWare and one on Hyper-V using CARP and PPPoE failover between the two (WAN and PPPoE Bridge are two different interfaces as per the CARP set up for PPPoE failover). The client OS is a Windows 10 machine, and others generally have the same problem, i.e. Ubuntu.

msmith100

I would rather not revive an old topic, but this post is exactly the problem I was having. Some websites would fail to load (e.g. yahoo.com), or take a long time/timeout (wiki.archlinux.org), and setting the MSS to 1472 makes everything work just fine. I have a PPPoE connection with native IPv4 and IPv6, no HE.net tunnel or anything.

I'm running 2.4.5-RELEASE-p1 and this problem still appears to exist.

viktor_g

I can only see such issue in pfSense 1.2.3 :https://redmine.pfsense.org/issues/1886

2.5.0-RC sets MTU of the interface and doesn't change the MSS value:

# ifconfig pppoe0
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
	description: SYNC
	inet 10.54.57.0 --> 10.54.54.6 netmask 0xffffffff
	inet6 fe80::5a9c:fcff:fe10:ffe5%pppoe0 prefixlen 64 scopeid 0x8
	inet6 fc00:6868::8 prefixlen 64
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
# pfctl -sr | grep mss

msmith100

@viktor_g Thanks for looking into this.

My current (working) setup:

[2.4.5-RELEASE][root@pfSense]/root: ifconfig pppoe0
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 104.163.xxxx.xxxx--> 96.127.xxxx.xxxx netmask 0xffffffff
        inet6 fe80::2e0:4cff:fe20:1476%pppoe0 prefixlen 64 scopeid 0x9
        inet6 2606:6d00:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
[2.4.5-RELEASE][root@pfSense]/root: pfctl -sr | grep mss
scrub on pppoe0 all max-mss 1432 fragment reassemble

Issue #1886 seems to be different from mine, as far as I can tell. I suppose my problem can be broken down into two questions:

I don't know if PMTUD is working or not. As I understand it, by default pfsense does not block any essential IPv6-related ICMP. I have passed tests such as http://icmpcheckv6.popcount.org or verifying that tracepath shows pmtu 1492 on Ubuntu but I'm a little suspicious it's giving me a false positive. Is there a surefire way of checking?
Setting the MSS explicitly to 1472 on the WAN was the only solution that resulted in 100% reliable web browsing. However, like the original post I'm not crazy about forcing the IPv4 MSS 20 bytes smaller than it should be - is there a way to set the IPv4 and IPv6 MSS separately?

viktor_g

Please create a bugreport:
https://docs.netgate.com/pfsense/en/latest/development/bug-reports.html

JKnott

@viktor_g said in IPv6 PPPoE MSS incorrect:

2.5.0-RC sets MTU of the interface and doesn't change the MSS value:

My understanding is the MTU sets the maximum size for packets created by pfsense, whereas MSS limits the size of packets passing through it. If the previous MTU is too big to pass through pfsense, either the packet will be fragmented or a too big ICMP message will be sent back to the source. The MSS will tell the source to not send such a big packet.

viktor_g

@msmith100 have you configured MSS manually on the interfaces page?

viktor_g

@bm118 said in IPv6 PPPoE MSS incorrect:

Looking at the logs and traces I have, it appears that PFsense is incorrectly applying the IPv4 calculations to IPv6 packets; 1500 MTU - 8 PPPoE - 20 TCP - 20 IPv4 = 1452 MSS. Therefore IPv6 packets have the IPv4 40-byte subtraction instead of the required IPv6 60-byte subtraction when calculating MSS and will, therefore, use the 1440 MSS as offered by the client as it is lower.

seems to be true:
https://github.com/pfsense/pfsense/blob/11fd7da72502c991b1f1c0e886ea212235f4a505/src/etc/inc/filter.inc#L640

msmith100

@viktor_g Yes I have set MSS manually, to 1472.

A screenshot of my settings is attached. I'll also create a bug report shortly.

viktor_g

@msmith100 The PPPoE interfaces already have a TCP MSS Fix option: https://docs.netgate.com/pfsense/en/latest/interfaces/ppp.html#advanced-ppp-options
Could you try to remove MSS settings from the interface and check again?

msmith100

@viktor_g I tried removing the MSS setting, and leaving TCP MSS fix on - i.e. not checking the disable box for it in the GUI. I can confirm that my issues accessing websites such as yahoo.com re-appear.

If I disable IPv6 on my client computer, accessing the same websites works just fine.

msmith100

@viktor_g @BM118 Bug report created as requested: https://redmine.pfsense.org/issues/11409

BM118

@viktor_g this is 100% the problem. It’s applying 40 bytes no matter what from my quick check. If that value was updated to 60 bytes it would alleviate needing to set the MSS to 20bytes lower than MTU.

However this is a workaround that would prevent people having to apply their own workaround, and the ideal fix would be to have different MSS applied based on IPv4 or IPv6. Mind you, I don’t know how how other vendors do it, or if they simply apply the standard 40 byte, and then change that to 60byte the moment IPv6 is enabled on the interface which achieves the same as the above.

For others reading this, as a general rule, don’t ever disable TCP MSS adjust settings, in this age of the internet you will have problems somewhere at some point, even if you don’t see it straight away. Also keep in mind that adjusting MSS only affects TCP and not UDP, which is even more reliant on a correct MTU than TCP is.

BM118

@msmith100 said in IPv6 PPPoE MSS incorrect:

@viktor_g Thanks for looking into this.
As I understand it, by default pfsense does not block any essential IPv6-related ICMP.

Regarding point 1, in a sense yes it does. IPv6 requires certain ICMP messages to be sent back to the source of a connection, which is an inbound new connection from a firewalls perspective. Specifically “Packet Too Big” messages amongst a couple of others that are critical to IPv6 working properly. Firewalls deliberately will block these packets unless you allow them inbound on your Outside/WAN interface.

IPv6 does not allow packets to be fragmented by intermediate routers like IPv4 does, and therefore the source will never know that the packet needs to be smaller to get through hop C when going from A to D if you don’t allow that packet inbound.
In IPv4 land, router C could have fragmented it and the packet would get there but this causes scalability and efficiency issues in the internet in general.

When using IPv6-test.com, it has a test to see if your are accepting ICMP messages or not, and even has a screen where it sends a constant ping against your IPv6 and IPv4 address which is very useful to see if ICMPv6 messages are making it all the way to your system.

Hint: if they aren’t succeeding, check your local system firewall as well as the PFSense rules.

msmith100

@bm118 Looking through the raw rules in pfsense, I see this:

pass quick inet6 proto ipv6-icmp all icmp6-type unreach keep state
pass quick inet6 proto ipv6-icmp all icmp6-type toobig keep state
pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state
pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state

These are not anything I created, and I believe part of the default pfsense rules. If I understand them correctly, they mean that by default pfsense will pass along "packet too big" messages without issue. I also allow pings with one of my own rules, and I pass the IPv6-test.com tests no problem. Doesn't help me on other sites though, as I think some sites like yahoo.com don't do PMTUD properly. Even for sites that do, the delay while it figures out the proper size is very noticeable and undesirable.

viktor_g

@bm118

Could you test this patch: 135.diff

You need to install System Patches pkg:
https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

msmith100

@viktor_g Applied on 2.4.5 without issue and confirmed working on all my problem websites with 1492 entered as MSS into WAN configuration page.

[2.4.5-RELEASE][root@pfSense]/root: pfctl -sr | grep mss
scrub on pppoe0 inet all max-mss 1452 fragment reassemble
scrub on pppoe0 inet6 all max-mss 1432 fragment reassemble

Thank you very much!

EDIT: May also want to update the description on the interfaces page:
If a value is entered in this field, then MSS clamping for TCP connections to the value entered above minus 40 (TCP/IP header size) will be in effect.

dem

Perhaps slightly off topic, but my IPv6 is via 6rd over PPPoE and when I was having PMTUD issues what I did was patch the radvd config to advertise a lower MTU.

viktor_g

@dem please try the updated patch with 6rd/6to4 support: 135 (1).diff

dem

@viktor_g Thanks, but I'm unable test with my production system right now.