pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link)

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Unless you have complete control over the entire data path from source to destination, high probability is the data will get fragmented somewhere along the way.
Normally fragmentation isn't a big deal because they get reassmbled at some point.
Unless the traffic is UDP. UDP is "send and don't care if it actually gets there".
A lot of video traffic is UDP.

As I mentioned, the world is moving to PMTUD, not fragmentation. With IPv6, it's mandatory and is now used for much of IPv4. This means the sending host will be told the MTU is too big and will have to retransmit. Even with fragmentation, the packet does get reassembled, whether UDP or not. This is not a new issue and IP has been able to handle it from the start. As I pointed out, at one time, dial up connections used 576, while much of the world was 1500 and token ring was a few K. It was the routers between networks that enabled this to work smoothly.

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

TCP automatically retransmits lost packets which may result in a loss of usable bandwidth simply because packets are being retransmitted.

With UDP, it's up to the application to request retransmission. Often, it just accepts the loss. A loss packet in something like audio or video is less disruptive than the delay caused by retransmission. In some cases, forward error correction might be used, which means an occasional lost packet is irrelevant.

mer

@JKnott I don't completely disagree, but if a lost packet is a key frame of video, that is often pretty disruptive.
I agree on PMTUD, but sometimes it takes a little while for every one to get on the same page.

I also was agreeing that large packets/MTU shouldn't be an issue, but there may be cases it is.
That's all. No more, no less.

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

I also was agreeing that large packets/MTU shouldn't be an issue, but there may be cases it is.
That's all. No more, no less.

Of course, there's nothing that says large frames have to be used. VoIP is an example, where small packets are used, to minimize latency. Even with 1500, the amount of time it would take to fill a frame would cause a noticeable delay.

Bottom line, the OP can use 9000 MTU on his LAN if he wants. However, I'm curious as to how well WiFi works with it. I believe 802.11ac supports aggregation. I'm not sure about n.

RobbieTT

@JKnott
I run 1508 MTU on my WAN, 1500 MTU on my maintenance LAN and 9k set on my main LAN and VLAN at 10 GbE.

In this mixed environment I have 2 main switches, a couple of tertiary switches, 5 APs, numerous servers, clients and sundry networked items. Everything purrs along as it should.

️

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

9k set on my main LAN and VLAN at 10 GbE.

Have you tried Wifi on it? I'm curious if it will handle the larger frames. I believe you'll need to run 802.11ac or later to get the aggregation. I can't do that here, as my notebook computer is 12 years old and only runs 802.11n on 2.4 GHz.

stephenw10

But upstream on a WAN where you have no idea what is in the path I would always use 1500 unless whoever does control that hop has explicitly told you to use something larger. Yes, path MTU discovery should catch it but when it doesn't it sucks!

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Have you tried Wifi on it? I'm curious if it will handle the larger frames. I believe you'll need to run 802.11ac or later to get the aggregation. I can't do that here, as my notebook computer is 12 years old and only runs 802.11n on 2.4 GHz.

The 10 GbE / 9k MTU has access points on it but wifi is not ethernet - encapsulated packets, additional headers, inherently simplex, pause frames, beacons, air time et al - so nothing native is carried across unadulterated on the RF segment of a link.

RobbieTT

@stephenw10 said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

But upstream on a WAN where you have no idea what is in the path I would always use 1500 unless whoever does control that hop has explicitly told you to use something larger. Yes, path MTU discovery should catch it but when it doesn't it sucks!

Does anyone even try to run jumbo frames over a WAN - in almost all scenarios you would not make it past the first hop?

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

so nothing native is carried across unadulterated on the RF segment of a link.

What happens if you just connect your own AP to the LAN side?

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Does anyone even try to run jumbo frames over a WAN - in almost all scenarios you would not make it past the first hop?

If the other side of the router is just 1500, then you'd experience fragmentation or ICMP too big messages. Either way, it should still work. This is part of the function of every router.

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

What happens if you just connect your own AP to the LAN side?

I do have my own APs on the LAN (& a VLAN) but wifi is still wifi. Even protocols can change in function over a wifi segment - for example, UDP packets are subject to ACKs over a wifi segment in a similar fashion to that of TCP over a wired segment.

️

JKnott

@RobbieTT

You seem to be missing my point. Can you ping various sizes beyond 1500, to see if they're successfully passed, without fragmentation or PMTUD? You may need to run Wireshark to see what's happening. I just want to know if WiFi is usable with jumbo frames. WiFi, without aggregation, has a larger MTU than 1500. I think it's around 2300 or so. What happens if you try to send a large file through WiFi?

I know the operation of WiFi is different than Ethernet, but it is effectively a bridge with all the dirty details hidden.

RobbieTT

@JKnott
If you are looking for the book figure* then the MTU you are seeking is 2312 bytes, so yes, it can easily be seen as jumbo frames as it is above the normal 1500 (well 1514) byte frame. But those 2312 bytes are a bit of a misnomer and most wifi boffins would not recognise it.

It is more of an air-time space for transmission to take place, inclusive of all the additional overheads. Some may even argue that 2312 MTU should be cut in half due to the simplex nature of wifi; but as we are really talking about an airtime opportunity I think that is poor analogy.

WiFi is just RF and RF is fun!

️

*Often quoted as such but not actually the case in modern wifi but who cares

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

WiFi is just RF and RF is fun!

I know. I just borrowed this book from the library. It's the first physical book I've read in years, as they didn't have it in an ebook.

I have also done some work with cell phone sites and short haul microwave and got my amateur radio licence almost 52 years ago.

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

I have also done some work with cell phone sites and short haul microwave and got my amateur radio licence almost 52 years ago.

5G seemed to go with simplicity and defined requirements; albeit they had to be steered in that direction in the early days. I don't think you could pay them now to do some of the stuff they originally had in mind. It's all about stability as that = printing money.

My background was more on the exploiting, spoofing, denying, intercepting or generally being on the aggressive side of RF. I did also have to sit on the UK board that allocates and manages the RF spectrum. That was the 'not fun' part of RF.

Regarding your thoughts on testing beyond 1500 MTU on a wifi segment. It's harder than you may think with commodity hardware as most have barriers in place to stop you from doing so. Many wifi interfaces deny the opportunity to fettle the interface at all. Some look like you can tweak them but under the hood nothing changes, either due to the OS or the driver/firmware/hardware itself.

There are exceptions of course - macOS lets you control and pass MTU changes to the wifi interface but in more recent version this became limited to a range of 1280 to 1436 MTU. The point to note here is that it will not allow anywhere near the minimum or maximum MTU unless 'auto' is in place (which caps the interface at 1514 MTU). Ironically the only commodity hardware that used to use a wifi MTU above 1514 was also Apple in the guise of AirPort Time Capsules and Express units.

Meanwhile we are seeing a drift to a new-normal MTU of 1280 (1294), so even the 1500 (1514) MTU will start to look relatively capacious.

️

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Meanwhile we are seeing a drift to a new-normal MTU of 1280 (1294),

Where is that? It's definitely a step in the wrong direction.

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Where is that? It's definitely a step in the wrong direction.

We are partially there already due to Google's interpretation of TLS 1.3, HTTPS/3 and QUIC. YouTube for example:

2023-10-28 at 15.47.30.png

The joys of GAFAM.

️

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

We are partially there already due to Google's interpretation of TLS 1.3, HTTPS/3 and QUIC. YouTube for example:

That's not the same as layer 2 technology. VoIP uses even smaller packets. I don't know enough about QUIC etc. to comment.

RobbieTT

@JKnott
L2 spends a lot of time moving formerly-L3 traffic around.

️