pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link)

mauro.tridici

Thank you, Steve.
You helped me once again.

Mauro

JKnott

@mauro-tridici said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

I read that "When traffic path is internet, path MTU will always be max 1500". Is it true? If yes, should I set MTU to 1500 also for a 10Gbps WAN interface?
I know that MTU 9000 is for faster and more efficient transmission, what will happen if I will use MTU 1500 on a 10Gbps iface connected to the 10Gbps ISP link?

A larger MTU will help if you're moving a lot of data. However, if your ISP provides only 1500, then it won't do much for data over the Internet.

In my not so humble opinion, it's high time to move to a much larger MTU for the Internet in general. The current 1500 MTU was set 43 years ago, with the original Ethernet spec. Other network tech, such as token ring, has used much larger MTUs and let the routers sort things out. Technology has advance a lot in the mean time, with telecom connections much faster and more reliable, which means the reasons for going with a smaller MTU (at one time, 576 was used for dial up) are no longer valid. As it is, 9000 is often used in data centres, so why not move the Internet to it? The smaller MTU wastes network bandwidth. It takes just as much effort for a switch to handle a 9000 byte frame as 1500, as they switch entire frames, not individual bytes.
BTW, I have a cheap, our favourite, TP-Link switch that will do 16000!

One other issue is WiFi, but I believe later 802.11 specs use frame aggregation, to improve efficiency, which means they should be able to handle 9000 MTU. However, I have no experience with that.

JKnott

@stephenw10 said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

It should work fine. And that is what you should use unless you know for sure that whatever upstream router you're connected to and any switches in between are also using a larger MTU, like 9000.

Routers are able to handle the difference, to ensure larger frames are reduced to fit. Originally, that was with fragmentation. These days, path MTU discovery (PMTUD) is used for most IPv4 and all IPv6 traffic.

Bottom line, using 9000 will not cause problems, with the possible exception of WiFi (see my note above). However, it won't provide an improvement over the Internet.

mer

Unless you have complete control over the entire data path from source to destination, high probability is the data will get fragmented somewhere along the way.
Normally fragmentation isn't a big deal because they get reassmbled at some point.
Unless the traffic is UDP. UDP is "send and don't care if it actually gets there".
A lot of video traffic is UDP.
What happens if you lose a packet of video? Glitches, blocky, bad video.
Most video codecs are "key frame periodically, incremental frames (deltas) in between". Key frame is basically a full frame of the video, so a large packet, the deltas are small. Key frames typically cause the receiver to synch.
Big packets are more likely to be fragmented, fragments can be lost causing the whole packet to get lost, causing a user to see "bad video quality"

TCP automatically retransmits lost packets which may result in a loss of usable bandwidth simply because packets are being retransmitted.

If you look back at the timeframe of "birth of the Internet", most businesses had less than 100M internally, T1 to the external world, most home users were on a dial up serial modem at 115200bps. Your "worst" speeds from Comcast are better than what businesses used to have.

I think the WAN port should negotiate with the upstream for things like MTU as others have said and "should not cause problems, but there may be cases where it does".

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Unless you have complete control over the entire data path from source to destination, high probability is the data will get fragmented somewhere along the way.
Normally fragmentation isn't a big deal because they get reassmbled at some point.
Unless the traffic is UDP. UDP is "send and don't care if it actually gets there".
A lot of video traffic is UDP.

As I mentioned, the world is moving to PMTUD, not fragmentation. With IPv6, it's mandatory and is now used for much of IPv4. This means the sending host will be told the MTU is too big and will have to retransmit. Even with fragmentation, the packet does get reassembled, whether UDP or not. This is not a new issue and IP has been able to handle it from the start. As I pointed out, at one time, dial up connections used 576, while much of the world was 1500 and token ring was a few K. It was the routers between networks that enabled this to work smoothly.

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

TCP automatically retransmits lost packets which may result in a loss of usable bandwidth simply because packets are being retransmitted.

With UDP, it's up to the application to request retransmission. Often, it just accepts the loss. A loss packet in something like audio or video is less disruptive than the delay caused by retransmission. In some cases, forward error correction might be used, which means an occasional lost packet is irrelevant.

mer

@JKnott I don't completely disagree, but if a lost packet is a key frame of video, that is often pretty disruptive.
I agree on PMTUD, but sometimes it takes a little while for every one to get on the same page.

I also was agreeing that large packets/MTU shouldn't be an issue, but there may be cases it is.
That's all. No more, no less.

JKnott

@mer said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

I also was agreeing that large packets/MTU shouldn't be an issue, but there may be cases it is.
That's all. No more, no less.

Of course, there's nothing that says large frames have to be used. VoIP is an example, where small packets are used, to minimize latency. Even with 1500, the amount of time it would take to fill a frame would cause a noticeable delay.

Bottom line, the OP can use 9000 MTU on his LAN if he wants. However, I'm curious as to how well WiFi works with it. I believe 802.11ac supports aggregation. I'm not sure about n.

RobbieTT

@JKnott
I run 1508 MTU on my WAN, 1500 MTU on my maintenance LAN and 9k set on my main LAN and VLAN at 10 GbE.

In this mixed environment I have 2 main switches, a couple of tertiary switches, 5 APs, numerous servers, clients and sundry networked items. Everything purrs along as it should.

️

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

9k set on my main LAN and VLAN at 10 GbE.

Have you tried Wifi on it? I'm curious if it will handle the larger frames. I believe you'll need to run 802.11ac or later to get the aggregation. I can't do that here, as my notebook computer is 12 years old and only runs 802.11n on 2.4 GHz.

stephenw10

But upstream on a WAN where you have no idea what is in the path I would always use 1500 unless whoever does control that hop has explicitly told you to use something larger. Yes, path MTU discovery should catch it but when it doesn't it sucks!

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Have you tried Wifi on it? I'm curious if it will handle the larger frames. I believe you'll need to run 802.11ac or later to get the aggregation. I can't do that here, as my notebook computer is 12 years old and only runs 802.11n on 2.4 GHz.

The 10 GbE / 9k MTU has access points on it but wifi is not ethernet - encapsulated packets, additional headers, inherently simplex, pause frames, beacons, air time et al - so nothing native is carried across unadulterated on the RF segment of a link.

RobbieTT

@stephenw10 said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

But upstream on a WAN where you have no idea what is in the path I would always use 1500 unless whoever does control that hop has explicitly told you to use something larger. Yes, path MTU discovery should catch it but when it doesn't it sucks!

Does anyone even try to run jumbo frames over a WAN - in almost all scenarios you would not make it past the first hop?

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

so nothing native is carried across unadulterated on the RF segment of a link.

What happens if you just connect your own AP to the LAN side?

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

Does anyone even try to run jumbo frames over a WAN - in almost all scenarios you would not make it past the first hop?

If the other side of the router is just 1500, then you'd experience fragmentation or ICMP too big messages. Either way, it should still work. This is part of the function of every router.

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

What happens if you just connect your own AP to the LAN side?

I do have my own APs on the LAN (& a VLAN) but wifi is still wifi. Even protocols can change in function over a wifi segment - for example, UDP packets are subject to ACKs over a wifi segment in a similar fashion to that of TCP over a wired segment.

️

JKnott

@RobbieTT

You seem to be missing my point. Can you ping various sizes beyond 1500, to see if they're successfully passed, without fragmentation or PMTUD? You may need to run Wireshark to see what's happening. I just want to know if WiFi is usable with jumbo frames. WiFi, without aggregation, has a larger MTU than 1500. I think it's around 2300 or so. What happens if you try to send a large file through WiFi?

I know the operation of WiFi is different than Ethernet, but it is effectively a bridge with all the dirty details hidden.

RobbieTT

@JKnott
If you are looking for the book figure* then the MTU you are seeking is 2312 bytes, so yes, it can easily be seen as jumbo frames as it is above the normal 1500 (well 1514) byte frame. But those 2312 bytes are a bit of a misnomer and most wifi boffins would not recognise it.

It is more of an air-time space for transmission to take place, inclusive of all the additional overheads. Some may even argue that 2312 MTU should be cut in half due to the simplex nature of wifi; but as we are really talking about an airtime opportunity I think that is poor analogy.

WiFi is just RF and RF is fun!

️

*Often quoted as such but not actually the case in modern wifi but who cares

JKnott

@RobbieTT said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

WiFi is just RF and RF is fun!

I know. I just borrowed this book from the library. It's the first physical book I've read in years, as they didn't have it in an ebook.

I have also done some work with cell phone sites and short haul microwave and got my amateur radio licence almost 52 years ago.

RobbieTT

@JKnott said in pfSense as router: question about MTU to be set on a 10Gbps interface (WAN interface to 10Gbps ISP link):

I have also done some work with cell phone sites and short haul microwave and got my amateur radio licence almost 52 years ago.

5G seemed to go with simplicity and defined requirements; albeit they had to be steered in that direction in the early days. I don't think you could pay them now to do some of the stuff they originally had in mind. It's all about stability as that = printing money.

My background was more on the exploiting, spoofing, denying, intercepting or generally being on the aggressive side of RF. I did also have to sit on the UK board that allocates and manages the RF spectrum. That was the 'not fun' part of RF.

Regarding your thoughts on testing beyond 1500 MTU on a wifi segment. It's harder than you may think with commodity hardware as most have barriers in place to stop you from doing so. Many wifi interfaces deny the opportunity to fettle the interface at all. Some look like you can tweak them but under the hood nothing changes, either due to the OS or the driver/firmware/hardware itself.

There are exceptions of course - macOS lets you control and pass MTU changes to the wifi interface but in more recent version this became limited to a range of 1280 to 1436 MTU. The point to note here is that it will not allow anywhere near the minimum or maximum MTU unless 'auto' is in place (which caps the interface at 1514 MTU). Ironically the only commodity hardware that used to use a wifi MTU above 1514 was also Apple in the guise of AirPort Time Capsules and Express units.

Meanwhile we are seeing a drift to a new-normal MTU of 1280 (1294), so even the 1500 (1514) MTU will start to look relatively capacious.

️