Mixed MTUs on different NIC's interfaces on same pfSense bare metal

[Sergei_Shablovsky]

Dear pfSense Gurus and Networking Engineers!

SETUP

1. pfSense (CE or Plus) on bare metal 2xCPU pack server with several NICs (1G copper, SFP/SFP+);
2. separate LANs for internal monitoring and for other (DB cluster, backend servers, branch office, etc...) physically on separate NICs hardware interfaces;

QUESTION
How different MTU size impact (or may improve) network load on each of Interfaces:

1. Jumbo MTU 9000 frames on Database servers cluster (so we able to set in pfSense MTU 9000 and offloading);
2. MTU 1440 (and even less) for Monitoring LAN (Syslog and SNMP packets are typically small (100-500 bytes) (so we able to set in pfSense MTU 1440 and offloading);
3. MTU 1500 for any other LANs (so we able to set in pfSense MTU 1500 and offloading);

Extremely happy to read Your opinions and suggestions, especially if You have experience in enterprise / High-Loading environment in DCs!

Thank You so much for Your time and have a nice sunny days, Merry Christmas to all of You and families!

[Sergei_Shablovsky]

How mixed MTUs impact on FreeBSD overall performance and throughput as this server are BORDER firewall ?
(PCI bus pressure, RAM pressure, etc...)

[JKnott]

@Sergei_Shablovsky

You normally set MTU to 1500. Larger MTU certainly help when you move lots of data around, as in a data centre. There's no point in setting a smaller MTU for snmp, etc., as the packet is only as large as needed, up to the MTU. While a larger MTU may help on the local network, you have to be aware of things such as WiFi MTU, which can be smaller than Ethernet supports. You don't want to mix MTU on the same network.

I think a large MTU could be beneficial. MTU size was determined by balancing throughput vs data loss, back in the days of half duplex Ethernet, where collisions were expected. It was even smaller, at 576 bytes, on early dial up connections, before error correcting modems, for the same reason. However, these days, networks are far more reliable so data loss is not the issue it used to be. Even 30 years ago, token ring networks had much larger MTU than Ethernet.

[JKnott]

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

How mixed MTUs impact on FreeBSD overall performance and throughput as this server are BORDER firewall ?
(PCI bus pressure, RAM pressure, etc...)

You can have different MTU on different sides of a router, as they are separate networks and the router will handle the MTU difference with Path MTU Discovery (PMTUD) or sometimes with fragmentation.

[Sergei_Shablovsky]

Thank You for answering!

@JKnott said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

@Sergei_Shablovsky

You normally set MTU to 1500. Larger MTU certainly help when you move lots of data around, as in a data centre. There's no point in setting a smaller MTU for snmp, etc., as the packet is only as large as needed, up to the MTU. While a larger MTU may help on the local network, you have to be aware of things such as WiFi MTU, which can be smaller than Ethernet supports.

My question come from the point where for internal Management LAN (where delay need to be shortest and speed - quickest as possible) and Backup LAN (where delay not so important, but speed need to be as fast as possible and network overhead need to be as little as possible).
So, for example, 1350 MTU for Management LAN and 9000 Jumbo MTU for Backup LAN looks like reasonable...

Where I am wrong?

You don't want to mix MTU on the same network.

Agree.

I think a large MTU could be beneficial. MTU size was determined by balancing throughput vs data loss, back in the days of half duplex Ethernet, where collisions were expected. It was even smaller, at 576 bytes, on early dial up connections, before error correcting modems, for the same reason. However, these days, networks are far more reliable so data loss is not the issue it used to be. Even 30 years ago, token ring networks had much larger MTU than Ethernet.

[Sergei_Shablovsky]

@JKnott said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

How mixed MTUs impact on FreeBSD overall performance and throughput as this server are BORDER firewall ?
(PCI bus pressure, RAM pressure, etc...)

You can have different MTU on different sides of a router, as they are separate networks and the router will handle the MTU difference with Path MTU Discovery (PMTUD) or sometimes with fragmentation.

Thank You for explanation!

How different MTUs on physically different interfaces (if NIC are 2- or 4- head model) impact on NIC's overall performance (overall throughput, numbers of IRQs, etc...) ?

[JKnott]

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

Where I am wrong?

You're assuming the MTU sets the size for all frames, when it only limits the maximum size. If a device only needs a small frame, then that's all that will be used, down to the minimum 64 bytes. So, on your 9000 byte jumbo frame network, you could easily see those smaller SNMP frames you mentioned. Remember MTU is just the maximum size allowed.

[JKnott]

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

How different MTUs on physically different interfaces (if NIC are 2- or 4- head model) impact on NIC's overall performance (overall throughput, numbers of IRQs, etc...) ?

The only impact is when larger packets are being sent from a large MTU network to smaller MTU. In that case, path MTU discovery or fragmentation will limit the size being passed to the smaller MTU network. Smaller packets, for the same amount of data, mean for work for the routers and switches. Fragmentation requires more work, as each oversize packet has to be fragmented, whereas PMTUD determines the largest packet that can be passed to the smaller MTU network and the sending device uses that size.

[JKnott]

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

How different MTUs on physically different interfaces (if NIC are 2- or 4- head model) impact on NIC's overall performance (overall throughput, numbers of IRQs, etc...) ?

As mentioned before, there is no effect between different NICs. The only issue is there will be more work with smaller packets on the computer/switch/router. This is because those devices handle Ethernet frames as a whole. So, the smaller the MTU, the more frames that have to be handled and the more work for the CPU in those devices.