Mixed MTUs on different NIC's interfaces on same pfSense bare metal

[Sergei_Shablovsky]

Dear pfSense Gurus and Networking Engineers!

SETUP

1. pfSense (CE or Plus) on bare metal 2xCPU pack server with several NICs (1G copper, SFP/SFP+);
2. separate LANs for internal monitoring and for other (DB cluster, backend servers, branch office, etc...) physically on separate NICs hardware interfaces;

QUESTION
How different MTU size impact (or may improve) network load on each of Interfaces:

1. Jumbo MTU 9000 frames on Database servers cluster (so we able to set in pfSense MTU 9000 and offloading);
2. MTU 1440 (and even less) for Monitoring LAN (Syslog and SNMP packets are typically small (100-500 bytes) (so we able to set in pfSense MTU 1440 and offloading);
3. MTU 1500 for any other LANs (so we able to set in pfSense MTU 1500 and offloading);

Extremely happy to read Your opinions and suggestions, especially if You have experience in enterprise / High-Loading environment in DCs!

Thank You so much for Your time and have a nice sunny days, Merry Christmas to all of You and families!

[Sergei_Shablovsky]

How mixed MTUs impact on FreeBSD overall performance and throughput as this server are BORDER firewall ?
(PCI bus pressure, RAM pressure, etc...)

[JKnott]

@Sergei_Shablovsky

You normally set MTU to 1500. Larger MTU certainly help when you move lots of data around, as in a data centre. There's no point in setting a smaller MTU for snmp, etc., as the packet is only as large as needed, up to the MTU. While a larger MTU may help on the local network, you have to be aware of things such as WiFi MTU, which can be smaller than Ethernet supports. You don't want to mix MTU on the same network.

I think a large MTU could be beneficial. MTU size was determined by balancing throughput vs data loss, back in the days of half duplex Ethernet, where collisions were expected. It was even smaller, at 576 bytes, on early dial up connections, before error correcting modems, for the same reason. However, these days, networks are far more reliable so data loss is not the issue it used to be. Even 30 years ago, token ring networks had much larger MTU than Ethernet.

[JKnott]

@Sergei_Shablovsky said in Mixed MTUs on different NIC's interfaces on same pfSense bare metal:

How mixed MTUs impact on FreeBSD overall performance and throughput as this server are BORDER firewall ?
(PCI bus pressure, RAM pressure, etc...)

You can have different MTU on different sides of a router, as they are separate networks and the router will handle the MTU difference with Path MTU Discovery (PMTUD) or sometimes with fragmentation.