[Solved] Jumbo Frames not working on L3in 10Gbit network
-
Hi guys,
I'm running the following 10Gbit-setup in which I want to enable Jumbo frames in order to increase the copy performance of large files.- NAS: Qnap TVS-863+ with a 10Gbit Tehuti TN9210 10GBase-T NIC - VLAN 200
- 2 Clients: Win10, CPU: i7-6850K, Intel X550 NIC - VLAN 10
- Switch: Netgear XS708E, 10Gbit
- pfSense: Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz, 8GB RAM, 2.4.4-RELEASE-p2 (amd64), Intel X550 NIC
Now, the reason for my intention to enable Jumbo frames is a poor performance while copying larger files (NAS -> Client: ~ 250MB/s)
I enabled jumbo frames (9000 Byte) on the Windows10 client, the NAS and the pfSense.
On the Windows clients simply in the NIC configuration (Jumbo packet = 9014 Byte), on the NAS in the NIC config as well and on the pfSense by setting the MTU to 9000.
The result was that I could not get a ping from the Windows client neither to the pfSense nor to the NAS (ping TARGET -f -l 8972).So I put the Windows clients and NAS on the same VLAN, and voila the ping works and the copy performance is increase to ~ 500MB/s.
My conclusion is that the pfSense config with regards to the jumbo packets is not correct as it works on L2 (same VLAN) but not L3.Do you have any clue how to fix this?
Cheers
-
I enabled jumbo frames (9000 Byte) on the Windows10 client, the NAS and the pfSense.>
I don't see where you configured the switch for jumbo frames. Was this done?
So I put the Windows clients and NAS on the same VLAN, and voila the ping works and the copy performance is increase to ~ 500MB/s.
My conclusion is that the pfSense config with regards to the jumbo packets is not correct as it works on L2 (same VLAN) but not L3.
500 MB/sec is still only 4 Gbit. I would expect to see double that in ideal situations. You might want to double check the config on your switch. Also, the initial 250 MB/sec throughput you were seeing may be a symptom of the traffic being filtered as it traverses the firewall. I'm not so certain that an i3 desktop is capable of sustaining 10 Gbit/sec of firewalled throughput.
Is there a requirement to firewall your VLANs? If not, and you're looking for performance, you may want to consider a different design. For example, Add a transit network between PFsense and your core switch. Then terminate your VLANs on an L3 switch. This way, all inter-VLAN traffic is handled by the switch instead of being filtered as it traverses the firewall.
-
@marvosa thx for your hints. I must confess that I forgot to set the MTU size at the VLANs
I just configured it at the LAN interface which obviously is not enough.
Now I got at least the basic issue solved. Next I will address the speed.Therefore I have some angles:
- Replace the i3 as you already implied
- Deactivate/uninstall WinPcap and other low-level network tools
I need the VLANs for security reasons. The switch itself can not be configured unfortunately with regards to the packet size. But I'll see...thx tough for your valuable input.
Cheers!
-
Hi @LaUs3r - have you tried running an iperf3 test across the firewall (i.e. between two VLAN's or subnets) to see how many packets per second it handle with PF enabled? That might be a good first step to see where the theoretical transfer limits are (and would leave out any impact storage might have on slowing down the transfer speed). Check out this link:
https://bsdrp.net/documentation/technical_docs/performance#where_is_the_bottleneck
You can use netstat to monitor number of packets being transferred while running an iperf3 test across the firewall (i.e. between two hosts in different VLAN's or subnets). Then reduce the MSS and see where you hit a bottleneck (i.e. the number of packets no longer increase as you increase the number of parallel iperf3 streams)
Hope this helps.
-
@LaUs3r said in Jumbo Frames not working on L3in 10Gbit network:
I need the VLANs for security reasons. The switch itself can not be configured unfortunately with regards to the packet size. But I'll see...thx tough for your valuable input.
Cheers!Anytime!
Also, for clarity, I wasn't suggesting getting rid of your VLANs. The thought was, if there isn't a need to firewall your VLANs, then my suggestion was to move your VLANs from PFsense to an L3 switch so that inter-VLAN traffic can be routed at wire speed without being filtered. You would still have your VLANs, but you'd just be creating them on the switch instead of PFsense. You'd gain performance but would lose the ability to add firewall rules to each VLAN. It was just a thought if you're looking for more performance.
-
@tman222 said in Jumbo Frames not working on L3in 10Gbit network:
Hi @LaUs3r - have you tried running an iperf3 test across the firewall (i.e. between two VLAN's or subnets) to see how many packets per second it handle with PF enabled? That might be a good first step to see where the theoretical transfer limits are (and would leave out any impact storage might have on slowing down the transfer speed). Check out this link:
https://bsdrp.net/documentation/technical_docs/performance#where_is_the_bottleneck
You can use netstat to monitor number of packets being transferred while running an iperf3 test across the firewall (i.e. between two hosts in different VLAN's or subnets). Then reduce the MSS and see where you hit a bottleneck (i.e. the number of packets no longer increase as you increase the number of parallel iperf3 streams)
Hope this helps.
So, today I performed some iperf tests. my i3 cannot take more than 8Gbit/s which is fine for me. I now have transfer rates of approx. 800MB/s :-)
Interestingly, on my other PC where WinPcap is installed, I only get around 550MB/s.So, thank you all for your valuable inputs and help!
Cheers guys