Gigabit devices slow over 10GB firewall
-
Got a weird one for you. I have a HP SFF PC with a Core i5 3470 and 8GB RAM, latest firmware, and a dual 10GBE Intel x500-T2 NIC. pfSense 2.6 is stock configuration with pfBlockerNG, Snort, and OpenVPN with one server running.
I have 1.2GB internet from Comcast connected to an Arris S33 modem with a 2.5GB connection to the WAN interface (autoselect to 2.5GB), and LAN is connected to a multi-speed SFP+ transceiver on a MikroTik CRS354 switch capable of 1, 2.5, 5, and 10GBE.
On the switch I have a handful of gigabit devices including two WIFI6-Pro APs, a couple laptops, TVs, etc. I have a gaming PC on a 10GB SFP+ port and a R710 server on another SFP+ port.
In this configuration, I have no problem getting ~1.4GB down to my gaming PC and server over pfSense, but I'm only getting about 200mb down to any of my gigabit devices. That's through either copper or WIFI. File download speeds reflect this as well.
HOWEVER...if I do one of the following things, I get 950mb down to all devices:
- Move LAN interface from SFP+ to a gigabit port
- Keep LAN interface on SFP+ port but change it to 1GB in pfSense interface settings
- Keep LAN interface on SFP+ 10GB but change the WAN port to 1GB in pfSense interface settings
The jist of it is that while both x550-t2 NICs are at above 1GB (10GBE LAN and 2.5/10GBE WAN), my 10GB devices get full bandwidth to the internet while my 1GB devices only get ~200mb down. Latency is the same on all devices.
Here is what I've tried so far:
- Packet captures - nothing jumped out but I do need to look at them closer. I saw some incorrect checksums until I shut off checksum offloading, which fixed that problem
- CPU load is ~5% during gigabit device SpeedTests (200mb) and closer to 30% during 10GB device SpeedTests (1.4gb)
- Checked buffer status using netstat -m. Everything looked good
- Disabled pfBlockerNG and Snort
- Disabled flow-control on the NIC
- Confirmed TSO/TRO disabled
- Introduced a 10GB switch in between the modem and WAN interface so that the pfSense WAN interface could sit at 10GB instead of 2.5GB
- Tried pfSense 2.7-devel
I'm at a loss, does anyone have any ideas?
-
Hmm, that is an odd set of results.
Some of those interfaces using a large MTU perhaps?
I assume you can see full Gigabit line rate between internal devices on the same switch whatever the uplink to the LAN is?
Maybe try an iperf test between LAN side clients and pfSense itself so you're testing only one NIC.
Some things require flow control to be active.
Steve
-
Thanks for the reply. I forgot to mention I did iPerf tests between 10GB > 1GB nodes and the router - all got full speed. That being said, I think I've found the issue. The MikroTik switch is the problem. I am running it in SwitchOS mode, but when I change to RouterOS everything works as expected. So I'll open a ticket with them. I appreciate the help!
