Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.
That is why I recommended that to you in the first place. The larger buffers don't completely resolve the issue, but makes it a lot better:
sudo ethtool enp110s0 | grep Speed Speed: 1000Mb/sspeedtest -s 1783 Speedtest by Ookla Server: Comcast - San Francisco, CA (id: 1783) ISP: Comcast Cable Idle Latency: 13.45 ms (jitter: 1.66ms, low: 10.66ms, high: 14.05ms) Download: 827.00 Mbps (data used: 743.6 MB) 16.85 ms (jitter: 10.94ms, low: 8.75ms, high: 273.13ms) Upload: 353.04 Mbps (data used: 384.2 MB) 16.75 ms (jitter: 1.10ms, low: 12.49ms, high: 35.54ms) Packet Loss: Not available. -
An interesting datapoint that makes me point the finger to DOCSIS is that I have a secondary WAN connection through Sail Internet. If I force the traffic for this 1GbE client through my WAN2 with a rule--it reaches 940/940Mbps
speedtest Speedtest by Ookla Server: Sail Internet - Santa Clara, CA (id: 56367) ISP: Sail Internet Idle Latency: 1.34 ms (jitter: 0.06ms, low: 1.14ms, high: 1.38ms) Download: 937.62 Mbps (data used: 423.4 MB) 34.14 ms (jitter: 7.94ms, low: 0.81ms, high: 289.02ms) Upload: 938.43 Mbps (data used: 422.4 MB) 142.21 ms (jitter: 75.57ms, low: 0.90ms, high: 825.07ms) Packet Loss: 0.0% -
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
I am going to try this MTU thing here in a sec, curious to what happens.
Doesn't do anything. I already tried everything on the pfSense that I could think of:
- Disable Hardware Checksum Offloading
- Changing the State Type from Keep to Loose
- Setting MSS & MTUs to 1420
- Disabling pf Scrub
The only solution that I don't like (which is not supported on the Cisco) is Ethernet Flow Control on both the WAN and LAN interfaces of the pfSense. Again a very ugly blunt solution that pauses the entire interface impacting other clients on your network.
-
@lnguyen said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
@stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?
I don't have either. I'm not in the US.
That is interesting that adding the switch would make any difference to a docsis issue though. Hard to see how that could happen.
-
I just noticed something odd, when I moved the PF LAN nic from 1Gb to 10Gb the GUI is no longer showing Flow Control being enabled. I would expect to see rxpause & txpause.
-
What NIC is it? Not all drivers report it so it can be displayed there.
-
@stephenw10 Using a switch with larger buffers only mask the symptoms of broken TCP Flow Control. This isn't a pfSense issue. It's a DOCSIS issue when the WAN speed is greater than a LAN client, ergo 2.35Gbps WAN to 940Mbps LAN
-
My LAN MTU is now 1460 in PF, so far no change is certain speedtests like fast.com when PF lan @ 10Gb. Speeds still slowly tapering down to 500Mbps. However with MTU at 1460 and PF Lan at 1Gb performance is still solid.
-
@ngr2001 Yes what NIC are you using on the pfSense?
-
Intel X550-T2
When that same LAN NIC is moved to 1Gb is reports (rxpause & txpause) when I move the patch cable over to 10Gb these values disappear in the GUI.
-
@ngr2001 Can you do a
show int 10GigabitEthernet1/2/8? -
Ah, that seems like a pretty big clue....
-
The 7250 doesn't have any 10GbaseT ports though so I assume that's a different Intel NIC?
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
My LAN MTU is now 1460 in PF, so far no change is certain speedtests
Did you set MSS like I suggested though? MTU does nothing to pf.
-
@stephenw10 Copper 10G-BaseT SFP+ modules most likely being used
-
Yes I have SPF+ RJ45 adapters in the 7250, thus I can easily move the CAT8 from one jack to the over for this test.
-
@ngr2001 Can you do a show int 10GigabitEthernet1/2/8?
-
I did not, but I just corrected that. I reset the states, waited a few mins and so far there may be a small boost in performance. I seem to be averaging now 700Mpbs instead of 500Mbps on fast.com, however if I switch ports we get the full 900+. So better but not perfect.
-
1/2/8 - My 10Gb uplink to PF LAN Nic.
I just noticed that 2 of my main clients currently at 1Gb on the switch have a lot of dropped packets. However right now I am running speedtests and I am not seeing the dropped packet counter increasing, so who know under which setup all these packets were dropped.
1Gb Win 11 Client
1Gb Win 11 Client
-
Should I disable flow control on my WAN NIC as a long term production setup being I am on Cable Internet ?
