Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
ah, thank you for chiming back in and the previous help.
In regards to my service, I have Xfinity Branded 2Gb/300Mb Cable Internet. I do not have any of the ISP gear. I have a single RG6 drop in the basement which is connected to my own private Netgear CM3000 DOCSIS 3.1 Cable modem.
https://www.netgear.com/home/wifi/modems/cm3000/That Modem has a 2.5Gb NIC that is connected to my PFSense WAN @ 2.5Gb. I used DHCP on the WAN to get an IP from Xfinity that for the most part is fairly static, rarely changes. I also have IPV6 enabled and working very well, all my internal clients are getting IPV6 addresses and IPV6 connectivity has been verified.
Going back to what you just stated though, If you are saying that a DOCSIS connection does not support flow-control, then would it make sense to disable flow control on only the PFSense WAN NIC, but then leave enabled on the PF LAN NIC and also on all my switchports ?
I also scored a Cisco WS-C3850-12X48U-S 48x (12x MultiGB) on ebay last night for $125 bucks, at this point I have a small collection of switches. I figure with this switch I can run WAN @ 2.5Gb Lan at 2.5Gb and my Win 11 Gamming PC's at 2.5Gb with a few stragglers still at 1Gb. Then if I run into more issues I can use the command you gave me before to max out the buffers.
I am going to try this MTU thing here in a sec, curious to what happens.
-
@stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.
That is why I recommended that to you in the first place. The larger buffers don't completely resolve the issue, but makes it a lot better:
sudo ethtool enp110s0 | grep Speed Speed: 1000Mb/sspeedtest -s 1783 Speedtest by Ookla Server: Comcast - San Francisco, CA (id: 1783) ISP: Comcast Cable Idle Latency: 13.45 ms (jitter: 1.66ms, low: 10.66ms, high: 14.05ms) Download: 827.00 Mbps (data used: 743.6 MB) 16.85 ms (jitter: 10.94ms, low: 8.75ms, high: 273.13ms) Upload: 353.04 Mbps (data used: 384.2 MB) 16.75 ms (jitter: 1.10ms, low: 12.49ms, high: 35.54ms) Packet Loss: Not available. -
An interesting datapoint that makes me point the finger to DOCSIS is that I have a secondary WAN connection through Sail Internet. If I force the traffic for this 1GbE client through my WAN2 with a rule--it reaches 940/940Mbps
speedtest Speedtest by Ookla Server: Sail Internet - Santa Clara, CA (id: 56367) ISP: Sail Internet Idle Latency: 1.34 ms (jitter: 0.06ms, low: 1.14ms, high: 1.38ms) Download: 937.62 Mbps (data used: 423.4 MB) 34.14 ms (jitter: 7.94ms, low: 0.81ms, high: 289.02ms) Upload: 938.43 Mbps (data used: 422.4 MB) 142.21 ms (jitter: 75.57ms, low: 0.90ms, high: 825.07ms) Packet Loss: 0.0% -
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
I am going to try this MTU thing here in a sec, curious to what happens.
Doesn't do anything. I already tried everything on the pfSense that I could think of:
- Disable Hardware Checksum Offloading
- Changing the State Type from Keep to Loose
- Setting MSS & MTUs to 1420
- Disabling pf Scrub
The only solution that I don't like (which is not supported on the Cisco) is Ethernet Flow Control on both the WAN and LAN interfaces of the pfSense. Again a very ugly blunt solution that pauses the entire interface impacting other clients on your network.
-
@lnguyen said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
@stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?
I don't have either. I'm not in the US.
That is interesting that adding the switch would make any difference to a docsis issue though. Hard to see how that could happen.
-
I just noticed something odd, when I moved the PF LAN nic from 1Gb to 10Gb the GUI is no longer showing Flow Control being enabled. I would expect to see rxpause & txpause.
-
What NIC is it? Not all drivers report it so it can be displayed there.
-
@stephenw10 Using a switch with larger buffers only mask the symptoms of broken TCP Flow Control. This isn't a pfSense issue. It's a DOCSIS issue when the WAN speed is greater than a LAN client, ergo 2.35Gbps WAN to 940Mbps LAN
-
My LAN MTU is now 1460 in PF, so far no change is certain speedtests like fast.com when PF lan @ 10Gb. Speeds still slowly tapering down to 500Mbps. However with MTU at 1460 and PF Lan at 1Gb performance is still solid.
-
@ngr2001 Yes what NIC are you using on the pfSense?
-
Intel X550-T2
When that same LAN NIC is moved to 1Gb is reports (rxpause & txpause) when I move the patch cable over to 10Gb these values disappear in the GUI.
-
@ngr2001 Can you do a
show int 10GigabitEthernet1/2/8? -
Ah, that seems like a pretty big clue....
-
The 7250 doesn't have any 10GbaseT ports though so I assume that's a different Intel NIC?
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
My LAN MTU is now 1460 in PF, so far no change is certain speedtests
Did you set MSS like I suggested though? MTU does nothing to pf.
-
@stephenw10 Copper 10G-BaseT SFP+ modules most likely being used
-
Yes I have SPF+ RJ45 adapters in the 7250, thus I can easily move the CAT8 from one jack to the over for this test.
-
@ngr2001 Can you do a show int 10GigabitEthernet1/2/8?
-
I did not, but I just corrected that. I reset the states, waited a few mins and so far there may be a small boost in performance. I seem to be averaging now 700Mpbs instead of 500Mbps on fast.com, however if I switch ports we get the full 900+. So better but not perfect.
