Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

lnguyen

@stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?

lnguyen

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.

That is why I recommended that to you in the first place. The larger buffers don't completely resolve the issue, but makes it a lot better:

sudo ethtool enp110s0 | grep Speed
	Speed: 1000Mb/s

speedtest -s 1783

   Speedtest by Ookla

      Server: Comcast - San Francisco, CA (id: 1783)
         ISP: Comcast Cable
Idle Latency:    13.45 ms   (jitter: 1.66ms, low: 10.66ms, high: 14.05ms)
    Download:   827.00 Mbps (data used: 743.6 MB)                                                   
                 16.85 ms   (jitter: 10.94ms, low: 8.75ms, high: 273.13ms)
      Upload:   353.04 Mbps (data used: 384.2 MB)                                                   
                 16.75 ms   (jitter: 1.10ms, low: 12.49ms, high: 35.54ms)
 Packet Loss: Not available.

lnguyen

An interesting datapoint that makes me point the finger to DOCSIS is that I have a secondary WAN connection through Sail Internet. If I force the traffic for this 1GbE client through my WAN2 with a rule--it reaches 940/940Mbps

speedtest

   Speedtest by Ookla

      Server: Sail Internet - Santa Clara, CA (id: 56367)
         ISP: Sail Internet
Idle Latency:     1.34 ms   (jitter: 0.06ms, low: 1.14ms, high: 1.38ms)
    Download:   937.62 Mbps (data used: 423.4 MB)                                                   
                 34.14 ms   (jitter: 7.94ms, low: 0.81ms, high: 289.02ms)
      Upload:   938.43 Mbps (data used: 422.4 MB)                                                   
                142.21 ms   (jitter: 75.57ms, low: 0.90ms, high: 825.07ms)
 Packet Loss:     0.0%

lnguyen

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

I am going to try this MTU thing here in a sec, curious to what happens.

Doesn't do anything. I already tried everything on the pfSense that I could think of:

Disable Hardware Checksum Offloading
Changing the State Type from Keep to Loose
Setting MSS & MTUs to 1420
Disabling pf Scrub

The only solution that I don't like (which is not supported on the Cisco) is Ethernet Flow Control on both the WAN and LAN interfaces of the pfSense. Again a very ugly blunt solution that pauses the entire interface impacting other clients on your network.

stephenw10

@lnguyen said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

@stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?

I don't have either. I'm not in the US.

That is interesting that adding the switch would make any difference to a docsis issue though. Hard to see how that could happen.

ngr2001

@lnguyen

I just noticed something odd, when I moved the PF LAN nic from 1Gb to 10Gb the GUI is no longer showing Flow Control being enabled. I would expect to see rxpause & txpause.

stephenw10

What NIC is it? Not all drivers report it so it can be displayed there.

lnguyen

@stephenw10 Using a switch with larger buffers only mask the symptoms of broken TCP Flow Control. This isn't a pfSense issue. It's a DOCSIS issue when the WAN speed is greater than a LAN client, ergo 2.35Gbps WAN to 940Mbps LAN

ngr2001

@stephenw10

My LAN MTU is now 1460 in PF, so far no change is certain speedtests like fast.com when PF lan @ 10Gb. Speeds still slowly tapering down to 500Mbps. However with MTU at 1460 and PF Lan at 1Gb performance is still solid.

lnguyen

@ngr2001 Yes what NIC are you using on the pfSense?

ngr2001

@stephenw10

Intel X550-T2

When that same LAN NIC is moved to 1Gb is reports (rxpause & txpause) when I move the patch cable over to 10Gb these values disappear in the GUI.

lnguyen

@ngr2001 Can you do a show int 10GigabitEthernet1/2/8?

stephenw10

Ah, that seems like a pretty big clue....

stephenw10

The 7250 doesn't have any 10GbaseT ports though so I assume that's a different Intel NIC?

stephenw10

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

My LAN MTU is now 1460 in PF, so far no change is certain speedtests

Did you set MSS like I suggested though? MTU does nothing to pf.

lnguyen

@stephenw10 Copper 10G-BaseT SFP+ modules most likely being used

ngr2001

@lnguyen

Yes I have SPF+ RJ45 adapters in the 7250, thus I can easily move the CAT8 from one jack to the over for this test.

lnguyen

@ngr2001 Can you do a show int 10GigabitEthernet1/2/8?

ngr2001

@stephenw10

I did not, but I just corrected that. I reset the states, waited a few mins and so far there may be a small boost in performance. I seem to be averaging now 700Mpbs instead of 500Mbps on fast.com, however if I switch ports we get the full 900+. So better but not perfect.

ngr2001

@lnguyen

1/2/8 - My 10Gb uplink to PF LAN Nic.

I just noticed that 2 of my main clients currently at 1Gb on the switch have a lot of dropped packets. However right now I am running speedtests and I am not seeing the dropped packet counter increasing, so who know under which setup all these packets were dropped.

1Gb Win 11 Client