Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

stephenw10

Yes I would start to suspect the module. It should negotiate that over base-T but....
For other media types it's fixed like:

ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN4
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:08:a2:12:e2:cb
	inet6 fe80::208:a2ff:fe12:e2cb%ix1 prefixlen 64 scopeid 0x6
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 1000Base-KX
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: ix1
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-H10GB-CU5M SN: CSS51F70287 DATE: 2015-07-01

There it's just set to 'enabled' in the switch. I would try setting both ends as enabled rather than negotiated.

ngr2001

@stephenw10

I dont see any other options other than negotiate on the brocade side, are you suggesting I tweak the PF Side with a tunable ?

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

I also found this in the manual:

So what the heck does that mean, the manual does not offer any means or clues on if you can manually force flow control on a 10Gb port. To me this simply reads as not supported ?

stephenw10

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

Auto-negotiation of flow control is not supported on 10 Gbps and 40 Gbps ports, fiber ports, and copper or fiber combination ports.

So you need to set a 10G port to enabled not neg-enabled.

stephenw10

There are multiple options there:

SSH@icx7250(config-if-e10000-1/2/1)#flow-control 
  both            Flow Control in PAUSE generation and honoring mode.
  generate-only   Flow Control in PAUSE generation only mode.
  honor-only      Flow Control in  PAUSE honoring (Default) mode.
  neg-on          Enable Flow Control with negotiation enabled
  <cr>

You probably want both

ngr2001

@stephenw10

I did just see that too.

I just set it to both, so far not seeing any change on the PF Side. Going to reboot everything and check again.

ngr2001

@stephenw10

Set to Both on Brocade Port

No change on PF side after rebooting both PF and Switch.

ngr2001

@stephenw10

On the transceiver side I am using:

10Gtek 80-Meter, 10GBase-T SFP+ to RJ45 Transceiver
https://www.amazon.com/gp/product/B094N9YKN9/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

ngr2001

@stephenw10

Found this gem, I think someone suggested its my transceiver, I bet it doesn't support flow control.

stephenw10

Urgh, well that would do it I guess.

lnguyen

@ngr2001 Did you buy the single speed 10GbE model or the quad speed 1/2.5/5/10GbE model?

ngr2001

@lnguyen

The 80M single speed 10GbE model which to what I am researching is unlikely to support flow control, what a kick in the pants.

lnguyen

@ngr2001 That is likely the case since the only factor that changes is moving the NIC from RJ45 switchport to SFP+ switchport. When does this Cisco switch you ordered arrive?

ngr2001

@lnguyen

Not for a few days, going away for the weekend though. I'm debating getting a new SFP+ adapter just to test this brocade out for educational purposes.

To your point, I'm 100% going to swap over to the Cisco 3850 and simply run WAN & LAN at 2.5Gb, hopefully my 1Gb clients will not have an issue but I more confident in my ability to fix it in Cisco land.

lnguyen

@ngr2001 If you use the global setting qos queue-softmax-multiplier 1200, you should get results like I shared earlier and below:

sudo ethtool enp110s0 | grep Speed
	Speed: 1000Mb/s

speedtest 

   Speedtest by Ookla

      Server: Sonic.net, Inc. - San Jose, CA (id: 17846)
         ISP: Comcast Cable
Idle Latency:    16.36 ms   (jitter: 3.65ms, low: 10.12ms, high: 20.24ms)
    Download:   897.11 Mbps (data used: 1.6 GB)                                                   
                 15.93 ms   (jitter: 5.82ms, low: 7.80ms, high: 257.78ms)
      Upload:   313.54 Mbps (data used: 179.2 MB)                                                   
                 17.45 ms   (jitter: 2.46ms, low: 13.94ms, high: 47.53ms)
 Packet Loss:     0.0%

Result URL: https://www.speedtest.net/result/c/50a2ddfa-c2b9-489c-ba91-5e7b4db52191

speedtest

   Speedtest by Ookla

      Server: Acreto - San Jose, CA (id: 56175)
         ISP: Comcast Cable
Idle Latency:    10.91 ms   (jitter: 5.79ms, low: 7.68ms, high: 20.91ms)
    Download:   931.50 Mbps (data used: 1.1 GB)                                                   
                 26.69 ms   (jitter: 30.95ms, low: 12.98ms, high: 295.44ms)
      Upload:   312.45 Mbps (data used: 287.8 MB)                                                   
                 16.41 ms   (jitter: 4.14ms, low: 12.00ms, high: 65.82ms)
 Packet Loss:     0.0%

Result URL: https://www.speedtest.net/result/c/a7bf33ea-a2e0-45a9-891e-4ad0abd4bbb0

You shouldn't have to use 802.3x FC as the larger buffers will mostly mask the symptoms of broken TCP FC. It just won't give you root cause resolution.

ngr2001

@lnguyen

Makes sense, that's the plan.

You mentioned you have been chasing this down for 3+ years with no end in sight. If one wanted to throw stupid money at it, what is the proper solution appose to just masking the symptoms.

Are we talking $10K data center switches ? I have to imagine in 2025 there has to a product that can properly handle this type of mixed network speed architecture?

stephenw10

You tried bumping the qos buffer values on the 7250?

lnguyen

@ngr2001 I have been thinking about this issue and discussing with other Network Engineering colleagues about this. When Comcast introduced the Gigabit Extra/Plus plans it was 1.2Gbps (provisioned 1440Mbps DS). This was the first time DOCSIS internet services surpassed the mainstream 1GbE LAN clients.

I had that plan for several years and never noticed an issue with buffer overflow on my 1GbE LAN clients--likely due to the buffer in my switch masking the broken TCP FC. However in December of 2022, when they released Gigabit x2 (provisioned as 2.35Gbps DS) I immediately experienced 500Mbps download/speedtest on 1GbE connected LAN clients-- yet full 2.35Gbps DS for 2.5GbE/5GbE/10GbE LAN clients.

I sent you a DM to a ChatGPT link. I asked it some questions to answer that might help you understand the way 802.3x FC works vs TCP FC. As well as how DOCSIS handles traffic congestion and its possible impact on TCP FC.

lnguyen

@ngr2001 Would you mind doing a test for me? Would you mind connecting a 1GbE client directly to your modem, reboot to get an IP and then test? If you still 500-800Mbps then it is not the pfSense, but if its 940/940Mbps, then its something in pfSense.

lnguyen

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

If one wanted to throw stupid money at it, what is the proper solution appose to just masking the symptoms.

Are we talking $10K data center switches ?

You can go with their Gigabit Pro/x10 (up to 10Gbps symmetrical) service plan for $300/mo if you are within reach of their fiber nodes. This is basically the same as their metro ethernet services that is sold to business enterprises. In fact it is that team that gets it installed and provides ongoing support. I would have done that already if I was within reach of the fiber node in my neighborhood.

https://www.reddit.com/r/Comcast_Xfinity/comments/14t9bph/gigabit_pro_availability_inquiry/

ChuckLasher

@lnguyen

I too have this problem.

I have pfSense as my edge.

HW configuration is as follows:

ISP XB8 @ 2.5 GbE -> pfSense @ 10GbE
pfSense (10G) -> Netgear XS724EM (10G)
Netgear XS724EM -> Netgear GS110EMX (10G -> 1G) (for residual 1GbE devices)

Several 1GbE-only devices are in the 10GbE switch

The only solution I currently have is to use 802.3x Ethernet Flow Control on
the LAN side of the pfSense -AND- XS724EM "input" port.

This is my "big hammer" approach.

I cannot afford enterprise grade equipment (I'm retired)