Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

ngr2001

So I believe I upgraded to the Layer 3 firmware and enabled it from what I recall.

HW: Stackable ICX7250-48

UNIT 1: SL 1: ICX7250-48 48-port Management Module

  Software Package: **ICX7250_L3_SOFT_PACKAGE**
  Current License: l3-prem-8X10G
  P-ASIC  0: type B344, rev 01  Chip BCM56344_A0

stephenw10

Right. As long as it's not actually routing that shouldn't matter.

It sure seems like it's failing to negotiate flow control though. What does: ifconfig -vvm ix1 show in pfSense?

ngr2001

This post is deleted!

ngr2001

@stephenw10

When connected at 10Gb I get this:

When connected at 1Gb I get this:

lnguyen

@ngr2001 Maybe the 10GBASE-T SFP+ module you have is not passing along the 802.3x FC negotiation to the SFP+ port on the Brocade 7250

stephenw10

Yes I would start to suspect the module. It should negotiate that over base-T but....
For other media types it's fixed like:

ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN4
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:08:a2:12:e2:cb
	inet6 fe80::208:a2ff:fe12:e2cb%ix1 prefixlen 64 scopeid 0x6
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 1000Base-KX
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: ix1
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-H10GB-CU5M SN: CSS51F70287 DATE: 2015-07-01

There it's just set to 'enabled' in the switch. I would try setting both ends as enabled rather than negotiated.

ngr2001

@stephenw10

I dont see any other options other than negotiate on the brocade side, are you suggesting I tweak the PF Side with a tunable ?

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

I also found this in the manual:

So what the heck does that mean, the manual does not offer any means or clues on if you can manually force flow control on a 10Gb port. To me this simply reads as not supported ?

stephenw10

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

Auto-negotiation of flow control is not supported on 10 Gbps and 40 Gbps ports, fiber ports, and copper or fiber combination ports.

So you need to set a 10G port to enabled not neg-enabled.

stephenw10

There are multiple options there:

SSH@icx7250(config-if-e10000-1/2/1)#flow-control 
  both            Flow Control in PAUSE generation and honoring mode.
  generate-only   Flow Control in PAUSE generation only mode.
  honor-only      Flow Control in  PAUSE honoring (Default) mode.
  neg-on          Enable Flow Control with negotiation enabled
  <cr>

You probably want both

ngr2001

@stephenw10

I did just see that too.

I just set it to both, so far not seeing any change on the PF Side. Going to reboot everything and check again.

ngr2001

@stephenw10

Set to Both on Brocade Port

No change on PF side after rebooting both PF and Switch.

ngr2001

@stephenw10

On the transceiver side I am using:

10Gtek 80-Meter, 10GBase-T SFP+ to RJ45 Transceiver
https://www.amazon.com/gp/product/B094N9YKN9/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

ngr2001

@stephenw10

Found this gem, I think someone suggested its my transceiver, I bet it doesn't support flow control.

stephenw10

Urgh, well that would do it I guess.

lnguyen

@ngr2001 Did you buy the single speed 10GbE model or the quad speed 1/2.5/5/10GbE model?

ngr2001

@lnguyen

The 80M single speed 10GbE model which to what I am researching is unlikely to support flow control, what a kick in the pants.

lnguyen

@ngr2001 That is likely the case since the only factor that changes is moving the NIC from RJ45 switchport to SFP+ switchport. When does this Cisco switch you ordered arrive?

ngr2001

@lnguyen

Not for a few days, going away for the weekend though. I'm debating getting a new SFP+ adapter just to test this brocade out for educational purposes.

To your point, I'm 100% going to swap over to the Cisco 3850 and simply run WAN & LAN at 2.5Gb, hopefully my 1Gb clients will not have an issue but I more confident in my ability to fix it in Cisco land.

lnguyen

@ngr2001 If you use the global setting qos queue-softmax-multiplier 1200, you should get results like I shared earlier and below:

sudo ethtool enp110s0 | grep Speed
	Speed: 1000Mb/s

speedtest 

   Speedtest by Ookla

      Server: Sonic.net, Inc. - San Jose, CA (id: 17846)
         ISP: Comcast Cable
Idle Latency:    16.36 ms   (jitter: 3.65ms, low: 10.12ms, high: 20.24ms)
    Download:   897.11 Mbps (data used: 1.6 GB)                                                   
                 15.93 ms   (jitter: 5.82ms, low: 7.80ms, high: 257.78ms)
      Upload:   313.54 Mbps (data used: 179.2 MB)                                                   
                 17.45 ms   (jitter: 2.46ms, low: 13.94ms, high: 47.53ms)
 Packet Loss:     0.0%

Result URL: https://www.speedtest.net/result/c/50a2ddfa-c2b9-489c-ba91-5e7b4db52191

speedtest

   Speedtest by Ookla

      Server: Acreto - San Jose, CA (id: 56175)
         ISP: Comcast Cable
Idle Latency:    10.91 ms   (jitter: 5.79ms, low: 7.68ms, high: 20.91ms)
    Download:   931.50 Mbps (data used: 1.1 GB)                                                   
                 26.69 ms   (jitter: 30.95ms, low: 12.98ms, high: 295.44ms)
      Upload:   312.45 Mbps (data used: 287.8 MB)                                                   
                 16.41 ms   (jitter: 4.14ms, low: 12.00ms, high: 65.82ms)
 Packet Loss:     0.0%

Result URL: https://www.speedtest.net/result/c/a7bf33ea-a2e0-45a9-891e-4ad0abd4bbb0

You shouldn't have to use 802.3x FC as the larger buffers will mostly mask the symptoms of broken TCP FC. It just won't give you root cause resolution.

ngr2001

@lnguyen

Makes sense, that's the plan.

You mentioned you have been chasing this down for 3+ years with no end in sight. If one wanted to throw stupid money at it, what is the proper solution appose to just masking the symptoms.

Are we talking $10K data center switches ? I have to imagine in 2025 there has to a product that can properly handle this type of mixed network speed architecture?