Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

lnguyen

@ngr2001 I am not sure you are catching what I have said a few times across the two threads. Ethernet Flow Control is done on L2, which is what you have been configuring. TCP Flow Control is done at L4 which seems to not function when pfSense WAN is connected to DOCSIS cable modem. If you want to use Ethernet Flow control, which IMO is ugly, then you should leave it enabled.

With my setup, I don't use ethernet flow control.

stephenw10

Mmm, try with the MTU at the default 1500 so it matches the switch and client.

200k input errors is pretty significant though.

ngr2001

@stephenw10

Just tried, same diff, speeds quickly taper down to 5-600Mbps.

I clear my stats on switch, right now at least the dropped packet count is not increasing.

stephenw10

Just to be clear you are running the switch at layer2 only right?

It's hard to see how adding the switch, or just changing the port speed, can have any effect on the tcp dircetly. It could be that is some how broken upstream such that when introducing some lower lever issue it's very badly effected though.

ngr2001

@stephenw10

I am not using any layer 3 features that I am aware of right now, thus I guess it should be in layer 2. I do have flow control enabled on the switch though.

SSH@romulus#show run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
module 1 icx7250-48-port-management-module
module 2 icx7250-sfp-plus-8port-80g-module
stack-port 1/2/1
stack-port 1/2/3
!

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
symmetrical-flow-control enable
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname romulus
ip dhcp-client disable
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server
!

clock timezone us Eastern
!
!
ntp
disable serve
server time.cloudflare.com
!
!
no web-management http
!
manager disable
!
!
manager port-list 987
!
!
interface ethernet 1/1/4
flow-control neg-on
!
interface ethernet 1/1/8
flow-control neg-on
!
interface ethernet 1/1/48
flow-control neg-on
!
interface ethernet 1/2/1
flow-control neg-on
!
interface ethernet 1/2/8
flow-control neg-on
!
interface ve 1
ip address 10.0.0.3 255.255.255.0
!

ip ssh idle-time 120
!
end
SSH@romulus#

ngr2001

@stephenw10

So I believe I upgraded to the Layer 3 firmware and enabled it from what I recall.

HW: Stackable ICX7250-48

UNIT 1: SL 1: ICX7250-48 48-port Management Module

  Software Package: **ICX7250_L3_SOFT_PACKAGE**
  Current License: l3-prem-8X10G
  P-ASIC  0: type B344, rev 01  Chip BCM56344_A0

stephenw10

Right. As long as it's not actually routing that shouldn't matter.

It sure seems like it's failing to negotiate flow control though. What does: ifconfig -vvm ix1 show in pfSense?

ngr2001

This post is deleted!

ngr2001

@stephenw10

When connected at 10Gb I get this:

When connected at 1Gb I get this:

lnguyen

@ngr2001 Maybe the 10GBASE-T SFP+ module you have is not passing along the 802.3x FC negotiation to the SFP+ port on the Brocade 7250

stephenw10

Yes I would start to suspect the module. It should negotiate that over base-T but....
For other media types it's fixed like:

ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN4
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:08:a2:12:e2:cb
	inet6 fe80::208:a2ff:fe12:e2cb%ix1 prefixlen 64 scopeid 0x6
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 1000Base-KX
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: ix1
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-H10GB-CU5M SN: CSS51F70287 DATE: 2015-07-01

There it's just set to 'enabled' in the switch. I would try setting both ends as enabled rather than negotiated.

ngr2001

@stephenw10

I dont see any other options other than negotiate on the brocade side, are you suggesting I tweak the PF Side with a tunable ?

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

I also found this in the manual:

So what the heck does that mean, the manual does not offer any means or clues on if you can manually force flow control on a 10Gb port. To me this simply reads as not supported ?

stephenw10

@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

https://docs.commscope.com/bundle/fastiron-08095-managementguide/page/GUID-A5971868-1051-4807-8ED2-D3BC6B10AA3B.html

Auto-negotiation of flow control is not supported on 10 Gbps and 40 Gbps ports, fiber ports, and copper or fiber combination ports.

So you need to set a 10G port to enabled not neg-enabled.

stephenw10

There are multiple options there:

SSH@icx7250(config-if-e10000-1/2/1)#flow-control 
  both            Flow Control in PAUSE generation and honoring mode.
  generate-only   Flow Control in PAUSE generation only mode.
  honor-only      Flow Control in  PAUSE honoring (Default) mode.
  neg-on          Enable Flow Control with negotiation enabled
  <cr>

You probably want both

ngr2001

@stephenw10

I did just see that too.

I just set it to both, so far not seeing any change on the PF Side. Going to reboot everything and check again.

ngr2001

@stephenw10

Set to Both on Brocade Port

No change on PF side after rebooting both PF and Switch.

ngr2001

@stephenw10

On the transceiver side I am using:

10Gtek 80-Meter, 10GBase-T SFP+ to RJ45 Transceiver
https://www.amazon.com/gp/product/B094N9YKN9/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

ngr2001

@stephenw10

Found this gem, I think someone suggested its my transceiver, I bet it doesn't support flow control.

stephenw10

Urgh, well that would do it I guess.

lnguyen

@ngr2001 Did you buy the single speed 10GbE model or the quad speed 1/2.5/5/10GbE model?