XG-7100 Not accepting 10GBase-SR speed

erasedhammer

Bought a pair of QSFPTEK Intel E10GSFPSR 10G SFP+ modules to connect my xg-7100 SFP+ to another pfsense box with an Intel X710-DA4 NIC.
When I plugged it in, they defaulted to 1000base-sx unfortunately. Checked intels compatibility guide, and yep the modules are officially supported by the x710-da4.
Checked the logs of my xg-7100 and when I try to set the speed manually (10base-sr) I get:

/interfaces.php: The command '/sbin/ifconfig 'ix0' media '10Gbase-SR'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Invalid argument' 
ix0: Invalid media type!

Not sure what that means.
When I set the other pfsense box with the x710-da4 to manually do 10base-sr I get:

Traffic rate for "ixl2" higher than set maximum 1000 Mbit (20s->2673868800, r4293874187 t4289457058, 64bit:0), syncing. 
/interfaces.php: The command '/sbin/ifconfig 'ixl2' media '10Gbase-SR'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device' 
 ixl2: Use 'advertise_speed' sysctl to change advertised speeds

On the xg-7100 here is the output of "ifconfig -v ix0":

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:a2:xx:xx:xx
	hwaddr 00:08:a2:xx:xx:xx
	inet x.x.x.x netmask 0xfffffffc broadcast x.x.x.x
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseSX <full-duplex,rxpause,txpause>)
	status: active
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AJB0Q5T0593 DATE: 2020-08-17
	module temperature: 43.81 C Voltage: 3.34 Volts
	RX: 0.68 mW (-1.67 dBm) TX: 0.54 mW (-2.63 dBm)

Since the modules are officially supported by intel, I can only assume there is a problem with the netgate appliance.

stephenw10

Try running ifconfig -vvvm ix0 to get more info. What media types does it support at either end?

Steve

erasedhammer

@stephenw10
Looks like both sides support 10Gbase-SR. Not sure what the problem is.

On the XG-7100:

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:a2:xx:xx:xx
	hwaddr 00:08:a2:xx:xx:xx
	inet6 fe80::208:a2ff:fe0f:13b0%ix0 prefixlen 64 scopeid 0x1
	inet x.x.x.x netmask 0xfffffffc broadcast x.x.x.x
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseSX <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 1000baseSX
		media 10Gbase-SR
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AJB0Q5T0593 DATE: 2020-08-17
	module temperature: 38.94 C Voltage: 3.35 Volts
	RX: 0.68 mW (-1.67 dBm) TX: 0.56 mW (-2.50 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 01 00 00 00 00 06 67 02 00 00
	08 03 00 1E 49 6E 74 65 6C 20 43 6F 72 70 20 20
	20 20 20 20 00 00 1B 21 46 54 4C 58 38 35 37 31
	44 33 42 43 56 2D 49 54 41 20 20 20 03 52 00 88
	00 3A 00 00 41 4A 42 30 51 35 54 30 35 39 33 20
	20 20 20 20 32 30 30 38 31 37 20 20 68 FA 03 59
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

On the other pfsense box with x710-da4:

ixl2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=7507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 68:05:ca:xx:xx:xx
	hwaddr 68:05:ca:xx:xx:xx
	inet6 fe80::6a05:caff:fe32:345e%ixl2 prefixlen 64 scopeid 0x3
	inet x.x.x.x. netmask 0xfffffffc broadcast x.x.x.x
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseSX <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 10Gbase-LR
		media 10Gbase-SR
		media 10Gbase-Twinax
		media 1000baseLX
		media 1000baseSX
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AJB0Q5T0476 DATE: 2020-08-17
	module temperature: 46.66 C Voltage: 3.37 Volts
	RX: 0.64 mW (-1.93 dBm) TX: 0.80 mW (-0.93 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 01 00 00 00 00 06 67 02 00 00
	08 03 00 1E 49 6E 74 65 6C 20 43 6F 72 70 20 20
	20 20 20 20 00 00 1B 21 46 54 4C 58 38 35 37 31
	44 33 42 43 56 2D 49 54 41 20 20 20 03 52 00 88
	00 3A 00 00 41 4A 42 30 51 35 54 30 34 37 36 20
	20 20 20 20 32 30 30 38 31 37 20 20 68 FA 03 59
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

stephenw10

Are both sides set to advertise 10G:

[2.4.5-RELEASE][admin@7100.stevew.lan]/root: sysctl -d dev.ix.1.advertise_speed
dev.ix.1.advertise_speed: 
Control advertised link speed using these flags:
	0x1 - advertise 100M
	0x2 - advertise 1G
	0x4 - advertise 10G
	0x8 - advertise 10M
	100M and 10M are only supported on certain adapters.

So if that is 4 or 6 it is.

What distance are they connected over? Have you tried a different fiber? Cleaning it? The power levels on each look quite different.

Steve

erasedhammer

@stephenw10

dev.ix.0.advertise_speed: 7

How do I set the advertise speed manually?
The link is over a 1.5 meter om3 patch cable.
I could try a different cable but the one in use is brand new.
Wouldn't I see errors if the cable were the issue?

stephenw10

Well 0x7 (0111) includes 10G so you shouldn't need to but you can set that sysctl:

sysctl dev.ix.0.advertise_speed=4

If the module decides the link cannot support 10G and drops to 1G you might not see errors.
It's unlikely though I agree.

Steve

erasedhammer

@stephenw10
Setting the advertise speed fixed it. Not sure why I couldn't set it manually in the GUI though.
What did you say about the power levels being different on each end?

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:a2:xx:xx:xx
	hwaddr 00:08:a2:xx:xx:xx
	inet6 fe80::208:a2ff:fe0f:13b0%ix0 prefixlen 64 scopeid 0x1
	inet x.x.x.x netmask 0xfffffffc broadcast x.x.x.x
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 1000baseSX
		media 10Gbase-SR
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AJB0Q5T0593 DATE: 2020-08-17
	module temperature: 39.34 C Voltage: 3.35 Volts
	RX: 0.68 mW (-1.67 dBm) TX: 0.56 mW (-2.50 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 01 00 00 00 00 06 67 02 00 00
	08 03 00 1E 49 6E 74 65 6C 20 43 6F 72 70 20 20
	20 20 20 20 00 00 1B 21 46 54 4C 58 38 35 37 31
	44 33 42 43 56 2D 49 54 41 20 20 20 03 52 00 88
	00 3A 00 00 41 4A 42 30 51 35 54 30 35 39 33 20
	20 20 20 20 32 30 30 38 31 37 20 20 68 FA 03 59
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

stephenw10

Ah, nice. Were you able to add that as a system tunable so it applies at boot?

If you look are the extended ifconfig output at both ends it shows the TX a RX power on the modules. Both are receiving at about the same single level but one side is transmitting a lot higher to achieve that. Since I imagine the fiber is the same in both directions it implied one end might be dirty or something similar. It's not something I know in depth though so it could be nothing.

Steve

erasedhammer

@stephenw10
Yep, I added dev.ix.0.advertise_speed with value of 4 to the tunables.