10G SPF+ ports are not working on my Supermicro E300-9A

grim-reaper

I recently got myself a Supermicro E300-9A and I installed pfsense and followed the instructions to upgrade it to pfsense+.

Everything has been running fine so far. But my 10G SFP+ ports doesn't seem to work no matter what I do.

I tried selecting speed for both ports to 10Gbase-Twinax and my Ubiquiti USW-Aggregation ports 10G but when I check the interface status I keep seeing "no carrier".

I tried auto-negotiation on one side to see if that changes anything but that did not work either, same status on the interface status page showing "no carrier".

Here are some screenshots:

I tried ubiquiti's DAC cable that did not work, I also tried 10Gtek Twinax passive cable from amazon but that didn't work. Although if I connect any of the cables to any two ports in my ubiquiti USW-Aggregation ports it works. So I don't think the cable is the problem. I also tried connecting the cable to both the ports in my supermicro hoping something would show up but nothing. I also tried ix interface tuning guide from Netgate hoping this would fix but it did not help either.

I don't know what to do here. Any help is highly appreciated!!

stephenw10

Hmm DAC cables like that usually 'just work' or don't. There are usually no available speed/duplex options presented.

What does ifconfig -vvvm ix2 show?

If you connect ix2 to ix3 directly do you see a link then? With any of the cables?

Steve

grim-reaper

@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:

ifconfig -vvvm ix2

If I try to connect any of the DAC cables to ix2 and ix3 it still shows the same error message "no carrier" and I also tried rebooting just in case if that changes anything but it did not.

Here's the output of ifconfig -vvvm ix2 and ifconfig -vvvm ix3:

ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN4
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:4d:83:5a
	inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
------------------------------------------------------------------------------------

ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN5
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:4d:83:5b
	inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Sorry can you re-run those with a DAC cable attached. That should show everything the NIC can see about the inserted module

grim-reaper

@stephenw10

No problem at all, running with one of the cables attached to ix2 and ix3

ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN4
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:4d:83:5a
	inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-10G-DAC-005 SN: 20221013C0177 DATE: 2022-10-17

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00
	00 00 01 00 4F 45 4D 20 20 20 20 20 20 20 20 20
	20 20 20 20 00 00 40 20 53 46 50 2D 31 30 47 2D
	44 41 43 2D 30 30 35 20 52 20 20 20 01 00 00 BD
	00 00 00 00 32 30 32 32 31 30 31 33 43 30 31 37
	37 20 20 20 32 32 31 30 31 37 20 20 00 00 00 6A
	80 00 11 D8 44 0D 71 AB 4E 1F 15 4E 4B 4D D8 8E
	E2 C3 C9 00 00 00 00 00 00 00 00 00 68 09 5B 9B
-----
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN5
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:4d:83:5b
	inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
	vendor: OEM PN: SFP-10G-DAC-005 SN: 20221013C0177 DATE: 2022-10-17

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00
	00 00 01 00 4F 45 4D 20 20 20 20 20 20 20 20 20
	20 20 20 20 00 00 40 20 53 46 50 2D 31 30 47 2D
	44 41 43 2D 30 30 35 20 52 20 20 20 01 00 00 BD
	00 00 00 00 32 30 32 32 31 30 31 33 43 30 31 37
	37 20 20 20 32 32 31 30 31 37 20 20 00 00 00 6A
	80 00 11 D8 44 0D 71 AB 4E 1F 15 4E 4B 4D D8 8E
	E2 C3 C9 00 00 00 00 00 00 00 00 00 68 09 5B 9B

Screenshot of interface status page with cable attached:
Screenshot 2022-12-29 at 10.47.37 PM.png

pete35

@grim-reaper

have you seen this one, just unplug power ?
https://forum.netgate.com/topic/141117/intel-x552-onboard-sfp-no-carrier-issues

stephenw10

Hmm, OK that looks as expected. Except it's not linking.

We sometimes see modules that will not link if they are inserted after boot.
Try running:

ifconfig ix2 down
ifconfig ix2 up

grim-reaper

@pete35 No I haven't tried that, let me try it out right now.

I tried that and now when I connect ix2 and ix3 the ports are working and it lights up. This is so amazing and I am so stupid at the same time

grim-reaper

@pete35 Removing the power code completely after a system halt from the UI seem to fix this issue. I am not sure what was causing this. I don't know what the issue was but this baffles me on how "turning off and on again" just seem to work.

But I would actually like to know what what the reason behind this. Is it ACPI power settings or something or is it port power settings...

stephenw10

We have seen issues with incompatible modules that require a full power cycle to clear. The NICs can because stuck in a mode the driver cannot recover from and simply rebooting doesn't reset them.

If running ifconfig down/up allows it to link though a reboot, or booting with the modules inserted, should also.

Steve

grim-reaper

@stephenw10

So now even though my cables work and I have set 10G-Twinax in pfsense UI, the connection to USW-Aggregation is somehow 1G. Not sure what is going on...

stephenw10

Hmm, that's odd. It's 10G or nothing with DAC cables. Is it showing that at both ends?

grim-reaper

@stephenw10 IKR, right?

Here's the output for the ifconfig -vvvm ix2

ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN5
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:4d:83:5a
	inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 10Gbase-Twinax
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector)
	vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00
	00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63
	2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31
	30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE
	00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39
	31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Screenshot of the interface status page:
Screenshot 2022-12-30 at 1.19.42 AM.png

Now, if I force 10G on the 1st interface of the USW-Agg the link stops working
Screenshot 2022-12-30 at 1.21.21 AM.png

stephenw10

If you set the switch to 10G fixed auto-negotiation will be disabled so you would also need to set the pfSense end to 10G fixed also. And you might need to down/up the interface again.

Steve

grim-reaper

@stephenw10 So I tried making both sides 10G and did a ifconfig ix2 down and then ifconfig ix2 up but that doesn't seem to be working. I also tried the power off, remove the power adapter, wait for 2-3 min and plug back in the adapter, turn on the machine but that didn't change anything.

I tried to see the link speeds using sysctl dev.ix.2.advertise_speed and sysctl dec.ix.3.advertise_speed and one of them was showing value 0 and other was showing value 4.

I tried changing it using sysctl dev.ix.2.advertise_speed:4 on the one which was showing 0 but it gave an error saying that I can't do that and this is only supported on multi-speed link thingy which I can't seem to remember properly. (sorry if this is not very clear it like 2:42AM here).

I don't know what is causing this, I also tried both DAC cables I have with me, but it still wants to stick to 1G on switch side and doesn't work when both are set to 10G.

stephenw10

The speeds available to limit advertisements to should be determined by the module. Did you have the same DAC cable in each?

grim-reaper

@stephenw10 Actually they are different cables. One from ubiquiti and other one from amazon (some brand I am not very familiar with).

stephenw10

Ah, well that probably explains the difference there then.

If you need 10G there you may have to use fiber modules instead. They usually present more options.

grim-reaper

@stephenw10 Thanks for all the help, I will get fiber modules for both the spf+ ports. Do you know if this will work with pfsense https://store-ui.in/collections/unifi-accessories/products/10-gbps-multi-mode-optical-module

stephenw10

No, sorry, I've never tested that. I would expect it to though if the NIC accepts it.