10G SPF+ ports are not working on my Supermicro E300-9A
-
I recently got myself a Supermicro E300-9A and I installed pfsense and followed the instructions to upgrade it to pfsense+.
Everything has been running fine so far. But my 10G SFP+ ports doesn't seem to work no matter what I do.
I tried selecting speed for both ports to 10Gbase-Twinax and my Ubiquiti USW-Aggregation ports 10G but when I check the interface status I keep seeing "no carrier".
I tried auto-negotiation on one side to see if that changes anything but that did not work either, same status on the interface status page showing "no carrier".
Here are some screenshots:
I tried ubiquiti's DAC cable that did not work, I also tried 10Gtek Twinax passive cable from amazon but that didn't work. Although if I connect any of the cables to any two ports in my ubiquiti USW-Aggregation ports it works. So I don't think the cable is the problem. I also tried connecting the cable to both the ports in my supermicro hoping something would show up but nothing. I also tried ix interface tuning guide from Netgate hoping this would fix but it did not help either.
I don't know what to do here. Any help is highly appreciated!!
-
Hmm DAC cables like that usually 'just work' or don't. There are usually no available speed/duplex options presented.
What does
ifconfig -vvvm ix2
show?If you connect ix2 to ix3 directly do you see a link then? With any of the cables?
Steve
-
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
ifconfig -vvvm ix2
If I try to connect any of the DAC cables to ix2 and ix3 it still shows the same error message "no carrier" and I also tried rebooting just in case if that changes anything but it did not.
Here's the output of
ifconfig -vvvm ix2
andifconfig -vvvm ix3
:ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect status: no carrier supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ------------------------------------------------------------------------------------ ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
-
Sorry can you re-run those with a DAC cable attached. That should show everything the NIC can see about the inserted module
-
No problem at all, running with one of the cables attached to
ix2
andix3
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail) vendor: OEM PN: SFP-10G-DAC-005 SN: 20221013C0177 DATE: 2022-10-17 SFF8472 DUMP (0xA0 0..127 range): 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00 00 00 01 00 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 40 20 53 46 50 2D 31 30 47 2D 44 41 43 2D 30 30 35 20 52 20 20 20 01 00 00 BD 00 00 00 00 32 30 32 32 31 30 31 33 43 30 31 37 37 20 20 20 32 32 31 30 31 37 20 20 00 00 00 6A 80 00 11 D8 44 0D 71 AB 4E 1F 15 4E 4B 4D D8 8E E2 C3 C9 00 00 00 00 00 00 00 00 00 68 09 5B 9B ----- ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail) vendor: OEM PN: SFP-10G-DAC-005 SN: 20221013C0177 DATE: 2022-10-17 SFF8472 DUMP (0xA0 0..127 range): 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00 00 00 01 00 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 40 20 53 46 50 2D 31 30 47 2D 44 41 43 2D 30 30 35 20 52 20 20 20 01 00 00 BD 00 00 00 00 32 30 32 32 31 30 31 33 43 30 31 37 37 20 20 20 32 32 31 30 31 37 20 20 00 00 00 6A 80 00 11 D8 44 0D 71 AB 4E 1F 15 4E 4B 4D D8 8E E2 C3 C9 00 00 00 00 00 00 00 00 00 68 09 5B 9B
Screenshot of interface status page with cable attached:
-
have you seen this one, just unplug power ?
https://forum.netgate.com/topic/141117/intel-x552-onboard-sfp-no-carrier-issues -
Hmm, OK that looks as expected. Except it's not linking.
We sometimes see modules that will not link if they are inserted after boot.
Try running:ifconfig ix2 down ifconfig ix2 up
-
@pete35 No I haven't tried that, let me try it out right now.
I tried that and now when I connect
ix2
andix3
the ports are working and it lights up. This is so amazing and I am so stupid at the same time -
@pete35 Removing the power code completely after a system halt from the UI seem to fix this issue. I am not sure what was causing this. I don't know what the issue was but this baffles me on how "turning off and on again" just seem to work.
But I would actually like to know what what the reason behind this. Is it ACPI power settings or something or is it port power settings...
-
We have seen issues with incompatible modules that require a full power cycle to clear. The NICs can because stuck in a mode the driver cannot recover from and simply rebooting doesn't reset them.
If running ifconfig down/up allows it to link though a reboot, or booting with the modules inserted, should also.
Steve
-
So now even though my cables work and I have set 10G-Twinax in pfsense UI, the connection to USW-Aggregation is somehow 1G. Not sure what is going on...
-
Hmm, that's odd. It's 10G or nothing with DAC cables. Is it showing that at both ends?
-
@stephenw10 IKR, right?
Here's the output for the
ifconfig -vvvm ix2
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (Unknown <rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Screenshot of the interface status page:
Now, if I force 10G on the 1st interface of the USW-Agg the link stops working
-
If you set the switch to 10G fixed auto-negotiation will be disabled so you would also need to set the pfSense end to 10G fixed also. And you might need to down/up the interface again.
Steve
-
@stephenw10 So I tried making both sides 10G and did a
ifconfig ix2 down
and thenifconfig ix2 up
but that doesn't seem to be working. I also tried the power off, remove the power adapter, wait for 2-3 min and plug back in the adapter, turn on the machine but that didn't change anything.I tried to see the link speeds using
sysctl dev.ix.2.advertise_speed
andsysctl dec.ix.3.advertise_speed
and one of them was showing value0
and other was showing value4
.I tried changing it using
sysctl dev.ix.2.advertise_speed:4
on the one which was showing0
but it gave an error saying that I can't do that and this is only supported on multi-speed link thingy which I can't seem to remember properly. (sorry if this is not very clear it like 2:42AM here).I don't know what is causing this, I also tried both DAC cables I have with me, but it still wants to stick to 1G on switch side and doesn't work when both are set to 10G.
-
The speeds available to limit advertisements to should be determined by the module. Did you have the same DAC cable in each?
-
@stephenw10 Actually they are different cables. One from ubiquiti and other one from amazon (some brand I am not very familiar with).
-
Ah, well that probably explains the difference there then.
If you need 10G there you may have to use fiber modules instead. They usually present more options.
-
@stephenw10 Thanks for all the help, I will get fiber modules for both the spf+ ports. Do you know if this will work with pfsense https://store-ui.in/collections/unifi-accessories/products/10-gbps-multi-mode-optical-module
-
No, sorry, I've never tested that. I would expect it to though if the NIC accepts it.