10G SPF+ ports are not working on my Supermicro E300-9A
-
Using SFP+ DAC modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix2, ix3]
:ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Connection from pfsense to USW-Agg via 10G DAC without being a part of BRIDGE
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Using SFP+ fiber modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix3, ix2]
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 35.81 C Voltage: 3.34 Volts RX: 0.60 mW (-2.17 dBm) TX: 0.63 mW (-1.94 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 41.13 C Voltage: 3.37 Volts RX: 0.56 mW (-2.47 dBm) TX: 0.69 mW (-1.56 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
Connection from pfsense to USW-Agg using SPF+ fiber modules without being a part of BRIDGE:
Output of
ifconfig -vvvm [ix3, ix2]
:I tried
ifconfig ix3/ix2 up/down
but that doesn't seem to have any effect...ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 41.54 C Voltage: 3.34 Volts RX: 0.49 mW (-3.04 dBm) TX: 0.64 mW (-1.87 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 44.41 C Voltage: 3.37 Volts RX: 0.00 mW (-40.00 dBm) TX: 0.70 mW (-1.49 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
-
Hmm, so none of those tests include ix2/3 as part of a bridge?
-
@stephenw10 yeah... I am so confused...
-
Hmm, so you give two sets of output above where the link is up correctly. Can you not use either of those setups?
I'm not sure what the difference is in the last output that fails... -
@stephenw10 I am not sure either. But I want the interfaces to be a part of the BRIDGE group. And connecting from SM to USW-Agg never works, it always auto negotiates to 1G instead of 10G
-
Hmm, so when the ports are part of a bridge they link to the switch but only at 1G? Presumably as shown above with pfSense showing 'Unknown' and the switch showing 1G?
But when removed from the bridge they are able to link at 10G?
You're running 2.6 here? You might try a 2.7 snapshot as a test. The behaviour of bridges inherited port settings changed there recently.
Steve
-
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
Hmm, so when the ports are part of a bridge they link to the switch but only at 1G? Presumably as shown above with pfSense showing 'Unknown' and the switch showing 1G?
No, the connection to the switch never work no matter if the ports in SM are part of BRIDGE or not. At most they auto-negotiate to 1G with both DAC and Fiber module.
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
But when removed from the bridge they are able to link at 10G?
The link between SM work when removed from BRIDGE with both fiber and DAC cable.
So I still give 2.7 a try?
-
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
You're running 2.6 here?
I'm running pfsense+ and the version is 22.05.
-
Then try 23.01. If you're running ZFS you can roll back the BE snap to 22.05 if required.
-
@stephenw10 Tried the 23.01 build but same issue no luck there either. Tested all similar cases with it but none worked.
-
So unable to get a link to the switch at all or only able to link 1G?
You haven't seen a 10G link to the switch at any time?
-
@stephenw10 Unfortunately no ... I tried all cables and all other suggestions mentioned in this thread.
-
Hmm. What options does the switch give you for link speeds on the port?
-
@stephenw10 These are the options I see on all 10G ports on both my Ubiquiti SWs
-
Ok so if you set both ends to 10G fixed does it still not link?
-
@stephenw10 If I try specifying 10G on both or either one the link doesn't work.
-
Hmm, not sure what more to suggest.
You could try booting a live Linux distro and see if it links there. At least confirm it's not a hardware issue.
-
@stephenw10 Yeah I guess that's the last thing to try.
The weird part is the port work fine when connected to each other like ix2 <---> ix3 but don't when connected to ubiquiti sw
Thanks for all the suggestions and help though, really appreciate it!