10G SPF+ ports are not working on my Supermicro E300-9A
-
Hmm, that's odd. It's 10G or nothing with DAC cables. Is it showing that at both ends?
-
@stephenw10 IKR, right?
Here's the output for the
ifconfig -vvvm ix2
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (Unknown <rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Screenshot of the interface status page:
Now, if I force 10G on the 1st interface of the USW-Agg the link stops working
-
If you set the switch to 10G fixed auto-negotiation will be disabled so you would also need to set the pfSense end to 10G fixed also. And you might need to down/up the interface again.
Steve
-
@stephenw10 So I tried making both sides 10G and did a
ifconfig ix2 down
and thenifconfig ix2 up
but that doesn't seem to be working. I also tried the power off, remove the power adapter, wait for 2-3 min and plug back in the adapter, turn on the machine but that didn't change anything.I tried to see the link speeds using
sysctl dev.ix.2.advertise_speed
andsysctl dec.ix.3.advertise_speed
and one of them was showing value0
and other was showing value4
.I tried changing it using
sysctl dev.ix.2.advertise_speed:4
on the one which was showing0
but it gave an error saying that I can't do that and this is only supported on multi-speed link thingy which I can't seem to remember properly. (sorry if this is not very clear it like 2:42AM here).I don't know what is causing this, I also tried both DAC cables I have with me, but it still wants to stick to 1G on switch side and doesn't work when both are set to 10G.
-
The speeds available to limit advertisements to should be determined by the module. Did you have the same DAC cable in each?
-
@stephenw10 Actually they are different cables. One from ubiquiti and other one from amazon (some brand I am not very familiar with).
-
Ah, well that probably explains the difference there then.
If you need 10G there you may have to use fiber modules instead. They usually present more options.
-
@stephenw10 Thanks for all the help, I will get fiber modules for both the spf+ ports. Do you know if this will work with pfsense https://store-ui.in/collections/unifi-accessories/products/10-gbps-multi-mode-optical-module
-
No, sorry, I've never tested that. I would expect it to though if the NIC accepts it.
-
@stephenw10 Ohh no worries, I will try and talk to supermicro folks and update this thread on what I ended up doing.
-
@stephenw10 so I got SPF+ fiber modules from super micro and they work fine within ubiquiti to ubiquiti connection with LC type cable.
But when I connect my supermicro SFP+ to my ubiquiti USW it doesn't
Here are the output of same commands:
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 39.91 C Voltage: 3.37 Volts RX: 0.41 mW (-3.83 dBm) TX: 0.69 mW (-1.56 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
I don't know what I am doing wrong, supermicro support said these modules are compatible with E300-9A.
-
The switch ports are at 10G?
Do you see link if you connect two ix ports on the SM?
You tried ifconfig down/up on the interface?
-
@stephenw10 I will try it today. And also I just wanted to mention that all the interfaces except two WAN interfaces are in bridge mode.
-
Using SFP+ DAC modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix2, ix3]
:ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Connection from pfsense to USW-Agg via 10G DAC without being a part of BRIDGE
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Using SFP+ fiber modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix3, ix2]
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 35.81 C Voltage: 3.34 Volts RX: 0.60 mW (-2.17 dBm) TX: 0.63 mW (-1.94 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 41.13 C Voltage: 3.37 Volts RX: 0.56 mW (-2.47 dBm) TX: 0.69 mW (-1.56 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
Connection from pfsense to USW-Agg using SPF+ fiber modules without being a part of BRIDGE:
Output of
ifconfig -vvvm [ix3, ix2]
:I tried
ifconfig ix3/ix2 up/down
but that doesn't seem to have any effect...ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 41.54 C Voltage: 3.34 Volts RX: 0.49 mW (-3.04 dBm) TX: 0.64 mW (-1.87 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 44.41 C Voltage: 3.37 Volts RX: 0.00 mW (-40.00 dBm) TX: 0.70 mW (-1.49 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
-
Hmm, so none of those tests include ix2/3 as part of a bridge?
-
@stephenw10 yeah... I am so confused...
-
Hmm, so you give two sets of output above where the link is up correctly. Can you not use either of those setups?
I'm not sure what the difference is in the last output that fails... -
@stephenw10 I am not sure either. But I want the interfaces to be a part of the BRIDGE group. And connecting from SM to USW-Agg never works, it always auto negotiates to 1G instead of 10G
-
Hmm, so when the ports are part of a bridge they link to the switch but only at 1G? Presumably as shown above with pfSense showing 'Unknown' and the switch showing 1G?
But when removed from the bridge they are able to link at 10G?
You're running 2.6 here? You might try a 2.7 snapshot as a test. The behaviour of bridges inherited port settings changed there recently.
Steve
-
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
Hmm, so when the ports are part of a bridge they link to the switch but only at 1G? Presumably as shown above with pfSense showing 'Unknown' and the switch showing 1G?
No, the connection to the switch never work no matter if the ports in SM are part of BRIDGE or not. At most they auto-negotiate to 1G with both DAC and Fiber module.
@stephenw10 said in 10G SPF+ ports are not working on my Supermicro E300-9A:
But when removed from the bridge they are able to link at 10G?
The link between SM work when removed from BRIDGE with both fiber and DAC cable.
So I still give 2.7 a try?