10G SPF+ ports are not working on my Supermicro E300-9A
-
@pete35 No I haven't tried that, let me try it out right now.
I tried that and now when I connect
ix2
andix3
the ports are working and it lights up. This is so amazing and I am so stupid at the same time -
@pete35 Removing the power code completely after a system halt from the UI seem to fix this issue. I am not sure what was causing this. I don't know what the issue was but this baffles me on how "turning off and on again" just seem to work.
But I would actually like to know what what the reason behind this. Is it ACPI power settings or something or is it port power settings...
-
We have seen issues with incompatible modules that require a full power cycle to clear. The NICs can because stuck in a mode the driver cannot recover from and simply rebooting doesn't reset them.
If running ifconfig down/up allows it to link though a reboot, or booting with the modules inserted, should also.
Steve
-
So now even though my cables work and I have set 10G-Twinax in pfsense UI, the connection to USW-Aggregation is somehow 1G. Not sure what is going on...
-
Hmm, that's odd. It's 10G or nothing with DAC cables. Is it showing that at both ends?
-
@stephenw10 IKR, right?
Here's the output for the
ifconfig -vvvm ix2
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (Unknown <rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Screenshot of the interface status page:
Now, if I force 10G on the 1st interface of the USW-Agg the link stops working
-
If you set the switch to 10G fixed auto-negotiation will be disabled so you would also need to set the pfSense end to 10G fixed also. And you might need to down/up the interface again.
Steve
-
@stephenw10 So I tried making both sides 10G and did a
ifconfig ix2 down
and thenifconfig ix2 up
but that doesn't seem to be working. I also tried the power off, remove the power adapter, wait for 2-3 min and plug back in the adapter, turn on the machine but that didn't change anything.I tried to see the link speeds using
sysctl dev.ix.2.advertise_speed
andsysctl dec.ix.3.advertise_speed
and one of them was showing value0
and other was showing value4
.I tried changing it using
sysctl dev.ix.2.advertise_speed:4
on the one which was showing0
but it gave an error saying that I can't do that and this is only supported on multi-speed link thingy which I can't seem to remember properly. (sorry if this is not very clear it like 2:42AM here).I don't know what is causing this, I also tried both DAC cables I have with me, but it still wants to stick to 1G on switch side and doesn't work when both are set to 10G.
-
The speeds available to limit advertisements to should be determined by the module. Did you have the same DAC cable in each?
-
@stephenw10 Actually they are different cables. One from ubiquiti and other one from amazon (some brand I am not very familiar with).
-
Ah, well that probably explains the difference there then.
If you need 10G there you may have to use fiber modules instead. They usually present more options.
-
@stephenw10 Thanks for all the help, I will get fiber modules for both the spf+ ports. Do you know if this will work with pfsense https://store-ui.in/collections/unifi-accessories/products/10-gbps-multi-mode-optical-module
-
No, sorry, I've never tested that. I would expect it to though if the NIC accepts it.
-
@stephenw10 Ohh no worries, I will try and talk to supermicro folks and update this thread on what I ended up doing.
-
@stephenw10 so I got SPF+ fiber modules from super micro and they work fine within ubiquiti to ubiquiti connection with LC type cable.
But when I connect my supermicro SFP+ to my ubiquiti USW it doesn't
Here are the output of same commands:
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 39.91 C Voltage: 3.37 Volts RX: 0.41 mW (-3.83 dBm) TX: 0.69 mW (-1.56 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
I don't know what I am doing wrong, supermicro support said these modules are compatible with E300-9A.
-
The switch ports are at 10G?
Do you see link if you connect two ix ports on the SM?
You tried ifconfig down/up on the interface?
-
@stephenw10 I will try it today. And also I just wanted to mention that all the interfaces except two WAN interfaces are in bridge mode.
-
Using SFP+ DAC modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix2, ix3]
:ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Connection from pfsense to USW-Agg via 10G DAC without being a part of BRIDGE
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-Twinax nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 1X Copper Passive (No separable connector) vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: BA22043002591 DATE: 2022-04-27 SFF8472 DUMP (0xA0 0..127 range): 03 04 23 01 00 00 04 41 84 80 D5 00 67 00 00 00 00 00 01 00 55 62 69 71 75 69 74 69 20 49 6E 63 2E 20 20 20 00 24 5A 4C 44 41 43 2D 53 46 50 31 30 2D 30 2E 35 4D 20 20 41 31 20 20 01 00 00 CE 00 00 00 00 42 41 32 32 30 34 33 30 30 32 35 39 31 20 20 20 32 32 30 34 32 37 20 20 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Using SFP+ fiber modules while ix2 and ix3 not part of BRIDGE interfaces
Output of
ifconfig -vvvm [ix3, ix2]
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 35.81 C Voltage: 3.34 Volts RX: 0.60 mW (-2.17 dBm) TX: 0.63 mW (-1.94 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>) status: active supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 41.13 C Voltage: 3.37 Volts RX: 0.56 mW (-2.47 dBm) TX: 0.69 mW (-1.56 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
Connection from pfsense to USW-Agg using SPF+ fiber modules without being a part of BRIDGE:
Output of
ifconfig -vvvm [ix3, ix2]
:I tried
ifconfig ix3/ix2 up/down
but that doesn't seem to have any effect...ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN4 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5b inet6 fe80::ae1f:6bff:fe4d:835b%ix3 prefixlen 64 scopeid 0x8 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0257 DATE: 2022-11-30 module temperature: 41.54 C Voltage: 3.34 Volts RX: 0.49 mW (-3.04 dBm) TX: 0.64 mW (-1.87 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 37 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1D 00 00 11 98 5C 37 04 BB 03 86 91 6F C4 46 7D 3C 44 D5 BA 00 00 00 00 00 00 00 00 00 1D C2 8C 47 ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN5 options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether ac:1f:6b:4d:83:5a inet6 fe80::ae1f:6bff:fe4d:835a%ix2 prefixlen 64 scopeid 0x7 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: OEM PN: AXS85-192-M3 SN: WT101MB0256 DATE: 2022-11-30 module temperature: 44.41 C Voltage: 3.37 Volts RX: 0.00 mW (-40.00 dBm) TX: 0.70 mW (-1.49 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 08 02 00 1E 4F 45 4D 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 00 41 58 53 38 35 2D 31 39 32 2D 4D 33 20 20 20 20 30 31 20 20 03 52 00 79 00 1A 00 00 57 54 31 30 31 4D 42 30 32 35 36 20 20 20 20 20 32 32 31 31 33 30 20 20 68 F0 08 1C 00 00 11 58 CF 59 31 EE E8 1C B2 EB 27 D8 3A 1F 3B 6C FF 00 00 00 00 00 00 00 00 00 D9 89 F0 FE
-
Hmm, so none of those tests include ix2/3 as part of a bridge?
-
@stephenw10 yeah... I am so confused...