NBASE-T Support for Intel X550

daplumber

Just another data point:

pfsense 2.5.1-RELEASE (amd64)

Intel X550-T2 connected to the 2.5Gb port on an Arris Surfboard S33 Cable Modem support page:

Standard Specification Compliant	DOCSIS 3.1
Hardware Version	                V1.0
Software Version	                TB01.01.001.14_102120_192.S3

Energy Efficient Ethernet               Enabled

Connected with a 10m Cat8 Cable.

ifconfig -m ix1 reports:

media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 100baseTX
		media 1000baseT
		media 10Gbase-T

From the transfer speeds it definitely seems to be connected at greater than 1Gb and seems to be 2.5Gb from what I can tell. Obviously it's not paused.

The latest Intel Driver is 3.3.24

The README still says:

This driver supports the following media type options:

Media Type          Description
----------          -----------
autoselect          Enables auto-negotiation for speed and duplex.
10baseT/UTP         Sets speed to 10 Mbps. Use the ifconfig mediaopt
                    option to select full-duplex mode.
100baseTX           Sets speed to 100 Mbps. Use the ifconfig mediaopt
                    option to select full-duplex mode.
1000baseTX          Sets speed to 1000 Mbps. In this case, the driver
                    supports only full-duplex mode.
1000baseSX          Sets speed to 1000 Mbps. In this case, the driver
                    supports only full-duplex mode.

Here's the dmesg output from boot:

...
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0xfcc00000-0xfcffffff,0xfcbfc000-0xfcbfffff irq 19 at device 0.0 on pci2
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: a0:36:9f:28:1f:a6
ix0: PCI Express Bus: Speed 5.0GT/s Width x4
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0xfc400000-0xfc7fffff,0xfcbf8000-0xfcbfbfff irq 18 at device 0.1 on pci2
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: a0:36:9f:28:1f:a4
ix1: PCI Express Bus: Speed 5.0GT/s Width x4
ix1: netmap queues/slots: TX 8/2048, RX 8/2048
...
lo0: link state changed to UP
ix1: link state changed to UP
ix1: link state changed to DOWN
igb1: link state changed to UP
ix1: link state changed to UP
pflog0: promiscuous mode enabled
re0: link state changed to DOWN
igb1: promiscuous mode enabled
...

WAN is on ix1, LAN is on igb1, ix0, igb0, and re0 (motherboard) are not currently connected. Eventually ix0 will connected to a 10GBASE-T SFP+ transceiver on a 10G switch as soon as I get around to installing the switch with another 10G Cat8.

So the hardware still seems happy to auto-negotiate to 2500Mbps, even though the FreeBSD ix driver can't report it. ¯_(ツ)_/¯ Annoying, but usable. Where do I sign the "Dear Intel" "metoo" petition? ;-)

stephenw10

The presence of <rxpause,txpause> there indicates that flow control is active on that link. Not that it's actually paused at that moment.

daplumber

@stephenw10

I knew that! :-)

Sorry, I should have rephrased that as “Flow Control appears to be working.” Rather than saying the NIC does not appear to be paused. Doh! ;-)

Any guess on when/if this could be addressed by Intel or someone else?

stephenw10

No new information there I'm afraid.

Brian Cunnie

2.5/5G added to the kernel's ixgbe driver with this commit: https://cgit.freebsd.org/src/commit/sys/dev/ixgbe/ixgbe.h?id=d381c807510de2ebb453a563540bd17e344a2aab

daplumber

@brian-cunnie said in NBASE-T Support for Intel X550:

https://cgit.freebsd.org/src/commit/sys/dev/ixgbe/ixgbe.h?id=d381c807510de2ebb453a563540bd17e344a2aab

Any idea on approximately when this should show up in a pfsense RELEASE? Would I have to change anything?

stephenw10

It's only in 13-stable right now so at the very least it would need either to brought into 12 stable or pfSense moved to 13 stable (or newer).

Brian Cunnie

@daplumber I don't have any idea when it would show up in a pfsense release.

Also, I've been struggling to get this to work on my non-pfsense straight-FreeBSD 13 firewall, and in spite of several attempts the Intel X550 negotiates a 1Gbe not 2.5Gbe connection. When I get it working I'll update the thread.

daplumber

@brian-cunnie I have found that the connection needs to be pretty clean. I use a Cat8 factory made cable. The chipset also runs pretty warm, I had to ensure good thermal grease and adequate air flow.

stephenw10

Interesting. One of the big advantages of 2.5G is that it runs over cat5e. I've never had an issue using igc. Not tried an x550 though.

Steve

daplumber

@stephenw10 I know that’s the spec. I just had zero auto negotiation issues when I switched to a Cat8 cable, I went from Cat5 straight to Cat8 as they were weirdly cheaper at the time, so I skipped Cat5e, Cat6, and Cat7. I just wanted it to work. The other end is an Arris S33 Cable modem which I think is some kind of Broadcom chipset? I don’t know if different chipsets/drivers are better at auto-negotiating?

Brian Cunnie

Solution: the reason I've been having trouble getting my FreeBSD-based firewall running on an Atom C3000 SoC to negotiate 2.5Gbe with my Arris S33 modem is that the custom Intel 10G X550 on the SoC does not do NBASE-T—either you get 1Gbe or 10Gbe and nothing in between. Sigh.

On the bright side, I learned a lot about building the FreeBSD kernel and a reasonable amount about the ixgbe driver source code and about reading Intel datasheets. I must admit it was kind of fun.

I'll revert to plugging the S33 & the firewall into a 1/2.5/5/10 switch, and that'll make it work. I was hoping for a direct connect, but that's not possible.

daplumber

@brian-cunnie Well that’s a bummer. I guess they simplified the network part on the SoC version. I didn’t even realize that existed!

stephenw10

Hmm, X553 claims to support 2.5G, but not autonegotiate at that:
https://www.intel.co.uk/content/dam/www/public/us/en/documents/manuals/atom-c3000-family-programmer-reference-manual.pdf#G2.1229696

I imagine the external components would also need to.

Steve

Brian Cunnie

@stephenw10 Good catch! I didn't realize that C3000 SoC X553 allows 2.5Gbe.

I hacked the ixgbe driver to allow 2.5Gbe for my card, ran sudo ifconfig ix0 media 2500Base-T to set it to 2.5Gbe, and then attached it to the Arris S33, but the carrier never came up, and I couldn't communicate with the Arris.

stephenw10

Hmm, could be a number of things. You think your code hack was good though?

Brian Cunnie

@stephenw10 I would say I'm 80% confident my hack was correct. Here's the code diff:

--- a/sys/dev/ixgbe/if_ix.c
+++ b/sys/dev/ixgbe/if_ix.c
@@ -1286,7 +1286,8 @@ ixgbe_add_media_types(if_ctx_t ctx)
        if (layer & IXGBE_PHYSICAL_LAYER_10BASE_T)
                ifmedia_add(sc->media, IFM_ETHER | IFM_10_T, 0, NULL);

-       if (hw->mac.type == ixgbe_mac_X550) {
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a) {
                ifmedia_add(sc->media, IFM_ETHER | IFM_2500_T, 0, NULL);
                ifmedia_add(sc->media, IFM_ETHER | IFM_5000_T, 0, NULL);
        }
@@ -1438,8 +1439,6 @@ ixgbe_config_link(if_ctx_t ctx)
                         * autonegotiation preferences and the default
                         * set of speeds should exclude 2.5G and 5G.
                         */
-                       autoneg &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                           IXGBE_LINK_SPEED_5GB_FULL);
                }

                if (hw->mac.ops.setup_link)
@@ -2265,7 +2264,8 @@ ixgbe_if_media_status(if_ctx_t ctx, struct ifmediareq * ifmr)
                        ifmr->ifm_active |= IFM_10_T | IFM_FDX;
                        break;
                }
-       if (hw->mac.type == ixgbe_mac_X550)
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a)
                switch (sc->link_speed) {
                case IXGBE_LINK_SPEED_5GB_FULL:
                        ifmr->ifm_active |= IFM_5000_T | IFM_FDX;
@@ -4283,8 +4283,6 @@ ixgbe_get_default_advertise(struct ixgbe_softc *sc)
                 * are disabled by default due to reported
                 * interoperability issues with some switches.
                 */
-               link_caps &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                   IXGBE_LINK_SPEED_5GB_FULL);
        }

        speed =
--- a/sys/dev/ixgbe/ixgbe_x550.c
+++ b/sys/dev/ixgbe/ixgbe_x550.c
@@ -1945,6 +1945,8 @@ s32 ixgbe_get_link_capabilities_X550em(struct ixgbe_hw *hw,
                        /* fall through */
                default:
                        *speed = IXGBE_LINK_SPEED_10GB_FULL |
+                                IXGBE_LINK_SPEED_2_5GB_FULL |
+                                IXGBE_LINK_SPEED_5GB_FULL |
                                 IXGBE_LINK_SPEED_1GB_FULL;
                        break;
                }

stephenw10

Mmm, well that doesn't look wrong.
But I couldn't tell you if that's all that's required there.

nater

@Brian-Cunnie I have an HPE Ethernet 10Gb 2-port 562T Adapter which is an Intel X550-T2 under the hood, and it was working at 2500Base-T in pre-2.7.0 pfSense releases (such as 2.6.0) as long as I left Speed and Duplex at "Default", because if I changed it to autoselect, I could not get link when connecting to a 2500Base-T device (such as a switch or cable modem). The caveat was that the speed would always show as "unknown" in pfSense, but when set to "Default" it would negotiate to 2500Base-T and I'd get 2.5Gbps performance despite the "unknown" speed.

pfSense 2.7.0 is not like that however. "Default" or "autoselect" now cause the link to negotiate to 1000Base-T but then the link drops, and re-negotiates to 1000Base-T, drops, over and over again every few seconds for the cable modem. I do see this warning in the pfSense UI: "MUST be set to autoselect (automatically negotiate speed) unless the port this interface connects to has its speed and duplex forced." I haven't tried forcing the cable modem as well as the ethernet card in pfSense to 2500Base-T at the same time, but I'm not sure it's worth it to me, I don't have a lot of 2500Base-T devices to fail over to if things go south with my pfSense config and this HPE Ethernet 10Gb 2-port 562T Adapter. So I've simply set the WAN interface to 1000Base-T for now, and plugged my LAN connection into a gigabit switch port.

Is there a dual 2.5Gbps PCIe ethernet card that will reliably auto-negotiate at 2500Base-T in pfSense 2.7.0?

w0w

@nater
I have similar Dell card. On the latest Dell firmware and pfSense plus 23.05.1, which is very close to 2.7 I have had similar problems.
In my case I have dumb 2.5Gbit switch on one port and that caused the same issue with negotiation.
I have set the speed manually on the card/pfSense side to 2.5 and now it's working just fine.