NBASE-T Support for Intel X550

[stephenw10]

It's only in 13-stable right now so at the very least it would need either to brought into 12 stable or pfSense moved to 13 stable (or newer).

[Brian Cunnie]

@daplumber I don't have any idea when it would show up in a pfsense release.

Also, I've been struggling to get this to work on my non-pfsense straight-FreeBSD 13 firewall, and in spite of several attempts the Intel X550 negotiates a 1Gbe not 2.5Gbe connection. When I get it working I'll update the thread.

[daplumber]

@brian-cunnie I have found that the connection needs to be pretty clean. I use a Cat8 factory made cable. The chipset also runs pretty warm, I had to ensure good thermal grease and adequate air flow.

[stephenw10]

Interesting. One of the big advantages of 2.5G is that it runs over cat5e. I've never had an issue using igc. Not tried an x550 though.

Steve

[daplumber]

@stephenw10 I know that’s the spec. I just had zero auto negotiation issues when I switched to a Cat8 cable, I went from Cat5 straight to Cat8 as they were weirdly cheaper at the time, so I skipped Cat5e, Cat6, and Cat7. I just wanted it to work. The other end is an Arris S33 Cable modem which I think is some kind of Broadcom chipset? I don’t know if different chipsets/drivers are better at auto-negotiating?

[Brian Cunnie]

Solution: the reason I've been having trouble getting my FreeBSD-based firewall running on an Atom C3000 SoC to negotiate 2.5Gbe with my Arris S33 modem is that the custom Intel 10G X550 on the SoC does not do NBASE-T—either you get 1Gbe or 10Gbe and nothing in between. Sigh.

On the bright side, I learned a lot about building the FreeBSD kernel and a reasonable amount about the ixgbe driver source code and about reading Intel datasheets. I must admit it was kind of fun.

I'll revert to plugging the S33 & the firewall into a 1/2.5/5/10 switch, and that'll make it work. I was hoping for a direct connect, but that's not possible.

[daplumber]

@brian-cunnie Well that’s a bummer. I guess they simplified the network part on the SoC version. I didn’t even realize that existed!

[stephenw10]

Hmm, X553 claims to support 2.5G, but not autonegotiate at that:
https://www.intel.co.uk/content/dam/www/public/us/en/documents/manuals/atom-c3000-family-programmer-reference-manual.pdf#G2.1229696

I imagine the external components would also need to.

Steve

[Brian Cunnie]

@stephenw10 Good catch! I didn't realize that C3000 SoC X553 allows 2.5Gbe.

I hacked the ixgbe driver to allow 2.5Gbe for my card, ran sudo ifconfig ix0 media 2500Base-T to set it to 2.5Gbe, and then attached it to the Arris S33, but the carrier never came up, and I couldn't communicate with the Arris.

[stephenw10]

Hmm, could be a number of things. You think your code hack was good though?

[Brian Cunnie]

@stephenw10 I would say I'm 80% confident my hack was correct. Here's the code diff:

--- a/sys/dev/ixgbe/if_ix.c
+++ b/sys/dev/ixgbe/if_ix.c
@@ -1286,7 +1286,8 @@ ixgbe_add_media_types(if_ctx_t ctx)
        if (layer & IXGBE_PHYSICAL_LAYER_10BASE_T)
                ifmedia_add(sc->media, IFM_ETHER | IFM_10_T, 0, NULL);

-       if (hw->mac.type == ixgbe_mac_X550) {
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a) {
                ifmedia_add(sc->media, IFM_ETHER | IFM_2500_T, 0, NULL);
                ifmedia_add(sc->media, IFM_ETHER | IFM_5000_T, 0, NULL);
        }
@@ -1438,8 +1439,6 @@ ixgbe_config_link(if_ctx_t ctx)
                         * autonegotiation preferences and the default
                         * set of speeds should exclude 2.5G and 5G.
                         */
-                       autoneg &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                           IXGBE_LINK_SPEED_5GB_FULL);
                }

                if (hw->mac.ops.setup_link)
@@ -2265,7 +2264,8 @@ ixgbe_if_media_status(if_ctx_t ctx, struct ifmediareq * ifmr)
                        ifmr->ifm_active |= IFM_10_T | IFM_FDX;
                        break;
                }
-       if (hw->mac.type == ixgbe_mac_X550)
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a)
                switch (sc->link_speed) {
                case IXGBE_LINK_SPEED_5GB_FULL:
                        ifmr->ifm_active |= IFM_5000_T | IFM_FDX;
@@ -4283,8 +4283,6 @@ ixgbe_get_default_advertise(struct ixgbe_softc *sc)
                 * are disabled by default due to reported
                 * interoperability issues with some switches.
                 */
-               link_caps &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                   IXGBE_LINK_SPEED_5GB_FULL);
        }

        speed =
--- a/sys/dev/ixgbe/ixgbe_x550.c
+++ b/sys/dev/ixgbe/ixgbe_x550.c
@@ -1945,6 +1945,8 @@ s32 ixgbe_get_link_capabilities_X550em(struct ixgbe_hw *hw,
                        /* fall through */
                default:
                        *speed = IXGBE_LINK_SPEED_10GB_FULL |
+                                IXGBE_LINK_SPEED_2_5GB_FULL |
+                                IXGBE_LINK_SPEED_5GB_FULL |
                                 IXGBE_LINK_SPEED_1GB_FULL;
                        break;
                }

[stephenw10]

Mmm, well that doesn't look wrong.
But I couldn't tell you if that's all that's required there.

[nater]

@Brian-Cunnie I have an HPE Ethernet 10Gb 2-port 562T Adapter which is an Intel X550-T2 under the hood, and it was working at 2500Base-T in pre-2.7.0 pfSense releases (such as 2.6.0) as long as I left Speed and Duplex at "Default", because if I changed it to autoselect, I could not get link when connecting to a 2500Base-T device (such as a switch or cable modem). The caveat was that the speed would always show as "unknown" in pfSense, but when set to "Default" it would negotiate to 2500Base-T and I'd get 2.5Gbps performance despite the "unknown" speed.

pfSense 2.7.0 is not like that however. "Default" or "autoselect" now cause the link to negotiate to 1000Base-T but then the link drops, and re-negotiates to 1000Base-T, drops, over and over again every few seconds for the cable modem. I do see this warning in the pfSense UI: "MUST be set to autoselect (automatically negotiate speed) unless the port this interface connects to has its speed and duplex forced." I haven't tried forcing the cable modem as well as the ethernet card in pfSense to 2500Base-T at the same time, but I'm not sure it's worth it to me, I don't have a lot of 2500Base-T devices to fail over to if things go south with my pfSense config and this HPE Ethernet 10Gb 2-port 562T Adapter. So I've simply set the WAN interface to 1000Base-T for now, and plugged my LAN connection into a gigabit switch port.

Is there a dual 2.5Gbps PCIe ethernet card that will reliably auto-negotiate at 2500Base-T in pfSense 2.7.0?

[w0w]

@nater
I have similar Dell card. On the latest Dell firmware and pfSense plus 23.05.1, which is very close to 2.7 I have had similar problems.
In my case I have dumb 2.5Gbit switch on one port and that caused the same issue with negotiation.
I have set the speed manually on the card/pfSense side to 2.5 and now it's working just fine.

[Brian Cunnie]

Is there a dual 2.5Gbps PCIe ethernet card that will reliably auto-negotiate at 2500Base-T in pfSense 2.7.0?

I don't know.

You're lucky: you must have the flavor of the X550-T2 card that includes the X550-BT2 chipset, which supports NBASE-T. The other flavor includes the X550-AT2 chipset, no NBASE-T.

[w0w]

@Brian-Cunnie
From link you provided for x550-t2
333369-009 43
Intel Ethernet Controller X550 Datasheet
Introduction
1.5 Overview: New Capabilities Beyond the X540
1.5.1 NBASE-T Support
Support for 2.5GBASE-T and 5GBASE-T is added to the X550

Also, there is revision description that states that support for NBASE-T was added in revision 2.0, so any card that bough from 2016 rev 2.0 supports NBASE.

Am I reading something wrong?

[Brian Cunnie]

@w0w said:

Am I reading something wrong?

No, you're not reading anything wrong. In fact, it's in line with what I said: there are two flavors of the X550-T2 controller: one that does NBASE-T, and another that doesn't.

It can be confusing: there's the X550-T2 controller (card), and there are the chipsets, X550-AT2 and X550-BT2. One or the other of the chipsets can be incorporated in the card. The X550-AT2 chipset doesn't do NBASE-T, and the X550-BT2 does.

[w0w]

@Brian-Cunnie
https://www.intel.com/content/www/us/en/products/sku/84329/intel-ethernet-controller-x550at2/specifications.html

Data Rate Per Port
10/5/2.5/1GbE (NBASE-T in Linux Only)

No, I think, AT2 and BT2 is just different “form-factor” for the same chipset
Compare
source
As you can see, all those iterations support NBASE-T in linux only (FREEBSD too, not by intel anyway)

[Brian Cunnie]

@w0w said:

No, I think, AT2 and BT2 is just different “form-factor” for the same chipset
Compare
source
As you can see, all those iterations support NBASE-T in linux only (FREEBSD too, not by intel anyway)

You're absolutely right, and I was completely wrong.

I had confused the X557-AT2 (no NBASE-T) chipset with the X550-AT2 chipset.

Thanks for the correction!

[nater]

@w0w Setting the speed manually to 2500Base-T with pfSense 2.7.0 and my HPE Ethernet 10Gb 2-port 562T Adapter definitely does not work for establishing a stable link to my Motorola cable modem. 1000Base-T is totally stable. I assume this is a FreeBSD bug, so I should be filing a FreeBSD bug report? Anyone know the best way to do that?

I did spend a little time looking into alternatives, and they all seem worse. Realtek can't be trusted for 24/7 heavy network duties, and while I thought maybe a different Intel 2.5Gbps solution might work like this:

https://www.startech.com/en-us/networking-io/pr42gi-network-card

Word is that the I225-V (and probably the I226) are dumpster fires:

https://www.reddit.com/r/hardware/comments/wi4soc/the_new_intel_i226v_i226lm_i226it_nics/

So then I thought maybe something Marvell AQtion AQC113C based could be good:

https://www.qnap.com/en/product/qxg-10g2tb/specs/hardware

Not for FreeBSD right now:

https://forums.freebsd.org/threads/support-for-marvell-r-aqc113cs-b1-c-10gbps-lan.89179/