NBASE-T Support for Intel X550

Brian Cunnie

@stephenw10 I would say I'm 80% confident my hack was correct. Here's the code diff:

--- a/sys/dev/ixgbe/if_ix.c
+++ b/sys/dev/ixgbe/if_ix.c
@@ -1286,7 +1286,8 @@ ixgbe_add_media_types(if_ctx_t ctx)
        if (layer & IXGBE_PHYSICAL_LAYER_10BASE_T)
                ifmedia_add(sc->media, IFM_ETHER | IFM_10_T, 0, NULL);

-       if (hw->mac.type == ixgbe_mac_X550) {
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a) {
                ifmedia_add(sc->media, IFM_ETHER | IFM_2500_T, 0, NULL);
                ifmedia_add(sc->media, IFM_ETHER | IFM_5000_T, 0, NULL);
        }
@@ -1438,8 +1439,6 @@ ixgbe_config_link(if_ctx_t ctx)
                         * autonegotiation preferences and the default
                         * set of speeds should exclude 2.5G and 5G.
                         */
-                       autoneg &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                           IXGBE_LINK_SPEED_5GB_FULL);
                }

                if (hw->mac.ops.setup_link)
@@ -2265,7 +2264,8 @@ ixgbe_if_media_status(if_ctx_t ctx, struct ifmediareq * ifmr)
                        ifmr->ifm_active |= IFM_10_T | IFM_FDX;
                        break;
                }
-       if (hw->mac.type == ixgbe_mac_X550)
+       if (hw->mac.type == ixgbe_mac_X550 ||
+               hw->mac.type == ixgbe_mac_X550EM_a)
                switch (sc->link_speed) {
                case IXGBE_LINK_SPEED_5GB_FULL:
                        ifmr->ifm_active |= IFM_5000_T | IFM_FDX;
@@ -4283,8 +4283,6 @@ ixgbe_get_default_advertise(struct ixgbe_softc *sc)
                 * are disabled by default due to reported
                 * interoperability issues with some switches.
                 */
-               link_caps &= ~(IXGBE_LINK_SPEED_2_5GB_FULL |
-                   IXGBE_LINK_SPEED_5GB_FULL);
        }

        speed =
--- a/sys/dev/ixgbe/ixgbe_x550.c
+++ b/sys/dev/ixgbe/ixgbe_x550.c
@@ -1945,6 +1945,8 @@ s32 ixgbe_get_link_capabilities_X550em(struct ixgbe_hw *hw,
                        /* fall through */
                default:
                        *speed = IXGBE_LINK_SPEED_10GB_FULL |
+                                IXGBE_LINK_SPEED_2_5GB_FULL |
+                                IXGBE_LINK_SPEED_5GB_FULL |
                                 IXGBE_LINK_SPEED_1GB_FULL;
                        break;
                }

stephenw10

Mmm, well that doesn't look wrong.
But I couldn't tell you if that's all that's required there.

nater

@Brian-Cunnie I have an HPE Ethernet 10Gb 2-port 562T Adapter which is an Intel X550-T2 under the hood, and it was working at 2500Base-T in pre-2.7.0 pfSense releases (such as 2.6.0) as long as I left Speed and Duplex at "Default", because if I changed it to autoselect, I could not get link when connecting to a 2500Base-T device (such as a switch or cable modem). The caveat was that the speed would always show as "unknown" in pfSense, but when set to "Default" it would negotiate to 2500Base-T and I'd get 2.5Gbps performance despite the "unknown" speed.

pfSense 2.7.0 is not like that however. "Default" or "autoselect" now cause the link to negotiate to 1000Base-T but then the link drops, and re-negotiates to 1000Base-T, drops, over and over again every few seconds for the cable modem. I do see this warning in the pfSense UI: "MUST be set to autoselect (automatically negotiate speed) unless the port this interface connects to has its speed and duplex forced." I haven't tried forcing the cable modem as well as the ethernet card in pfSense to 2500Base-T at the same time, but I'm not sure it's worth it to me, I don't have a lot of 2500Base-T devices to fail over to if things go south with my pfSense config and this HPE Ethernet 10Gb 2-port 562T Adapter. So I've simply set the WAN interface to 1000Base-T for now, and plugged my LAN connection into a gigabit switch port.

Is there a dual 2.5Gbps PCIe ethernet card that will reliably auto-negotiate at 2500Base-T in pfSense 2.7.0?

w0w

@nater
I have similar Dell card. On the latest Dell firmware and pfSense plus 23.05.1, which is very close to 2.7 I have had similar problems.
In my case I have dumb 2.5Gbit switch on one port and that caused the same issue with negotiation.
I have set the speed manually on the card/pfSense side to 2.5 and now it's working just fine.

Brian Cunnie

Is there a dual 2.5Gbps PCIe ethernet card that will reliably auto-negotiate at 2500Base-T in pfSense 2.7.0?

I don't know.

You're lucky: you must have the flavor of the X550-T2 card that includes the X550-BT2 chipset, which supports NBASE-T. The other flavor includes the X550-AT2 chipset, no NBASE-T.

w0w

@Brian-Cunnie
From link you provided for x550-t2
333369-009 43
Intel Ethernet Controller X550 Datasheet
Introduction
1.5 Overview: New Capabilities Beyond the X540
1.5.1 NBASE-T Support
Support for 2.5GBASE-T and 5GBASE-T is added to the X550

Also, there is revision description that states that support for NBASE-T was added in revision 2.0, so any card that bough from 2016 rev 2.0 supports NBASE.

Am I reading something wrong?

Brian Cunnie

@w0w said:

Am I reading something wrong?

No, you're not reading anything wrong. In fact, it's in line with what I said: there are two flavors of the X550-T2 controller: one that does NBASE-T, and another that doesn't.

It can be confusing: there's the X550-T2 controller (card), and there are the chipsets, X550-AT2 and X550-BT2. One or the other of the chipsets can be incorporated in the card. The X550-AT2 chipset doesn't do NBASE-T, and the X550-BT2 does.

w0w

@Brian-Cunnie
https://www.intel.com/content/www/us/en/products/sku/84329/intel-ethernet-controller-x550at2/specifications.html

Data Rate Per Port
10/5/2.5/1GbE (NBASE-T in Linux Only)

No, I think, AT2 and BT2 is just different “form-factor” for the same chipset
Compare
source
As you can see, all those iterations support NBASE-T in linux only (FREEBSD too, not by intel anyway)

Brian Cunnie

@w0w said:

No, I think, AT2 and BT2 is just different “form-factor” for the same chipset
Compare
source
As you can see, all those iterations support NBASE-T in linux only (FREEBSD too, not by intel anyway)

You're absolutely right, and I was completely wrong.

I had confused the X557-AT2 (no NBASE-T) chipset with the X550-AT2 chipset.

Thanks for the correction!

nater

@w0w Setting the speed manually to 2500Base-T with pfSense 2.7.0 and my HPE Ethernet 10Gb 2-port 562T Adapter definitely does not work for establishing a stable link to my Motorola cable modem. 1000Base-T is totally stable. I assume this is a FreeBSD bug, so I should be filing a FreeBSD bug report? Anyone know the best way to do that?

I did spend a little time looking into alternatives, and they all seem worse. Realtek can't be trusted for 24/7 heavy network duties, and while I thought maybe a different Intel 2.5Gbps solution might work like this:

https://www.startech.com/en-us/networking-io/pr42gi-network-card

Word is that the I225-V (and probably the I226) are dumpster fires:

https://www.reddit.com/r/hardware/comments/wi4soc/the_new_intel_i226v_i226lm_i226it_nics/

So then I thought maybe something Marvell AQtion AQC113C based could be good:

https://www.qnap.com/en/product/qxg-10g2tb/specs/hardware

Not for FreeBSD right now:

https://forums.freebsd.org/threads/support-for-marvell-r-aqc113cs-b1-c-10gbps-lan.89179/

w0w

@nater said in NBASE-T Support for Intel X550:

Realtek can't be trusted for 24/7 heavy network duties

Same as those 225/226 cards that are not stable on me. I am not sure, maybe some intel driver issue. Realtek works quite well when you are not using some netmap mode like suricata or snort on it. You just need to install the latest drivers, manually, do it before card insertion and you will be fine.
Realtek kmod install

By the way, what exactly happens when you set speed manually on pfSense to 2500?
What is firmware version is on the card now?

stephenw10

I've never seen any of the reported issues in the i225/226 NICs. Even on the one box I have that has the supposedly terrible rev1/2 i225-V NICs.

The only quirk I've seen is the the igc NICs can only link use auto-negotiate. If you set a specific speed yopu are in fact only limiting he advertised speeds it tries to negotiate. That means it will have issues linking to something that is truly fixed speed and not negotiating.

Steve

w0w

@stephenw10 said in NBASE-T Support for Intel X550:

've never seen any of the reported issues in the i225/226 NICs. Even on the one box I have that has the supposedly terrible rev1/2 i225-V NICs.

I understand that this question is absolutely off-topic, but did you buy these cards separately or are we talking about built-in cards in Netgate devices?
For pci-e cards, there are different assembly options, single-port ones use the pcie bus directly, two or more ports use pcie splitters, usually asmedia. At the moment I have 3 variants of 225 cards purchased separately and 1 version of 226. They all work, yes. Perhaps I do not rule out that there will not even be any difference with realtek in specific conditions. Not mine, unfortunately. At least for now, I will refrain from recommending these cards for purchase.

But for @nater, it is very possible that 226 or 225 will be quite suitable.

I'm not sure that everything is lost with HPE Ethernet 10Gb 2-port 562T Adapter.

stephenw10

Almost all the testing I've done is with the on-board NICs on our appliances but I do also have a single port i225 NIC on a card I tested in the 7100 and that was fine too.

w0w

I have played a bit with NBASE-T on my Intel original X550-T2 card.
EDIT: This card is not on the latest firmware, but some 1.93 stated. Will re-test this with latest later.
Port connected to 2.5Gbit switch. When set pfSense autoselect on that port it can SOMETIMES just crash the firewall and on the next boot it stops saying "Configuring interface xxx "
To boot I need to disconnect the cable.
When set to default it stops for a minute or two on boot and then comes 1000Base-T and not getting IP via DHCP.
If I set manually to 2500Base-T everything is working fine.
Booted to Kali-linux some latest available today.
There is no more 2.5G speed available and card works only on 10GBase-T, any other link – no IP or no media connected. Same for the latest Ubuntu version.
According to this Intel forum link to get the 2.5G it is needed to run 18.04 Ubuntu, so I downloaded it and voilà, no need to set up anything, both ports are working on desired speed. 10G and 2.5 are working “out of the box”.

EDIT2: Similar behavior on the latest 3.6 NVM.
Well, it looks like we have at least possibility to get this all working, just a broken driver need to be fixed to get everything to work.

Should it be reported? I don't know…

w0w

Crasdumps crash_x550.txt

Also reading FreeBSD forum provides more information about what is wrong with that card
For example this answer on Intel forums

stephenw10

One of those look likes this: https://redmine.pfsense.org/issues/14431
Probably a symptom of the link failure rather than a cause of it.
Yes, likely a driver update required to get auto-select working.

w0w

@stephenw10 said in NBASE-T Support for Intel X550:

Yes, likely a driver update required to get auto-select working.

It not just auto-select not working, but I can not get any IP by DHCP on the second, ix0 port, even if it up and running, if I do packet capture I see at least that some packets are moving, but don't see any DHCP offerings. That's strange. Will do further testings soon, may be I've missed something.

w0w

When mode is autoselect and speed is 1G, then pcap shows only my PPPoE PADI connection retries and Errors IN constantly grows UP, whatever it means and no packets going in. Does anyone else have the same behavior?

w0w

Booted into 2.6, using recovery mode. Got both ports running, one port shows "unknown" speed, but got IP address, so it's like @nater said just don't show the correct NBASE-T speed. Second shows correct speed and also works fine. Should I create redmine ticket or it's already known problem?