Can't connect X710 with SFP+ to SFP port via passive DAC.

gertty

Re: 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface

I'm having a similar issue with a much more simple device, a passive DAC. With pfSense CE 2.7.2, I have some on board X710 SFP+ ports. I am trying to use a passive DAC to connect them to an SFP (not SFP PLUS ) port on a Ubiquiti 16 port switch. Interrogating the X710 from the pfSense side with the DAC cable inserted:

[2.7.2-RELEASE][admin@pfSense.lan]/root: ifconfig -vvvmm ixl1
ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: TEST_3099
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 64:62:66:22:c5:66
        inet 172.30.99.254 netmask 0xffffffff broadcast 172.30.99.254
        inet6 fe80::6662:66ff:fe22:c566%ixl1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
                media 10Gbase-LR
                media 10Gbase-SR
                media 10Gbase-Twinax
                media 1000baseLX
                media 1000baseSX
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
        vendor: Protectli PN: WPP-PC192-3010 SN: HS23121400020 DATE: 2023-12-25

        SFF8472 DUMP (0xA0 0..127 range):
         03 04 21 01 00 00 00 41 04 80 d5 06 67 00 00 00
         00 00 01 00 50 72 6f 74 65 63 74 6c 69 20 20 20
         20 20 20 20 00 00 00 00 57 50 50 2d 50 43 31 39
         32 2d 33 30 31 30 20 20 31 20 20 20 01 00 00 dd
         00 00 00 00 48 53 32 33 31 32 31 34 30 30 30 32
         30 20 20 20 32 33 31 32 32 35 20 20 00 00 00 89
         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

As with the other post, I cannot use ifconfig or the webUI to select anything but "autoselect" for media. When I try, dmesg gives me:
ixl1: Media change is not supported.
The general advice seems to be: "turn off auto-negotiation on both sides" but at least on the version of pfSense I am running, that doesn't seem to be supported. I have no problem switching the UniFi switch into 1000-FDX mode.

I did find the Netgate docs about system tunables that mentions the dev.ixl.1.advertise_speeds OID. I changed that from 0x6 (10G and 1G) to 0x2 (just 1G). When I do that, dmesg tells me:
ixl1: Link failed because an unqualified module was detected!

After reading the ixl driver man page, I see that this driver does not support a way to allow unsupported modules via sysctl, but I'm not entirely sure that is the problem or how to proceed. On this page Intel says this about DAC compatibility:

Which direct attach cables are compatible?
Intel validated cables are listed in the Intel Product Compatibility Tool.
SFP+ passive or active limiting direct attach copper cables that comply with the SFF-8431 v4.1 and SFF-8472 v10.4 specifications.
SFF-8472 identifier must have value 03h (SFP or SFP Plus). You can verify the value with the cable manufacturer.
Maximum cable length for passive cables is 7 meters.
Support for active cables requires Intel Network Connections software version 22.01 or later.

These seems to indicate that non-Intel DACs should work. Is it possible to read the SFF-8472 identifier ( to check for 03h) from the pfSense command line?

stephenw10

@gertty said in Can't connect X710 with SFP+ to SFP port via passive DAC.:

I did find the Netgate docs about system tunables that mentions the dev.ixl.1.advertise_speeds OID. I changed that from 0x6 (10G and 1G) to 0x2 (just 1G). When I do that, dmesg tells me:
ixl1: Link failed because an unqualified module was detected!

Can you set it to 0x4, 10G only?

What does sysctl dev.ixl.1.supported_speeds show?

I assume the switch is 1G only?

gertty

@stephenw10

I assume the switch is 1G only?

Oh! I forgot to say. Yes. It is a SFP port (not plus) 1G only. My only option when forcing a speed/duplex instead of auto-negotiate is 1Gbps FDX

What does sysctl dev.ixl.1.supported_speeds show?

> sysctl dev.ixl.1.supported_speeds
dev.ixl.1.supported_speeds: 6

Can you set it to 0x4, 10G only?

I set advertise_speed to x04, then unplugged/re-seated the DAC on the pfSense box, no change.

I played more with this setting and discovered that when setting advertise_speed to 0x4 or 0x6 produces no error but I still get no link. When I transition from either 4 or 6 to just 2 (when I unset the bit for 10G speed) I get the unqualified module message:
ixl1: Link failed because an unqualified module was detected!

stephenw10

Mmm, it's interesting. DAC cables usually don't present multiple link options like that.

[24.11-BETA][admin@7100.stevew.lan]/root: ifconfig -vvm ixl0
ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG>
	capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:e0:ed:86:a6:8c
	inet6 fe80::208:a2ff:fe0e:a591%ixl0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (10GBase-AOC <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: ixl0
	plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
	vendor: BROCADE PN: 58-1000026-01 SN: CAX116410001093 DATE: 2016-10-07

And it seems as though that cable you have claims to support numerous media types but I'm not sure it actually does.

Try to link two ixl ports in pfSense and make sure they at least link. Then try to force them to some other speed and see it that can work.

gertty

@stephenw10
Thanks for that idea.
The cable plugged into two ixl ports on the same box work does show a link status: active in ifconfig. This works as long as the advertise_speed OID I set includes the 10G bit (so a value of either 0x4 or 0x6). I still can't choose anything other than the autonegotiate "media" in the pfSense webUI or via ifconfig. It feels like this cable doesn't support it? Or maybe that's the X710, dunno. I cannot see a way to force a speed and duplex.

In any case, now I'm wondering whether this is an SFP+ only DAC. I'm very new to SFP-land and I've just come across a few things that make me think that what I really need is an 1G SFP DAC, because then my SFP+ port on pfSense would be backwards compatible with that, but I could never expect an the plain SFP port on the Ubiquiti switch to be forward compatible with the modules in an SFP+ DAC.

stephenw10

Yup, DAC cables commonly don't support multiple connection rates. I was surprised to see yours does. More so that it reports several connection types at the same speed.

gertty

@stephenw10
I think this "supports different rates" may have been some weird state. When I first plug in a DAC after booting, I get only two options in ifconfig (and of course the webUI) autoselect and 10Gbase-Twinax. I can't seem to do either with a "media" keyword in ifconfig, I get Media change is not supported.

Over the last few days, I've learned more about SFP and SFP+. Apparently DACs that support SFP+ to SFP are rare. Amazon has some cables that support SFP-to-SFP or SFP+-to-SFP+, but explicitly not both. Ubiquiti sells a DAC that they explicitly claim works with SFP and SFP+ ports, but I can't tell if that means: "SFP+ to SFP ports ONLY on their equipment". In my case it's a Ubiquity SFP port and an Intel X710 SFP+ in a Protectli box. I got hold of one of these cables and it didn't work either.

So far, the following works:

SFP toSFP on the Ubiquity switch: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable
SFP+ to SFP+ on the X710 ports on the Protectli router: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable. Confirmed ifconfig gives me a 10G speed.

From what I have read, it appears what I need to be able to do is force speed and duplex on both side, but I can't find a way to do that with my current pfSense install. This "media change is not supported" error is maybe the key symptom? I might try booting a Linux LiveCD to see if it's a driver issue.

gertty

Huh. Looking at the FreeBSD source for the ixl driver, I found this in freebsd-src/sys/dev/ixl /if_ixl.c:

static int
ixl_if_media_change(if_ctx_t ctx)
{
	struct ifmedia *ifm = iflib_get_media(ctx);

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(iflib_get_ifp(ctx), "Media change is not supported.\n");
	return (ENODEV);
}

I interpret this as doing:

Get the media that was just selected for this interface
If the media isn't in the ETHER family (all the familiar SFP Twinax, 10Gbase optics, etc are in this family), error out with an invalid value error
Otherwise, just don't change the media and warn the user the changing the media isn't supported.

So, this looks to me like the FreeBSD (both 14.0 and top of tree in main) ixl driver just doesn't support this. So there's no way to force a given speed and duplex setting for the IXL driver?

gertty

I see that Intel has their own driver packages for these cards, any idea what it would take to get that compiled and installed on a pfSense machine?

I'm comfortable at the command line and compiling things, but have really limited experience with FreeBSD. I assume I'd need a FreeBSD VM. How close does it need to match the exact FreeBSD version pfSense was made from? 14.1 ok? Or do I need to do 14.0? How can I ensure the new driver is loaded vs the built-in ixl driver? How will it survive reboots and upgrades?

stephenw10

Hmm, interesting. I'm pretty sure I've seen ixl NICs running at different rates but maybe you can't change media at run time.

There's a kmod port for it. https://www.freshports.org/net/intel-ixl-kmod/

But I'd expect to need to build that for the 2.7.2 kernel specifically.

gertty

seen ixl NICs running at different rates but maybe you can't change media at run time.

If anyone knows of a way to force speed and duplex to 1Gbps FDX on these ports, at boot time or otherwise, I would love to hear it. The code I posted from the driver looks to me like a slightly dressed up "NOT IMPLEMENTED".

stephenw10

Yup, it does.

gertty

TL;DR: I am now convinced the Intel X710 chipset will not allow you to force a speed or duplex. It is possible to get it to advertise only 10G, only 1G, or both during auto-negotiation, but the other side has to be able to negotiate.

Ok, I had some time last night and over lunch to grab Intel's latest (1.14.2) driver for FreeBSD, compile it on a FreeBDF 14.0 VM, and copy it over to pfSense. It didn't work either. I should have dug through the Intel code first, but when I did an hour ago, I found this:

/*
 * NOTE: Fortville does not support forcing media speeds. Instead,
 * use the set_advertise sysctl to set the speeds Fortville
 * will advertise or be allowed to operate at.
 */
int
ixl_media_change(struct ifnet * ifp)
{
	struct ixl_vsi *vsi = ifp->if_softc;
	struct ifmedia *ifm = &vsi->media;

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(ifp, "Use 'advertise_speed' sysctl to change advertised speeds\n");

	return (ENODEV);
}

Fortville is the Intel codename for the X710 and related adapters. So, this card just will not do a forced speed setup. AFAICT, SFP+ to SFP links even over a DAC that supports this will not work with auto-negotiation. Everyone's recommendation is to force the speed/duplex settings on both side. Since that can be done with these adapters, this link over DAC isn't possible. At least, that's my current conclusion.

Since this was only going to be a 1G link anyway and I was doing it opportunistically to free up an Ethernet port on the switch, I'm going to see if I can avoid that, maybe do SFP and SFP+ copper modules or something. I'll have to price it out.

Thanks @stephenw10 for your help here. The information about SFP, SFP+ around various forums is not very consistent, hopefully this thread will help others.

gertty

Hopefully one last update:
I booted a Linux live cd just now so I could try its driver and ethtool. I found similar behavior. With a DAC cable inserted, I was not able to change speed to anything but 10000. It gave me errors about the chosen speed (1000) not being supported. When the cable was not inserted, I could force the speed with ethtool but but when I re-inserted the cable, I got an error similar to the one I got earlier about a the module not being recognized.

I'm putting this down for now. I've rearranged things so I don't need to free up the 1Gbe port just yet, but when I do, I may buy copper SFP modules that will do 1G.

stephenw10

It would be interesting to know if Linux sees more than one speed as supported. As I said previously most DAC cables do not offer more than speed in FreeBSD/pfSense.

gertty

@stephenw10

It would be interesting to know if Linux sees more than one speed as supported.

It didn't. Just the 10G-Twinax media. I think when I was seeing multiple options earlier on pfSense it was some weird artifact from loading the web UI before plugging in a module. What I saw in ifconfig on pfSense was the with no module inserted, I got a range of fiber and copper options. With a DAC cable inserted, I got only 10G-Twinax

stephenw10

Ah OK. Yup that's what I might expect.

Using fiber modules will often give you more options for connection type.