Can't connect X710 with SFP+ to SFP port via passive DAC.

gertty

@stephenw10
I think this "supports different rates" may have been some weird state. When I first plug in a DAC after booting, I get only two options in ifconfig (and of course the webUI) autoselect and 10Gbase-Twinax. I can't seem to do either with a "media" keyword in ifconfig, I get Media change is not supported.

Over the last few days, I've learned more about SFP and SFP+. Apparently DACs that support SFP+ to SFP are rare. Amazon has some cables that support SFP-to-SFP or SFP+-to-SFP+, but explicitly not both. Ubiquiti sells a DAC that they explicitly claim works with SFP and SFP+ ports, but I can't tell if that means: "SFP+ to SFP ports ONLY on their equipment". In my case it's a Ubiquity SFP port and an Intel X710 SFP+ in a Protectli box. I got hold of one of these cables and it didn't work either.

So far, the following works:

SFP toSFP on the Ubiquity switch: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable
SFP+ to SFP+ on the X710 ports on the Protectli router: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable. Confirmed ifconfig gives me a 10G speed.

From what I have read, it appears what I need to be able to do is force speed and duplex on both side, but I can't find a way to do that with my current pfSense install. This "media change is not supported" error is maybe the key symptom? I might try booting a Linux LiveCD to see if it's a driver issue.

gertty

Huh. Looking at the FreeBSD source for the ixl driver, I found this in freebsd-src/sys/dev/ixl /if_ixl.c:

static int
ixl_if_media_change(if_ctx_t ctx)
{
	struct ifmedia *ifm = iflib_get_media(ctx);

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(iflib_get_ifp(ctx), "Media change is not supported.\n");
	return (ENODEV);
}

I interpret this as doing:

Get the media that was just selected for this interface
If the media isn't in the ETHER family (all the familiar SFP Twinax, 10Gbase optics, etc are in this family), error out with an invalid value error
Otherwise, just don't change the media and warn the user the changing the media isn't supported.

So, this looks to me like the FreeBSD (both 14.0 and top of tree in main) ixl driver just doesn't support this. So there's no way to force a given speed and duplex setting for the IXL driver?

gertty

I see that Intel has their own driver packages for these cards, any idea what it would take to get that compiled and installed on a pfSense machine?

I'm comfortable at the command line and compiling things, but have really limited experience with FreeBSD. I assume I'd need a FreeBSD VM. How close does it need to match the exact FreeBSD version pfSense was made from? 14.1 ok? Or do I need to do 14.0? How can I ensure the new driver is loaded vs the built-in ixl driver? How will it survive reboots and upgrades?

stephenw10

Hmm, interesting. I'm pretty sure I've seen ixl NICs running at different rates but maybe you can't change media at run time.

There's a kmod port for it. https://www.freshports.org/net/intel-ixl-kmod/

But I'd expect to need to build that for the 2.7.2 kernel specifically.

gertty

seen ixl NICs running at different rates but maybe you can't change media at run time.

If anyone knows of a way to force speed and duplex to 1Gbps FDX on these ports, at boot time or otherwise, I would love to hear it. The code I posted from the driver looks to me like a slightly dressed up "NOT IMPLEMENTED".

stephenw10

Yup, it does.

gertty

TL;DR: I am now convinced the Intel X710 chipset will not allow you to force a speed or duplex. It is possible to get it to advertise only 10G, only 1G, or both during auto-negotiation, but the other side has to be able to negotiate.

Ok, I had some time last night and over lunch to grab Intel's latest (1.14.2) driver for FreeBSD, compile it on a FreeBDF 14.0 VM, and copy it over to pfSense. It didn't work either. I should have dug through the Intel code first, but when I did an hour ago, I found this:

/*
 * NOTE: Fortville does not support forcing media speeds. Instead,
 * use the set_advertise sysctl to set the speeds Fortville
 * will advertise or be allowed to operate at.
 */
int
ixl_media_change(struct ifnet * ifp)
{
	struct ixl_vsi *vsi = ifp->if_softc;
	struct ifmedia *ifm = &vsi->media;

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(ifp, "Use 'advertise_speed' sysctl to change advertised speeds\n");

	return (ENODEV);
}

Fortville is the Intel codename for the X710 and related adapters. So, this card just will not do a forced speed setup. AFAICT, SFP+ to SFP links even over a DAC that supports this will not work with auto-negotiation. Everyone's recommendation is to force the speed/duplex settings on both side. Since that can be done with these adapters, this link over DAC isn't possible. At least, that's my current conclusion.

Since this was only going to be a 1G link anyway and I was doing it opportunistically to free up an Ethernet port on the switch, I'm going to see if I can avoid that, maybe do SFP and SFP+ copper modules or something. I'll have to price it out.

Thanks @stephenw10 for your help here. The information about SFP, SFP+ around various forums is not very consistent, hopefully this thread will help others.

gertty

Hopefully one last update:
I booted a Linux live cd just now so I could try its driver and ethtool. I found similar behavior. With a DAC cable inserted, I was not able to change speed to anything but 10000. It gave me errors about the chosen speed (1000) not being supported. When the cable was not inserted, I could force the speed with ethtool but but when I re-inserted the cable, I got an error similar to the one I got earlier about a the module not being recognized.

I'm putting this down for now. I've rearranged things so I don't need to free up the 1Gbe port just yet, but when I do, I may buy copper SFP modules that will do 1G.

stephenw10

It would be interesting to know if Linux sees more than one speed as supported. As I said previously most DAC cables do not offer more than speed in FreeBSD/pfSense.

gertty

@stephenw10

It would be interesting to know if Linux sees more than one speed as supported.

It didn't. Just the 10G-Twinax media. I think when I was seeing multiple options earlier on pfSense it was some weird artifact from loading the web UI before plugging in a module. What I saw in ifconfig on pfSense was the with no module inserted, I got a range of fiber and copper options. With a DAC cable inserted, I got only 10G-Twinax

stephenw10

Ah OK. Yup that's what I might expect.

Using fiber modules will often give you more options for connection type.