Can't connect X710 with SFP+ to SFP port via passive DAC.

gertty

Huh. Looking at the FreeBSD source for the ixl driver, I found this in freebsd-src/sys/dev/ixl /if_ixl.c:

static int
ixl_if_media_change(if_ctx_t ctx)
{
	struct ifmedia *ifm = iflib_get_media(ctx);

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(iflib_get_ifp(ctx), "Media change is not supported.\n");
	return (ENODEV);
}

I interpret this as doing:

Get the media that was just selected for this interface
If the media isn't in the ETHER family (all the familiar SFP Twinax, 10Gbase optics, etc are in this family), error out with an invalid value error
Otherwise, just don't change the media and warn the user the changing the media isn't supported.

So, this looks to me like the FreeBSD (both 14.0 and top of tree in main) ixl driver just doesn't support this. So there's no way to force a given speed and duplex setting for the IXL driver?

gertty

I see that Intel has their own driver packages for these cards, any idea what it would take to get that compiled and installed on a pfSense machine?

I'm comfortable at the command line and compiling things, but have really limited experience with FreeBSD. I assume I'd need a FreeBSD VM. How close does it need to match the exact FreeBSD version pfSense was made from? 14.1 ok? Or do I need to do 14.0? How can I ensure the new driver is loaded vs the built-in ixl driver? How will it survive reboots and upgrades?

stephenw10

Hmm, interesting. I'm pretty sure I've seen ixl NICs running at different rates but maybe you can't change media at run time.

There's a kmod port for it. https://www.freshports.org/net/intel-ixl-kmod/

But I'd expect to need to build that for the 2.7.2 kernel specifically.

gertty

seen ixl NICs running at different rates but maybe you can't change media at run time.

If anyone knows of a way to force speed and duplex to 1Gbps FDX on these ports, at boot time or otherwise, I would love to hear it. The code I posted from the driver looks to me like a slightly dressed up "NOT IMPLEMENTED".

stephenw10

Yup, it does.

gertty

TL;DR: I am now convinced the Intel X710 chipset will not allow you to force a speed or duplex. It is possible to get it to advertise only 10G, only 1G, or both during auto-negotiation, but the other side has to be able to negotiate.

Ok, I had some time last night and over lunch to grab Intel's latest (1.14.2) driver for FreeBSD, compile it on a FreeBDF 14.0 VM, and copy it over to pfSense. It didn't work either. I should have dug through the Intel code first, but when I did an hour ago, I found this:

/*
 * NOTE: Fortville does not support forcing media speeds. Instead,
 * use the set_advertise sysctl to set the speeds Fortville
 * will advertise or be allowed to operate at.
 */
int
ixl_media_change(struct ifnet * ifp)
{
	struct ixl_vsi *vsi = ifp->if_softc;
	struct ifmedia *ifm = &vsi->media;

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(ifp, "Use 'advertise_speed' sysctl to change advertised speeds\n");

	return (ENODEV);
}

Fortville is the Intel codename for the X710 and related adapters. So, this card just will not do a forced speed setup. AFAICT, SFP+ to SFP links even over a DAC that supports this will not work with auto-negotiation. Everyone's recommendation is to force the speed/duplex settings on both side. Since that can be done with these adapters, this link over DAC isn't possible. At least, that's my current conclusion.

Since this was only going to be a 1G link anyway and I was doing it opportunistically to free up an Ethernet port on the switch, I'm going to see if I can avoid that, maybe do SFP and SFP+ copper modules or something. I'll have to price it out.

Thanks @stephenw10 for your help here. The information about SFP, SFP+ around various forums is not very consistent, hopefully this thread will help others.

gertty

Hopefully one last update:
I booted a Linux live cd just now so I could try its driver and ethtool. I found similar behavior. With a DAC cable inserted, I was not able to change speed to anything but 10000. It gave me errors about the chosen speed (1000) not being supported. When the cable was not inserted, I could force the speed with ethtool but but when I re-inserted the cable, I got an error similar to the one I got earlier about a the module not being recognized.

I'm putting this down for now. I've rearranged things so I don't need to free up the 1Gbe port just yet, but when I do, I may buy copper SFP modules that will do 1G.

stephenw10

It would be interesting to know if Linux sees more than one speed as supported. As I said previously most DAC cables do not offer more than speed in FreeBSD/pfSense.

gertty

@stephenw10

It would be interesting to know if Linux sees more than one speed as supported.

It didn't. Just the 10G-Twinax media. I think when I was seeing multiple options earlier on pfSense it was some weird artifact from loading the web UI before plugging in a module. What I saw in ifconfig on pfSense was the with no module inserted, I got a range of fiber and copper options. With a DAC cable inserted, I got only 10G-Twinax

stephenw10

Ah OK. Yup that's what I might expect.

Using fiber modules will often give you more options for connection type.