Can't connect X710 with SFP+ to SFP port via passive DAC.

gertty

I assume the switch is 1G only?

Oh! I forgot to say. Yes. It is a SFP port (not plus) 1G only. My only option when forcing a speed/duplex instead of auto-negotiate is 1Gbps FDX

What does sysctl dev.ixl.1.supported_speeds show?

> sysctl dev.ixl.1.supported_speeds
dev.ixl.1.supported_speeds: 6

Can you set it to 0x4, 10G only?

I set advertise_speed to x04, then unplugged/re-seated the DAC on the pfSense box, no change.

I played more with this setting and discovered that when setting advertise_speed to 0x4 or 0x6 produces no error but I still get no link. When I transition from either 4 or 6 to just 2 (when I unset the bit for 10G speed) I get the unqualified module message:
ixl1: Link failed because an unqualified module was detected!

stephenw10

Mmm, it's interesting. DAC cables usually don't present multiple link options like that.

[24.11-BETA][admin@7100.stevew.lan]/root: ifconfig -vvm ixl0
ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG>
	capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:e0:ed:86:a6:8c
	inet6 fe80::208:a2ff:fe0e:a591%ixl0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (10GBase-AOC <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: ixl0
	plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
	vendor: BROCADE PN: 58-1000026-01 SN: CAX116410001093 DATE: 2016-10-07

And it seems as though that cable you have claims to support numerous media types but I'm not sure it actually does.

Try to link two ixl ports in pfSense and make sure they at least link. Then try to force them to some other speed and see it that can work.

gertty

@stephenw10
Thanks for that idea.
The cable plugged into two ixl ports on the same box work does show a link status: active in ifconfig. This works as long as the advertise_speed OID I set includes the 10G bit (so a value of either 0x4 or 0x6). I still can't choose anything other than the autonegotiate "media" in the pfSense webUI or via ifconfig. It feels like this cable doesn't support it? Or maybe that's the X710, dunno. I cannot see a way to force a speed and duplex.

In any case, now I'm wondering whether this is an SFP+ only DAC. I'm very new to SFP-land and I've just come across a few things that make me think that what I really need is an 1G SFP DAC, because then my SFP+ port on pfSense would be backwards compatible with that, but I could never expect an the plain SFP port on the Ubiquiti switch to be forward compatible with the modules in an SFP+ DAC.

stephenw10

Yup, DAC cables commonly don't support multiple connection rates. I was surprised to see yours does. More so that it reports several connection types at the same speed.

gertty

@stephenw10
I think this "supports different rates" may have been some weird state. When I first plug in a DAC after booting, I get only two options in ifconfig (and of course the webUI) autoselect and 10Gbase-Twinax. I can't seem to do either with a "media" keyword in ifconfig, I get Media change is not supported.

Over the last few days, I've learned more about SFP and SFP+. Apparently DACs that support SFP+ to SFP are rare. Amazon has some cables that support SFP-to-SFP or SFP+-to-SFP+, but explicitly not both. Ubiquiti sells a DAC that they explicitly claim works with SFP and SFP+ ports, but I can't tell if that means: "SFP+ to SFP ports ONLY on their equipment". In my case it's a Ubiquity SFP port and an Intel X710 SFP+ in a Protectli box. I got hold of one of these cables and it didn't work either.

So far, the following works:

SFP toSFP on the Ubiquity switch: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable
SFP+ to SFP+ on the X710 ports on the Protectli router: Link lights with both the Ubiquiti SFP/SFP+ cable and the my original SFP+ Protectli cable. Confirmed ifconfig gives me a 10G speed.

From what I have read, it appears what I need to be able to do is force speed and duplex on both side, but I can't find a way to do that with my current pfSense install. This "media change is not supported" error is maybe the key symptom? I might try booting a Linux LiveCD to see if it's a driver issue.

gertty

Huh. Looking at the FreeBSD source for the ixl driver, I found this in freebsd-src/sys/dev/ixl /if_ixl.c:

static int
ixl_if_media_change(if_ctx_t ctx)
{
	struct ifmedia *ifm = iflib_get_media(ctx);

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(iflib_get_ifp(ctx), "Media change is not supported.\n");
	return (ENODEV);
}

I interpret this as doing:

Get the media that was just selected for this interface
If the media isn't in the ETHER family (all the familiar SFP Twinax, 10Gbase optics, etc are in this family), error out with an invalid value error
Otherwise, just don't change the media and warn the user the changing the media isn't supported.

So, this looks to me like the FreeBSD (both 14.0 and top of tree in main) ixl driver just doesn't support this. So there's no way to force a given speed and duplex setting for the IXL driver?

gertty

I see that Intel has their own driver packages for these cards, any idea what it would take to get that compiled and installed on a pfSense machine?

I'm comfortable at the command line and compiling things, but have really limited experience with FreeBSD. I assume I'd need a FreeBSD VM. How close does it need to match the exact FreeBSD version pfSense was made from? 14.1 ok? Or do I need to do 14.0? How can I ensure the new driver is loaded vs the built-in ixl driver? How will it survive reboots and upgrades?

stephenw10

Hmm, interesting. I'm pretty sure I've seen ixl NICs running at different rates but maybe you can't change media at run time.

There's a kmod port for it. https://www.freshports.org/net/intel-ixl-kmod/

But I'd expect to need to build that for the 2.7.2 kernel specifically.

gertty

seen ixl NICs running at different rates but maybe you can't change media at run time.

If anyone knows of a way to force speed and duplex to 1Gbps FDX on these ports, at boot time or otherwise, I would love to hear it. The code I posted from the driver looks to me like a slightly dressed up "NOT IMPLEMENTED".

stephenw10

Yup, it does.

gertty

TL;DR: I am now convinced the Intel X710 chipset will not allow you to force a speed or duplex. It is possible to get it to advertise only 10G, only 1G, or both during auto-negotiation, but the other side has to be able to negotiate.

Ok, I had some time last night and over lunch to grab Intel's latest (1.14.2) driver for FreeBSD, compile it on a FreeBDF 14.0 VM, and copy it over to pfSense. It didn't work either. I should have dug through the Intel code first, but when I did an hour ago, I found this:

/*
 * NOTE: Fortville does not support forcing media speeds. Instead,
 * use the set_advertise sysctl to set the speeds Fortville
 * will advertise or be allowed to operate at.
 */
int
ixl_media_change(struct ifnet * ifp)
{
	struct ixl_vsi *vsi = ifp->if_softc;
	struct ifmedia *ifm = &vsi->media;

	INIT_DEBUGOUT("ixl_media_change: begin");

	if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER)
		return (EINVAL);

	if_printf(ifp, "Use 'advertise_speed' sysctl to change advertised speeds\n");

	return (ENODEV);
}

Fortville is the Intel codename for the X710 and related adapters. So, this card just will not do a forced speed setup. AFAICT, SFP+ to SFP links even over a DAC that supports this will not work with auto-negotiation. Everyone's recommendation is to force the speed/duplex settings on both side. Since that can be done with these adapters, this link over DAC isn't possible. At least, that's my current conclusion.

Since this was only going to be a 1G link anyway and I was doing it opportunistically to free up an Ethernet port on the switch, I'm going to see if I can avoid that, maybe do SFP and SFP+ copper modules or something. I'll have to price it out.

Thanks @stephenw10 for your help here. The information about SFP, SFP+ around various forums is not very consistent, hopefully this thread will help others.

gertty

Hopefully one last update:
I booted a Linux live cd just now so I could try its driver and ethtool. I found similar behavior. With a DAC cable inserted, I was not able to change speed to anything but 10000. It gave me errors about the chosen speed (1000) not being supported. When the cable was not inserted, I could force the speed with ethtool but but when I re-inserted the cable, I got an error similar to the one I got earlier about a the module not being recognized.

I'm putting this down for now. I've rearranged things so I don't need to free up the 1Gbe port just yet, but when I do, I may buy copper SFP modules that will do 1G.

stephenw10

It would be interesting to know if Linux sees more than one speed as supported. As I said previously most DAC cables do not offer more than speed in FreeBSD/pfSense.

gertty

@stephenw10

It would be interesting to know if Linux sees more than one speed as supported.

It didn't. Just the 10G-Twinax media. I think when I was seeing multiple options earlier on pfSense it was some weird artifact from loading the web UI before plugging in a module. What I saw in ifconfig on pfSense was the with no module inserted, I got a range of fiber and copper options. With a DAC cable inserted, I got only 10G-Twinax

stephenw10

Ah OK. Yup that's what I might expect.

Using fiber modules will often give you more options for connection type.