XG-1537 SFP+ ports

Derelict

As far as I know we have not tested any modules other than standard 10G, 10G/1G, and 1G SMF and MMF so the only suggestion I can offer is to try another one or call Mosaic and ask why the one they brand as intel-specific isn't accepted by the FreeBSD ix driver.

dotdash

Had one of these in for a client build, and some stray SFP transceivers, so I decided to do some testing for science. I added the hw.ix.unsupported_sfp=1 and tested modules from a cold boot. I had four optical SFP modules and one copper. All of the optical modules worked fine: HP J4858-A MMF, an addon clone of the HP, a Cisco CLC-SX-MMD, and a Cisco MGBSX1 100SX. Box booted fine and linked up with a switch with a similar SFP. The copper module, a startech branded HP J8177C clone, caused the driver to complain about it being unsupported and not initialize the interface. This caused the box to drop to interface mismatch. Intel doesn't list any copper modules as compatible, so one would probably be wise to avoid them. It did however, work fine with all the old garbage optical SFP modules I threw at it. Disclaimer- I didn't use the ix interface, just verified it linked up on both ends.
TL,DR- Works fine with non-Intel optical SFP modules, just not copper ones.

Derelict

@dotdash At least based on your sample size.

dotdash

Sample size limited to what was on the shelf. No warranties implied or offered. May expire at any time. Mileage may vary depending on conditions. Discontinue use if you experience headaches, nausea, or bleeding about the eyes.

desrux2

@dotdash thanks for sharing.
Was it only SFP modules or also SFP+?
Do you remember what module types you tried besides T (copper) like LR, SR, etc.?

I looked through the intel driver code:
https://github.com/freebsd/freebsd/blob/master/sys/dev/ixgbe/
And looks to me like there is a number of scenarios where you can get:

hw->phy.type = ixgbe_phy_sfp_unsupported;

leading to an error status
and where

if (hw->allow_unsupported_sfp == TRUE) {
					EWARN(hw, "WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.\n");
					status = IXGBE_SUCCESS;

Is NOT implemented/applied. My guess would be when module type is something else than LR or SR and perhaps in combination with 10Gbe then the allow unsupported sfp flag does not have an effect.

dotdash

The only modules I had on hand were two Cisco SX SFP modules, an HP SX SFP, an HP clone SX SFP, and the clone HP copper GB SFP. I didn't have any SFP+ modules handy. I wanted to verify the box would boot with SFP (not SFP+) modules. From my limited research, copper modules of any kind are not supported on this card. If you want 10G optical, Netgate only charges around $50 to add a fully supported one to your order. Direct attach cables are slightly cheaper/slightly more expensive depending on length. By no means do you need to get an overpriced genuine Intel module.

desrux2

@dotdash Thanks, if you see my earlier post I have specified the module I need (what I currently have from the ISP):

“If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful”

So far no replies on this.

dotdash

Intel lists the E10GSFPLR as supported. Search for something compatible with that. In the States, you can get a used Intel one, or a new clone pretty cheap. (<$50)

desrux2

Update for anyone googling and looking at this thread. (Now updated with info from Steven)

With newer PFSense based on FreeBSD 11.2 the mentioned bugs with the Intel ix driver and unsupported sfp+ modules have finally been fixed, hurray!!

TL;DR you could have a working setup with a unsupported sfp+ module but could not power cycle system without loosing the interface. Workaround was to physically remove sfp+ module and reinsert after boot.. Now this workaround is no longer required!
(Our ISP requires us to use a CWDM module which does not exist in native Intel so we are forced to use unsupported sfp+ modules.)

How
To allow unsupported sfp+ modules to function, in the /boot/loader.conf.local add the line:

hw.ix.unsupported_sfp="1"

Shared below for reference.

~~and set this option in System Tunables~~ has no effect see next post:

It seems like is not enough only having the System Tunables option.

Important note
Please note that if you bootup with a unsupported sfp+ module before adding these options - it is possible that the 10gbe NIC enters a ‘broken’ state where network is sluggish. It may look like the optics is working but effectively no traffic is transmitted. Only way to get out of this ‘broken’ state is completely powering down the XG 1537.

Copy of /boot/loader.conf.local

/boot/loader.conf.local

kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
legal.intel_wpi.license_ack="1"
legal.intel_ipw.license_ack="1"
legal.intel_iwi.license_ack="1"
hw.ix.unsupported_sfp="1"
boot_verbose="YES"
autoboot_delay="3"

stephenw10

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.

[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

Steve

desrux2

@stephenw10 said in XG-1537 SFP+ ports:

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.
[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf
Steve

Thanks for the clarification.

moxlotus

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

desrux2

@moxlotus said in XG-1537 SFP+ ports:

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

@moxlotus did you try powering completely down after adding the line to the loader.conf.local ?
If the module has been inserted during boot without the line present in the loader.conf.local then nic will be rendered inoperable until power cycle - reboot is not enough.

moxlotus

@desrux2 yes i did power cycle. And just a note for the rest, it broke right after i upgraded from 2.5.2 to 2.6.

stephenw10

Why are you running 2.6 in an XG-1537? Do you see the same thing in 22.01?

What module are you using?

Steve