XG-1537 SFP+ ports

desrux2

Yes. Booting with something that the driver recognizes as unsupported can render the NIC unusable until the unit is power cycled. My advice is to use modules that the driver accepts.

Yes it kind of makes the allow unsupported sfp setting useless in practice. Not the best driver design choice. I follow your advice, no other choice*

Once you get working modules in the ports and they are in place and configured, all of this should go away.

Offending module is in place and working, just cannot reboot with it installed. But as I mentioned before it was received from ISP and I am currently waiting for a intel compatible variant.

But right, so my issue is that I cannot get link on the intel compatible 1G SFP RJ45 module.
a) Just unlucky with particular modules others will work?
b) XG-1537 does not like copper modules in general?
c) Only likes 10Gbit media and not 1Gbit in the SFP+ ports? (I was unfortunately not able to check link against something with the 1Gb SFP LR modules)
d) ?

BR Kristian

*no other choice, that is except modifying the ixgbe driver

desrux2

@desrux2 now received intel compatible module from ISP. But I unfortunatly still get “Unsupported SFP+ module type was detected.”

Excactly which module types are supported by the XG-1537 ?

Module is a 10G Base-ER ( CWDM 40km 1510nm ) is anyone else out there running something similar?

Also is there any limitations on which PCIe adapters and brands there can be inserted? Any dual 10gb port SFP+ recommendations?

Output of ifconfig -vvvvvvvvm

nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	plugged: SFP/SFP+/SFP28 10G Base-ER (LC)
	vendor: MOSAIC PN: I-FT3940D-51 SN: INB190418001 DATE: 2019-04-18
Class: 10G Base-ER
Length: (null)
Tech: (null)
Media: (null)
Speed: (null)
	module temperature: 53.70 C Voltage: 3.32 Volts
	RX: 0.11 mW (-9.24 dBm) TX: 1.62 mW (2.12 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 80 00 00 00 00 00 00 00 06 67 00 28 FF
	00 00 00 00 4D 4F 53 41 49 43 20 20 20 20 20 20
	20 20 20 20 00 00 00 00 49 2D 46 54 33 39 34 30
	44 2D 35 31 20 20 20 20 20 20 20 20 05 E6 00 C0
	60 0A 00 00 49 4E 42 31 39 30 34 31 38 30 30 31
	20 20 20 20 31 39 30 34 31 38 20 20 68 90 01 FB
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Derelict

What is the exact module in use? Sorry but we can't go by what is reported by the module itself because counterfeiting and fake results there are pretty much the norm.

I find it hard to believe that anything that is actually an Intel module is reporting as unsupported. Not impossible, but...

desrux2

@Derelict What exact specs are you missing from my description?

You seem to keep taking about original intel modules.

Let me be clear: I am using intel compatible modules or at least what should be intel compatible modules. I am not using original intel modules.

Before buying the XG-1537 i got an official Netgate response to an inquiry that intel compatible modules are required, as in not original intel modules. Is that incorrect?

I am asking about module types, so is there a distinction between SR, LR, ER, T etc.? Or what is considered the “module type”. Perhaps there could be some (in my opinion) stupid driver limitation that only SR and LR works but not ER and T (RJ45).

If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful.

All of this is simply to be able to survive reboot of XG-1537 without being physically present.
(As the connection with the unsupported module works perfectly. Problem is that interface disappears on reboot due to (in my opinion) an intentional bad intel driver design.
If I could compile a new slightly modified driver with relative ease and make sure there is no risk to it being overwritten by pfsense upgrades then I had probably done that by now.)

Derelict

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

You are saying yours does not.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

desrux2

@Derelict said in XG-1537 SFP+ ports:

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

Does work mean that it will survive reboot or just that link will work?

You are saying yours does not.

Link works. I have network connection. Traffic flows.
Problem is I get the unsupported module type message which means that on boot the intel driver will refuse to load the interface e.g. ix1 and because the interface is missing pfsense will boot into interface assignment mode requiring physical presence to cancel - remove the module from ix1 boot up normally so ix1 exsists and then insert module into ix1 again.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

I would love to know of any specific module that fits the specs and does not get the unsupported module type message. Intel org or otherwise. Preferably available in Europe.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

I already have that in place.

Derelict

As far as I know we have not tested any modules other than standard 10G, 10G/1G, and 1G SMF and MMF so the only suggestion I can offer is to try another one or call Mosaic and ask why the one they brand as intel-specific isn't accepted by the FreeBSD ix driver.

dotdash

Had one of these in for a client build, and some stray SFP transceivers, so I decided to do some testing for science. I added the hw.ix.unsupported_sfp=1 and tested modules from a cold boot. I had four optical SFP modules and one copper. All of the optical modules worked fine: HP J4858-A MMF, an addon clone of the HP, a Cisco CLC-SX-MMD, and a Cisco MGBSX1 100SX. Box booted fine and linked up with a switch with a similar SFP. The copper module, a startech branded HP J8177C clone, caused the driver to complain about it being unsupported and not initialize the interface. This caused the box to drop to interface mismatch. Intel doesn't list any copper modules as compatible, so one would probably be wise to avoid them. It did however, work fine with all the old garbage optical SFP modules I threw at it. Disclaimer- I didn't use the ix interface, just verified it linked up on both ends.
TL,DR- Works fine with non-Intel optical SFP modules, just not copper ones.

Derelict

@dotdash At least based on your sample size.

dotdash

Sample size limited to what was on the shelf. No warranties implied or offered. May expire at any time. Mileage may vary depending on conditions. Discontinue use if you experience headaches, nausea, or bleeding about the eyes.

desrux2

@dotdash thanks for sharing.
Was it only SFP modules or also SFP+?
Do you remember what module types you tried besides T (copper) like LR, SR, etc.?

I looked through the intel driver code:
https://github.com/freebsd/freebsd/blob/master/sys/dev/ixgbe/
And looks to me like there is a number of scenarios where you can get:

hw->phy.type = ixgbe_phy_sfp_unsupported;

leading to an error status
and where

if (hw->allow_unsupported_sfp == TRUE) {
					EWARN(hw, "WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.\n");
					status = IXGBE_SUCCESS;

Is NOT implemented/applied. My guess would be when module type is something else than LR or SR and perhaps in combination with 10Gbe then the allow unsupported sfp flag does not have an effect.

dotdash

The only modules I had on hand were two Cisco SX SFP modules, an HP SX SFP, an HP clone SX SFP, and the clone HP copper GB SFP. I didn't have any SFP+ modules handy. I wanted to verify the box would boot with SFP (not SFP+) modules. From my limited research, copper modules of any kind are not supported on this card. If you want 10G optical, Netgate only charges around $50 to add a fully supported one to your order. Direct attach cables are slightly cheaper/slightly more expensive depending on length. By no means do you need to get an overpriced genuine Intel module.

desrux2

@dotdash Thanks, if you see my earlier post I have specified the module I need (what I currently have from the ISP):

“If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful”

So far no replies on this.

dotdash

Intel lists the E10GSFPLR as supported. Search for something compatible with that. In the States, you can get a used Intel one, or a new clone pretty cheap. (<$50)

desrux2

Update for anyone googling and looking at this thread. (Now updated with info from Steven)

With newer PFSense based on FreeBSD 11.2 the mentioned bugs with the Intel ix driver and unsupported sfp+ modules have finally been fixed, hurray!!

TL;DR you could have a working setup with a unsupported sfp+ module but could not power cycle system without loosing the interface. Workaround was to physically remove sfp+ module and reinsert after boot.. Now this workaround is no longer required!
(Our ISP requires us to use a CWDM module which does not exist in native Intel so we are forced to use unsupported sfp+ modules.)

How
To allow unsupported sfp+ modules to function, in the /boot/loader.conf.local add the line:

hw.ix.unsupported_sfp="1"

Shared below for reference.

~~and set this option in System Tunables~~ has no effect see next post:

It seems like is not enough only having the System Tunables option.

Important note
Please note that if you bootup with a unsupported sfp+ module before adding these options - it is possible that the 10gbe NIC enters a ‘broken’ state where network is sluggish. It may look like the optics is working but effectively no traffic is transmitted. Only way to get out of this ‘broken’ state is completely powering down the XG 1537.

Copy of /boot/loader.conf.local

/boot/loader.conf.local

kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
legal.intel_wpi.license_ack="1"
legal.intel_ipw.license_ack="1"
legal.intel_iwi.license_ack="1"
hw.ix.unsupported_sfp="1"
boot_verbose="YES"
autoboot_delay="3"

stephenw10

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.

[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

Steve

desrux2

@stephenw10 said in XG-1537 SFP+ ports:

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.
[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf
Steve

Thanks for the clarification.

moxlotus

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

desrux2

@moxlotus said in XG-1537 SFP+ ports:

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

@moxlotus did you try powering completely down after adding the line to the loader.conf.local ?
If the module has been inserted during boot without the line present in the loader.conf.local then nic will be rendered inoperable until power cycle - reboot is not enough.

moxlotus

@desrux2 yes i did power cycle. And just a note for the rest, it broke right after i upgraded from 2.5.2 to 2.6.