XG-1537 SFP+ ports

beatvjiking

Hi, I was just wondering - are the SFP+ ports in the XG-1537 dual-personality? As in, can they take SFPs and SFP+s or SFP+ only?

beatvjiking

It appears from testing that they are not. They are 10Gb only.

desrux2

@beatvjiking said in XG-1537 SFP+ ports:

It appears from testing that they are not. They are 10Gb only.

I have a XG-1537 expecting to be able to use both SFP and SFP+ modules. I have the same issue.
I have not been able to get SFP modules to work (tried both with intel and Cisco compatible) - is this by design or a sw bug / hw bug?

Any official status on this issue?

stephenw10

What modules exactly have you tested? I'm not aware of any particular restriction on the XG-1537.

Steve

desrux2

@stephenw10 said in XG-1537 SFP+ ports:

What modules exactly have you tested? I'm not aware of any particular restriction on the XG-1537.

Steve

Hi Steve

I have not been able to get any of these SPF modules to work:

I-FT3420D
I-FT3401

C-FT3401G
C-FT320D
From Mosaic Optics.

BR Kristian

Derelict

When you have one of those Intel-compatible single-mode modules inserted what is the output of:

ifconfig -vm ixX

Should go without saying but ixX there is ix0, ix1, etc.

beatvjiking

This is an old thread, and the XG-1537s I got were from the initial production run. I seem to recall early units having a restriction to 10Gb only, while later ones were dual-personality. I don't have personal experience with later units, so I don't know how accurate that is. Might be worth looking at though.

stephenw10

That applied to the XG-7100 but not, as far as I know, to the XG-1537.

It would be interesting to see that ifconfig output.

Steve

desrux2

I have been busy (onsite) testing SFP module combinations the last 4 hours. Currently output copy pasted in a large excel sheet - need to clean it up, make sense of it and then share it here.

Until then here is some initial ifconfig output:

I-FT3401 (SFP - RJ45)
ix0: Unsupported SFP+ module type was detected.

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:8d:00:3c
	hwaddr ac:1f:6b:8d:00:3c
	inet6 xxxxx0 prefixlen 64 scopeid 0x1
	inet6 xxxxxx:: prefixlen 50
	inet 172.16.0.1 netmask 0xffff0000 broadcast 172.16.255.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	plugged: SFP/SFP+/SFP28 1000BASE-T (RJ45)
	vendor: MOSAIC PN: I-FT3401 SN: INB180201001 DATE: 2018-02-02

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 22 00 00 00 08 00 00 00 00 01 00 00 00 00
	00 00 64 00 4D 4F 53 41 49 43 20 20 20 20 20 20
	20 20 20 20 00 00 00 00 49 2D 46 54 33 34 30 31
	20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 EA
	00 1A 00 00 49 4E 42 31 38 30 32 30 31 30 30 31
	20 20 20 20 31 38 30 32 30 32 00 00 00 00 00 5D
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20

C-FT3401GDN (SFP RJ45)
ix0: Unsupported SFP+ module type was detected.

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:8d:00:3c
	hwaddr ac:1f:6b:8d:00:3c
	inet6 xxxxxx prefixlen 64 scopeid 0x1
	inet6 xxxxxxx prefixlen 50
	inet 172.16.0.1 netmask 0xffff0000 broadcast 172.16.255.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax

These where done with the ix 3.3.6 driver. But so far I believe it looks like the same output I got with the stock driver.

Anyway to confirm if I happen to have an XG-1537 from the initial production run?

BR Kristian

Derelict

We are not tracking anything like that from the XG-1537. As @stevew pointed out, the XG-1537 is not the XG-7100.

Looks like you are plugging a copper module but it is reporting its media as a Twinaxial direct attach cable.

desrux2

@Derelict said in XG-1537 SFP+ ports:

Looks like you are plugging a copper module but it is reporting its media as a Twinaxial direct attach cable.

Yes, glad you noticed, that seems to sometimes happen if I had that 10G Twinax cable inserted before inserting a different module. Somehow the “10Gbase-Twinax” media information lingers on the interface.

Derelict

It might also be the driver not being updated due to unsupported modules being detected.

Just a question: have you tried an actual intel-branded SFP module?

I have about three boxes of test modules but, unfortunately, no access to an XG-1537.

desrux2

@Derelict said in XG-1537 SFP+ ports:

It might also be the driver not being updated due to unsupported modules being detected.

I did a reboot without any SFP modules installed in ix0 or ix1. 1) Then I installed a unsupported SFP+ 10gbe LR module into ix1 and got link up and working fine. 2) Then the 10gbe DAC in ix0
3) I removed the DAC from ix0
4) Installed the two copper modules above one after the other and recoded the output.

If I do a reboot/boot with the unsupported SFP+ LR the interface (ix1) is killed during boot and never comes up. PFsense then cannot start normally and initiates interface assignment due to one interface missing which I do not want as it requires WAN to be assigned and that is the interface missing. (I have ordered a intel compatible replacement that I expect will not have this issue - the current one was given by the ISP - Cisco compatible...)
Need to reboot/boot after this (first removing any SFP in ix1 or ix0), and here the ix1 will sometimes not recognize/load the SFP+ LR in a manner where link is possible. (No green lights). Taking ix1 down and up does not help. But I did find that installing and uninstalling the 10gbe DAC (twinax) in a loop between ix1 and ix0 would put the interface i a mode where it is possible to get link again.

Just a question: have you tried an actual intel-branded SFP module?

You mean one of those org intel that cost about 680 USD vs the intel compatible ones that cost 80 USD?
No (Which is why I got confirmation that the official requirement is only intel compatible before purchasing the XG-1537).

I have about three boxes of test modules but, unfortunately, no access to an XG-1537.
Too bad, I guess it would make a lot of sense for you to have one easily available.

Derelict

Yes. Booting with something that the driver recognizes as unsupported can render the NIC unusable until the unit is power cycled. My advice is to use modules that the driver accepts.

Once you get working modules in the ports and they are in place and configured, all of this should go away.

desrux2

@Derelict said in XG-1537 SFP+ ports:

Yes. Booting with something that the driver recognizes as unsupported can render the NIC unusable until the unit is power cycled. My advice is to use modules that the driver accepts.

Yes it kind of makes the allow unsupported sfp setting useless in practice. Not the best driver design choice. I follow your advice, no other choice*

Once you get working modules in the ports and they are in place and configured, all of this should go away.

Offending module is in place and working, just cannot reboot with it installed. But as I mentioned before it was received from ISP and I am currently waiting for a intel compatible variant.

But right, so my issue is that I cannot get link on the intel compatible 1G SFP RJ45 module.
a) Just unlucky with particular modules others will work?
b) XG-1537 does not like copper modules in general?
c) Only likes 10Gbit media and not 1Gbit in the SFP+ ports? (I was unfortunately not able to check link against something with the 1Gb SFP LR modules)
d) ?

BR Kristian

*no other choice, that is except modifying the ixgbe driver

desrux2

@desrux2 now received intel compatible module from ISP. But I unfortunatly still get “Unsupported SFP+ module type was detected.”

Excactly which module types are supported by the XG-1537 ?

Module is a 10G Base-ER ( CWDM 40km 1510nm ) is anyone else out there running something similar?

Also is there any limitations on which PCIe adapters and brands there can be inserted? Any dual 10gb port SFP+ recommendations?

Output of ifconfig -vvvvvvvvm

nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	plugged: SFP/SFP+/SFP28 10G Base-ER (LC)
	vendor: MOSAIC PN: I-FT3940D-51 SN: INB190418001 DATE: 2019-04-18
Class: 10G Base-ER
Length: (null)
Tech: (null)
Media: (null)
Speed: (null)
	module temperature: 53.70 C Voltage: 3.32 Volts
	RX: 0.11 mW (-9.24 dBm) TX: 1.62 mW (2.12 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 80 00 00 00 00 00 00 00 06 67 00 28 FF
	00 00 00 00 4D 4F 53 41 49 43 20 20 20 20 20 20
	20 20 20 20 00 00 00 00 49 2D 46 54 33 39 34 30
	44 2D 35 31 20 20 20 20 20 20 20 20 05 E6 00 C0
	60 0A 00 00 49 4E 42 31 39 30 34 31 38 30 30 31
	20 20 20 20 31 39 30 34 31 38 20 20 68 90 01 FB
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Derelict

What is the exact module in use? Sorry but we can't go by what is reported by the module itself because counterfeiting and fake results there are pretty much the norm.

I find it hard to believe that anything that is actually an Intel module is reporting as unsupported. Not impossible, but...

desrux2

@Derelict What exact specs are you missing from my description?

You seem to keep taking about original intel modules.

Let me be clear: I am using intel compatible modules or at least what should be intel compatible modules. I am not using original intel modules.

Before buying the XG-1537 i got an official Netgate response to an inquiry that intel compatible modules are required, as in not original intel modules. Is that incorrect?

I am asking about module types, so is there a distinction between SR, LR, ER, T etc.? Or what is considered the “module type”. Perhaps there could be some (in my opinion) stupid driver limitation that only SR and LR works but not ER and T (RJ45).

If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful.

All of this is simply to be able to survive reboot of XG-1537 without being physically present.
(As the connection with the unsupported module works perfectly. Problem is that interface disappears on reboot due to (in my opinion) an intentional bad intel driver design.
If I could compile a new slightly modified driver with relative ease and make sure there is no risk to it being overwritten by pfsense upgrades then I had probably done that by now.)

Derelict

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

You are saying yours does not.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

desrux2

@Derelict said in XG-1537 SFP+ ports:

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

Does work mean that it will survive reboot or just that link will work?

You are saying yours does not.

Link works. I have network connection. Traffic flows.
Problem is I get the unsupported module type message which means that on boot the intel driver will refuse to load the interface e.g. ix1 and because the interface is missing pfsense will boot into interface assignment mode requiring physical presence to cancel - remove the module from ix1 boot up normally so ix1 exsists and then insert module into ix1 again.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

I would love to know of any specific module that fits the specs and does not get the unsupported module type message. Intel org or otherwise. Preferably available in Europe.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

I already have that in place.

Derelict

As far as I know we have not tested any modules other than standard 10G, 10G/1G, and 1G SMF and MMF so the only suggestion I can offer is to try another one or call Mosaic and ask why the one they brand as intel-specific isn't accepted by the FreeBSD ix driver.

dotdash

Had one of these in for a client build, and some stray SFP transceivers, so I decided to do some testing for science. I added the hw.ix.unsupported_sfp=1 and tested modules from a cold boot. I had four optical SFP modules and one copper. All of the optical modules worked fine: HP J4858-A MMF, an addon clone of the HP, a Cisco CLC-SX-MMD, and a Cisco MGBSX1 100SX. Box booted fine and linked up with a switch with a similar SFP. The copper module, a startech branded HP J8177C clone, caused the driver to complain about it being unsupported and not initialize the interface. This caused the box to drop to interface mismatch. Intel doesn't list any copper modules as compatible, so one would probably be wise to avoid them. It did however, work fine with all the old garbage optical SFP modules I threw at it. Disclaimer- I didn't use the ix interface, just verified it linked up on both ends.
TL,DR- Works fine with non-Intel optical SFP modules, just not copper ones.

Derelict

@dotdash At least based on your sample size.

dotdash

Sample size limited to what was on the shelf. No warranties implied or offered. May expire at any time. Mileage may vary depending on conditions. Discontinue use if you experience headaches, nausea, or bleeding about the eyes.

desrux2

@dotdash thanks for sharing.
Was it only SFP modules or also SFP+?
Do you remember what module types you tried besides T (copper) like LR, SR, etc.?

I looked through the intel driver code:
https://github.com/freebsd/freebsd/blob/master/sys/dev/ixgbe/
And looks to me like there is a number of scenarios where you can get:

hw->phy.type = ixgbe_phy_sfp_unsupported;

leading to an error status
and where

if (hw->allow_unsupported_sfp == TRUE) {
					EWARN(hw, "WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.\n");
					status = IXGBE_SUCCESS;

Is NOT implemented/applied. My guess would be when module type is something else than LR or SR and perhaps in combination with 10Gbe then the allow unsupported sfp flag does not have an effect.

dotdash

The only modules I had on hand were two Cisco SX SFP modules, an HP SX SFP, an HP clone SX SFP, and the clone HP copper GB SFP. I didn't have any SFP+ modules handy. I wanted to verify the box would boot with SFP (not SFP+) modules. From my limited research, copper modules of any kind are not supported on this card. If you want 10G optical, Netgate only charges around $50 to add a fully supported one to your order. Direct attach cables are slightly cheaper/slightly more expensive depending on length. By no means do you need to get an overpriced genuine Intel module.

desrux2

@dotdash Thanks, if you see my earlier post I have specified the module I need (what I currently have from the ISP):

“If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful”

So far no replies on this.

dotdash

Intel lists the E10GSFPLR as supported. Search for something compatible with that. In the States, you can get a used Intel one, or a new clone pretty cheap. (<$50)

desrux2

Update for anyone googling and looking at this thread. (Now updated with info from Steven)

With newer PFSense based on FreeBSD 11.2 the mentioned bugs with the Intel ix driver and unsupported sfp+ modules have finally been fixed, hurray!!

TL;DR you could have a working setup with a unsupported sfp+ module but could not power cycle system without loosing the interface. Workaround was to physically remove sfp+ module and reinsert after boot.. Now this workaround is no longer required!
(Our ISP requires us to use a CWDM module which does not exist in native Intel so we are forced to use unsupported sfp+ modules.)

How
To allow unsupported sfp+ modules to function, in the /boot/loader.conf.local add the line:

hw.ix.unsupported_sfp="1"

Shared below for reference.

~~and set this option in System Tunables~~ has no effect see next post:

It seems like is not enough only having the System Tunables option.

Important note
Please note that if you bootup with a unsupported sfp+ module before adding these options - it is possible that the 10gbe NIC enters a ‘broken’ state where network is sluggish. It may look like the optics is working but effectively no traffic is transmitted. Only way to get out of this ‘broken’ state is completely powering down the XG 1537.

Copy of /boot/loader.conf.local

/boot/loader.conf.local

kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
legal.intel_wpi.license_ack="1"
legal.intel_ipw.license_ack="1"
legal.intel_iwi.license_ack="1"
hw.ix.unsupported_sfp="1"
boot_verbose="YES"
autoboot_delay="3"

stephenw10

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.

[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

Steve

desrux2

@stephenw10 said in XG-1537 SFP+ ports:

That value is a loader variable, you need to set it in loader.conf.local.

It's actually read-only in the sysctls so that Systems Tunables entry is not doing anything.
[22.01-BETA][admin@6100.stevew.lan]/root: sysctl hw.ix.unsupported_sfp=1
sysctl: oid 'hw.ix.unsupported_sfp' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf
Steve

Thanks for the clarification.

moxlotus

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

desrux2

@moxlotus said in XG-1537 SFP+ ports:

@stephenw10 Re: XG-1537 SFP+ ports

The issue is now back in pfsense 2.6 and changing the config in /boot/loader.conf.local does not seem to help.

I tried to boot up the machine without the module installed. Upon inserting the sfp+ module, it will not get ip on the wan interface.

If i were to boot with the module inserted, the port will fail to load

@moxlotus did you try powering completely down after adding the line to the loader.conf.local ?
If the module has been inserted during boot without the line present in the loader.conf.local then nic will be rendered inoperable until power cycle - reboot is not enough.

moxlotus

@desrux2 yes i did power cycle. And just a note for the rest, it broke right after i upgraded from 2.5.2 to 2.6.

stephenw10

Why are you running 2.6 in an XG-1537? Do you see the same thing in 22.01?

What module are you using?

Steve