XG-1537 SFP+ ports

[beatvjiking]

Hi, I was just wondering - are the SFP+ ports in the XG-1537 dual-personality? As in, can they take SFPs and SFP+s or SFP+ only?

[beatvjiking]

It appears from testing that they are not. They are 10Gb only.

[desrux2]

@beatvjiking said in XG-1537 SFP+ ports:

It appears from testing that they are not. They are 10Gb only.

I have a XG-1537 expecting to be able to use both SFP and SFP+ modules. I have the same issue.
I have not been able to get SFP modules to work (tried both with intel and Cisco compatible) - is this by design or a sw bug / hw bug?

Any official status on this issue?

[stephenw10]

What modules exactly have you tested? I'm not aware of any particular restriction on the XG-1537.

Steve

[desrux2]

@stephenw10 said in XG-1537 SFP+ ports:

What modules exactly have you tested? I'm not aware of any particular restriction on the XG-1537.

Steve

Hi Steve

I have not been able to get any of these SPF modules to work:

I-FT3420D
I-FT3401

C-FT3401G
C-FT320D
From Mosaic Optics.

BR Kristian

[Derelict]

When you have one of those Intel-compatible single-mode modules inserted what is the output of:

ifconfig -vm ixX

Should go without saying but ixX there is ix0, ix1, etc.

[beatvjiking]

This is an old thread, and the XG-1537s I got were from the initial production run. I seem to recall early units having a restriction to 10Gb only, while later ones were dual-personality. I don't have personal experience with later units, so I don't know how accurate that is. Might be worth looking at though.

[stephenw10]

That applied to the XG-7100 but not, as far as I know, to the XG-1537.

It would be interesting to see that ifconfig output.

Steve

[desrux2]

I have been busy (onsite) testing SFP module combinations the last 4 hours. Currently output copy pasted in a large excel sheet - need to clean it up, make sense of it and then share it here.

Until then here is some initial ifconfig output:

I-FT3401 (SFP - RJ45)
ix0: Unsupported SFP+ module type was detected.

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:8d:00:3c
	hwaddr ac:1f:6b:8d:00:3c
	inet6 xxxxx0 prefixlen 64 scopeid 0x1
	inet6 xxxxxx:: prefixlen 50
	inet 172.16.0.1 netmask 0xffff0000 broadcast 172.16.255.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	plugged: SFP/SFP+/SFP28 1000BASE-T (RJ45)
	vendor: MOSAIC PN: I-FT3401 SN: INB180201001 DATE: 2018-02-02

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 22 00 00 00 08 00 00 00 00 01 00 00 00 00
	00 00 64 00 4D 4F 53 41 49 43 20 20 20 20 20 20
	20 20 20 20 00 00 00 00 49 2D 46 54 33 34 30 31
	20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 EA
	00 1A 00 00 49 4E 42 31 38 30 32 30 31 30 30 31
	20 20 20 20 31 38 30 32 30 32 00 00 00 00 00 5D
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
	20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20

C-FT3401GDN (SFP RJ45)
ix0: Unsupported SFP+ module type was detected.

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:8d:00:3c
	hwaddr ac:1f:6b:8d:00:3c
	inet6 xxxxxx prefixlen 64 scopeid 0x1
	inet6 xxxxxxx prefixlen 50
	inet 172.16.0.1 netmask 0xffff0000 broadcast 172.16.255.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax

These where done with the ix 3.3.6 driver. But so far I believe it looks like the same output I got with the stock driver.

Anyway to confirm if I happen to have an XG-1537 from the initial production run?

BR Kristian

[Derelict]

We are not tracking anything like that from the XG-1537. As @stevew pointed out, the XG-1537 is not the XG-7100.

Looks like you are plugging a copper module but it is reporting its media as a Twinaxial direct attach cable.

[desrux2]

@Derelict said in XG-1537 SFP+ ports:

Looks like you are plugging a copper module but it is reporting its media as a Twinaxial direct attach cable.

Yes, glad you noticed, that seems to sometimes happen if I had that 10G Twinax cable inserted before inserting a different module. Somehow the “10Gbase-Twinax” media information lingers on the interface.

[Derelict]

It might also be the driver not being updated due to unsupported modules being detected.

Just a question: have you tried an actual intel-branded SFP module?

I have about three boxes of test modules but, unfortunately, no access to an XG-1537.

[desrux2]

@Derelict said in XG-1537 SFP+ ports:

It might also be the driver not being updated due to unsupported modules being detected.

I did a reboot without any SFP modules installed in ix0 or ix1. 1) Then I installed a unsupported SFP+ 10gbe LR module into ix1 and got link up and working fine. 2) Then the 10gbe DAC in ix0
3) I removed the DAC from ix0
4) Installed the two copper modules above one after the other and recoded the output.

If I do a reboot/boot with the unsupported SFP+ LR the interface (ix1) is killed during boot and never comes up. PFsense then cannot start normally and initiates interface assignment due to one interface missing which I do not want as it requires WAN to be assigned and that is the interface missing. (I have ordered a intel compatible replacement that I expect will not have this issue - the current one was given by the ISP - Cisco compatible...)
Need to reboot/boot after this (first removing any SFP in ix1 or ix0), and here the ix1 will sometimes not recognize/load the SFP+ LR in a manner where link is possible. (No green lights). Taking ix1 down and up does not help. But I did find that installing and uninstalling the 10gbe DAC (twinax) in a loop between ix1 and ix0 would put the interface i a mode where it is possible to get link again.

Just a question: have you tried an actual intel-branded SFP module?

You mean one of those org intel that cost about 680 USD vs the intel compatible ones that cost 80 USD?
No (Which is why I got confirmation that the official requirement is only intel compatible before purchasing the XG-1537).

I have about three boxes of test modules but, unfortunately, no access to an XG-1537.
Too bad, I guess it would make a lot of sense for you to have one easily available.

[Derelict]

Yes. Booting with something that the driver recognizes as unsupported can render the NIC unusable until the unit is power cycled. My advice is to use modules that the driver accepts.

Once you get working modules in the ports and they are in place and configured, all of this should go away.

[desrux2]

@Derelict said in XG-1537 SFP+ ports:

Yes. Booting with something that the driver recognizes as unsupported can render the NIC unusable until the unit is power cycled. My advice is to use modules that the driver accepts.

Yes it kind of makes the allow unsupported sfp setting useless in practice. Not the best driver design choice. I follow your advice, no other choice*

Once you get working modules in the ports and they are in place and configured, all of this should go away.

Offending module is in place and working, just cannot reboot with it installed. But as I mentioned before it was received from ISP and I am currently waiting for a intel compatible variant.

But right, so my issue is that I cannot get link on the intel compatible 1G SFP RJ45 module.
a) Just unlucky with particular modules others will work?
b) XG-1537 does not like copper modules in general?
c) Only likes 10Gbit media and not 1Gbit in the SFP+ ports? (I was unfortunately not able to check link against something with the 1Gb SFP LR modules)
d) ?

BR Kristian

*no other choice, that is except modifying the ixgbe driver

[desrux2]

@desrux2 now received intel compatible module from ISP. But I unfortunatly still get “Unsupported SFP+ module type was detected.”

Excactly which module types are supported by the XG-1537 ?

Module is a 10G Base-ER ( CWDM 40km 1510nm ) is anyone else out there running something similar?

Also is there any limitations on which PCIe adapters and brands there can be inserted? Any dual 10gb port SFP+ recommendations?

Output of ifconfig -vvvvvvvvm

nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	plugged: SFP/SFP+/SFP28 10G Base-ER (LC)
	vendor: MOSAIC PN: I-FT3940D-51 SN: INB190418001 DATE: 2019-04-18
Class: 10G Base-ER
Length: (null)
Tech: (null)
Media: (null)
Speed: (null)
	module temperature: 53.70 C Voltage: 3.32 Volts
	RX: 0.11 mW (-9.24 dBm) TX: 1.62 mW (2.12 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 80 00 00 00 00 00 00 00 06 67 00 28 FF
	00 00 00 00 4D 4F 53 41 49 43 20 20 20 20 20 20
	20 20 20 20 00 00 00 00 49 2D 46 54 33 39 34 30
	44 2D 35 31 20 20 20 20 20 20 20 20 05 E6 00 C0
	60 0A 00 00 49 4E 42 31 39 30 34 31 38 30 30 31
	20 20 20 20 31 39 30 34 31 38 20 20 68 90 01 FB
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[Derelict]

What is the exact module in use? Sorry but we can't go by what is reported by the module itself because counterfeiting and fake results there are pretty much the norm.

I find it hard to believe that anything that is actually an Intel module is reporting as unsupported. Not impossible, but...

[desrux2]

@Derelict What exact specs are you missing from my description?

You seem to keep taking about original intel modules.

Let me be clear: I am using intel compatible modules or at least what should be intel compatible modules. I am not using original intel modules.

Before buying the XG-1537 i got an official Netgate response to an inquiry that intel compatible modules are required, as in not original intel modules. Is that incorrect?

I am asking about module types, so is there a distinction between SR, LR, ER, T etc.? Or what is considered the “module type”. Perhaps there could be some (in my opinion) stupid driver limitation that only SR and LR works but not ER and T (RJ45).

If anyone can tell me specifically which SFP+ 10Gb 40km/15dB 1510nm (CWDM) LR/ER module will not trigger the unsupported SFP module type in the intel driver then please let me know. I will be very grateful.

All of this is simply to be able to survive reboot of XG-1537 without being physically present.
(As the connection with the unsupported module works perfectly. Problem is that interface disappears on reboot due to (in my opinion) an intentional bad intel driver design.
If I could compile a new slightly modified driver with relative ease and make sure there is no risk to it being overwritten by pfsense upgrades then I had probably done that by now.)

[Derelict]

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

You are saying yours does not.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

[desrux2]

@Derelict said in XG-1537 SFP+ ports:

I am looking for the exact model of the exact module you are using.

Yes, any intel-compatible module should work.

Does work mean that it will survive reboot or just that link will work?

You are saying yours does not.

Link works. I have network connection. Traffic flows.
Problem is I get the unsupported module type message which means that on boot the intel driver will refuse to load the interface e.g. ix1 and because the interface is missing pfsense will boot into interface assignment mode requiring physical presence to cancel - remove the module from ix1 boot up normally so ix1 exsists and then insert module into ix1 again.

There are no specific requirements as Steve stated, though it would not be a surprise to be if anything not specifically marked as Intel-compatible had issues.

I would love to know of any specific module that fits the specs and does not get the unsupported module type message. Intel org or otherwise. Preferably available in Europe.

You can try putting this in /boot/loader.conf.local:

hw.ix.unsupported_sfp=1

I already have that in place.