SFP incompatibility on Netgate XG-7100

[WillCT1]

Our building just got lit for Verizon 1Gig fiber service and when I tried to use an SFP module in the SFP+ port to connect the Verizon fiber I had no success. I used an Add-On Networks Cisco compatible SFP (GLC-LX-SM-RGD-AO) and a generic 1Gig SMF SFP that I picked up locally to try and isolate the problem but neither worked.

When I plugged the same SFP into my switch (separate VLAN to isolate) and use a 10G Twinax to that same switch to the pfsense port everything works fine. I'd rather not have to use my switch as a media converter so any suggestions on how to make it work? I have already tried enabling "hw.ixgbe.allow_unsupported_sfp" in the system tuneables without success.

Output from the shell of "ifconfig -vvvvv ix1" is: (IPs,MAC and SN changed)

ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:XX:XX:XX:06
	hwaddr 00:08:XX:XX:XX:06
	inet6 fe80::XXX:XXX:fe0e:~~XX~~06%ix1 prefixlen 64 scopeid 0x2
        inet 11.22.33.74 netmask 0xfffffffc broadcast 11.22.33.75  
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	plugged: SFP/SFP+/SFP28 1000BASE-LX (LC)
	vendor: CISCO-ADDON PN: GLC-LX-SM-RGD-AO SN: ADXXXXXXX27 DATE: 2018-08-07
Class: 1000BASE-LX
Length: (null)
Tech: (null)
Media: (null)
Speed: (null)
module temperature: 20.03 C Voltage: 3.30 Volts
       RX: 0.20 mW (-6.86 dBm) TX: 0.24 mW (-6.04 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 00 00 00 02 00 00 00 00 01 0C 00 0A 64 
	37 37 00 00 43 49 53 43 4F 2D 41 44 44 4F 4E 20 
	20 20 20 20 00 44 D5 A5 47 4C 43 2D 4C 58 2D 53 
	4D 2D 52 47 44 2D 41 4F 44 31 20 20 05 1E 00 6E 
	00 1A 00 00 41 44 44 34 38 41 30 46 37 38 32 37 
	20 20 20 20 31 38 30 38 30 37 20 20 68 F0 01 2F 
	00 00 11 0C 33 9A 21 77 D4 56 43 0B B0 8E D9 E8 
	82 E6 D8 00 00 00 00 00 00 00 00 00 D3 70 54 D3

If I used a Ubiquiti 10G SMF SFP+ in the SFP+ slot I got carrier on my side immediately but the Verizon tech said his equipment still showed as no carrier.

I thought it might be this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=150249 but that bug was back in 2010.

I bought a few more SFPs from fs.com to see if a different module might work but any help would be appreciated.

[WillCT1]

OK so I just saw this thread: https://forum.netgate.com/topic/131725/xg-7100-sfp-module-1gbps/7

Time to contact support even though this unit was purchased just over a month ago.

[stephenw10]

Were you able to get any further info on this?

Digging a bit it doesn't look like your boards are affected by that issue.

What does the ifconfig output from the other SFP module look like?

Are you able to test those modules direct to your switch?

I assume the switch shows it linked at 1G to Verizon when it's connected that way?

Steve

[WillCT1]

@stephenw10 Not yet, I am working with support on this. With the fiber connected it shows:
media: Ethernet autoselect (Unknown <rxpause,txpause>)
status: active
supported media:
media autoselect

with the fiber disconnected it shows:
media: Ethernet autoselect
status: no carrier
supported media:
media autoselect

There doesn't seem to be traffic passing through when connected. The SFP functions fine in a D-Link DGS-3630 managed switch

[stephenw10]

The generic SFP module also shows similar ifconfig verbose output?

Steve

[WillCT1]

@stephenw10 that was the output of the generic SFP (IC INTRACOM PN: 506724). This was with two of those linked together.

I bought a few other SFPs from fs.com so I can try the same test with those SFPs and see if anything changes. I'll post after I run the tests.

[WillCT1]

Same failure with the following SFPs:

vendor: CISCO-ADDON PN: GLC-LX-SM-RGD-AO SN: XXXXXX DATE: 2018-08-07
module temperature: 23.93 C Voltage: 3.29 Volts
RX: 0.20 mW (-6.91 dBm) TX: 0.24 mW (-6.07 dBm)

vendor: IC INTRACOM PN: 506724 SN: XXXXXX DATE: 2018-02-08
module temperature: 18.00 C Voltage: 3.24 Volts
RX: 0.21 mW (-6.60 dBm) TX: 0.31 mW (-5.00 dBm)

vendor: FS PN: SFP1G-SX-31 SN: XXXXXX DATE: 2018-11-09
module temperature: 22.05 C Voltage: 3.24 Volts
RX: 0.24 mW (-6.10 dBm) TX: 0.26 mW (-5.76 dBm)

vendor: FS PN: SFP1G-SX-31 SN: XXXXXX DATE: 2019-02-19
module temperature: 14.73 C Voltage: 3.30 Volts
RX: 0.31 mW (-4.96 dBm) TX: 0.23 mW (-6.26 dBm)

vendor: FS PN: SFP1G-LX-31 SN: XXXXXX DATE: 2018-06-29
module temperature: 21.85 C Voltage: 3.31 Volts
RX: 0.24 mW (-6.17 dBm) TX: 0.25 mW (-5.97 dBm)

There is RX and TX power so you know that there is handshaking but for some reason FreeBSD / pfSense just cannot make use of it. I know the SFP isn't faulty since I am using the same fiber / SFP in my switch to act as a media converter until support can help me fix this. The CISCO-ADDON SFP functions whether it is in a SFP or SFP+ port so I don't think this is a case of a SFP speed issue.

[stephenw10]

And by the same failure you mean they all show:

media: Ethernet autoselect (Unknown <rxpause,txpause>)
status: active

And seemingly pass no traffic?

Steve

[WillCT1]

Correct. I also might have had the "unsupported SFP" flag set incorrectly so I changed my loader.conf.local to:

hw.ixgbe.unsupported_sfp=1
hw.ix.unsupported_sfp=1

I also put those same items in the system tunables.

The output of the ifconfig was the same and I still cannot ping the gateway or any other IP that goes through that interface.

[stephenw10]

I doubt that was an issue, it's very rare that does anything.

If you do get an unsupported module you will find the interface disappears entirely if you boot with it in. The driver will fail to attach and you would see that reported in the boot log.

Steve