Intel 82599ES; GPON SFP module not working

milindhvijay · Jan 12, 2024, 6:02 PM

Hello,

My firewall detects the sfp card (Intel 82599ES) but when I connect GPON sfp modules and attach an interface, it does not auto-negotiate and ends up showing (rxpause, txpause). Also, I am unable to get fiber rx signal. GPON sfp is working fine. I have 2 units tested with a media converter.

ix0@pci0:6:0:0:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0xffff subdevice=0xffff
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
ix1@pci0:6:0:1:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0xffff subdevice=0xffff
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet

At first, I thought it's happening because of unsupported sfp, so I tried adding hw.ixgbe.unsupported_sfp=1 to loader.conf.local. Also, I flipped the bit in IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP to make unsupported sfp work, but still in the same situation.

Any help would be appreciated.

stephenw10 · Jan 12, 2024, 6:24 PM

What does ifconfig -vvvm ix0 show when it's connected?

milindhvijay · Jan 12, 2024, 7:16 PM

When ISP fiber not connected into sfp module:

ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: OPT8
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether d4:20:00:b1:2b:3d
	inet6 fe80::d620:ff:feb1:2b3d%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (SC)
	vendor: Hisense-Leox PN: LXT-010S-H SN: LEOXD10002EF DATE: 2022-11-04
	module temperature: 32.20 C voltage: 3.22 Volts
	lane 1: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 02 00 00 00 00 03 0d 00 14 c8
	 00 00 00 00 48 69 73 65 6e 73 65 2d 4c 65 6f 78
	 20 20 20 20 00 00 00 00 4c 58 54 2d 30 31 30 53
	 2d 48 20 20 20 20 20 20 31 2e 30 20 05 1e 00 1a
	 00 1a 00 00 4c 45 4f 58 44 31 30 30 30 32 45 46
	 20 20 20 20 32 32 31 31 30 34 20 20 68 f0 08 5e
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00	
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ISP fiber connected to sfp module:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT8
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether d4:20:00:b1:2b:3d
	inet6 fe80::d620:ff:feb1:2b3d%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (SC)
	vendor: Hisense-Leox PN: LXT-010S-H SN: LEOXD10002EF DATE: 2022-11-04
	module temperature: 37.05 C voltage: 3.25 Volts
	lane 1: RX power: 0.01 mW (-21.74 dBm) TX bias: 0.00 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 02 00 00 00 00 03 0d 00 14 c8
	 00 00 00 00 48 69 73 65 6e 73 65 2d 4c 65 6f 78
	 20 20 20 20 00 00 00 00 4c 58 54 2d 30 31 30 53
	 2d 48 20 20 20 20 20 20 31 2e 30 20 05 1e 00 1a
	 00 1a 00 00 4c 45 4f 58 44 31 30 30 30 32 45 46
	 20 20 20 20 32 32 31 31 30 34 20 20 68 f0 08 5e
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

stephenw10 · Jan 12, 2024, 8:00 PM

Hmm, it shows as active only when the fiber is not connected?

It only shows whatever the module is passing. Those GPON SFP modules are a full router/modem in the module.
Do you know how it's expected to be configured? It's probably got a user interface on some default IP.

milindhvijay · Jan 13, 2024, 5:01 AM

No, the Hisense-Leox LXT-010S-H GPON SFP does not have any routing functionality. It functions in pure transparent mode, converts optical to eletric. VLAN tagging, PPPoE is done on pfSense.

I tested with another GPON SPF:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:e2:69:56:98:dc
	hwaddr d4:20:00:b1:2b:3d
	inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
	inet6 fe80::2e2:69ff:fe56:98dc%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 BASE-PX (SC)
	vendor: HSGQ PN: HSGQ-XPON-Stick SN: XPON23120861 DATE: 2023-12-21
	module temperature: 46.43 C voltage: 3.15 Volts
	lane 1: RX power: 0.01 mW (-22.15 dBm) TX bias: 14.60 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 80 00 00 00 00 01 0c 00 14 c8
	 00 00 00 00 48 53 47 51 20 20 20 20 20 20 20 20
	 20 20 20 20 00 00 00 00 48 53 47 51 2d 58 50 4f
	 4e 2d 53 74 69 63 6b 20 30 30 30 20 05 1e 00 e7
	 00 1c 00 00 58 50 4f 4e 32 33 31 32 30 38 36 31
	 20 20 20 20 32 33 31 32 32 31 20 20 68 b0 03 fe
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This one has both RX and TX signal when connecting ISP fiber (which means OLT is able to authenticate the GPON module, but with or without ISP fiber connected the status is always no carrier with no lights.

Both these GPON SFP modules get authenticated by the ISP OLT as I am able to get them working with a media converter.

This concludes that the issue is not with SFP modules, but with the compatibility of 82599ES with pfSense or some setting change that I am not aware of.

If nothing works can you suggest a good replacement x4 SFP PCI card that has high compatibility with pfSense?

stephenw10 · Jan 13, 2024, 4:51 PM

That Hisense device is a router running Linux with 32MB of RAM: https://hack-gpon.org/ont-leox-lxt-010s-h/

It may be configured to pass the connection transparently but it may not. It looks like it has a default admin page at 192.168.100.1.

You might try setting the NIC to 1G fixed rate to get link. Some modules will only negotiate at 10G.

milindhvijay · Jan 13, 2024, 5:46 PM

This particular Hisense device, although has a default admin page, it is only used for firmware upgrades, and monitoring PON signal strength. There is no routing function on it. Also, this module only negotiates at either 1G or 2.5G and I have set it to 1G via telnet.

My ISP uses PPPoE, and when I use this GPON SFP with a media converter, I tag VLAN and authenticate PPP on pfSense. Gets public IP on pfsense, and no double NAT.

stephenw10 · Jan 13, 2024, 6:15 PM

@milindhvijay said in Intel 82599ES; GPON SFP module not working:

this module only negotiates at either 1G or 2.5G

Actually negotiate or use a fixed speed? If it's fixed the NIC won't link to it if it's set to auto.

milindhvijay · Jan 13, 2024, 6:29 PM

This is what I have set it to:

To change the link mode use this command:

# /etc/scripts/flash get LAN_SDS_MODE
LAN_SDS_MODE=0
# /etc/scripts/flash set LAN_SDS_MODE 1

Value 	cat /proc/kmsg 	Mode 	Behavior
1 	<4>change mode to 1(Fiber 1G) 	FIBER 	1GbaseX with auto-neg on

stephenw10 · Jan 13, 2024, 6:32 PM

Hmm, and that's definitely the SFP side of the module?

How are you accessing that, in the media converter?

milindhvijay · Jan 13, 2024, 6:35 PM

login-to-view

Yes, currently sfp connected to a 1G media converter-> connected with rj-45 to pfsense. Parameter (LAN_SDS_MODE=1) set via telnet.

stephenw10 · Jan 13, 2024, 6:49 PM

Hmm, yet neither module presents as 1G capable in the NIC. Auto-only.

When you boot with the module inserted does it show any errors in the boot log?

milindhvijay · Jan 13, 2024, 7:03 PM

ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x3020-0x303f mem 0x50c20000-0x50c3ffff,0x50c44000-0x50c47fff at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: d4:20:00:b1:2b:3d
ix0: PCI Express Bus: Speed 5.0GT/s Unknown
ix0: eTrack 0x800003de PHY FW V65535
ix0: netmap queues/slots: TX 4/2048, RX 4/2048

stephenw10 · Jan 13, 2024, 8:47 PM

Hmm, no error there. There's not really much that can be set for those NICs beyond what you tried already.

Does really only show an active link when the upstream fiber is disconnected?

milindhvijay · Jan 15, 2024, 9:41 AM

@stephenw10 Yes, only shows active when ISP fiber is disconnected?

Would updating the driver help? If so, how can I do it?

stephenw10 · Jan 15, 2024, 12:43 PM

I doubt it would make any difference but you could try to either build it yourself or install the kmod port for ixgbe. It may be marginally newer. Finding a version that loads can be challenging though since it has to closely match the kernel.

When the optic is disconnected do both modules show as active?

In that state can you connect to their cli through pfSense?

milindhvijay · Jan 15, 2024, 12:59 PM

No, Hisense-Leox LXT-010S-H shows active and power(green) and activity(amber) indication light is constantly ON when optic is disconnected. With optic connected, lights come ON in constant intervals, stays ON for 2-3 seconds and turns OFF. Without optics I am able to get into the admin page and telnet works.

HSGQ-XPON-Stick shows no carrier with or without optic. Even the power/activity lights are off always. Never even seen (rxpause, txpause) in interfaces. If I remember right, it always stay on autoselect.

I think HSGQ-XPON-Stick does not work because it is a 2.5G stick and AFAIK Intel x520 supports only 1G and 10G. I've tried forcing it to 1G but no luck.

stephenw10 · Jan 15, 2024, 1:19 PM

Indeed only the x550 support N-Base and I think it's only N-BaseT. I don't have one to test with.

You can try setting the advertise speeds to 1G only in the negotiation:

[24.03-DEVELOPMENT][root@8200.stevew.lan]/root: sysctl -d dev.ix.0.advertise_speed
dev.ix.0.advertise_speed: 
Control advertised link speed using these flags:
	0x1 - advertise 100M
	0x2 - advertise 1G
	0x4 - advertise 10G
	0x8 - advertise 10M

	0x10  - advertise 2.5G
	0x20  - advertise 5G

	100M and 10M are only supported on certain adapters.

That still requires a compatible module though so may not allow it.

milindhvijay · Jan 15, 2024, 1:32 PM

[2.7.2-RELEASE][admin@Gaia.home]/root: sysctl -d dev.ix.0.advertise_speed
dev.ix.0.advertise_speed:
Control advertised link speed using these flags:
	0x1 - advertise 100M
	0x2 - advertise 1G
	0x4 - advertise 10G
	0x8 - advertise 10M

	0x10  - advertise 2.5G
	0x20  - advertise 5G

	100M and 10M are only supported on certain adapters.

How do I set the flag?

stephenw10 · Jan 15, 2024, 1:42 PM

Like:

[24.03-DEVELOPMENT][root@8200.stevew.lan]/root: sysctl dev.ix.2.advertise_speed=1
dev.ix.2.advertise_speed: 11 -> 1

But it has to have a device that supports those speeds. ix2 there is a combo port so it supports 1G, 100M, 10M baseT.