Intel 82599ES; GPON SFP module not working

milindhvijay

When ISP fiber not connected into sfp module:

ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: OPT8
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether d4:20:00:b1:2b:3d
	inet6 fe80::d620:ff:feb1:2b3d%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect (Unknown <rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (SC)
	vendor: Hisense-Leox PN: LXT-010S-H SN: LEOXD10002EF DATE: 2022-11-04
	module temperature: 32.20 C voltage: 3.22 Volts
	lane 1: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 02 00 00 00 00 03 0d 00 14 c8
	 00 00 00 00 48 69 73 65 6e 73 65 2d 4c 65 6f 78
	 20 20 20 20 00 00 00 00 4c 58 54 2d 30 31 30 53
	 2d 48 20 20 20 20 20 20 31 2e 30 20 05 1e 00 1a
	 00 1a 00 00 4c 45 4f 58 44 31 30 30 30 32 45 46
	 20 20 20 20 32 32 31 31 30 34 20 20 68 f0 08 5e
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00	
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ISP fiber connected to sfp module:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: OPT8
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether d4:20:00:b1:2b:3d
	inet6 fe80::d620:ff:feb1:2b3d%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (SC)
	vendor: Hisense-Leox PN: LXT-010S-H SN: LEOXD10002EF DATE: 2022-11-04
	module temperature: 37.05 C voltage: 3.25 Volts
	lane 1: RX power: 0.01 mW (-21.74 dBm) TX bias: 0.00 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 02 00 00 00 00 03 0d 00 14 c8
	 00 00 00 00 48 69 73 65 6e 73 65 2d 4c 65 6f 78
	 20 20 20 20 00 00 00 00 4c 58 54 2d 30 31 30 53
	 2d 48 20 20 20 20 20 20 31 2e 30 20 05 1e 00 1a
	 00 1a 00 00 4c 45 4f 58 44 31 30 30 30 32 45 46
	 20 20 20 20 32 32 31 31 30 34 20 20 68 f0 08 5e
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

stephenw10

Hmm, it shows as active only when the fiber is not connected?

It only shows whatever the module is passing. Those GPON SFP modules are a full router/modem in the module.
Do you know how it's expected to be configured? It's probably got a user interface on some default IP.

milindhvijay

No, the Hisense-Leox LXT-010S-H GPON SFP does not have any routing functionality. It functions in pure transparent mode, converts optical to eletric. VLAN tagging, PPPoE is done on pfSense.

I tested with another GPON SPF:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:e2:69:56:98:dc
	hwaddr d4:20:00:b1:2b:3d
	inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
	inet6 fe80::2e2:69ff:fe56:98dc%ix0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 BASE-PX (SC)
	vendor: HSGQ PN: HSGQ-XPON-Stick SN: XPON23120861 DATE: 2023-12-21
	module temperature: 46.43 C voltage: 3.15 Volts
	lane 1: RX power: 0.01 mW (-22.15 dBm) TX bias: 14.60 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 01 00 00 00 80 00 00 00 00 01 0c 00 14 c8
	 00 00 00 00 48 53 47 51 20 20 20 20 20 20 20 20
	 20 20 20 20 00 00 00 00 48 53 47 51 2d 58 50 4f
	 4e 2d 53 74 69 63 6b 20 30 30 30 20 05 1e 00 e7
	 00 1c 00 00 58 50 4f 4e 32 33 31 32 30 38 36 31
	 20 20 20 20 32 33 31 32 32 31 20 20 68 b0 03 fe
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This one has both RX and TX signal when connecting ISP fiber (which means OLT is able to authenticate the GPON module, but with or without ISP fiber connected the status is always no carrier with no lights.

Both these GPON SFP modules get authenticated by the ISP OLT as I am able to get them working with a media converter.

This concludes that the issue is not with SFP modules, but with the compatibility of 82599ES with pfSense or some setting change that I am not aware of.

If nothing works can you suggest a good replacement x4 SFP PCI card that has high compatibility with pfSense?

stephenw10

That Hisense device is a router running Linux with 32MB of RAM: https://hack-gpon.org/ont-leox-lxt-010s-h/

It may be configured to pass the connection transparently but it may not. It looks like it has a default admin page at 192.168.100.1.

You might try setting the NIC to 1G fixed rate to get link. Some modules will only negotiate at 10G.

milindhvijay

This particular Hisense device, although has a default admin page, it is only used for firmware upgrades, and monitoring PON signal strength. There is no routing function on it. Also, this module only negotiates at either 1G or 2.5G and I have set it to 1G via telnet.

My ISP uses PPPoE, and when I use this GPON SFP with a media converter, I tag VLAN and authenticate PPP on pfSense. Gets public IP on pfsense, and no double NAT.

stephenw10

@milindhvijay said in Intel 82599ES; GPON SFP module not working:

this module only negotiates at either 1G or 2.5G

Actually negotiate or use a fixed speed? If it's fixed the NIC won't link to it if it's set to auto.

milindhvijay

This is what I have set it to:

To change the link mode use this command:

# /etc/scripts/flash get LAN_SDS_MODE
LAN_SDS_MODE=0
# /etc/scripts/flash set LAN_SDS_MODE 1

Value 	cat /proc/kmsg 	Mode 	Behavior
1 	<4>change mode to 1(Fiber 1G) 	FIBER 	1GbaseX with auto-neg on

stephenw10

Hmm, and that's definitely the SFP side of the module?

How are you accessing that, in the media converter?

milindhvijay

c0dde983-a428-4964-a352-044a77643d94-Screenshot 2024-01-14 at 00.03.22.png

Yes, currently sfp connected to a 1G media converter-> connected with rj-45 to pfsense. Parameter (LAN_SDS_MODE=1) set via telnet.

stephenw10

Hmm, yet neither module presents as 1G capable in the NIC. Auto-only.

When you boot with the module inserted does it show any errors in the boot log?

milindhvijay

ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0x3020-0x303f mem 0x50c20000-0x50c3ffff,0x50c44000-0x50c47fff at device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: d4:20:00:b1:2b:3d
ix0: PCI Express Bus: Speed 5.0GT/s Unknown
ix0: eTrack 0x800003de PHY FW V65535
ix0: netmap queues/slots: TX 4/2048, RX 4/2048

stephenw10

Hmm, no error there. There's not really much that can be set for those NICs beyond what you tried already.

Does really only show an active link when the upstream fiber is disconnected?

milindhvijay

@stephenw10 Yes, only shows active when ISP fiber is disconnected?

Would updating the driver help? If so, how can I do it?

stephenw10

I doubt it would make any difference but you could try to either build it yourself or install the kmod port for ixgbe. It may be marginally newer. Finding a version that loads can be challenging though since it has to closely match the kernel.

When the optic is disconnected do both modules show as active?

In that state can you connect to their cli through pfSense?

milindhvijay

No, Hisense-Leox LXT-010S-H shows active and power(green) and activity(amber) indication light is constantly ON when optic is disconnected. With optic connected, lights come ON in constant intervals, stays ON for 2-3 seconds and turns OFF. Without optics I am able to get into the admin page and telnet works.

HSGQ-XPON-Stick shows no carrier with or without optic. Even the power/activity lights are off always. Never even seen (rxpause, txpause) in interfaces. If I remember right, it always stay on autoselect.

I think HSGQ-XPON-Stick does not work because it is a 2.5G stick and AFAIK Intel x520 supports only 1G and 10G. I've tried forcing it to 1G but no luck.

stephenw10

Indeed only the x550 support N-Base and I think it's only N-BaseT. I don't have one to test with.

You can try setting the advertise speeds to 1G only in the negotiation:

[24.03-DEVELOPMENT][root@8200.stevew.lan]/root: sysctl -d dev.ix.0.advertise_speed
dev.ix.0.advertise_speed: 
Control advertised link speed using these flags:
	0x1 - advertise 100M
	0x2 - advertise 1G
	0x4 - advertise 10G
	0x8 - advertise 10M

	0x10  - advertise 2.5G
	0x20  - advertise 5G

	100M and 10M are only supported on certain adapters.

That still requires a compatible module though so may not allow it.

milindhvijay

[2.7.2-RELEASE][admin@Gaia.home]/root: sysctl -d dev.ix.0.advertise_speed
dev.ix.0.advertise_speed:
Control advertised link speed using these flags:
	0x1 - advertise 100M
	0x2 - advertise 1G
	0x4 - advertise 10G
	0x8 - advertise 10M

	0x10  - advertise 2.5G
	0x20  - advertise 5G

	100M and 10M are only supported on certain adapters.

How do I set the flag?

stephenw10

Like:

[24.03-DEVELOPMENT][root@8200.stevew.lan]/root: sysctl dev.ix.2.advertise_speed=1
dev.ix.2.advertise_speed: 11 -> 1

But it has to have a device that supports those speeds. ix2 there is a combo port so it supports 1G, 100M, 10M baseT.

milindhvijay

[2.7.2-RELEASE][admin@Gaia.home]/root: sysctl dev.ix.0.advertise_speed=1
dev.ix.0.advertise_speed: 0
sysctl: dev.ix.0.advertise_speed=1: Invalid argument

stephenw10

Mmm, it may not work. You would need to set it to 2 there though. 1 is 100M which is almost certainly not supported on that NIC.