XG-1537 + FS.com media converter = No DHCP

iklier

I'm curious if anyone has gotten an FS.com media converter to work with the XG-1537. I'm seeing an issue where pfsense never gets a DHCP address on the WAN interface.

Shared Setup Info

Netgate XG-1537 (pfsense 22.05-RELEASE)
LAN (ix1) SFP+ DAC
WAN (ix0)
WAN connecting to Adtran 622 ONT
Sonic.com 10Gbps service (IPv4 only service at the moment)

Working Setup:
WAN (ix0) using Ubiquiti SFP+ 10GbE SFP+
-[Cat6]-> ONT 10GbE port

[22.05-RELEASE][admin@pfSense.home.arpa]/root: ifconfig -vvvv ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 3c:ec:ef:c0:34:a6
	inet6 fe80::3eec:efff:fec0:34a6%ix0 prefixlen 64 scopeid 0x1
	inet <redacted> netmask 0xfffff800 broadcast 135.180.239.255
	media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 10G Base-SR (RJ45)
	vendor: Ubiquiti Inc. PN: CM-RJ45-10G SN: AX21104201795 DATE: 2021-09-23

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 22 10 00 00 00 20 40 04 80 06 67 00 00 00 
	00 00 1E 00 55 62 69 71 75 69 74 69 20 49 6E 63 
	2E 20 20 20 00 24 5A 4C 43 4D 2D 52 4A 34 35 2D 
	31 30 47 20 20 20 20 20 41 31 20 20 00 00 00 6F 
	00 1A 00 00 41 58 32 31 31 30 34 32 30 31 37 39 
	35 20 20 20 32 31 30 39 32 33 20 20 00 00 00 B4 
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Failing Setup:
WAN (ix0) using FS.COM Intel or GE 10GBASE-SR SFP+
-[OM4]-> FS.com Unmanaged SFP+ to 10GbE media converter using FS.COM FS or GE 10GBASE-SR SFP+ module
-[Cat6]-> ONT 10GbE port
I should not that initially I didn't get a carrier signal with fiber SFP+ module, but power cycling the XG-1537 (halt + 10sec power disconnect + reboot) resolved that specific issue.

[22.05-RELEASE][admin@pfSense.home.arpa]/root: ifconfig -vvvv ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 3c:ec:ef:c0:34:a6
	inet6 fe80::3eec:efff:fec0:34a6%ix0 prefixlen 64 scopeid 0x1
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
	media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: FS PN: SFP-10GSR-85 SN: G2130577363 DATE: 2022-01-13
	module temperature: 32.79 C Voltage: 3.32 Volts
	RX: 0.50 mW (-2.98 dBm) TX: 0.57 mW (-2.39 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 00 00 00 00 00 06 67 02 00 00 
	08 03 00 1E 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 53 
	52 2D 38 35 20 20 20 20 41 20 20 20 03 52 00 BE 
	00 3A 00 00 47 32 31 33 30 35 37 37 33 36 33 20 
	20 20 20 20 32 32 30 31 31 33 20 20 68 FA 03 F4 
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Working Test Setup: (Yes, this is double-NAT, but it was a experiment to test the SFPs, but it work, pfsense pulls an address from the UDM Pro DHCP pool and routing work.):
WAN (ix0) using FS.COM Intel or GE 10GBASE-SR SFP+
-[OM4]-> Ubiquiti UDM Pro LAN SFP+ using FS.com GE 10GBASE-SR SFP+ module
UDM Pro WAN Ubiquiti SFP+ 10GbE SFP+
-[Cat6]-> ONT 10GbE port

Additional Working Setup: (using Ubiquiti UDM Pro and FS.com SFP+ modules)
UDM Pro WAN SFP+ using FS.COM GE 10GBASE-SR SFP+
-[OM4]-> FS.com Unmanaged SFP+ to 10GbE media converter using FS.COM FS 10GBASE-SR SFP+ module
-[Cat6]-> ONT 10GbE port

stephenw10

Hmm, interesting. If anything I would have expected the 10Gbase-T SFP+ module to give more problems.

Try running a pcap on ix0 in the failing setup and see if anything is coming in at all.

If you do see traffic check what tagging is on those packets.

It's possible one of those combinations is adding or incorrectly tags which is know to prevent the dhcp client working. Though I would not expect it with something that should simply be a media converter.

Steve

iklier

I did a non-promiscuous packet capture on ix0 with the failing setup and I only see the ix0 interface trying to get DHCP by sending requests to broadcast.

I suspect there is just some weird auto-negotiation issue with between the SFP+ modules when one is plugged into the media converter.

I ordered a SFP+ DAC to see if that works between ix0 and the media converter box. While not ideal, it should at least be slightly cooler than the 10GbE SFP+ module.

stephenw10

Hmm, odd. Possibly some flow control issue? It's enabled in both cases though.
Any reason you used non-promiscuous mode? I would usually enable that to be sure I'm not missing anything.

iklier

Only had a brief window for the network outage and so I just used the defaults. If I can manage some more downtime, I'll try promiscuous mode.

iklier

Got a long enough SFP+ DAC cable to try, and it works fine between the 1537 and FS.com media converter.
So the failure with using 10GBASE-SR must just be some auto-negotiation issue with those SFP+ modules, that only occurs with the media converter setup.