10G Odyssey

Voidrunner

Hi folks,

I am trying since a few weeks to upgrade my Pfsense box to 10G. Before buying adapters, I looked up the support in the freeBSD documentation. I am running Pfsense 2.4.5-RELEASE-p1.

When I used an HP NC523, based on the qle3242, I got a link, the interface went up, the packets would be forwarded. Managing the box through the 10G interface was possible.

BUT. No DHCP responses were sent. None at all, even when I just switched the interface on the console menu - without touching the DHCP config.

Plus when I rebooted, the driver would throw errors that basically froze the whole box.

Next try. Intel X520, based on the 82599 Intel chipset. Card is recognized as ix0. But whatever I do, no carrier. Changed the card to another 82599 based card, same story recognized as Intel card - no carrier. I tried generic FS.com SFP+s and DAC, I tried Intel compatible FS.com SFP+s. No luck. (the HP card was running fine with fs.com generic DAC).

Bildschirmfoto 2021-01-23 um 11.42.40.png

Bildschirmfoto 2021-01-23 um 11.40.37.png

Pfsense runs on a HP Microserver G8, I use Mikrotik CSS610-8G-2S+IN and CRS309-1G-8S+IN switches, both on the latest release of SwitchOS.

What am I missing?

TIA,
-V.

stephenw10

Try running at the command line ifconfig -vvvvvvm ix0

Does it show the SFP module or DAC as present?

What are you connecting it to?

Steve

Voidrunner

@stephenw10

It says the SFP is present:

ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 64:9d:99:b1:67:aa
	hwaddr 64:9d:99:b1:67:aa
	inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 tentative scopeid 0x1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-SR
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: FS PN: SFP-10GSR-85 SN: F2020575588 DATE: 2020-07-14
Class: 10G Base-SR
Length: short distance
Tech: (null)
Media: Multimode, 50um
Speed: (null)
	module temperature: 28.56 C Voltage: 3.31 Volts
	RX: 0.54 mW (-2.64 dBm) TX: 0.00 mW (-40.00 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 
	1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 00 00 53 46 50 2D 31 30 47 53 
	52 2D 38 35 20 20 20 20 20 20 20 20 03 52 00 AF 
	00 1A 0A 58 46 32 30 32 30 35 37 35 35 38 38 20 
	20 20 20 20 32 30 30 37 31 34 20 20 68 F0 03 35 
	00 00 08 80 6D A0 B8 90 B4 AB F0 FB 5E 05 4C D7 
	F8 01 CA 00 00 00 00 00 00 00 00 00 8A 8B 53 70

The card is connected via 3m OC4 to the SFP on the Switch.

The switch recognizes the SFP.

stephenw10

It looks to be logically down.

It's not transmitting: TX: 0.00 mW (-40.00 dBm)

Try running ifconfig ix0 up then rechecking. Or assigning and enabling the interface in pfSense.

Steve

Voidrunner

@stephenw10

No change after I configure it as OPT1 in Pfsense.

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 64:9d:99:b1:67:aa
	hwaddr 64:9d:99:b1:67:aa
	inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1
	inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-SR
	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
	vendor: FS PN: SFP-10GSR-85 SN: F2020575588 DATE: 2020-07-14
Class: 10G Base-SR
Length: short distance
Tech: (null)
Media: Multimode, 50um
Speed: (null)
	module temperature: 30.99 C Voltage: 3.31 Volts
	RX: 0.54 mW (-2.64 dBm) TX: 0.68 mW (-1.62 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 
	1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 00 00 53 46 50 2D 31 30 47 53 
	52 2D 38 35 20 20 20 20 20 20 20 20 03 52 00 AF 
	00 1A 0A 58 46 32 30 32 30 35 37 35 35 38 38 20 
	20 20 20 20 32 30 30 37 31 34 20 20 68 F0 03 35 
	00 00 08 80 6D A0 B8 90 B4 AB F0 FB 5E 05 4C D7

stephenw10

@voidrunner said in 10G Odyssey:

TX: 0.68 mW (-1.62 dBm)

Except that it is now actually transmitting. You might try reconnecting the fibre and/or the SFP module.

Does the switch show the link as UP? Do you see any link LEDs?

An x520 NIC would usually 'just work' unless the remote side is set to 1G or otherwise not set to autonegotiate the speed.

Steve

Voidrunner

@stephenw10 said in 10G Odyssey:

@voidrunner said in 10G Odyssey:

TX: 0.68 mW (-1.62 dBm)

Except that it is now actually transmitting. You might try reconnecting the fibre and/or the SFP module.

Does the switch show the link as UP? Do you see any link LEDs?

An x520 NIC would usually 'just work' unless the remote side is set to 1G or otherwise not set to autonegotiate the speed.

Steve

I've tried using an intel compatible DAC - still no carrier

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 64:9d:99:b1:67:aa
	hwaddr 64:9d:99:b1:67:aa
	inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1
	inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10Gbase-Twinax
	plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
	vendor: FS PN: SFPP-PC01 SN: S1910031607-1 DATE: 2019-10-24
Class: 1X Copper Passive
Length: (null)
Tech: Passive Cable
Media: Twin Axial Pair
Speed: 100 MBytes/sec

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 21 01 00 00 04 00 04 80 D5 00 67 00 00 00 
	00 00 01 00 46 53 20 20 20 20 20 20 20 20 20 20 
	20 20 20 20 00 00 00 00 53 46 50 50 2D 50 43 30 
	31 20 20 20 20 20 20 20 44 20 20 20 0C 00 00 31 
	00 12 00 31 53 31 39 31 30 30 33 31 36 30 37 2D 
	31 20 20 20 31 39 31 30 32 34 20 20 00 00 00 C1 
	00 D8 20 20 20 20 20 20 66 61 20 20 20 20 20 20 
	20 20 20 20 FF FF FF FF FF FF FF FF FF FF FF FF

Voidrunner

Switch shows recognizes the DAC and shows the link to be up. Bildschirmfoto 2021-01-23 um 15.37.38.png

stephenw10

Is it hard set to 10G at the switch? You need to match both ends so if it try either setting it to auto in the switch or setting the speed to 10G fixed in pfSense.

Steve

Voidrunner

@stephenw10

Actually it's autoselect on the Switch.

I even tried setting the interface on PfSense to 10GB SR
Bildschirmfoto 2021-01-23 um 16.00.45.png

But the adapter seems to remain in auto select

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
ether 64:9d:99:b1:67:aa
hwaddr 64:9d:99:b1:67:aa
inet6 fe80::669d:99ff:feb1:67aa%ix0 prefixlen 64 scopeid 0x1
inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
supported media:
media autoselect
media 10Gbase-SR
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FS PN: SFP-10GSR-85 SN: F2020521292 DATE: 2020-07-15
Class: 10G Base-SR
Length: short distance
Tech: (null)
Media: Multimode, 50um
Speed: (null)
module temperature: 37.13 C Voltage: 3.33 Volts
RX: 0.69 mW (-1.61 dBm) TX: 0.67 mW (-1.72 dBm)

SFF8472 DUMP (0xA0 0..127 range):
03 04 07 10 00 00 00 40 00 0C 00 06 67 00 00 00 
1E 0F 00 00 46 53 20 20 20 20 20 20 20 20 20 20 
20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 53 
52 2D 38 35 20 20 20 20 41 20 20 20 03 52 00 0C 
00 1A 00 00 46 32 30 32 30 35 32 31 32 39 32 20 
20 20 20 20 32 30 30 37 31 35 20 20 68 90 01 61 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Voidrunner

Update - the problem was with the Mikrotik CSS610-8G-2S+. When switching with the very same config to the CRS309-1G-8S+IN, the links comes up fine.

stephenw10

Ah, nice catch! Weird though.