unsupported SFP+ error

[federicop]

I purchased this model https://www.amazon.it/gp/product/B0C4H4CFSQ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
and everything works perfectly.

After about a week of testing, I decided to purchase a second identical model to install, and that's when the problems started.

Configuration:
2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT

Hardware:
Intel(R) Pentium(R) Gold G5420 CPU @ 3.80GHz
Current: 3492 MHz, Max: 3800 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
QAT Crypto: No

Motherboard: Asus Prime B365-PLUS [link to Asus website]
(I had disassembled it some time ago) with two PCIe3 slots (16x).

Now, the first card works perfectly, while the second one doesn't. Same model, same everything... or rather, I correct the second one. If I start pfSense with the SFP+ connector, it gives an unsupported SFP+ error. So, I modified the /boot/loader.conf.local file by adding the line hw.ix.unsupported_sfp="1" as described in several discussions (without solving the problem). If I connect SFP+ after startup, it shows Unknown <rxpause,txpause> but it works.

Could someone help me with this issue?

The first card, when I connect SFP+, shows 10Gbase-Twinax <full-duplex,rxpause,txpause>.

Thank you

[stephenw10]

That's using the same SFP module?

If so those NICs must not be identical. Check the reported firmware versions in the boot log or sysctls.

Steve

[federicop]

@stephenw10

the cable is identical but not the same.

the dmesg log is this (I notice the PCI data which is different even though the manual says that the slots are identical - ix0: PCI Express Bus: Speed 5.0GT/s Width x8 - ix3: PCI Express Bus: Speed 5.0GT/s Width x4)

ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xe020-0xe03f mem 0xf0280000-0xf02fffff,0xf0304000-0xf0307fff irq 16 at device 0.0 on pci1
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: a0:36:9f:37:a9:0c
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: Option ROM V1-b1018-p0 eTrack 0x800006d1 PHY FW V65535
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xe000-0xe01f mem 0xf0200000-0xf027ffff,0xf0300000-0xf0303fff irq 17 at device 0.1 on pci1
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: a0:36:9f:37:a9:0d
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: Option ROM V1-b1018-p0 eTrack 0x800006d1 PHY FW V65535
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf4000000-0xf4ffffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video devi

login-to-view ix2: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xa020-0xa03f mem 0xf0080000-0xf00fffff,0xf0104000-0xf0107fff irq 16 at device 0.0 on pci5
ix2: Using 2048 TX descriptors and 2048 RX descriptors
ix2: Using 4 RX queues 4 TX queues
ix2: Using MSI-X interrupts with 5 vectors
ix2: allocated for 4 queues
ix2: allocated for 4 rx queues
ix2: Ethernet address: a0:36:9f:37:a9:ae
ix2: PCI Express Bus: Speed 5.0GT/s Width x4
ix2: Option ROM V1-b1018-p0 eTrack 0x800006d1 PHY FW V65535
ix2: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xa000-0xa01f mem 0xf0000000-0xf007ffff,0xf0100000-0xf0103fff irq 17 at device 0.1 on pci5
ix3: Using 2048 TX descriptors and 2048 RX descriptors
ix3: Using 4 RX queues 4 TX queues
ix3: Using MSI-X interrupts with 5 vectors
ix3: allocated for 4 queues
ix3: allocated for 4 rx queues
ix3: Ethernet address: a0:36:9f:37:a9:af
ix3: PCI Express Bus: Speed 5.0GT/s Width x4
ix3: Option ROM V1-b1018-p0 eTrack 0x800006d1 PHY FW V65535
ix3: netmap queues/slots: TX 4/2048, RX 4/2048
pcib7: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 29.3 on pci0
pci6: <ACPI PCI bus> on pcib8

[stephenw10]

Hmm, OK that looks like the same Option ROM and firmware version.

OK I would swapping the cables so you are actually using the same cable. There's a chance that's coded differently.

[federicop]

@stephenw10 said in unsupported SFP+ error:

Hmm, OK that looks like the same Option ROM and firmware version.

OK I would swapping the cables so you are actually using the same cable. There's a chance that's coded differently.

so the cable could have different codecs? Meaning what?
I use this cable https://www.amazon.it/gp/product/B01MRPJ60I/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1

the question ix0: PCI Express Bus: Speed ​​5.0GT/s Width x8 - ix3: PCI Express Bus: Speed ​​5.0GT/s Width x4) you say it has no effect?

[stephenw10]

SFP modules, and hence the ends of DAC cables, have some electronics in them to identify themselves. They can often be coded to work with specific switches for example. The cables you have may not be coded the same.

[federicop]

@stephenw10

nice discovery ! so it could depend on the cable!

a curiosity.... but can I connect and disconnect the cables while hot or do I always have to turn off the devices?

tks

[stephenw10]

They should be hot swappable but if you only see that error when booting with it already connected you'd need to do that to test.

[federicop]

@stephenw10

OK. Changed the cable and it seems to work. I see speed in pfsense dashboad and I see speed in vmware esxi.

but if I do the IPFER speed test I don't go beyond 5 Gb/s.

login-to-view

test pfsense 2.7.2 Vs Winserver2019 VM vmwareEsxi 6.5

[stephenw10]

Can that Win Server VM push more than that?

Try testing through pfSense instead of to/from it directly; that will always give a lower result.

One iperf instance is single threaded so you may be hitting a limit on one CPU core.

[federicop]

@stephenw10

What do you mean by testing directly from pfsense?

I installed operf on pfsese and run the command to reach the win server.

The WIN server (VM) is mounted on server a 4 CPUs x Intel(R) Xeon(R) CPU E3-1225 v6 @ 3.30GHz and configured with
CPU Cores 4
Sockets 2
Cores for Socket 2
Memory 50GB
MSSQL server (Heavier service)

[stephenw10]

Right, if you are running iperf3 on pfSense you will always see a lower result because it's optimised as a router not a server.

To see what it's capable of you need to test through it. So run the iperf server and client on two different devices connected to two separate interfaces in pfSense.