SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8

lifespeed

@patient0 impressive! I would be tempted to get the 25Gbps service, but I doubt there is a small form factor motherboard suitable for pFsense router duty that supports more than 10Gbps SFP+. So probably a huge expense for appropriate hardware, not to mention power consumption. So probably doesn't make sense, as you said.

lifespeed

The FS 1G/2.5G/5G/10Gbps NBASE-T SFP+ USXGMII Copper 30m RJ-45 Transceiver Module arrived. I powered down the Supermicro/pFsense router, plugged it in, and configured pFsense to use ixl3 (2nd SFP+ 1/10Gb SFP+ cage) for the WAN port. There was no internet access, nor did the LEDs on for the 2nd SFP+ port light up like they do for the 1st SFP+ port for the 10Gb direct-attach cable for the LAN. The new FS 1/2.5/5/10Gb transceiver was powered and warm, but apparently is not compatible with the older 1\10Gb motherboard.

So I guess I need to get a converter like the FS Mini Unmanaged 1x 100M/1G/2.5G/5G/10GBase-T RJ45 to 1x 10GBase-X SFP+ Slot 10Gigabit Ethernet Media Converter. The question is, will I find myself in the same situation where the newer multi-gig protocol is incompatible with the older 1/10Gb protocol? The labeling of the converter does imply it is converting internally to 1/10Gb SFP+. If so, I can just use a direct-attach cable, so the cost is reasonable as no SFP+ transceivers need to be purchased.

I decided against link aggregation, I don't want to introduce any more complexity into pFsense. As it is, I am barely clever enough to make it work.

tman222

@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:

The FS 1G/2.5G/5G/10Gbps NBASE-T SFP+ USXGMII Copper 30m RJ-45 Transceiver Module arrived. I powered down the Supermicro/pFsense router, plugged it in, and configured pFsense to use ixl3 (2nd SFP+ 1/10Gb SFP+ cage) for the WAN port. There was no internet access, nor did the LEDs on for the 2nd SFP+ port light up like they do for the 1st SFP+ port for the 10Gb direct-attach cable for the LAN. The new FS 1/2.5.5.10Gb transceiver was powered and warm, but apparently is not compatible with the older 1\10Gb motherboard.

So I guess I need to get a converter like the FS Mini Unmanaged 1x 100M/1G/2.5G/5G/10GBase-T RJ45 to 1x 10GBase-X SFP+ Slot 10Gigabit Ethernet Media Converter. The question is, will I find myself in the same situation where the newer multi-gig protocol is incompatible with the older 1/10Gb protocol? The labeling of the converter does imply it is converting internally to 1/10Gb SFP+. If so, I can just use a direct-attach cable, so the cost is reasonable as no SFP+ transceivers need to be purchased.

I decided against link aggregation, I don't want to introduce any more complexity into pFsense. As it is, I am barely clever enough to make it work.

I think what you propose should work, but I would probably doublecheck with FS support to be certain before spending that kind of money. Apologies for not being able to provide a more direct answer - while I am currently using one of these media converters, it is on a symmetric 10gig/10gig link (10gig SFP+ to 10gig RJ-45), and for that it has worked great.

patient0

@lifespeed I would try a few more things with the FS transceiver module. Maybe just insert the module without changing WAN to it for the moment.

Was there anything related to this module in the logs, the reason it didn't accept it?

What is the output of ifconfig -vvvmm ixl3 and what does sysctl dev.ixl.3.supported_speeds show?

lifespeed

@patient0 good suggestion, although I suspect the final answer may remain the same.

ifconfig -vvvmm ixl3

ifconfig -vvvmm ixl3.png

sysctl dev.ixl.3.supported_speeds

sysctl dev.ixl.3.supported_speeds.png

ixl2 is the SFP+ DAC cable from pFsense to my 10Gb switch. ixl3 is the FS 1/2/5/5/10Gb transceiver. These two lines from the boot log appear to confirm it isn't going to work:

ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl3: ixl_attach_get_link_status get link status, status: I40E_ERR_UNKNOWN_PHY aq_err=I40E_AQ_RC_EIO

Here is the full boot log referencing ixl3.

ixl3: netmap queues/slots: TX 4/1024, RX 4/1024
ixl3: SR-IOV ready
ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl3: ixl_attach_get_link_status get link status, status: I40E_ERR_UNKNOWN_PHY aq_err=I40E_AQ_RC_EIO
ixl3: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl3: Ethernet address: 00:25:90:bb:bf:0d
ixl3: Using MSI-X interrupts with 5 vectors
ixl3: Using 4 RX queues 4 TX queues
ixl3: Using 1024 TX descriptors and 1024 RX descriptors
ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl3: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001006 oem 1.262.0
ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xf7000000-0xf7ffffff,0xfb000000-0xfb007fff irq 46 at device 0.3 numa-domain 0 on pci12
ixl2: netmap queues/slots: TX 4/1024, RX 4/1024
ixl2: SR-IOV ready
ixl2: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl2: Ethernet address: 00:25:90:bb:bf:0c
ixl2: Using MSI-X interrupts with 5 vectors
ixl2: Using 4 RX queues 4 TX queues
ixl2: Using 1024 TX descriptors and 1024 RX descriptors
ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl2: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001006 oem 1.262.0
ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xf8000000-0xf8ffffff,0xfb008000-0xfb00ffff irq 46 at device 0.2 numa-domain 0 on pci12

patient0

@lifespeed that "Error getting phy capabilities" indicate a driver issue or a hardware issue? Would you know that?

pfSense does recognise the transceiver correctly, the speed too (6 == 1G / 10G).

Would be interesting to boot up with a Ubuntu Live CD to see if it's a driver or hardware issue. But since it's your router that's probably not very practical.

lifespeed

@patient0 I would think I could boot using a live USB stick as this is a small form factor PC with accessible USB and display ports. However, I'm not a linux expert and don't know how to proceed as to identifying a driver vs hardware issue.

The transceiver is 1/2.5/5/10Gb, not 1/10Gb, so I'm not sure it is being recognized correctly.

Is pFsense capable of using a 1/2.5/5/10Gb interface?

patient0

@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:

The transceiver is 1/2.5/5/10Gb, not 1/10Gb, so I'm not sure it is being recognized correctly.

I would have assumed that the host links at 10GB with the transceiver and the transceiver negotiates with the client between 1/2.5/5/10Gbit (but the host<->transceiver stays at 10G).

Maybe we can be so bold and involve @stephenw10?

lifespeed

@patient0 The link speed must proceed as negotiated. This means a 2.5Gb link can't transceive at the physical layer at a different speed, nor can a 10Gb link. In order for a 2.5Gb link to communicate with a 10Gb link, it must buffer enough to fill a frame, then transmit at the 10Gb physical layer speed.

I'm almost certain buffering is not part of Ethernet chip functionality, and must happen externally. I don't think this functionality is included in an SFP+ transceiver, either. Probably that is the difference between a "media converter" and an SFP+ transceiver.

lifespeed

Having confirmed the link speed translation capability with FS I have ordered the media converter and will report back with results, hopefully a successful WAN link at 2.5Gb.