SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8
-
How do I configure pFsense to move the WAN port from my Intel 1Gbase-T copper port to the SFP+ port? Is this as simple as "assign interfaces" in the console, perhaps followed by a reboot?
I'll be switching from igb0 to ixl3 for WAN. -
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
How do I configure pFsense to move the WAN port from my Intel 1Gbase-T copper port to the SFP+ port? Is this as simple as "assign interfaces" in the console, perhaps followed by a reboot?
I'll be switching from igb0 to ixl3 for WAN.Yes, you just need to switch the assigned port for WAN from igb0 to ixl3 - can do this from the GUI as well (Interfaces -> Assignments). No need to reboot.
Hope this helps.
-
The LAN chip driving the SFP+ ports came out in 2015, from what I can tell. It is the Intel X557-AT2. The datasheet only mentions 10Gbe. Does that it mean it won't work with the multi-gig transceiver I ordered? Apparently IEEE 802.3bz came out in 2016. Does the LAN chip also need to support multi-gig, not just the SFP+transceiver? Sigh . . .
Perhaps I'll need to use the link aggregation option of the Netgear CM3000, as it has dual 1Gbase-T ports in addition to the 2.5Gbase-T port. I wonder how difficult link aggregation is to set up?
-
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
The LAN chip driving the SFP+ ports came out in 2015, from what I can tell. It is the Intel X557-AT2
I do have a 10Gbit SFP+ WAN uplink, but working at 10Gbit.
But I do have Mikrotk switch(es) with Mikrotik S+RJ10 copper modules. And AFAI see the module connects with 10Gbit to the switch and the transceiver will auto-neg the speed (1/2.5/5/10) with the client. The SFP+ don't run at different speeds (at least mine don't), but in your case it doesn't have to since the transceiver will do that.
Btw: According to the spec sheet from Supermicro that board got two SFP+ and two 10Gbase-T ports. The X557-AT2 is a two port 10Gbase-T card and it supports multi-speed but only 100/1000/10000. And the two SFP+ are connected to the SOC (couldn't find the model). I assume that's why you use the SFP+ port with a transceiver.
Long story short: the LAN chip won't have to support multi-gig, I'd say.
-
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
The LAN chip driving the SFP+ ports came out in 2015, from what I can tell. It is the Intel X557-AT2. The datasheet only mentions 10Gbe. Does that it mean it won't work with the multi-gig transceiver I ordered? Apparently IEEE 802.3bz came out in 2016. Does the LAN chip also need to support multi-gig, not just the SFP+transceiver? Sigh . . .
Perhaps I'll need to use the link aggregation option of the Netgear CM3000, as it has dual 1Gbase-T ports in addition to the 2.5Gbase-T port. I wonder how difficult link aggregation is to set up?
Another option (although a bit more expensive) would be to use a SFP+ to RJ45 media converter that supports multi-gig on the RJ45 side:
https://www.fs.com/c/copper-to-fiber-media-converters-1038
Hope this helps.
-
@tman222 I might aggregate the two 1GBase-T ports on the cable modem into my pFsense router before buying a media converter, as I only need 2Gb to support my internet download speed.
-
@patient0 what sort of internet setup do you have that provides a 10Gbit port, fiber? I wish they would run FTTH here, and I could just plug in an SFP+ "fiber stick" into pFsense for the WAN connection.
-
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
@tman222 wow that's a lot of $$$. No doubt it consumes significant power as well.
I think I would aggregate the two 1GBase-T ports on the cable modem into my pFsense router before buying a media converter, as I only need 2Gb to support my internet download speed.
Compared to the transceiver module you linked above, yes, about 1.25W more power consumption.
-
@tman222 my comment on the price was due to them serving me a foreign web page. The price is reasonable, although not inexpensive. We'll see how the multi-gig SFP+ transceiver works in my 10Gb SFP+ cage.
-
@patient0 said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
Btw: According to the spec sheet from Supermicro that board got two SFP+ and two 10Gbase-T ports. The X557-AT2 is a two port 10Gbase-T card and it supports multi-speed but only 100/1000/10000. And the two SFP+ are connected to the SOC (couldn't find the model). I assume that's why you use the SFP+ port with a transceiver.
Long story short: the LAN chip won't have to support multi-gig, I'd say.
That's a good point, the x557 chip is to drive copper 10GBase-T cables. The copper (or fiber) drive circuit for SFP+ is in the transceiver, the whole point of the SFP+cage allowing swappable hardware.
-
@lifespeed it's Switzerland, FTTH and a small ISP which let you bring your own router. I could get 25Gbit for the same price but that is just a bit overkill :)
They tell you what specs fiber optics you need (10G SFP+ BIDI LR, TX1270/RX1330 nm for 10Gbit) or you buy it from them. Then you plug it in and voila.
-
@patient0 impressive! I would be tempted to get the 25Gbps service, but I doubt there is a small form factor motherboard suitable for pFsense router duty that supports more than 10Gbps SFP+. So probably a huge expense for appropriate hardware, not to mention power consumption. So probably doesn't make sense, as you said.
-
The FS 1G/2.5G/5G/10Gbps NBASE-T SFP+ USXGMII Copper 30m RJ-45 Transceiver Module arrived. I powered down the Supermicro/pFsense router, plugged it in, and configured pFsense to use ixl3 (2nd SFP+ 1/10Gb SFP+ cage) for the WAN port. There was no internet access, nor did the LEDs on for the 2nd SFP+ port light up like they do for the 1st SFP+ port for the 10Gb direct-attach cable for the LAN. The new FS 1/2.5/5/10Gb transceiver was powered and warm, but apparently is not compatible with the older 1\10Gb motherboard.
So I guess I need to get a converter like the FS Mini Unmanaged 1x 100M/1G/2.5G/5G/10GBase-T RJ45 to 1x 10GBase-X SFP+ Slot 10Gigabit Ethernet Media Converter. The question is, will I find myself in the same situation where the newer multi-gig protocol is incompatible with the older 1/10Gb protocol? The labeling of the converter does imply it is converting internally to 1/10Gb SFP+. If so, I can just use a direct-attach cable, so the cost is reasonable as no SFP+ transceivers need to be purchased.
I decided against link aggregation, I don't want to introduce any more complexity into pFsense. As it is, I am barely clever enough to make it work.
-
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
The FS 1G/2.5G/5G/10Gbps NBASE-T SFP+ USXGMII Copper 30m RJ-45 Transceiver Module arrived. I powered down the Supermicro/pFsense router, plugged it in, and configured pFsense to use ixl3 (2nd SFP+ 1/10Gb SFP+ cage) for the WAN port. There was no internet access, nor did the LEDs on for the 2nd SFP+ port light up like they do for the 1st SFP+ port for the 10Gb direct-attach cable for the LAN. The new FS 1/2.5.5.10Gb transceiver was powered and warm, but apparently is not compatible with the older 1\10Gb motherboard.
So I guess I need to get a converter like the FS Mini Unmanaged 1x 100M/1G/2.5G/5G/10GBase-T RJ45 to 1x 10GBase-X SFP+ Slot 10Gigabit Ethernet Media Converter. The question is, will I find myself in the same situation where the newer multi-gig protocol is incompatible with the older 1/10Gb protocol? The labeling of the converter does imply it is converting internally to 1/10Gb SFP+. If so, I can just use a direct-attach cable, so the cost is reasonable as no SFP+ transceivers need to be purchased.
I decided against link aggregation, I don't want to introduce any more complexity into pFsense. As it is, I am barely clever enough to make it work.
I think what you propose should work, but I would probably doublecheck with FS support to be certain before spending that kind of money. Apologies for not being able to provide a more direct answer - while I am currently using one of these media converters, it is on a symmetric 10gig/10gig link (10gig SFP+ to 10gig RJ-45), and for that it has worked great.
-
@lifespeed I would try a few more things with the FS transceiver module. Maybe just insert the module without changing WAN to it for the moment.
Was there anything related to this module in the logs, the reason it didn't accept it?
What is the output of
ifconfig -vvvmm ixl3and what doessysctl dev.ixl.3.supported_speedsshow? -
@patient0 good suggestion, although I suspect the final answer may remain the same.
ifconfig -vvvmm ixl3sysctl dev.ixl.3.supported_speedsixl2 is the SFP+ DAC cable from pFsense to my 10Gb switch. ixl3 is the FS 1/2/5/5/10Gb transceiver. These two lines from the boot log appear to confirm it isn't going to work:
ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5 ixl3: ixl_attach_get_link_status get link status, status: I40E_ERR_UNKNOWN_PHY aq_err=I40E_AQ_RC_EIOHere is the full boot log referencing ixl3.
ixl3: netmap queues/slots: TX 4/1024, RX 4/1024 ixl3: SR-IOV ready ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5 ixl3: ixl_attach_get_link_status get link status, status: I40E_ERR_UNKNOWN_PHY aq_err=I40E_AQ_RC_EIO ixl3: Allocating 4 queues for PF LAN VSI; 4 queues active ixl3: Ethernet address: 00:25:90:bb:bf:0d ixl3: Using MSI-X interrupts with 5 vectors ixl3: Using 4 RX queues 4 TX queues ixl3: Using 1024 TX descriptors and 1024 RX descriptors ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C ixl3: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001006 oem 1.262.0 ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xf7000000-0xf7ffffff,0xfb000000-0xfb007fff irq 46 at device 0.3 numa-domain 0 on pci12 ixl2: netmap queues/slots: TX 4/1024, RX 4/1024 ixl2: SR-IOV ready ixl2: Allocating 4 queues for PF LAN VSI; 4 queues active ixl2: Ethernet address: 00:25:90:bb:bf:0c ixl2: Using MSI-X interrupts with 5 vectors ixl2: Using 4 RX queues 4 TX queues ixl2: Using 1024 TX descriptors and 1024 RX descriptors ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C ixl2: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001006 oem 1.262.0 ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xf8000000-0xf8ffffff,0xfb008000-0xfb00ffff irq 46 at device 0.2 numa-domain 0 on pci12 -
@lifespeed that "Error getting phy capabilities" indicate a driver issue or a hardware issue? Would you know that?
pfSense does recognise the transceiver correctly, the speed too (6 == 1G / 10G).
Would be interesting to boot up with a Ubuntu Live CD to see if it's a driver or hardware issue. But since it's your router that's probably not very practical.
-
@patient0 I would think I could boot using a live USB stick as this is a small form factor PC with accessible USB and display ports. However, I'm not a linux expert and don't know how to proceed as to identifying a driver vs hardware issue.
The transceiver is 1/2.5/5/10Gb, not 1/10Gb, so I'm not sure it is being recognized correctly.
Is pFsense capable of using a 1/2.5/5/10Gb interface?
-
@lifespeed said in SFP+ multi-gig copper twisted pair transceiver for Supermicro X11SDV-4C-TP8:
The transceiver is 1/2.5/5/10Gb, not 1/10Gb, so I'm not sure it is being recognized correctly.
I would have assumed that the host links at 10GB with the transceiver and the transceiver negotiates with the client between 1/2.5/5/10Gbit (but the host<->transceiver stays at 10G).
Maybe we can be so bold and involve @stephenw10?
-
@patient0 The link speed must proceed as negotiated. This means a 2.5Gb link can't transceive at the physical layer at a different speed, nor can a 10Gb link. In order for a 2.5Gb link to communicate with a 10Gb link, it must buffer enough to fill a frame, then transmit at the 10Gb physical layer speed.
I'm almost certain buffering is not part of Ethernet chip functionality, and must happen externally. I don't think this functionality is included in an SFP+ transceiver, either. Probably that is the difference between a "media converter" and an SFP+ transceiver.