1Gb SFP not coming up on Intel x710 10Gb SFP+ interface
-
I'm wondering if anyone can suggest if I can get this working.
I'm running pfSense on a Protectli Vault 6 with a pair of Intel X710 SFP+ interfaces.In one of the ports, I plan to use my ISP-provided SFP which runs at 1Gb speed. Normally, the SFP plugs into the router provided by my ISP but I want to use pfSense for OpenVPN capabilities and a set of other features.
The SFP works in a Cisco switch with 1Gb SFP ports once I enable the third-party vendor compatibility. The link comes up.
In my pfSense box though, no link comes up.
Here are the interfaces:
ixl0 64:62:66:22:88:35 (down) Intel(R) Ethernet Controller X710 for 10GbE SF ixl1 64:62:66:22:88:36 (down) Intel(R) Ethernet Controller X710 for 10GbE SF igc0 64:62:66:22:88:37 (up) Intel(R) Ethernet Controller I226-V igc1 64:62:66:22:88:38 (down) Intel(R) Ethernet Controller I226-V igc2 64:62:66:22:88:39 (down) Intel(R) Ethernet Controller I226-V igc3 64:62:66:22:88:3a (down) Intel(R) Ethernet Controller I226-V
I want to use either of the 10Gb SFP+ ports with the 1Gb SFP.
ifconfig ixl1 ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 64:62:66:22:88:36 inet6 fe80::6662:66ff:fe22:8836%ixl1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect status: no carrier nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Looking at the alleged media support:
ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 64:62:66:22:88:36 inet6 fe80::6662:66ff:fe22:8836%ixl1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-LR media 10Gbase-SR media 10Gbase-Twinax media 1000baseLX media 1000baseSX nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
I tried setting the speed manually, but it doesn't seem to like it.
ifconfig ixl1 media 1000baseLX ifconfig: SIOCSIFMEDIA (media): Operation not supported by device
The speed-setting command doesn't appear to be linked to the 1Gb SFP: I get the not supported message on either port, whether the SFP is inserted or not.
If I try setting things up with sysctl devices, I get similar "no can do" responses:
sysctl -d dev.ixl.1.advertise_speed dev.ixl.1.advertise_speed: Control advertised link speed. Flags: 0x1 - advertise 100M 0x2 - advertise 1G 0x4 - advertise 10G 0x8 - advertise 20G 0x10 - advertise 25G 0x20 - advertise 40G 0x40 - advertise 2.5G 0x80 - advertise 5G
and then:
sysctl dev.ixl.1.advertise_speed dev.ixl.1.advertise_speed: 6
which suggests it supports both 1G and 10G speeds.
But when I plug in the SFP, I get:
ixl1: Invalid advertised speed; valid flags are: 0x06
Does that suggest the SFP is running at a different. non-supported speed?
Any suggestions on what to try?
Thanks in advance.
-
Sorry, I forgot to specify. The SFP is an EZConn ETB43315-7S34-FR
-
ixl1: Invalid advertised speed; valid flags are: 0x06
It is perhaps a auto negotiation mismatch and could be
worked around if you set up not auto if it is able to
realize you may set it up to let us say 1000 MBit/s full -
@Dobby_ thanks. How would I do that?
-
The reported valid media should be dependent on the inserted module.
Do you see data back from the module if you run:
ifconfig -vvvm ixl1
? -
@stephenw10
It looks like it isn't picked up:ifconfig -vvvm ixl1 ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 64:62:66:22:88:36 inet6 fe80::6662:66ff:fe22:8836%ixl1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-LR media 10Gbase-SR media 10Gbase-Twinax media 1000baseLX media 1000baseSX nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
whereas if I plug an old Cisco 10Gb LR SFP in, it does appear:
ifconfig -vvvm ixl0 ixl0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 64:62:66:22:88:35 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-LR media 10Gbase-SR media 10Gbase-Twinax media 1000baseLX media 1000baseSX nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 10G Base-LR (LC) vendor: CISCO-FINISAR PN: FTLX1474D3BCL-CS SN: FNS16471S8X DATE: 2012-11-24 module temperature: 26.02 C voltage: 3.32 Volts lane 1: RX power: 0.00 mW (-25.38 dBm) TX bias: 30.30 mA SFF8472 DUMP (0xA0 0..127 range): 03 04 07 20 00 00 00 00 00 00 00 06 67 00 0a 64 00 00 00 00 43 49 53 43 4f 2d 46 49 4e 49 53 41 52 20 20 20 00 00 90 65 46 54 4c 58 31 34 37 34 44 33 42 43 4c 2d 43 53 41 20 20 20 05 1e 00 e5 00 1a 00 00 46 4e 53 31 36 34 37 31 53 38 58 20 20 20 20 20 31 32 31 31 32 34 20 20 68 e0 03 3d 00 00 02 36 3b 18 85 c7 13 1b 6d de b8 1c a9 da fa 21 3f 00 00 00 00 00 00 00 00 00 b7 14 a6 38
So I guess this suggests some incompatibility with the hardware or firmware or driver?
I was hoping that as the SFP was picked up in a Cisco switch with 1Gb SFP ports, it would work here.
It's true that in the list of supported media,
1000baseBX
isn't listed, but I don't know if that should be a problem or not. The Intel X710 documentation says 1000baseBX is supported. (it's a bidirectional / single-core SMF SFP). -
The fact it doesn't show BX usually isn't an issue. That module can only work in BiDi mode. It's not uncommon to see all 1G module shown as 1000SX for example. But it does have to see the module.
If you boot with the module inserted does it show any errors in the log when the driver attaches? Like 'unsupported module' for example.
-
@stephenw10 Hi Stephen, thanks for your answer.
Yes! I do get two things that pop up:ixl_admin_timer: i40e_get_link_status status I40_ERR_TIMEOUT, aq error I40E_AQ_RC_EAGAIN
and also
ixl1_set_link: Error getting phy capabilities -37, aq error: 8
I don't know if that means much and if there's a workaround. I did search a bit but couldn't find anything that seemed relevant.
-
Do you only see those logged when the module is present?
Mmm, I'm not sure what what we can do about that. I don't think ixl has any tunables that would help.
Hmm, actually what firmware version is on those NICs?
sysctl dev.ixl.1.fw_version
You could try changing the i2c access method
sysctl hw.ixl.i2c_access_method
I've never seen that required though.
-
sysctl dev.ixl.1.fw_version dev.ixl.1.fw_version: fw 6.1.49420 api 1.7 nvm 6.80 etid 80003cf2 oem 255.65535.255
and
hw.ixl.i2c_access_method hw.ixl.i2c_access_method: 0
-
Hmm, there's probably a newer firmware version available for that. The ixl card I have shows:
dev.ixl.0.fw_version: fw 7.0.50775 api 1.8 nvm 7.00 etid 8000521b oem 0.0.0
You'd probably need to boot it in Linux or Windows to upgrade that.You could try adding a loader value to select a different 12c method:
hw.ixl.i2c_access_method: I2C access method that driver will use: 0 - best available method 1 - bit bang via I2CPARAMS register 2 - register read/write via I2CCMD register 3 - Use Admin Queue command (best) Using the Admin Queue is only supported on 710 devices with FW version 1.7 or higher
Create the file /boot/loader.conf.local then add the line to it like:
hw.ixl.i2c_access_method=1
-
Aha. Some progress! :)
I updated the firmware to the latest version (9.40) and now the SFP is detected and shows up in the list!ifconfig -vvvm ixl1 ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 64:62:66:22:88:36 inet6 fe80::6662:66ff:fe22:8836%ixl1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect status: no carrier supported media: media autoselect media 10Gbase-LR media 10Gbase-SR media 10Gbase-Twinax media 1000baseLX media 1000baseSX nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> plugged: SFP/SFP+/SFP28 BASE-BX10 (SC) vendor: EZconn Corp. PN: ETB43315-7S34-FR SN: 618500117372 DATE: 2018-12-13 module temperature: 47.30 C voltage: 3.32 Volts lane 1: RX power: 0.07 mW (-11.26 dBm) TX bias: 13.00 mA SFF8472 DUMP (0xA0 0..127 range): 03 04 01 00 00 00 40 00 00 00 00 01 0d 00 04 28 00 00 00 00 45 5a 63 6f 6e 6e 20 43 6f 72 70 2e 20 20 20 20 00 00 00 00 45 54 42 34 33 33 31 35 2d 37 53 33 34 2d 46 52 00 00 00 00 05 1e 00 12 00 1a 00 00 36 31 38 35 30 30 31 31 37 33 37 32 20 20 20 20 31 38 31 32 31 33 20 20 68 f0 03 ce 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Still no link and "no carrier" as can be seen, but the SFP is now discovered and even picks up the TX and RX power with the fibre attached.
edited to add I still see the
ixl1_set_link: Error getting phy capabilities -37, aq error: 8
message. -
Also, to provide some more info, output of dmesg (I put sfp into ixl0)
dmesg |grep ixl ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0x6000800000-0x6000ffffff,0x6001008000-0x600100ffff at device 0.0 on pci2 ixl0: fw 9.140.76856 api 1.15 nvm 9.40 etid 8000ecd6 oem 255.65535.255 ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C ixl0: Using 1024 TX descriptors and 1024 RX descriptors ixl0: Using 10 RX queues 10 TX queues ixl0: Using MSI-X interrupts with 11 vectors ixl0: Ethernet address: 64:62:66:22:88:35 ixl0: Error getting supported media types, err -37, AQ error 8 ixl0: Allocating 16 queues for PF LAN VSI; 10 queues active ixl0: ixl_attach_get_link_status get link status, status: I40E_ERR_ADMIN_QUEUE_ERROR aq_err=I40E_AQ_RC_EAGAIN ixl0: PCI Express Bus: Speed 8.0GT/s Width x4 ixl0: ixl_set_advertised_speeds: Error getting phy capabilities -37, aq error: 8 ixl0: ixl_set_initial_advertised_speeds: ixl_set_advertised_speeds() error 5 ixl0: ixl_set_link: Error getting phy capabilities -37, aq error: 8 ixl0: ixl_if_attach_post get link status, status: I40E_ERR_ADMIN_QUEUE_ERROR aq_err=I40E_AQ_RC_EAGAIN ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 10/1024, RX 10/1024
I'm wondering if it's because the SFP isn't on the "compatibility list" so it's being forced down.
-
It could be. I'd expect it to show a specific 'unsupported' error though. Also there is no sysctl/loader value to force unsupported modules as far as I know.
Does it still refuse to set a link type?
-
@stephenw10 thank you for hanging in their with my noob abilities!
I'm not sure how to set that properly.
If I try viaifconfig ixl0 media 1000baseSX
for example, I get theOperation not supported by device
.
If I'm to try to set theadvertise_speed
via sysctl, how do I do that exactly ? -
@tfboy said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:
If I try via ifconfig ixl0 media 1000baseSX for example, I get the Operation not supported by device.
Yup, I'd expect that to work since it's listed in the ifconfig output.
I assume the setting in the GUI also fails since that does the same thing.
-
@stephenw10 said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:
@tfboy said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:
If I try via ifconfig ixl0 media 1000baseSX for example, I get the Operation not supported by device.
Yup, I'd expect that to work since it's listed in the ifconfig output.
I assume the setting in the GUI also fails since that does the same thing.
I think I've understood how that speeds listed works: it appears that the interface lists the supported speeds once the SFP has been inserted.
For example, when I insert my 10G-LR Cisco SFP, there are two options that are listed:autoselect
and10Gbase-LR
.
When I plug my 1Gb SFP, I only get theautoselect
option so my understanding is the values are those presented to the interfae by the SFP.
So the mention of all the speeds earlier was incorrect as it was before I had inserted the SFP so the interface was listing all it could do but not what the SFP was then limited to saying it could do.
I fear that unless there's a way to bypass that, then I'm effectively stuck It's even more infuriating now as since upgrading the firmware, the interfae can see and communicate with the SFP, it just doesn't bring the carrier link up.If there are any more ideas to try, then I'm all ears.
But maybe I'll just have to buy another SFP from fs.com which is likely to be more "compatible" than the bespoke one provided by my ISP.
I just can't figure out the RX wavelength as it's not listed. TX is 1310nm but no indication that the RX is. Either 1490 or 1550nm. Is there any way I can find out? -
Hmm, it clearly still shows all the media speeds above with the module inserted. Which is odd.
You could try setting the advertised speeds to 1G only again.
I'd expect to see the frequencies written on the module. But if not the spec sheet should show it.
-
-
@stephenw10 Yes, that's the one! I also wondered about the RX wavelength as mine doesn't have it listed. Good to know it's 1550nm.
I've been able to do some more testing with different SFP / SFP+ and the results are quite interesting.
Here's what I've tried:
1Gb MMF Cisco SFP:
- It's detected with an ifconfig -vvvm
- I can't actually get a link as I don't have any physical other hardware with SFP ports
1Gb RJ45 Cisco SFP:
- It's detected with an ifconfig -vvvm
- The link light does not come on however, the interface does come up at the other end (plugging a Cat6 cable into a server)! No traffic appears to pass
10Gb RJ45 FS SFP+:
- It's detected with an ifconfig -vvvm
- The link light does come on. The interface comes up. It's negotiated at 1Gb as the server on the other end is only 1Gb. There is traffic, interface appears to be functioning correctly, IP address obtained from remote DHCP server, etc.
When using pfSemse, if I try to set any speed with the ifconfig command, it just gets refused:
# ifconfig ixxl0 media 1000baseSX ifconfig: SIOCSIFMEDIA (media): Operation not supported by device
From the above tests, it appears that the interfaces will NOT operate with standard 1Gb SFP. They only work with 10Gb SFP+, even if the link is negotiated to a slower 1Gb speed. Any attempt to set / force the speed manually via ifconfig is visible not allowed.
As it stands, it's a shame. Despite the hardware being sold as SFP / SFP+ compatible, it appears it will only work with SFP+ modules and even if a link appears to come up with a 1Gb SFP module, it will still not allow the OS to access it. This now really feels more like a hardware limitation than an OS / software configuration issue.
I've raised this with Protectli who are looking into it. I don't know if there's anything more that can be done at the pfSense / FreeBSD level.