1Gb SFP not coming up on Intel x710 10Gb SFP+ interface

stephenw10

Do you only see those logged when the module is present?

Mmm, I'm not sure what what we can do about that. I don't think ixl has any tunables that would help.

Hmm, actually what firmware version is on those NICs?
sysctl dev.ixl.1.fw_version

You could try changing the i2c access method
sysctl hw.ixl.i2c_access_method

I've never seen that required though.

tfboy

sysctl dev.ixl.1.fw_version
dev.ixl.1.fw_version: fw 6.1.49420 api 1.7 nvm 6.80 etid 80003cf2 oem 255.65535.255

and

hw.ixl.i2c_access_method
hw.ixl.i2c_access_method: 0

stephenw10

Hmm, there's probably a newer firmware version available for that. The ixl card I have shows: dev.ixl.0.fw_version: fw 7.0.50775 api 1.8 nvm 7.00 etid 8000521b oem 0.0.0
You'd probably need to boot it in Linux or Windows to upgrade that.

You could try adding a loader value to select a different 12c method:

hw.ixl.i2c_access_method:
I2C access method that driver will use:
	0 - best available method
	1 - bit bang via I2CPARAMS register
	2 - register read/write via I2CCMD register
	3 - Use Admin Queue command (best)
Using the Admin Queue is only supported on 710 devices with FW version 1.7 or higher

Create the file /boot/loader.conf.local then add the line to it like:
hw.ixl.i2c_access_method=1

tfboy

Aha. Some progress! :)
I updated the firmware to the latest version (9.40) and now the SFP is detected and shows up in the list!

 ifconfig -vvvm ixl1
ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 64:62:66:22:88:36
        inet6 fe80::6662:66ff:fe22:8836%ixl1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
                media 10Gbase-LR
                media 10Gbase-SR
                media 10Gbase-Twinax
                media 1000baseLX
                media 1000baseSX
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 BASE-BX10 (SC)
        vendor: EZconn Corp. PN: ETB43315-7S34-FR SN: 618500117372 DATE: 2018-12-13
        module temperature: 47.30 C voltage: 3.32 Volts
        lane 1: RX power: 0.07 mW (-11.26 dBm) TX bias: 13.00 mA

        SFF8472 DUMP (0xA0 0..127 range):
         03 04 01 00 00 00 40 00 00 00 00 01 0d 00 04 28
         00 00 00 00 45 5a 63 6f 6e 6e 20 43 6f 72 70 2e
         20 20 20 20 00 00 00 00 45 54 42 34 33 33 31 35
         2d 37 53 33 34 2d 46 52 00 00 00 00 05 1e 00 12
         00 1a 00 00 36 31 38 35 30 30 31 31 37 33 37 32
         20 20 20 20 31 38 31 32 31 33 20 20 68 f0 03 ce
         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Still no link and "no carrier" as can be seen, but the SFP is now discovered and even picks up the TX and RX power with the fibre attached.

edited to add I still see the ixl1_set_link: Error getting phy capabilities -37, aq error: 8 message.

tfboy

Also, to provide some more info, output of dmesg (I put sfp into ixl0)

 dmesg |grep ixl
ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0x6000800000-0x6000ffffff,0x6001008000-0x600100ffff at device 0.0 on pci2
ixl0: fw 9.140.76856 api 1.15 nvm 9.40 etid 8000ecd6 oem 255.65535.255
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 10 RX queues 10 TX queues
ixl0: Using MSI-X interrupts with 11 vectors
ixl0: Ethernet address: 64:62:66:22:88:35
ixl0: Error getting supported media types, err -37, AQ error 8
ixl0: Allocating 16 queues for PF LAN VSI; 10 queues active
ixl0: ixl_attach_get_link_status get link status, status: I40E_ERR_ADMIN_QUEUE_ERROR aq_err=I40E_AQ_RC_EAGAIN
ixl0: PCI Express Bus: Speed 8.0GT/s Width x4
ixl0: ixl_set_advertised_speeds: Error getting phy capabilities -37, aq error: 8
ixl0: ixl_set_initial_advertised_speeds: ixl_set_advertised_speeds() error 5
ixl0: ixl_set_link: Error getting phy capabilities -37, aq error: 8
ixl0: ixl_if_attach_post get link status, status: I40E_ERR_ADMIN_QUEUE_ERROR aq_err=I40E_AQ_RC_EAGAIN
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 10/1024, RX 10/1024

I'm wondering if it's because the SFP isn't on the "compatibility list" so it's being forced down.

stephenw10

It could be. I'd expect it to show a specific 'unsupported' error though. Also there is no sysctl/loader value to force unsupported modules as far as I know.

Does it still refuse to set a link type?

tfboy

@stephenw10 thank you for hanging in their with my noob abilities!
I'm not sure how to set that properly.
If I try via ifconfig ixl0 media 1000baseSX for example, I get the Operation not supported by device.
If I'm to try to set the advertise_speed via sysctl, how do I do that exactly ?

stephenw10

@tfboy said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

If I try via ifconfig ixl0 media 1000baseSX for example, I get the Operation not supported by device.

Yup, I'd expect that to work since it's listed in the ifconfig output.

I assume the setting in the GUI also fails since that does the same thing.

tfboy

@stephenw10 said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

@tfboy said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

If I try via ifconfig ixl0 media 1000baseSX for example, I get the Operation not supported by device.

Yup, I'd expect that to work since it's listed in the ifconfig output.

I assume the setting in the GUI also fails since that does the same thing.

I think I've understood how that speeds listed works: it appears that the interface lists the supported speeds once the SFP has been inserted.
For example, when I insert my 10G-LR Cisco SFP, there are two options that are listed: autoselect and 10Gbase-LR.
When I plug my 1Gb SFP, I only get the autoselect option so my understanding is the values are those presented to the interfae by the SFP.
So the mention of all the speeds earlier was incorrect as it was before I had inserted the SFP so the interface was listing all it could do but not what the SFP was then limited to saying it could do.
I fear that unless there's a way to bypass that, then I'm effectively stuck It's even more infuriating now as since upgrading the firmware, the interfae can see and communicate with the SFP, it just doesn't bring the carrier link up.

If there are any more ideas to try, then I'm all ears.

But maybe I'll just have to buy another SFP from fs.com which is likely to be more "compatible" than the bespoke one provided by my ISP.
I just can't figure out the RX wavelength as it's not listed. TX is 1310nm but no indication that the RX is. Either 1490 or 1550nm. Is there any way I can find out?

stephenw10

Hmm, it clearly still shows all the media speeds above with the module inserted. Which is odd.

You could try setting the advertised speeds to 1G only again.

I'd expect to see the frequencies written on the module. But if not the spec sheet should show it.

stephenw10

@tfboy said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

ETB43315-7S34-FR

Like:

tfboy

@stephenw10 Yes, that's the one! I also wondered about the RX wavelength as mine doesn't have it listed. Good to know it's 1550nm.

I've been able to do some more testing with different SFP / SFP+ and the results are quite interesting.

Here's what I've tried:

1Gb MMF Cisco SFP:

It's detected with an ifconfig -vvvm
I can't actually get a link as I don't have any physical other hardware with SFP ports

1Gb RJ45 Cisco SFP:

It's detected with an ifconfig -vvvm
The link light does not come on however, the interface does come up at the other end (plugging a Cat6 cable into a server)! No traffic appears to pass

10Gb RJ45 FS SFP+:

It's detected with an ifconfig -vvvm
The link light does come on. The interface comes up. It's negotiated at 1Gb as the server on the other end is only 1Gb. There is traffic, interface appears to be functioning correctly, IP address obtained from remote DHCP server, etc.

When using pfSemse, if I try to set any speed with the ifconfig command, it just gets refused:

# ifconfig ixxl0 media 1000baseSX
ifconfig: SIOCSIFMEDIA (media): Operation not supported by device

From the above tests, it appears that the interfaces will NOT operate with standard 1Gb SFP. They only work with 10Gb SFP+, even if the link is negotiated to a slower 1Gb speed. Any attempt to set / force the speed manually via ifconfig is visible not allowed.

As it stands, it's a shame. Despite the hardware being sold as SFP / SFP+ compatible, it appears it will only work with SFP+ modules and even if a link appears to come up with a 1Gb SFP module, it will still not allow the OS to access it. This now really feels more like a hardware limitation than an OS / software configuration issue.

I've raised this with Protectli who are looking into it. I don't know if there's anything more that can be done at the pfSense / FreeBSD level.

stephenw10

Mmm, from a high level I would expect to need to set the link speed at 1G. And that may require a dual speed module.

I have a 1G BiDi module like that running in an 8200 but it will only run in the combo ports where the link is forced to 1G.

tfboy

OK, I now have some "supported" 1Gb SFPs I've bought from FS.com: I've found the actual RX and TX wavelengths and these SFPs do work. If I put them into my ISP's router, the link comes up and I have internet access.

But, if I plug them into my Protectli, the SFP is seen, but is just stuck with "no carrier".

ifconfig -vvvm ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:ca:e5:58:4f:a7
        hwaddr 64:62:66:22:88:35
        inet6 fe80::f6ca:e5ff:fe58:4fa7%ixl0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 BASE-BX10 (LC)
        vendor: FS PN: SFP-GE-BX SN: G2330160894 DATE: 2023-09-22
        module temperature: 32.98 C voltage: 3.24 Volts
        lane 1: RX power: 0.07 mW (-11.28 dBm) TX bias: 10.70 mA

        SFF8472 DUMP (0xA0 0..127 range):
         03 04 07 00 00 00 40 00 00 00 00 01 0d 00 0a 64
         00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20
         20 20 20 20 00 64 9d 99 53 46 50 2d 47 45 2d 42
         58 20 20 20 20 20 20 20 20 20 20 20 05 1e 20 c9
         20 0a 00 00 47 32 33 33 30 31 36 30 38 39 34 20
         20 20 20 20 32 33 30 39 32 32 20 20 68 90 01 80
         20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
         20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00

There's apparently no media or speed selection possible (other than autoselect).
Any other ideas?

stephenw10

Can you set the advertised speed to 1G only?

tfboy

@stephenw10 unfortunately I don't think I can. The SFP I have is supported. I no longer have any errors like I had before.

But with this SFP inserted, the only speed offered is "autoselect".

Interestingly, if I take the SFP out, then I get all the options shown:

ifconfig -vvvm ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:ca:e5:58:4f:a7
        hwaddr 64:62:66:22:88:35
        inet6 fe80::f6ca:e5ff:fe58:4fa7%ixl0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
                media 10Gbase-LR
                media 10Gbase-SR
                media 10Gbase-Twinax
                media 1000baseLX
                media 1000baseSX
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

But with no SFP inserted, if I try to set the speed, for example:

ifconfig ixl0 up debug media 1000baseSX
ifconfig: SIOCSIFMEDIA (media): Operation not supported by device

I get the same "not supported" message with the SFP inserted.

In the WebUI, with no SFP inserted, I get the same list of media options, but setting them then plugging the SFP in doesn't do anything. I guess it's not really setting the media type as that cannot be achieved from the shell.

However, a 1Gb SFP (RJ45) does work straight away (I didn't need to try to set a speed) and the SFP I want to use works in the ISP's router so I can't understand why it's not working here.

I've spent a long time searching why and the "no carrier" issues often appear to be related to speed or setting / forcing something, but I've not had any success so far.

stephenw10

It shows as invalid if you try to set it to advertise 1G only?

tfboy

@stephenw10 said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

It shows as invalid if you try to set it to advertise 1G only?

How would I do that?

stephenw10

sysctl dev.ixl.0.advertise_speed=2 probably

tfboy

@stephenw10 Thanks for sticking with me Stephen!

I'm wondering ig the issue is due to Intel being picky with their list of "qualified" modules. Although I wasn't getting any warnings unlike with the previous SFP, setting the advertised speed looked like it was OK. However I just noticed on the console (I was connecting sending commands over ssh) that it still comes up with

ixl0: Link fialed because an unqualified module was detected!

I've read that you can try to allow this to not be an issue with a tunable, but I'm not sure if it will work here. I will play around a little more, but my guess is if the speed needs to be set to 1Gb via advertising that speed only and that command will only really work if the system sees a "qualified" module, I might not be able to get any further.

It's rather annoying as having spoken to Protectli, they said preferably Intel or MSA coded, it should be fine. So I went ahead and purchased some allegedly compatible SFP (Generic MSA one). But it looks like that's not the case, at least with this Intel driver used in FreeBSD / pfSense. Maybe I'll have to buy an Intel-coded one specifically.

Can you think of anything else I can try? I'll have a go with the unsupported tweaks in the /boot/loader.conf.local file.