1Gb SFP not coming up on Intel x710 10Gb SFP+ interface

tfboy

OK, I now have some "supported" 1Gb SFPs I've bought from FS.com: I've found the actual RX and TX wavelengths and these SFPs do work. If I put them into my ISP's router, the link comes up and I have internet access.

But, if I plug them into my Protectli, the SFP is seen, but is just stuck with "no carrier".

ifconfig -vvvm ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:ca:e5:58:4f:a7
        hwaddr 64:62:66:22:88:35
        inet6 fe80::f6ca:e5ff:fe58:4fa7%ixl0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 BASE-BX10 (LC)
        vendor: FS PN: SFP-GE-BX SN: G2330160894 DATE: 2023-09-22
        module temperature: 32.98 C voltage: 3.24 Volts
        lane 1: RX power: 0.07 mW (-11.28 dBm) TX bias: 10.70 mA

        SFF8472 DUMP (0xA0 0..127 range):
         03 04 07 00 00 00 40 00 00 00 00 01 0d 00 0a 64
         00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20
         20 20 20 20 00 64 9d 99 53 46 50 2d 47 45 2d 42
         58 20 20 20 20 20 20 20 20 20 20 20 05 1e 20 c9
         20 0a 00 00 47 32 33 33 30 31 36 30 38 39 34 20
         20 20 20 20 32 33 30 39 32 32 20 20 68 90 01 80
         20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
         20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00

There's apparently no media or speed selection possible (other than autoselect).
Any other ideas?

stephenw10

Can you set the advertised speed to 1G only?

tfboy

@stephenw10 unfortunately I don't think I can. The SFP I have is supported. I no longer have any errors like I had before.

But with this SFP inserted, the only speed offered is "autoselect".

Interestingly, if I take the SFP out, then I get all the options shown:

ifconfig -vvvm ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:ca:e5:58:4f:a7
        hwaddr 64:62:66:22:88:35
        inet6 fe80::f6ca:e5ff:fe58:4fa7%ixl0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
                media 10Gbase-LR
                media 10Gbase-SR
                media 10Gbase-Twinax
                media 1000baseLX
                media 1000baseSX
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

But with no SFP inserted, if I try to set the speed, for example:

ifconfig ixl0 up debug media 1000baseSX
ifconfig: SIOCSIFMEDIA (media): Operation not supported by device

I get the same "not supported" message with the SFP inserted.

In the WebUI, with no SFP inserted, I get the same list of media options, but setting them then plugging the SFP in doesn't do anything. I guess it's not really setting the media type as that cannot be achieved from the shell.

However, a 1Gb SFP (RJ45) does work straight away (I didn't need to try to set a speed) and the SFP I want to use works in the ISP's router so I can't understand why it's not working here.

I've spent a long time searching why and the "no carrier" issues often appear to be related to speed or setting / forcing something, but I've not had any success so far.

stephenw10

It shows as invalid if you try to set it to advertise 1G only?

tfboy

@stephenw10 said in 1Gb SFP not coming up on Intel x710 10Gb SFP+ interface:

It shows as invalid if you try to set it to advertise 1G only?

How would I do that?

stephenw10

sysctl dev.ixl.0.advertise_speed=2 probably

tfboy

@stephenw10 Thanks for sticking with me Stephen!

I'm wondering ig the issue is due to Intel being picky with their list of "qualified" modules. Although I wasn't getting any warnings unlike with the previous SFP, setting the advertised speed looked like it was OK. However I just noticed on the console (I was connecting sending commands over ssh) that it still comes up with

ixl0: Link fialed because an unqualified module was detected!

I've read that you can try to allow this to not be an issue with a tunable, but I'm not sure if it will work here. I will play around a little more, but my guess is if the speed needs to be set to 1Gb via advertising that speed only and that command will only really work if the system sees a "qualified" module, I might not be able to get any further.

It's rather annoying as having spoken to Protectli, they said preferably Intel or MSA coded, it should be fine. So I went ahead and purchased some allegedly compatible SFP (Generic MSA one). But it looks like that's not the case, at least with this Intel driver used in FreeBSD / pfSense. Maybe I'll have to buy an Intel-coded one specifically.

Can you think of anything else I can try? I'll have a go with the unsupported tweaks in the /boot/loader.conf.local file.

stephenw10

You may as well try the unsupported modules loader line. Nothing to lose there. I'd expect it to fail to read the module entirely though in that case.

tfboy

@stephenw10 well it didn't change anything. I tried

hw.ix.unsupported_sfp=1
hw.ixl.unsupported_sfp=1

No change through. Inguess that's the only thing for "unqualified" modules?

stephenw10

That's the only thing I'm aware of. And as I said that usually only applies when modules are not recognised at all.

The only other thing is that i2c_access_method value I mentioned. But I've never seen that used.

tfboy

@stephenw10 Hi Stephen,
Just to say after so much trouble with this, I learnt that a work colleague of mine had the FS BOX which I just borrowed to recode the SFPs to Intel. As soon as I had done that and plugged them in, they came to life immediately without any need for tunables or other tweaks to get the link working. It's working on autoselect / autodetect with a fresh install of pfSense.

Thank you so much for helping me out with this, but I hope this thread is of use to others: any recent FreeBSD install or security appliance designed on top of FreeBSD using Intel SFPs, you really need Intel-coded SFPs otherwise it just won't work!