Intel X552 Onboard SFP+ No Carrier Issues



  • Hi all,

    I run pfSense on a Supermicro 5018D-F8NT server for a couple years without major issues until today:

    https://www.supermicro.com/products/system/1u/5018/SYS-5018D-FN8T.cfm

    The system is basically stock, but I did decide to install a Chelsio T520 dual 10Gbit fiber card in the system's only PCIe expansion slot. The Chelsio card works great and is connected to a Ubiquiti ES-16-XG 10Gbit switch on the other side. Today I wanted to use the two onboard SFP+ ports as well, but no matter what I try I cannot get a link established on either port; the status on ix0 and ix1 says "no carrier".

    Here's what I've tried so far:

    1. Check dmesg and system.log for errors - unfortunately I did not find anything there and the X552 is detected just fine on boot.
    2. Try different fiber patch cables - still no link. Connect fiber cables between switch and Chelsio card - link established, so cables are fine.
    3. Tried to connect onboard SFP+ (ix0 and ix1) to Chelsio SFP+ card - no link.
    4. Tried co connect ix0 to ix1 - no link.
    5. Check that SFP+ modules in switch are working by connecting cable between each module in the switch and Chelsio card - everything linked up fine.
    6. Checked to make sure the SFP+ modules are detected by the onboard Intel X552 ports via ifconfig -v ix0 and ifconfig -v ix1 -- both modules show up as detected and operating.
    7. Tried setting media type manually (instead of autoselect) via: ifconfig ix0 media 10GBase-SR and ifconfig ix1 media 10GBase-SR and received this back: ifconfig: SIOCSIFMEDIA (media): Invalid argument. In system.log, it also says "kernel: ix0: Invalid media type!" (similar for ix1). Doing a ifconfig -m ix0 and ifconfig -m ix1 shows 10Gbase-SR as supported media, so this is rather perplexing.

    I'm now pretty much out of ideas as to what else to try. I did have ix0 and ix1 working just fine in early 2018 (on an earlier version of pfSense) using the exact same SFP+ modules that are currently installed, so I really don't think it's the modules.

    I wanted to ask the community if anyone had any idea what to try next or whether they have experienced a similar problem with the ix driver in FreeBSD. Thanks in advance for your help, I really appreciate it.


  • Netgate Administrator

    What modules exactly are you using?

    I've seen issues with determining the link speed on ix but not with showing a carrier. Can we see the verbose output from ifconfig?

    I assume you're running 2.4.4p2?

    Steve



  • @stephenw10 said in Intel X552 Onboard SFP+ No Carrier Issues:

    What modules exactly are you using?

    I've seen issues with determining the link speed on ix but not with showing a carrier. Can we see the verbose output from ifconfig?

    I assume you're running 2.4.4p2?

    Steve

    Hi @stephenw10 - Thanks a lot for responding. I ended up figuring it out (and in hindsight I probably should have tried this earlier in the troubleshooting process): What I had to do is power down the firewall AND pull the power cord (i.e. completely remove power from the unit). Once I connected the power again and pressed the on button both onboard SFP+ ports lit up immediately and I had a connection. I did try just shutting down earlier and restarting WITHOUT cutting the power completely and that made no difference (i.e. I still didn't have a connection on those two ports).

    I'm not 100% sure why this was required, but maybe it has something to do with IPMI? I do recall moving the firewall a couple notches down in my rack at some point last year and when I turned everything back on I did not have anything to those two SFP+ ports. Perhaps the system automatically disables them then?

    In any case, I hope this helps others who might experiencing a similar problem. I guess sometimes sometimes "turning it off and back on again" is the best solution afterall. :)


  • Netgate Administrator

    Ah, interesting. I have seen that with an incompatible SFP module but never with just the connection like that. But yes in that case simply removing the module and rebooting was not sufficient to clear the state in the NIC. It had to be completely powered down to see the correct module again.

    Steve


Log in to reply