SFP+ ixgbe Network Interface Won't Come Back Up



  • Hi all,

    I've had some trouble over the last few months with the SFP+ onboard network interfaces on my Supermicro 5018D-F8NT pfSense system. These interfaces use the ixgbe driver that comes with FreeBSD. Occasionally upon a system reboot or even by just briefly dropping the link (by disconnecting and reconnecting the fiber cable) the interface will stay down and not come back up (i.e. there is no link). The only way to resolve this issue is to shutdown the system completely and remove the power. Upon restart, the link returns and everything is fine. The most recent occurrence of this came after upgrading switch firmware a few days ago. The SFP+ link was dropped briefly as the switch needed to reboot, but never came back up after the switch had rebooted. Here again, shutting down the machine and unplugging the power solved the problem. I have been doing some searching online, but all I could find was this, which suggests issues with the ixgbe driver:

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221317
    https://github.com/opnsense/core/issues/2591

    Has anyone seen this issue or have any suggestions on how I might be able to fix it? I also have a 4 port Chelsio SFP+ card in this same system which does not exhibit this same behavior. Thanks in advance for your help, I really appreciate it.


  • Netgate Administrator

    I have seen similar behaviour to that if you connect an unsupported SFP module to an ixgbe port. The chip somehow gets into a mode that can only be cleared by completely depowering it.

    With the wrong module in it the driver fails to attach so you can't do much at all. I assume here you can still see the NIC it just never shows link? In which case it might be possible to reset it in software.
    You've tried resaving the interface settings with the media type etc?

    Steve



  • @stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:

    I have seen similar behaviour to that if you connect an unsupported SFP module to an ixgbe port. The chip somehow gets into a mode that can only be cleared by completely depowering it.

    With the wrong module in it the driver fails to attach so you can't do much at all. I assume here you can still see the NIC it just never shows link? In which case it might be possible to reset it in software.
    You've tried resaving the interface settings with the media type etc?

    Steve

    Thanks @stephenw10 - I am currently using transceiver modules from FS.com with the ixgbe SFP+ ports vs. Intel modules. Perhaps using actual Intel SFP+ transceivers is worth trying out in this case. I just looked on Ebay and it seems like they are quite readily available and fairly affordable so I'm going to give this a shot.

    And yes, I can still the NIC fine after the link drops. Even trying to resave the interface unfortunately makes no difference.

    Thanks again for your help.



  • Got the new Intel SFP+ transceivers installed today. Tried to cycle the interface a few times by disconnecting and reconnecting the fiber cable into the transceiver and didn't lose the link. Also rebooted the firewall to see if I would lose the link, but all stayed up. Fingers crossed it was as simple as that :).



  • Well, unfortunately the SFP+ modules did not fix the problem. Rebooted the firewall the other day and ran into these same issues again (had to completely power cycle the unit to get the interface back). At this point I'm led to believe this may be a driver issue - hopefully it will be sorted in FreeBSD 12.x / pfSense 2.5.


  • Netgate Administrator

    Are you able to test a 2.5 snapshot?

    Now is the time to report issues with that if you can. It may be something upstream in the driver of course. We are limited in what we can do if that's the case.

    Steve



  • @stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:

    Are you able to test a 2.5 snapshot?

    Now is the time to report issues with that if you can. It may be something upstream in the driver of course. We are limited in what we can do if that's the case.

    Steve

    It might be a bit tough as this is my only production system - how stable are the snapshots right now? Thanks again.


  • Netgate Administrator

    Pretty stable. I've not seen a significant problem in months. But very hard to advise you run it in productions though.

    Steve



  • @stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:

    Pretty stable. I've not seen a significant problem in months. But very hard to advise you run it in productions though.

    Steve

    Thanks @stephenw10 - I suppose technically it's not really a "production" system since it's on my home network (and the number of people inconvenienced would be very low in case of any downtime). Would you say that for this kind of scenario the snapshots are probably stable enough? I suppose worst case I can always start over with a fresh 2.4.4 install and restore from backup. How easy is it to go back to the stable branch once 2.5.0 is released - is it just a matter of switching back over in the UI and installing the Release version over the last 2.5.0 snapshot (or more likely Release Candidate)? Thanks again.


  • Netgate Administrator

    I would say they are stable enough for that. As I said I've had numerous boxes up running 2.5 here for some time without any significant issues. Of course what issues there are might be more significant to you. 😉

    You can't easily downgrade in place, you have to reinstall 2.4.4p3. But if you have the media and a config standing by that's pretty easily accomplished. If you had to.

    Steve


Log in to reply