SFP+ ixgbe Network Interface Won't Come Back Up
-
Hi all,
I've had some trouble over the last few months with the SFP+ onboard network interfaces on my Supermicro 5018D-F8NT pfSense system. These interfaces use the ixgbe driver that comes with FreeBSD. Occasionally upon a system reboot or even by just briefly dropping the link (by disconnecting and reconnecting the fiber cable) the interface will stay down and not come back up (i.e. there is no link). The only way to resolve this issue is to shutdown the system completely and remove the power. Upon restart, the link returns and everything is fine. The most recent occurrence of this came after upgrading switch firmware a few days ago. The SFP+ link was dropped briefly as the switch needed to reboot, but never came back up after the switch had rebooted. Here again, shutting down the machine and unplugging the power solved the problem. I have been doing some searching online, but all I could find was this, which suggests issues with the ixgbe driver:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221317
https://github.com/opnsense/core/issues/2591Has anyone seen this issue or have any suggestions on how I might be able to fix it? I also have a 4 port Chelsio SFP+ card in this same system which does not exhibit this same behavior. Thanks in advance for your help, I really appreciate it.
-
I have seen similar behaviour to that if you connect an unsupported SFP module to an ixgbe port. The chip somehow gets into a mode that can only be cleared by completely depowering it.
With the wrong module in it the driver fails to attach so you can't do much at all. I assume here you can still see the NIC it just never shows link? In which case it might be possible to reset it in software.
You've tried resaving the interface settings with the media type etc?Steve
-
@stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:
I have seen similar behaviour to that if you connect an unsupported SFP module to an ixgbe port. The chip somehow gets into a mode that can only be cleared by completely depowering it.
With the wrong module in it the driver fails to attach so you can't do much at all. I assume here you can still see the NIC it just never shows link? In which case it might be possible to reset it in software.
You've tried resaving the interface settings with the media type etc?Steve
Thanks @stephenw10 - I am currently using transceiver modules from FS.com with the ixgbe SFP+ ports vs. Intel modules. Perhaps using actual Intel SFP+ transceivers is worth trying out in this case. I just looked on Ebay and it seems like they are quite readily available and fairly affordable so I'm going to give this a shot.
And yes, I can still the NIC fine after the link drops. Even trying to resave the interface unfortunately makes no difference.
Thanks again for your help.
-
Got the new Intel SFP+ transceivers installed today. Tried to cycle the interface a few times by disconnecting and reconnecting the fiber cable into the transceiver and didn't lose the link. Also rebooted the firewall to see if I would lose the link, but all stayed up. Fingers crossed it was as simple as that :).
-
Well, unfortunately the SFP+ modules did not fix the problem. Rebooted the firewall the other day and ran into these same issues again (had to completely power cycle the unit to get the interface back). At this point I'm led to believe this may be a driver issue - hopefully it will be sorted in FreeBSD 12.x / pfSense 2.5.
-
Are you able to test a 2.5 snapshot?
Now is the time to report issues with that if you can. It may be something upstream in the driver of course. We are limited in what we can do if that's the case.
Steve
-
@stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:
Are you able to test a 2.5 snapshot?
Now is the time to report issues with that if you can. It may be something upstream in the driver of course. We are limited in what we can do if that's the case.
Steve
It might be a bit tough as this is my only production system - how stable are the snapshots right now? Thanks again.
-
Pretty stable. I've not seen a significant problem in months. But very hard to advise you run it in productions though.
Steve
-
@stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:
Pretty stable. I've not seen a significant problem in months. But very hard to advise you run it in productions though.
Steve
Thanks @stephenw10 - I suppose technically it's not really a "production" system since it's on my home network (and the number of people inconvenienced would be very low in case of any downtime). Would you say that for this kind of scenario the snapshots are probably stable enough? I suppose worst case I can always start over with a fresh 2.4.4 install and restore from backup. How easy is it to go back to the stable branch once 2.5.0 is released - is it just a matter of switching back over in the UI and installing the Release version over the last 2.5.0 snapshot (or more likely Release Candidate)? Thanks again.
-
I would say they are stable enough for that. As I said I've had numerous boxes up running 2.5 here for some time without any significant issues. Of course what issues there are might be more significant to you.
You can't easily downgrade in place, you have to reinstall 2.4.4p3. But if you have the media and a config standing by that's pretty easily accomplished. If you had to.
Steve
-
@stephenw10 said in SFP+ ixgbe Network Interface Won't Come Back Up:
I would say they are stable enough for that. As I said I've had numerous boxes up running 2.5 here for some time without any significant issues. Of course what issues there are might be more significant to you.
You can't easily downgrade in place, you have to reinstall 2.4.4p3. But if you have the media and a config standing by that's pretty easily accomplished. If you had to.
Steve
Well, this issue got me again today. I had been noticing some random hotplug events over the last few days for the ix1 interface in the system logs (maybe once or twice per day), but they only lasted for about a second and the interface came right back up. Didn't make much of it until today when the ix1 interface did not come back up after another such hotplug event. This occurred after about 50 days of uptime. As usual, I had to shutdown the machine, pull the power, and then restart it for the interface to come back. Up until now I had 4 VLAN's configured on this physical interface, but decided to swap those around today with one of the Chelsio physical interfaces after finding this:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235918
Not sure if it's related, but will see if it makes any difference going forward. Looks like I may end up trying 2.5.0 sooner than later, or going back to an igb interface instead, as those work fine in the current 2.4.4 version.