Problem with NICs flapping at intervals of 5 mins
-
@stephenw10 said in Problem with NICs flapping at intervals of 5 mins:
Hmm, I wonder it's misreporting 'autoselect' there. If the switch side is set to 1G fixed the NIC should not be able to negotiate with it.
I guess we'll see if it makes any difference anyway.
When in doubt, look at the logs I guess. I should have looked sooner:
2024-10-15 15:29:09.000 kernel: kernel: ixl1: Media change is not supported. 2024-10-15 15:29:09.000 php-fpm[28896]: php-fpm[28896]: /interfaces.php: The command '/sbin/ifconfig 'ixl1' media '10Gbase-T'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device' -
@whosmatt said in Problem with NICs flapping at intervals of 5 mins:
but the list of possible values has been narrowed.
Yeah that very well could be - since gig came out, the preferred setting has been auto neg.. If you want a slower speed you should really just limit what is offered, etc.
While forcing with gig is an option, the best practice is this is only for temp troubleshooting to figure out why auto isn't working.. Been a really long time I have looked at the actual spec for 10ge, but I recall reading that there would be no option for hard set, had to be auto..
Makers don't always follow the spec ;) heheh but if your running fine on gig - but having issues with 10ge over copper - how long are the runs? Any way you could switch to fiber?
I would have to do some looking around on work network.. Only been at this gig going on a year and some of these locations were inherited when a company was bought, etc. We still have some old HPs that working on getting rid of, etc. But I believe the only copper 10ge we have is from the server in the rack to the TOR switch in that same rack. And the only reason those are copper is the server came with the 10ge copper connections or something.. We sure do not run any switch to switch over copper that I am aware of. But there are a lot of sites ;) and I haven't even been on some of the switches or routers in those sites..
-
@johnpoz said in Problem with NICs flapping at intervals of 5 mins:
Any way you could switch to fiber?
I intend to switch to DAC. I was using the X710-T4L because my AT&T fiber CPE has a multigig port (up to 5Gbps) and I could never get any igc card stable. I have since bypassed the AT&T CPE by using a SFP+ module that has the ONT built in to it. Right now that SFP+ module is plugged into a dumb switch acting as a media converter but my intention is to get a X520-DA2 and just go direct fiber for WAN and DAC for all the inside interfaces.
-
Ok, so I plugged ixl1 into a different switch at 10Gbps and the problem followed. Gonna force it down to 5Gbps and see if it does any better there.
-
So far so good at 5Gbps. To answer the earlier question about cabling, pfSense is connected directly to the switch with 14' Monoprice Slimrun Cat6A patch cables (orange if that matters).
-
Still good at 5Gbps. What I did, for anyone curious, is force my speeds at the switch, since they don't appear to apply on the pfSense side. Not complaining about pfSense since it can only support what the hardware supports. After evaluating ixl1 at 5Gbps on a different switch, I reconfigured my lagg on the pfSense side to two NICs (ixl0 and ixl1) and on the switch to port 7 and port 8. On the switch side, my Trendnet TEG-7124WS allowed me to to set the port speed on the "trunk" (their nomenclature for link aggregation) and it forced both ports down to 5Gbps and the ports on pfSense followed suit. So I'm currently at a 2 port lagg with each port auto negotiating to 5Gbps.
I'm starting to think that 10Gbps over twisted pair copper is a crapshoot, and it's not just because of this experience. I had an event last night with my TrueNAS box where both NICs in a lagg experienced up/down events about 10 minutes apart and I logged (smokeping) some packet loss. The up/down events on that box are rare and i've never seen any packet loss so going to keep watching that.
-
One more update relevant to this subject:
I checked a couple of pfSense VMs we have at work. Those have passed through X550 NICs and have also had negotiation issues. Setting the X550 to 10Gbase-T does in fact work there on the pfSense side so I have to assume it's something specific to the X710 or to the ixl driver.
-
Mmm, does seem like it's the driver. I expected that to work. I don't have anything base-T to test it with here currently I know it did work with X500 based NICs. Especially the X550-T that can do n-base speeds.
-
@stephenw10 said in Problem with NICs flapping at intervals of 5 mins:
Especially the X550-T that can do n-base speeds.
Yeah, that's what we have. The switches, on the other hand, are older Cisco Nexus 5000 series that don't support the intermediate speeds. We were seeing speed mismatches that caused all sorts of problems as you might imagine.
-
One more update on this subject. pfSense is still stable with zero up/down events since limiting to 5Gbps.
My TrueNAS box, on the other hand, started showing the same behavior (at the same 5 min intervals) after a software update. They were happening about 1x per day, but even though only one interface in the lagg would go down at any given time, I'd measure some packet loss, likely because smokeping is also on a 5 min interval. This was a change from previous observed behavior where the up/down events would occasionally happen, but not on the 5 min interval and no measured packet loss.
I picked up a Dell X710-T2L for a song on ebay (seriously, like $30 shipped) and cross-flashed it to the Intel firmware. I replaced the X540-T2. My intention was to also limit to 5Gbps but I haven't had to do that. The events just stopped and 0% loss since then. It's even more puzzling because the cabling here, while short, has multiple excursions into wall-mounted jacks with Cat6 (not Cat6a) running under the house through the crawl space. That was not the case with pfSense, which is directly cabled to the switch with Cat6a patch cables.
Don't really know what to think about this other than that 10GBase-T is a crapshoot.
