• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Problem with NICs flapping at intervals of 5 mins

Scheduled Pinned Locked Moved Hardware
30 Posts 3 Posters 1.7k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • W
    whosmatt @whosmatt
    last edited by Oct 14, 2024, 5:48 PM

    @whosmatt said in Problem with NICs flapping at intervals of 5 mins:

    @johnpoz said in Problem with NICs flapping at intervals of 5 mins:

    if you take the laggs out of the equation - do these interfaces still go up down?

    I have taken port 7 on the switch and ixl1 on pfSense out of the lagg but have left them physically connected to see what happens.

    It has already happened outside of the lagg:

    From switch:
    
    2024 Oct 14 17:30:09 TEG-7124WS CFA Slot0/7 Link Status [UP]
    2024 Oct 14 17:30:04 TEG-7124WS CFA Slot0/7 Link Status [DOWN]
    
    From pfSense:
    
    2024-10-14 17:30:09.000	kernel:
    kernel: ixl1: link state changed to UP
    2024-10-14 17:30:09.000	kernel:
    kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None
    2024-10-14 17:30:09.000	check_reload_status[430]:
    check_reload_status[430]: Linkup starting ixl1
    2024-10-14 17:30:05.000	kernel:
    kernel: ixl1: link state changed to DOWN
    2024-10-14 17:30:05.000	check_reload_status[430]:
    check_reload_status[430]: Linkup starting ixl1
    

    Guess I'll force speed and duplex on both sides and see what happens.

    1 Reply Last reply Reply Quote 1
    • W
      whosmatt @stephenw10
      last edited by Oct 14, 2024, 6:51 PM

      @stephenw10 said in Problem with NICs flapping at intervals of 5 mins:

      Can you set both sides to a fixed speed?

      Curious how to do this on the pfSense side with interfaces that are part of a lagg, or whether it's possible.

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Oct 14, 2024, 7:06 PM

        Yeah, not easily since you can't assign the member interfaces separately. You can add shell cmds to set them at boot.

        But you can test it with the NIC you removed from the lagg first.

        W 1 Reply Last reply Oct 14, 2024, 7:07 PM Reply Quote 0
        • W
          whosmatt @stephenw10
          last edited by Oct 14, 2024, 7:07 PM

          @stephenw10 said in Problem with NICs flapping at intervals of 5 mins:

          But you can test it with the NIC you removed from the lagg first.

          Yep, that's what I'm doing currently. Thanks!

          1 Reply Last reply Reply Quote 1
          • W
            whosmatt
            last edited by whosmatt Oct 15, 2024, 1:27 AM Oct 15, 2024, 1:18 AM

            Well, I was about to post that it happened again with forced speed and duplex but then I saw this in the pfSense log:

            kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None

            Which is odd because I definitely set 10G full. But then I realized I didn't enable the interface I assigned to ixl1. So I enabled the interface and will wait and see what happens.

            Edit:

            Actually it appears that the settings aren't correctly applying at least when I view the output of ifconfig. And I set up a second unassigned NIC and forced its speed and duplex in the UI just to see the difference:

            [2.7.2-RELEASE][root@pfsense]/root: ifconfig ixl1
            ixl1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
                    description: OPT10
                    options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG>
                    ether b4:96:91:b6:27:b5
                    inet6 fe80::b696:91ff:feb6:27b5%ixl1 prefixlen 64 scopeid 0x2
                    media: Ethernet autoselect (10Gbase-T <full-duplex>)
                    status: active
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            [2.7.2-RELEASE][root@pfsense]/root: ifconfig bge1
            bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                    description: OPT12
                    options=80098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
                    ether 00:0a:f7:8f:51:89
                    inet6 fe80::20a:f7ff:fe8f:5189%bge1 prefixlen 64 scopeid 0x7
                    media: Ethernet 1000baseT <full-duplex> (none)
                    status: no carrier
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            

            The media for ixl1 still shows 'autoselect' even when set to 10G full in the UI.

            J 1 Reply Last reply Oct 15, 2024, 1:45 AM Reply Quote 0
            • J
              johnpoz LAYER 8 Global Moderator @whosmatt
              last edited by Oct 15, 2024, 1:45 AM

              @whosmatt Can you even set manual 10ge, thought part of the spec was auto? You can set the speed down manual.. I have to call up the spec.. And we really don't run much copper 10ge at work.. I believe we do have some.. I will have to tool around tmrw and see..

              Might be able to set it 5 or 2.5, etc.

              An intelligent man is sometimes forced to be drunk to spend time with his fools
              If you get confused: Listen to the Music Play
              Please don't Chat/PM me for help, unless mod related
              SG-4860 24.11 | Lab VMs 2.8, 24.11

              W 2 Replies Last reply Oct 15, 2024, 2:01 AM Reply Quote 0
              • W
                whosmatt @johnpoz
                last edited by Oct 15, 2024, 2:01 AM

                @johnpoz said in Problem with NICs flapping at intervals of 5 mins:

                Can you even set manual 10ge

                It's an option in the UI, yes. It's also an option on the switch side.

                @johnpoz said in Problem with NICs flapping at intervals of 5 mins:

                And we really don't run much copper 10ge at work

                I'm beginning to understand why.

                1 Reply Last reply Reply Quote 0
                • W
                  whosmatt @johnpoz
                  last edited by whosmatt Oct 15, 2024, 2:12 AM Oct 15, 2024, 2:03 AM

                  @johnpoz said in Problem with NICs flapping at intervals of 5 mins:

                  Might be able to set it 5 or 2.5, etc.

                  I tried setting 5000Base-T and ifconfig still shows "media: Ethernet autoselect (10Gbase-T <full-duplex>)"

                  If I go through the various settings on the switch the NIC follows along, down as far as 1Gbps. There's also a 100M Full setting on the switch but the NIC won't link at that speed.

                  1 Reply Last reply Reply Quote 0
                  • S
                    stephenw10 Netgate Administrator
                    last edited by Oct 15, 2024, 4:00 AM

                    The options offered in the gui are what ifconfig -m returns. For example:

                    [admin@7100.stevew.lan]/root: ifconfig -vvm ixl0
                    ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
                    	options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG>
                    	capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
                    	ether 00:e0:ed:86:a6:8c
                    	inet6 fe80::208:a2ff:fe0e:a591%ixl0 prefixlen 64 scopeid 0x1
                    	media: Ethernet autoselect (10GBase-AOC <full-duplex>)
                    	status: active
                    	supported media:
                    		media autoselect
                    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                    	drivername: ixl0
                    	plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
                    	vendor: BROCADE PN: 58-1000026-01 SN: CAX116410001093 DATE: 2016-10-07
                    

                    DACs like that usually don't offer more than one speed.

                    J 1 Reply Last reply Oct 15, 2024, 12:51 PM Reply Quote 0
                    • J
                      johnpoz LAYER 8 Global Moderator @stephenw10
                      last edited by Oct 15, 2024, 12:51 PM

                      @stephenw10 so yeah that doesn't show any options.. My igb0 on the other hand does

                              media: Ethernet autoselect (1000baseT <full-duplex>)
                              status: active
                              supported media:
                                      media autoselect
                                      media 1000baseT
                                      media 1000baseT mediaopt full-duplex
                                      media 100baseTX mediaopt full-duplex
                                      media 100baseTX
                                      media 10baseT/UTP mediaopt full-duplex
                                      media 10baseT/UTP
                      

                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                      If you get confused: Listen to the Music Play
                      Please don't Chat/PM me for help, unless mod related
                      SG-4860 24.11 | Lab VMs 2.8, 24.11

                      W 1 Reply Last reply Oct 15, 2024, 2:24 PM Reply Quote 0
                      • W
                        whosmatt @johnpoz
                        last edited by Oct 15, 2024, 2:24 PM

                        Yeah I can see the list of supported speed / duplex, it's just that setting any of them in the UI doesn't seem to change the media from autoselect:

                        ixl1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
                                description: OPT10
                                options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG>
                                capabilities=4f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
                                ether b4:96:91:b6:27:b5
                                inet6 fe80::b696:91ff:feb6:27b5%ixl1 prefixlen 64 scopeid 0x2
                                media: Ethernet autoselect (1000baseT <full-duplex>)
                                status: active
                                supported media:
                                        media autoselect
                                        media 10Gbase-T
                                        media 5000Base-T
                                        media 2500Base-T
                                        media 1000baseT
                                        media 100baseTX
                                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                        

                        I've got it running at 1000Mbps right now because the port is forced to that speed on the switch. Incidentally, it hasn't gone down since I set that about 12 hours ago or so.

                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by Oct 15, 2024, 2:30 PM

                          Hmm, I wonder it's misreporting 'autoselect' there. If the switch side is set to 1G fixed the NIC should not be able to negotiate with it.

                          I guess we'll see if it makes any difference anyway.

                          W 2 Replies Last reply Oct 15, 2024, 2:33 PM Reply Quote 0
                          • W
                            whosmatt @stephenw10
                            last edited by Oct 15, 2024, 2:33 PM

                            @stephenw10 said in Problem with NICs flapping at intervals of 5 mins:

                            Hmm, I wonder it's misreporting 'autoselect' there. If the switch side is set to 1G fixed the NIC should not be able to negotiate with it.

                            I guess we'll see if it makes any difference anyway.

                            I'm wondering if the setting on the switch side is really forcing speed/duplex or just forcing it to auto negotiate to a predetermined speed. If that makes sense. In other words it's still autoselect, but the list of possible values has been narrowed.

                            J 1 Reply Last reply Oct 15, 2024, 3:37 PM Reply Quote 0
                            • S
                              stephenw10 Netgate Administrator
                              last edited by Oct 15, 2024, 2:44 PM

                              Yes, that could certainly be the case.

                              You can set that in pfSense using sysctl:

                              [admin@7100.stevew.lan]/root: sysctl -d dev.ixl.0.advertise_speed
                              dev.ixl.0.advertise_speed: 
                              Control advertised link speed.
                              Flags:
                              	 0x1 - advertise 100M
                              	 0x2 - advertise 1G
                              	 0x4 - advertise 10G
                              	 0x8 - advertise 20G
                              	0x10 - advertise 25G
                              	0x20 - advertise 40G
                              	0x40 - advertise 2.5G
                              	0x80 - advertise 5G
                              
                              Set to 0 to disable link.
                              Use "sysctl -x" to view flags properly.
                              

                              But if it is still negotiating and doesn't lose link at 1G that might also be a clue.

                              1 Reply Last reply Reply Quote 0
                              • W
                                whosmatt @stephenw10
                                last edited by Oct 15, 2024, 3:30 PM

                                @stephenw10 said in Problem with NICs flapping at intervals of 5 mins:

                                Hmm, I wonder it's misreporting 'autoselect' there. If the switch side is set to 1G fixed the NIC should not be able to negotiate with it.

                                I guess we'll see if it makes any difference anyway.

                                When in doubt, look at the logs I guess. I should have looked sooner:

                                2024-10-15 15:29:09.000	kernel:
                                kernel: ixl1: Media change is not supported.
                                2024-10-15 15:29:09.000	php-fpm[28896]:
                                php-fpm[28896]: /interfaces.php: The command '/sbin/ifconfig 'ixl1' media '10Gbase-T'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device'
                                
                                1 Reply Last reply Reply Quote 0
                                • J
                                  johnpoz LAYER 8 Global Moderator @whosmatt
                                  last edited by Oct 15, 2024, 3:37 PM

                                  @whosmatt said in Problem with NICs flapping at intervals of 5 mins:

                                  but the list of possible values has been narrowed.

                                  Yeah that very well could be - since gig came out, the preferred setting has been auto neg.. If you want a slower speed you should really just limit what is offered, etc.

                                  While forcing with gig is an option, the best practice is this is only for temp troubleshooting to figure out why auto isn't working.. Been a really long time I have looked at the actual spec for 10ge, but I recall reading that there would be no option for hard set, had to be auto..

                                  Makers don't always follow the spec ;) heheh but if your running fine on gig - but having issues with 10ge over copper - how long are the runs? Any way you could switch to fiber?

                                  I would have to do some looking around on work network.. Only been at this gig going on a year and some of these locations were inherited when a company was bought, etc. We still have some old HPs that working on getting rid of, etc. But I believe the only copper 10ge we have is from the server in the rack to the TOR switch in that same rack. And the only reason those are copper is the server came with the 10ge copper connections or something.. We sure do not run any switch to switch over copper that I am aware of. But there are a lot of sites ;) and I haven't even been on some of the switches or routers in those sites..

                                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                                  If you get confused: Listen to the Music Play
                                  Please don't Chat/PM me for help, unless mod related
                                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                                  W 1 Reply Last reply Oct 15, 2024, 3:50 PM Reply Quote 0
                                  • W
                                    whosmatt @johnpoz
                                    last edited by Oct 15, 2024, 3:50 PM

                                    @johnpoz said in Problem with NICs flapping at intervals of 5 mins:

                                    Any way you could switch to fiber?

                                    I intend to switch to DAC. I was using the X710-T4L because my AT&T fiber CPE has a multigig port (up to 5Gbps) and I could never get any igc card stable. I have since bypassed the AT&T CPE by using a SFP+ module that has the ONT built in to it. Right now that SFP+ module is plugged into a dumb switch acting as a media converter but my intention is to get a X520-DA2 and just go direct fiber for WAN and DAC for all the inside interfaces.

                                    1 Reply Last reply Reply Quote 0
                                    • W
                                      whosmatt
                                      last edited by Oct 15, 2024, 4:19 PM

                                      Ok, so I plugged ixl1 into a different switch at 10Gbps and the problem followed. Gonna force it down to 5Gbps and see if it does any better there.

                                      W 1 Reply Last reply Oct 16, 2024, 3:20 PM Reply Quote 0
                                      • W
                                        whosmatt @whosmatt
                                        last edited by Oct 16, 2024, 3:20 PM

                                        So far so good at 5Gbps. To answer the earlier question about cabling, pfSense is connected directly to the switch with 14' Monoprice Slimrun Cat6A patch cables (orange if that matters).

                                        W 1 Reply Last reply Oct 17, 2024, 11:33 PM Reply Quote 1
                                        • W
                                          whosmatt @whosmatt
                                          last edited by Oct 17, 2024, 11:33 PM

                                          Still good at 5Gbps. What I did, for anyone curious, is force my speeds at the switch, since they don't appear to apply on the pfSense side. Not complaining about pfSense since it can only support what the hardware supports. After evaluating ixl1 at 5Gbps on a different switch, I reconfigured my lagg on the pfSense side to two NICs (ixl0 and ixl1) and on the switch to port 7 and port 8. On the switch side, my Trendnet TEG-7124WS allowed me to to set the port speed on the "trunk" (their nomenclature for link aggregation) and it forced both ports down to 5Gbps and the ports on pfSense followed suit. So I'm currently at a 2 port lagg with each port auto negotiating to 5Gbps.

                                          I'm starting to think that 10Gbps over twisted pair copper is a crapshoot, and it's not just because of this experience. I had an event last night with my TrueNAS box where both NICs in a lagg experienced up/down events about 10 minutes apart and I logged (smokeping) some packet loss. The up/down events on that box are rare and i've never seen any packet loss so going to keep watching that.

                                          W 1 Reply Last reply Oct 18, 2024, 9:14 PM Reply Quote 1
                                          26 out of 30
                                          • First post
                                            26/30
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received