Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    LACP not working

    Scheduled Pinned Locked Moved General pfSense Questions
    113 Posts 2 Posters 27.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      cyberbot @stephenw10
      last edited by

      @stephenw10

      yes of course only there is no log now, nothing happening at all.

      1 Reply Last reply Reply Quote 0
      • C
        cyberbot
        last edited by

        when I do show log on the switch nothing really happens about the LACP or errors neither on the pfsense

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by stephenw10

          Ok but can we see ifconfig -vvv lagg0 from pfSense and show lag from the switch. The complete output from each.

          Steve

          C 1 Reply Last reply Reply Quote 0
          • C
            cyberbot @stephenw10
            last edited by cyberbot

            @stephenw10 said in LACP not working:

            Ok but can we see ifconfig -vvv lagg0 from pfSense and show lag from the switch. The complete output from each.

            Steve

            Hi Steve,
            see below, the engineer still confirm the switch is the cause.

            lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                    options=800008<VLAN_MTU>
                    ether e8:39:35:11:fa:ab
                    inet6 fe80::ea39:35ff:fe11:faab%lagg0 prefixlen 64 scopeid 0xb
                    inet 192.168.73.1 netmask 0xffffff00 broadcast 192.168.73.255
                    laggproto lacp lagghash l2,l3,l4
                    lagg options:
                            flags=90<LACP_STRICT>
                            flowid_shift: 16
                    lagg statistics:
                            active ports: 2
                            flapping: 64
                    lag id: [(8000,E8-39-35-11-FA-AB,016B,0000,0000),
                             (0001,60-9C-9F-4B-80-8C,4E21,0000,0000)]
                    laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3f<ACTIVITY,TIMEOUT,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                            [(8000,E8-39-35-11-FA-AB,016B,8000,0003),
                             (0001,60-9C-9F-4B-80-8C,4E21,0001,0002)]
                    laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3f<ACTIVITY,TIMEOUT,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                            [(8000,E8-39-35-11-FA-AB,016B,8000,0004),
                             (0001,60-9C-9F-4B-80-8C,4E21,0001,0102)]
                    groups: lagg
                    media: Ethernet autoselect
                    status: active
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            
            === LAG "LAN" ID 1 (dynamic Deployed) ===
            LAG Configuration:
               Ports:         e 1/1/2 e 2/1/2
               Port Count:    2
               Primary Port:  1/1/2
               Trunk Type:    hash-based
               LACP Key:      20001
            Deployment: HW Trunk ID 1
            Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
            1/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.105d  LAN1
            2/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.105d LAN2
            
            Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
            1/1/2           1        1   20001   Yes   S   Agg  Syn  Col  Dis  Def  No   Err
            2/1/2           1        1   20001   Yes   S   Agg  Syn  Col  Dis  Def  No   Err
            
            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              The timeouts are back to short on the switch but do also look like they're set short in pfSense.

              There are two additional sections in the 'show lag' output. The system lag info at the top and the partner info at the bottom. Both of which might be useful here.
              For example:

              SSH@ICX6450-24P Switch>show lag
              Total number of LAGs:          1
              Total number of deployed LAGs: 1
              Total number of trunks created:1 (123 available)
              LACP System Priority / ID:     1 / 609c.9f54.14f2
              LACP Long timeout:             90, default: 90
              LACP Short timeout:            3, default: 3
              
              === LAG "lacp1" ID 2047 (dynamic Deployed) ===
              LAG Configuration:
                 Ports:         e 1/2/1 e 1/2/3 
                 Port Count:    2
                 Primary Port:  1/2/1
                 Trunk Type:    hash-based
                 LACP Key:      22047
                 LACP Timeout:  long
              Deployment: HW Trunk ID 1
              Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
              1/2/1      Up      Forward Full 10G   2047  No  1    0   609c.9f54.150b                 
              1/2/3      Up      Forward Full 10G   2047  No  1    0   609c.9f54.150b                 
              
              Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
              1/2/1           1        1   22047   Yes   L   Agg  Syn  Col  Dis  No   No   Ope
              1/2/3           1        1   22047   Yes   L   Agg  Syn  Col  Dis  No   No   Ope
                                                                                
              
               Partner Info and PDU Statistics 
              Port          Partner         Partner     LACP      LACP     
                           System ID         Key     Rx Count  Tx Count  
              1/2/1    32768-00e0.ed86.a68c      690     5638     13113
              1/2/3    32768-00e0.ed86.a68c      690     5625     13138
              
              [2.4.5-RELEASE][admin@7100.stevew.lan]/root: ifconfig -vvv lagg0
              lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
              	options=500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
              	ether 00:e0:ed:86:a6:8c
              	inet6 fe80::2e0:edff:fe86:a68c%lagg0 prefixlen 64 scopeid 0x15
              	inet 172.21.16.206 netmask 0xffffff00 broadcast 172.21.16.255
              	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
              	media: Ethernet autoselect
              	status: active
              	groups: lagg
              	laggproto lacp lagghash l2,l3,l4
              	lagg options:
              		flags=10<LACP_STRICT>
              		flowid_shift: 16
              	lagg statistics:
              		active ports: 2
              		flapping: 0
              	lag id: [(8000,00-E0-ED-86-A6-8C,02B2,0000,0000),
              		 (0001,60-9C-9F-54-14-F2,561F,0000,0000)]
              	laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
              		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0001),
              		 (0001,60-9C-9F-54-14-F2,561F,0001,0041)]
              	laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
              		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0002),
              		 (0001,60-9C-9F-54-14-F2,561F,0001,0043)]
              
              1 Reply Last reply Reply Quote 0
              • C
                cyberbot
                last edited by

                Hi Steve,

                do you suggest to reconfigure the set up to long instead of short?
                i can do that is not a issue at all, but the switch still blocking it

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Well it should work with either as long as both sides are set the same. The switch ports are not showing Exp (expired) which I would expect if pfSense was still using Long timeouts.

                  I know Long works though.

                  The port operational state (Ope) is still showing Err:

                  Err: If there is a peer information mismatch, then that particular port is moved to the Error disable state (Err).
                  

                  https://docs.commscope.com/bundle/fastiron-08095-commandref/page/GUID-06AFF73D-6957-44A5-AF25-3527B2BE1580.html

                  I would have expected that to be logged still.

                  Can we see the full 'show lag' output?

                  Steve

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    cyberbot @stephenw10
                    last edited by cyberbot

                    @stephenw10 said in LACP not working:

                    Well it should work with either as long as both sides are set the same. The switch ports are not showing Exp (expired) which I would expect if pfSense was still using Long timeouts.

                    I know Long works though.

                    The port operational state (Ope) is still showing Err:

                    Err: If there is a peer information mismatch, then that particular port is moved to the Error disable state (Err).
                    

                    https://docs.commscope.com/bundle/fastiron-08095-commandref/page/GUID-06AFF73D-6957-44A5-AF25-3527B2BE1580.html

                    I would have expected that to be logged still.

                    Can we see the full 'show lag' output?

                    Steve

                    on the switch side I don't see any logs showing anything about the LACP but only about the when I logging using ssh ect...
                    I have change it to long but still shows blocked.

                    === LAG "LAN" ID 1 (dynamic Deployed) ===
                    LAG Configuration:
                       Ports:         e 1/1/2 e 2/1/2
                       Port Count:    2
                       Primary Port:  1/1/2
                       Trunk Type:    hash-based
                       LACP Key:      20001
                       LACP Timeout:  long
                    Deployment: HW Trunk ID 1
                    Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
                    1/1/2      Up      Blocked Full 1G    1     Yes N/A  0    LAN1
                    2/1/2      Up      Blocked Full 1G    1     Yes N/A  0    LAN2
                    
                    Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
                    1/1/2           1        1   20001   Yes   L   Agg  No   No   No   No   No   Ina
                    2/1/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  Def  No   Err
                    
                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      We need to see the full output, including the partner info.

                      The first port there is showing it's not seeing ant LACP packets from the other side.

                      C 1 Reply Last reply Reply Quote 0
                      • C
                        cyberbot @stephenw10
                        last edited by cyberbot

                        @stephenw10 said in LACP not working:

                        We need to see the full output, including the partner info.

                        The first port there is showing it's not seeing ant LACP packets from the other side.

                        Hi Steve

                        What full output are you referring to ? On the pfsense or switch ?
                        On the switch side it shows only information about who is connected from the ssh and what time nothing else .

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          The complete output from show lag on the switch contains the Partner info and PDU stats at the end which show what is connected and on which port. It also shows the system wide lag parameters at the top on my switch. It would be good to compare those.
                          Your switches are different to mine though, the output is very similar but not identical.

                          You posted a more complete output here: https://forum.netgate.com/post/947906

                          Steve

                          C 1 Reply Last reply Reply Quote 0
                          • C
                            cyberbot @stephenw10
                            last edited by cyberbot

                            @stephenw10 said in LACP not working:

                            The complete output from show lag on the switch contains the Partner info and PDU stats at the end which show what is connected and on which port. It also shows the system wide lag parameters at the top on my switch. It would be good to compare those.
                            Your switches are different to mine though, the output is very similar but not identical.

                            You posted a more complete output here: https://forum.netgate.com/post/947906

                            Steve

                            Hi Steve,

                            Are you referring to these?

                            === LAG "LAN" ID 1 (dynamic Deployed) ===
                            LAG Configuration:
                               Ports:         e 1/1/2 e 2/1/2
                               Port Count:    2
                               Primary Port:  1/1/2
                               Trunk Type:    hash-based
                               LACP Key:      20001
                               LACP Timeout:  long
                            Deployment: HW Trunk ID 1
                            Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
                            1/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.606d  LAN1
                            2/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.606d  LAN2
                            
                            Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
                            1/1/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  Def  No   Err
                            2/1/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  Def  No   Err
                            
                            
                             Partner Info and PDU Statistics
                            Port          Partner         Partner     LACP      LACP
                                         System ID         Key     Rx Count  Tx Count
                            1/1/2    32768-e839.3511.faab      363        0    237550
                            2/1/2    32768-e839.3511.faab      363        0    237550
                            
                            lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                    options=800008<VLAN_MTU>
                                    ether e8:39:35:11:fa:ab
                                    inet6 fe80::ea39:35ff:fe11:faab%lagg0 prefixlen 64 scopeid 0xb
                                    inet 192.168.73.1 netmask 0xffffff00 broadcast 192.168.73.255
                                    laggproto lacp lagghash l2,l3,l4
                                    lagg options:
                                            flags=10<LACP_STRICT>
                                            flowid_shift: 16
                                    lagg statistics:
                                            active ports: 2
                                            flapping: 291
                                    lag id: [(8000,E8-39-35-11-FA-AB,016B,0000,0000),
                                             (0001,60-9C-9F-4B-80-8C,4E21,0000,0000)]
                                    laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                                            [(8000,E8-39-35-11-FA-AB,016B,8000,0003),
                                             (0001,60-9C-9F-4B-80-8C,4E21,0001,0002)]
                                    laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                                            [(8000,E8-39-35-11-FA-AB,016B,8000,0004),
                                             (0001,60-9C-9F-4B-80-8C,4E21,0001,0102)]
                                    groups: lagg
                                    media: Ethernet autoselect
                                    status: active
                                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                            
                            stephenw10S 1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator @cyberbot
                              last edited by

                              Ok so from that we can see that the system ID the switch sees on both those ports matches the ID pfSense has set on it's lagg ports.

                              However we cannot see the switch system ID. pfSense sees it as: 0001,60-9C-9F-4B-80-8C

                              That info is at the top of the show lag output from the switch as I showed above:

                              SSH@ICX6450-24P Switch>show lag
                              Total number of LAGs:          1
                              Total number of deployed LAGs: 1
                              Total number of trunks created:1 (123 available)
                              LACP System Priority / ID:     1 / 609c.9f54.14f2
                              LACP Long timeout:             90, default: 90
                              LACP Short timeout:            3, default: 3
                              

                              It obviously should match but....

                              What we can see is that the switch has recorded precisely 0 lacpdus received.

                              Steve

                              C 1 Reply Last reply Reply Quote 0
                              • C
                                cyberbot @stephenw10
                                last edited by

                                @stephenw10 said in LACP not working:

                                Ok so from that we can see that the system ID the switch sees on both those ports matches the ID pfSense has set on it's lagg ports.

                                However we cannot see the switch system ID. pfSense sees it as: 0001,60-9C-9F-4B-80-8C

                                That info is at the top of the show lag output from the switch as I showed above:

                                SSH@ICX6450-24P Switch>show lag
                                Total number of LAGs:          1
                                Total number of deployed LAGs: 1
                                Total number of trunks created:1 (123 available)
                                LACP System Priority / ID:     1 / 609c.9f54.14f2
                                LACP Long timeout:             90, default: 90
                                LACP Short timeout:            3, default: 3
                                

                                It obviously should match but....

                                What we can see is that the switch has recorded precisely 0 lacpdus received.

                                Steve

                                what are you suggesting ?
                                if i connect a different switch the ports comes online and shows the LACP is fine.

                                stephenw10S 1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator @cyberbot
                                  last edited by

                                  To that switch or to pfSense?

                                  I have once seen a similar issue to this that was eventually resolved by simply rebooting the switch stack. Something no-one had thought to do because generally switches do not require that sort of thing.

                                  Beyond that I would try a lagg to ports on the same switch to remove the cross-chassis LACP as an issue.

                                  Then I'm out of suggestions. You probably need to get Brocade/Rukus/Commscope support involved at that point.

                                  Steve

                                  C 1 Reply Last reply Reply Quote 0
                                  • C
                                    cyberbot @stephenw10
                                    last edited by

                                    @stephenw10 said in LACP not working:

                                    To that switch or to pfSense?

                                    I have once seen a similar issue to this that was eventually resolved by simply rebooting the switch stack. Something no-one had thought to do because generally switches do not require that sort of thing.

                                    Beyond that I would try a lagg to ports on the same switch to remove the cross-chassis LACP as an issue.

                                    Then I'm out of suggestions. You probably need to get Brocade/Rukus/Commscope support involved at that point.

                                    Steve

                                    Hi Steve
                                    I can do the reboot tonight. When I am logged in shut reboot its will reboot both switches ?

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      I can't answer that I only have a single Brocade switch. I would hope there is some failover happens. I've never configured that.
                                      Be sure to have saved the running config. Have a recovery plan etc....

                                      Steve

                                      C 1 Reply Last reply Reply Quote 0
                                      • C
                                        cyberbot @stephenw10
                                        last edited by cyberbot

                                        @stephenw10 said in LACP not working:

                                        I can't answer that I only have a single Brocade switch. I would hope there is some failover happens. I've never configured that.
                                        Be sure to have saved the running config. Have a recovery plan etc....

                                        Steve

                                        I have rebooted the switch stack, but unfortunately it still shows blocked.

                                        1000 MHz ARM processor ARMv7 88 MHz bus
                                        8192 KB boot flash memory
                                        2048 MB code flash memory
                                        2048 MB DRAM
                                        STACKID 1  system uptime is 3 minute(s) 34 second(s)
                                        STACKID 2  system uptime is 3 minute(s) 31 second(s)
                                        The system : started=warm start   reloaded=by "reload"
                                        My stack unit ID = 1, bootup role = active
                                        
                                        === LAG "LAN" ID 10 (dynamic Deployed) ===
                                        LAG Configuration:
                                           Ports:         e 1/1/2 e 2/1/2
                                           Port Count:    2
                                           Primary Port:  1/1/2
                                           Trunk Type:    hash-based
                                           LACP Key:      20001
                                           LACP Timeout:  long
                                        Deployment: HW Trunk ID 1
                                        Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
                                        1/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.606d  LAN1
                                        2/1/2      Up      Blocked Full 1G    1     Yes N/A  0   609c.9f4b.606d  LAN2
                                        
                                        Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
                                        1/1/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  Def  No   Err
                                        2/1/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  Def  No   Err
                                        
                                        
                                         Partner Info and PDU Statistics
                                        Port          Partner         Partner     LACP      LACP
                                                     System ID         Key     Rx Count  Tx Count
                                        1/1/2    32768-e839.3511.faab      363        0        59
                                        2/1/2    32768-e839.3511.faab      363        0        66
                                        

                                        I saw the error of the mismatch

                                        Dec  7 23:47:37:I:Stack: Stack unit 2 has been assigned as STANDBY unit of the stack system
                                        Dec  7 23:46:41:I:System: Logical link on dynamic lag interface ethernet 2/1/2 is down.
                                        Dec  7 23:46:41:I:System: Logical link on dynamic lag interface ethernet 2/1/11 is up.
                                        Dec  7 23:46:41:I:System: Interface ethernet 2/1/11, state up
                                        Dec  7 23:46:40:I:System: Logical link on dynamic lag interface ethernet 2/1/12 is up.
                                        Dec  7 23:46:40:I:System: Interface ethernet 2/1/12, state up
                                        Dec  7 23:46:40:I:Trunk: Group (1/1/11, 1/1/12, 2/1/11, 2/1/12) created by 802.3ad link-aggregation module.
                                        Dec  7 23:46:38:I:System: Logical link on dynamic lag interface ethernet 2/1/2 is down.
                                        Dec  7 23:46:38:I:System: dynamic lag interface 2/1/2's peer info (priority=32768,id=e839.3511.faab,key=363) mis-matches with lag's peer info (priority=32768,id=609c.9f4b.808c,key=363), set to mismatch Error
                                        Dec  7 23:46:36:I:System: Interface ethernet 2/1/48, state up
                                        

                                        but after it disappear it didn't comes back.

                                        I know freebsd is using the key 363

                                        peer-info sys-mac MAC of the LACP sys-pri 32768 key 363
                                        

                                        fix the problem of the mismatch but the ports are still blocked

                                        I cannot seems to see if lag interface on the pfsense is using the fast or long timeout

                                        lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                                options=800008<VLAN_MTU>
                                                ether e8:39:35:11:fa:ab
                                                inet6 fe80::ea39:35ff:fe11:faab%lagg0 prefixlen 64 scopeid 0xb
                                                inet 192.168.73.1 netmask 0xffffff00 broadcast 192.168.73.255
                                                laggproto lacp lagghash l2,l3,l4
                                                laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
                                                laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
                                                groups: lagg
                                                media: Ethernet autoselect
                                                status: active
                                                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by stephenw10

                                          The only thing I have not seen from your switch is the LACP ID value. My switch shows that at the top of the show lag output but either yours doesn't or you're just not including it here.

                                          We can guess it's 609c.9f4b.808c because that's what the error logs show and the ifconfig from pfSense. But if it isn't that it would be mismatched.

                                          Steve

                                          C 1 Reply Last reply Reply Quote 0
                                          • C
                                            cyberbot @stephenw10
                                            last edited by

                                            @stephenw10 said in LACP not working:

                                            609c.9f4b.808c

                                            Thank you for your answer Steve,
                                            i have managed to get the 609c.9f4b.808c corrected. it was the second Mac of a different LAG.
                                            i've remvoed that lag and the issue is fixed.
                                            both Pfsense and Switch share the same LACP ID now.
                                            what information do you need from the switch?

                                            stephenw10S 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.