After upgrade from 24.11 to 25.0.7.1 lagg port did not work

along5664

@stephenw10 It showed the ports down in 25, well on switch it showed no bpdus and suspended. No special settings, standard lagg setup. Strange it worked on 24 but not 25. I will keep digging.

stephenw10

Hmm, well if both member ports were down I wouldn't expect any PDUs. If that's what you mean there. And if that is the case then the question becomes why didn't they negotiate a link and, presumably, what changed in the driver that the switch doesn't like. How are the ports configured?

along5664

@stephenw10
Here are logs from switch:
2025 Sep 12 01:52:56 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel31 is down (No operational members)
2025 Sep 12 01:52:56 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel31,bandwidth changed to 100000 Kbit
2025 Sep 12 01:52:56 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/31 is down (Link failure)
2025 Sep 12 01:52:56 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel31 is down (No operational members)
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-SPEED: Interface Ethernet1/31, operational speed changed to 10 Gbps
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/31, operational duplex mode changed to Full
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/31, operational Receive Flow Control state changed to off
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/31, operational Transmit Flow Control state changed to on
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-SPEED: Interface port-channel31, operational speed changed to 10 Gbps
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel31, operational duplex mode changed to Full
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel31, operational Receive Flow Control state changed to off
2025 Sep 12 01:53:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel31, operational Transmit Flow Control state changed to on
2025 Sep 12 01:53:11 SLC-SHRD-CORE-SW-01 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/31: Ethernet1/31 is suspended
2025 Sep 12 01:54:00 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/31 is down (Link failure)
2025 Sep 12 01:54:00 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel31 is down (No operational members)
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-SPEED: Interface Ethernet1/31, operational speed changed to 10 Gbps
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/31, operational duplex mode changed to Full
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/31, operational Receive Flow Control state changed to off
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/31, operational Transmit Flow Control state changed to on
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-SPEED: Interface port-channel31, operational speed changed to 10 Gbps
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel31, operational duplex mode changed to Full
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel31, operational Receive Flow Control state changed to off
2025 Sep 12 01:54:01 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel31, operational Transmit Flow Control state changed to on
2025 Sep 12 01:54:11 SLC-SHRD-CORE-SW-01 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/31: Ethernet1/31 is suspended
2025 Sep 12 02:01:49 SLC-SHRD-CORE-SW-01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/31 is down (Link failure)

The port came up , but then went suspended mode for no bpdu , that is what shows on the port when it was suspended.

Hers is port config on switch:
interface Ethernet1/31
description SLCHOSTEDPFS02 - WAN3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
channel-group 31 mode active
no shutdown

interface port-channel31
description SLCHOSTEDPFS02 - Bond 3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
vpc 31

stephenw10

Hmm, it's a lagg with one member port? Cross chassis?

along5664

@stephenw10
There are two ports in the port channel. I only showed one port off our 9k, the other 9k has the same port configuration.
Note this works fine in 24.11 but did not work on 25.07.01 when I did the upgrade.
This is the first pfsense I have upgraded using lagg ports.

Here are the configs:

SLC-SHRD-CORE-SW-01# sh run int e1/31

!Command: show running-config interface Ethernet1/31
!Running configuration last done at: Fri Sep 12 02:43:43 2025
!Time: Tue Sep 16 00:13:49 2025

version 9.3(9) Bios:version 05.51

interface Ethernet1/31
description SLCHOSTEDPFS02 - WAN3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
channel-group 31 mode active
no shutdown

SLC-SHRD-CORE-SW-01# sh run int po31

!Command: show running-config interface port-channel31
!Running configuration last done at: Fri Sep 12 02:43:43 2025
!Time: Tue Sep 16 00:14:03 2025

version 9.3(9) Bios:version 05.51

interface port-channel31
description SLCHOSTEDPFS02 - Bond 3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
vpc 31

SLC-SHRD-CORE-SW-01#

SLC-SHRD-CORE-SW-01# sh vpc 31

vPC status

Id Port Status Consistency Reason Active vlans

31 Po31 up success success 602,604,606,1105

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

SLC-SHRD-CORE-SW-01#

SLC-SHRD-CORE-SW-02# sh run int e1/31

!Command: show running-config interface Ethernet1/31
!Running configuration last done at: Fri Sep 12 02:21:29 2025
!Time: Tue Sep 16 00:12:38 2025

version 9.3(9) Bios:version 05.45

interface Ethernet1/31
description SLCHOSTEDPFS02 - WAN4
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
channel-group 31 mode active
no shutdown

SLC-SHRD-CORE-SW-02# sh run int po31

!Command: show running-config interface port-channel31
!Running configuration last done at: Fri Sep 12 02:21:29 2025
!Time: Tue Sep 16 00:12:51 2025

version 9.3(9) Bios:version 05.45

interface port-channel31
description SLCHOSTEDPFS02 - Bond 3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
vpc 31

SLC-SHRD-CORE-SW-02# sh vpc 31

vPC status

Id Port Status Consistency Reason Active vlans

31 Po31 up success success 602,604,606,1105

stephenw10

Hmm, did you try enabling the lacp debug?

I've not seen any issues with LACP in 25.07.1. I have a number running here and it's very widely used.

along5664

@stephenw10
I have a support ticket opened, but here is what I saw:
I will roll back to 24.11 as this is standby unit for production.

Here is all the information I collected while on 25.07.1, All ports are working except the lagg port. I tried slow and fast , no change
This is what I see at my switch. While on 25.07.1 on both switch ports
SLC-SHRD-CORE-SW-01# sh int e1/31
Ethernet1/31 is down (suspended(no LACP PDUs))

SLC-SHRD-CORE-SW-01# sh run int po31

!Command: show running-config interface port-channel31
!Running configuration last done at: Tue Sep 16 23:02:39 2025
!Time: Tue Sep 16 23:08:02 2025

version 9.3(9) Bios:version 05.51

interface port-channel31
description SLCHOSTEDPFS02 - Bond 3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
vpc 31

SLC-SHRD-CORE-SW-01# sh run int e1/31

!Command: show running-config interface Ethernet1/31
!Running configuration last done at: Tue Sep 16 23:02:39 2025
!Time: Tue Sep 16 23:08:07 2025

version 9.3(9) Bios:version 05.51

interface Ethernet1/31
description SLCHOSTEDPFS02 - WAN3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
channel-group 31 mode active
no shutdown

SLC-SHRD-CORE-SW-02# sh run int po31

!Command: show running-config interface port-channel31
!Running configuration last done at: Tue Sep 16 22:53:56 2025
!Time: Tue Sep 16 23:09:24 2025

version 9.3(9) Bios:version 05.45

interface port-channel31
description SLCHOSTEDPFS02 - Bond 3
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
vpc 31

SLC-SHRD-CORE-SW-02# sh run int e1/31

!Command: show running-config interface Ethernet1/31
!Running configuration last done at: Tue Sep 16 22:53:56 2025
!Time: Tue Sep 16 23:09:30 2025

version 9.3(9) Bios:version 05.45

interface Ethernet1/31
description SLCHOSTEDPFS02 - WAN4
switchport
switchport mode trunk
switchport trunk allowed vlan 602,604,606,1105
flowcontrol send on
channel-group 31 mode active
no shutdown

SLC-SHRD-CORE-SW-02#

[24.11-RELEASE][root@pslchostedpfs02.vesta.io]/root: ifconfig -vvv lagg0
resizewin: timeout reading from terminal
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: BOND3LAGG
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 90:ec:77:92:87:b2
hwaddr 00:00:00:00:00:00
inet6 fe80::92ec:77ff:fe92:87b2%lagg0 prefixlen 64 scopeid 0xd
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=14<USE_NUMA,LACP_STRICT>
flowid_shift: 16
lagg statistics:
active ports: 2
flapping: 0
lag id: [(8000,90-EC-77-92-87-B2,01B2,0000,0000),
(7F9B,00-23-04-EE-C0-58,801F,0000,0000)]
laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,90-EC-77-92-87-B2,01B2,8000,0005),
(7F9B,00-23-04-EE-C0-58,801F,8000,4179)]
laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,90-EC-77-92-87-B2,01B2,8000,0006),
(7F9B,00-23-04-EE-C0-58,801F,8000,0179)]
groups: lagg ALL_Interfaces
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: lagg0
[24.11-RELEASE][root@pslchostedpfs02.vesta.io]/root:

[25.07.1-RELEASE][root@pslchostedpfs02.vesta.io]/root: ifconfig -vvv lagg0
resizewin: timeout reading from terminal
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: BOND3LAGG
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 90:ec:77:92:87:b2
hwaddr 00:00:00:00:00:00
inet6 fe80::92ec:77ff:fe92:87b2%lagg0 prefixlen 64 scopeid 0xd
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=14<USE_NUMA,LACP_STRICT>
flowid_shift: 16
lagg statistics:
active ports: 0
flapping: 0
lag id: [(0000,00-00-00-00-00-00,0000,0000,0000),
(0000,00-00-00-00-00-00,0000,0000,0000)]
laggport: ix0 flags=0<> state=41<ACTIVITY,DEFAULTED>
[(8000,90-EC-77-92-87-B2,8005,8000,0005),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ix1 flags=0<> state=41<ACTIVITY,DEFAULTED>
[(8000,90-EC-77-92-87-B2,8006,8000,0006),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
groups: lagg ALL_Interfaces
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: lagg0
[25.07.1-RELEASE][root@pslchostedpfs02.vesta.io]/root:

sysctl net.link.lagg.lacp.debug=1

acp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
ix1: lacpdu receive
actor=(7F9B,00-23-04-EE-C0-58,801F,8000,0179)
actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
partner=(0000,00-00-00-00-00-00,0000,0000,0000)
partner.state=0
maxdelay=0
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator
lacp_select_tx_port_by_hash: no active aggregator

stephenw10

Can we assume that the ix0 and ix1 ports showed as UP and linked at 10G though when that happened?

Still slightly confused by the port down logs. We need to determine if this is some change in the ix driver causing link issues or in the lagg driver running over that.

As I say though both are commonly used and I'm not aware of any issues so this must be something subtle.

along5664

@stephenw10
I do see the ports come up on switch, then they go into susupend mode. Agree I think this is some driver issue or something causing pdu to not be sent. I am going to set this to a single port and run the upgrade without lagg to see if it works. Thanks for all the help.

along5664

This morning, I removed the lagg port and set the pfsense to use a single interface not in a lagg/lacp config.
The upgrade ran without any issues, and everything looks as it should.

For now we will change our devices to use only single ports and not lagg, until a solid resolution to the issues is discovered.
Thanks for all your help on this.

stephenw10

Hmm, so looks like a lagg/lacp issue then. Unless the ix driver is somehow dropping the pdus...

along5664

@stephenw10
We changed to using a sinlge port on our primary last night and the upgrade worked perfect, and port came up as expected. Def something with lagg/lacp.

stephenw10

OK I may have replicated this.... testing....

stephenw10

Hmm, maybe not. I was initially able see it fail to link but after rebooting it came up fine:

[25.07.1-RELEASE][root@8200-2.stevew.lan]/root: ifconfig -vvv lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAGG0
        options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 00:08:a2:12:e2:ca
        hwaddr 00:00:00:00:00:00
        inet6 fe80::208:a2ff:fe12:e2ca%lagg0 prefixlen 64 scopeid 0xe
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=14<USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,00-08-A2-12-E2-CA,01D2,0000,0000),
                 (0001,60-9C-9F-54-14-F2,561F,0000,0000)]
        laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-08-A2-12-E2-CA,01D2,8000,0005),
                 (0001,60-9C-9F-54-14-F2,561F,0001,0042)]
        laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-08-A2-12-E2-CA,01D2,8000,0006),
                 (0001,60-9C-9F-54-14-F2,561F,0001,0041)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        drivername: lagg0

I assume you rebooted and it still failed?

along5664

@stephenw10
Yes I rebooted several times. Did not seem to change anything on our side.
We are working fine now on single port.
Thnaks for looking into this further.

stephenw10

Mmm, if this is something more general I'd love to pin it down. Can't find the steps to replicate it again now though. I'll keep trying...

netboy

I upgraded from 24.11 to 25.0.7.1 on my netgate 2100 yesterday night without a hitch - off course i took screen shots of my various configuration and did have previous boot backup (and xml)