LAGG (LACP) - UniFi Switch (16XG)

kklouzal

I've tried every troubleshooting step with the LAGG in a bridge and as a standalone interface with appropriate firewall rules to allow traffic and no combination will allow packets to pass using LACP.

Currently ROUNDROBIN is working fine and in bridge mode however I would prefer to get it setup using LACP.

It is a bit troubling that simply changing the LAGG Protocol to LACP then back to ROUNDROBIN breaks the system again requiring me to fuss around with the switch and set two random unused ports as aggregate before packets will start passing once more.

chpalmer

LAGG (LACP) - UniFi Switch (16XG):

ifconfig -v lagg0

Will your Unifi Switch work with while your pfsense box has a MAC address on that LAGG of 00:00:00:00:00:00?

Yours-

Mine-

kklouzal

Is there a way to force the Lag ID? I tried directly setting the MAC Address on lagg0 however lag id stayed all zeros.

Derelict

Yeah, but that might be the switch.

chpalmer

@derelict

On my picture that is the MAC address that I spoofed on my WAN page. My modem is the other end of the LAGG in my case.

I would assume that his case would be similar.. ??

chpalmer

@kklouzal said in LAGG (LACP) - UniFi Switch (16XG):

Is there a way to force the Lag ID? I tried directly setting the MAC Address on lagg0 however lag id stayed all zeros.

Make sure the address you are trying does not exist anywhere else in your system..

The other issue I see is that both your ports appear to have the same MAC address.. Are you sure your ports are not in some kind of switch mode?

kklouzal

The only difference I can see between my output and yours from the image is that LAG ID is all 0's for mine and yours is set.

Both of your ports are using the same MAC Address too
lag id: -------------- 00-90-7f-88-b4-2e & 02-10-18-3a-41-f1
laggport: em0 - 00-90-7f-88-b4-2e & 02-10-18-3a-41-f1
laggport: em1 - 00-90-7f-88-b4-2e & 02-10-18-3a-41-f1

For your setup, I would assume that 00-90-7f-88-b4-2e is the physical address of em0/em1 on PfSense and 02-10-18-3a-41-f1 is the physical address of your modem, each device on both ends have multiple ports on the same adapter so they are sharing a physical address.
Mine is doing the same thing except with the Chelsio card and my UniFi 16XG switch

lag id: ------------------ 00-00-00-00-00-00 - 00-00-00-00-00-00
laggport: cxgbe0 - 98-be-94-12-d5-e0 - b4-fb-e4-50-50-16
laggport: cxgbe1 - 98-be-94-12-d5-e0 - b4-fb-e4-50-50-16

lag id of all 0's is telling me the link is not setting itself up properly. Switching over to ROUNDROBIN allows packets to pass but only after doing that tricky/hacky thing of going over to the switch and setting two unused ports as aggregate, which will kick off the link and get packets moving, then unaggregating those ports.

I'm leaning more towards the side of something being wrong on the UniFi side of things here. I can't find mention of this problem anywhere else on the netgate forums or unifi forums so in all reality I probably have something misconfigured. There aren't many dials to turn and switches to flip without digging into the CLI on our switch. LACP should just work out of the box after aggregating two ports on the switch side.

chpalmer

Im looking at your 1st picture at the top of the thread here.

That looks strange to me. Both ports should have an HW: address I believe. And they should be different.

Derelict

Two ports in LACP have the same MAC address. It's perfectly normal.

[2.4.4-RELEASE][root@fw]/root: ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:08:a2:0a:59:3f
        inet6 fe80::208:a2ff:fe0a:593f%lagg0 prefixlen 64 scopeid 0xb 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: lagg 
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=10<LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,00-08-A2-0A-59-3F,016B,0000,0000),
                 (0001,CC-4E-24-53-94-00,4E21,0000,0000)]
        laggport: igb4 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-08-A2-0A-59-3F,016B,8000,0005),
                 (0001,CC-4E-24-53-94-00,4E21,0001,0023)]
        laggport: igb5 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,00-08-A2-0A-59-3F,016B,8000,0006),
                 (0001,CC-4E-24-53-94-00,4E21,0001,0024)]
[2.4.4-RELEASE][root@fw]/root: ifconfig -v igb4
igb4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:08:a2:0a:59:3f
        hwaddr 00:08:a2:0a:59:3f
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
[2.4.4-RELEASE][root@fw]/root: ifconfig -v igb5
igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:08:a2:0a:59:3f
        hwaddr 00:08:a2:0a:59:40
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

chpalmer

The 2e address in my picture here is the MAC I spoofed on my WAN page.

Derelict

OK?

Is that em0 or em1?

What does ifconfig -v show for em0 and em1?

kklouzal

I was able to get Dynamic 802.3ad LACP working between the switch and a windows 10 machine with no problems at all. The only log entries I can find related to this issue are these here:

cxgbe0: Interface stopped DISTRIBUTING, possible flapping
cxgbe1: Interface stopped DISTRIBUTING, possible flapping

Derelict

And what does the switch say?

I can get LACP running between my Brocade, Cisco, and D-Link switches with no problems at all. If your experience points to pfSense, mine points to your switch.

kklouzal

I'm not trying to play a whose at fault game here, just trying to pin down the issue so it can be corrected.

Only option left to try is a different NIC and see if that changes things. There could be something physically wrong with the card or with the FreeBSD driver being used, it's an older T4 Chelsio adapter. I'll try one of the built in Intel adapters and report back.

chpalmer

@derelict

It says the same thing that the picture shows em0 ends with 26 em1 ends with 27 my spoofed MAC is 2e

em0:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 00:90:7f:88:b4:2e
hwaddr 00:90:7f:88:b4:26
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

em1:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 00:90:7f:88:b4:2e
hwaddr 00:90:7f:88:b4:27
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

kklouzal

So after using two of the integrated Intel ports to setup the LAG everything is working fine and in bridge mode too. It was super easy and straightforward, just aggregate the ports on the UniFi 16XG and setup the LAG interface on PfSense as LACP, add to bridge, done.

So this leaves us with the conclusion something is broken with the Chelsio card when attempting to configure a LAG. I have no way of knowing if it's the physical card at fault or if there is a driver issue here. I'd like to say this is a driver issue as there have been no troubles with this card thus far. It's also an older T4 adapter, most people will be using T5's and T6's which may not have any issues.

Can anyone else verify their T4 card works with LACP? I'd like to get another users confirmation before spending $500 on a new adapter.

Derelict

@chpalmer What interface is assigned to WAN and has the spoofed MAC address set?

chpalmer

em0 and em1 are assigned to a LAGG LACP

LAGG is assigned as WAN.

I have an MB8600 cable modem on the other end of the LAGG.

Derelict

OK then the MAC address should be spoofed. The MAC address on the LAGG should also be the spoofed MAC. That is exactly what would be expected.