LACP LAGG in Silicom NICs

DaddyGo

THX :)

"Does the LACP continue to show as UP?" = yes (in the DHCP logs you can see that, the interface (lagg0) goes down, but it never goes down)

on the other hand there is no traffic on it, I tried now, there is not even PING towards MikroTik, but it seems completely "alive"

"Both links good in ifconfig -vvv lagg0 ?"

Does it still do that if you do not have any custom loader variables or sysctls set?

just the usual, these:

and
strangely on the other side, MikroTik (CSS610-8G-2S+IN) does not display LAGG

the LAGG as you can see, are on igb3-4
maybe I have to remove everything from loader.conf.local and so examine this way...(?)

what's going on in my head:

-does the group is affected, if I configure the parent intefaces separately?
(but this is not logical, unless I configure the lagg0 interface only and not the parents

like:

loader.conf.local igb3-4 (disable EEE / FC) (parents)
and / or sys. tunables...?

instead:
dev.lagg0..........etc?

The final question is why the 10G NIC is behaving properly...(?)

stephenw10

Mmm, that lagg does not look good.
I expect to see something more like:

[2.4.5-RELEASE][admin@7100.stevew.lan]/root: ifconfig -vvv lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
	ether 00:e0:ed:86:a6:8c
	inet6 fe80::2e0:edff:fe86:a68c%lagg0 prefixlen 64 scopeid 0x15
	inet 172.21.16.206 netmask 0xffffff00 broadcast 172.21.16.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=10<LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 0
	lag id: [(8000,00-E0-ED-86-A6-8C,02B2,0000,0000),
		 (0001,60-9C-9F-54-14-F2,561F,0000,0000)]
	laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0001),
		 (0001,60-9C-9F-54-14-F2,561F,0001,0041)]
	laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0002),
		 (0001,60-9C-9F-54-14-F2,561F,0001,0043)]

You have 0 active ports. Those that are shown are defaulted.

Something is mis-matched there.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Something is mis-matched there.

since I saw somewhere that the Silicom, a Netgate partner or vica versa....

I was hoping you had come across a similar Silicom issue...
setting up a LAGG is not a big deal, so the weird behavior is likely to be sought deeper... (NIC FW or I dont know)

so I do remove everything (loader.conf..... sys tunables) which is related to the NIC and try to set up the LAGG again

BTW:
have you ever seen the LEDs flashes when you save the "sys tunables" setting?
this is the strangest fact of all

+++edit:
yes what still conspicuous:

and you saw that weird thing is that DHCPOFFER is present in the DHCP log, ergo there must be some traffic on the interface

stephenw10

Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1

It looks like there are probably no lacpdus arriving from the switch for some reason.

Silicom bought ADI who designed a number of our systems. I've never really used their stand alone NICs though.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Try enabling lacp debugging and see if anything is happening:

I don't do it today, a few days and I'll be back.
Somewhere here is a Cisco SG350X-24 in the lab, I will try it with this switch too...
Thank you for your guidance so far...

@stephenw10 "Silicom bought ADI who designed a number of our systems."

I love the Silicom stuffs and I have never had a problem with it before...
I didn't know about "marriage", - ADI

Have a nice Christmas

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1

What seems certain is that the Silicom i350-F4 is causing the LAGG problem.... (igb...)

because with an Intel X710-D4 (4 ports 10Gig SFP) everything works perfectly

sysctl net.link.lagg.lacp.debug=1

ifconfig -vvv lagg0

DHCP on the LAGG interface is also problem-free with this NIC
the problem now is that we don't need 10Gig in this pfSense box

stephenw10

Do you see any incoming traffic in the lacp debug messages when using the igb NICs?

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Do you see any incoming traffic

Yes the DHCP req. it arrives but the answer to it doesn't go out, due to....

In the meantime, I talked about this with Silicom support...
OP ROM is disabled at the factory on these cards, so Cisco CIMC does not recognize NIC MAC addresses...

SILICOM:
"Hi,
This product is shipped without the PXE boot room enabled and it is not programmed on it."

it is likely that the special behavior is caused by this, so this is not a pfSense problem (Cisco + Silicom w/o OP ROM)
if I don't use the LAGG setting on the ports (interfaces), everything behaves as expected

now it seems to me that it only affects the LAGG settings, don't ask why - I have not seen such a thing

The solution will be a dual rate SFP module loaded with Intel code, so we will have 1 / 10G at our disposal
https://www.fs.com/de-en/products/36431.html

stephenw10

Hmm, the only time I've seen anything similar to this was on the very early 7100 ix ports. They worked in every way but not as part of an LACP LAGG. It's been a while since we saw it but if I recall they would see traffic coming in but seemingly never sent anything out.
Completely different NIC type though.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Completely different NIC type though.

Yep, I think I'll wait a bit and test again under 2.5.(?)
although, if I read it correctly (somewhere), only "ixl" got a brand new driver under FB12