LACP LAGG in Silicom NICs

DaddyGo

@stephenw10 said in VPN tunnel is connected but packets are not getting transferred in pfsense 2.4.5, please suggest some solution.:

Steve

Hi Steve,

Sorry to ask you here...(-:
how much are you in the picture with LACP (LAG on pfSense) stuff and Silicom NICs?

I need to your help on this theme (if it is possible), not in pfSense, but I trust you: https://www.silicom-usa.com/pr/server-adapters/networking-adapters/gigabit-ethernet-networking-server-adapters/pe2g4sfpi35l-server-adapter/

btw:
as PM is not preferred by anyone,
sorry to ask you here

stephenw10

That should work. I know of no reason why it wouldn't unless you have one of the NICs with LAN bypass that require a special driver.

Do you have that NIC? Is it failing to connect?

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Do you have that NIC? Is it failing to connect?

Sorry for my late reply, now I get access to this hardware again...
-so this will be a pfSense which is designed for 10Gig and some direction 1Gig

The base is a Cisco UCS C220 M3, with the following network options:

2 ports LOM I350
PE2G4SFPI35L Server Adapter with I350AM4 (SFP 4x1Gig)
PE310G4I71L Server Adapter with Intel XL710BM1 (SFP 4x10Gig)

All interfaces are perfectly detected by the OS (that's a total of ten).
the problem is with the 4x1Gig card (PE2G4SFPI35L) when setting up 2 ports as LAGG with LACP...

the configuration is almost complete, but I faced the following under the interface tests:

step1: a test client (Intel i211) connected to the LAGG interface, the DHCP server on interface (LAGG0) detects a DHCP request, but returns an error that it is not possible to send 300 bytes of data on this interface...

something like this:

Notes:

configured for 10Gig NIC with the same setting LAGG / LACP - 2 ports), DHCP works without problems
strange short crashes, when adding 4x1Gig card settings in "System Tunables", such as EEE disable, FC disable
in this case the two LEDs on the current channel of the NIC flashes and the GUI is inaccessible for about 6 to 8 minutes
(this happens per interface, igb2 - 5)

there is no need to restart, after a while the GUI will recover and the LEDs will stop flashing

if I disassemble the LAGG all the interfaces here work perfectly...

stephenw10

Hmm, that is odd.

Does the LACP continue to show as UP?

Both links good in ifconfig -vvv lagg0 ?

Does it still do that if you do not have any custom loader variables or sysctls set?

Steve

DaddyGo

@stephenw10

THX :)

"Does the LACP continue to show as UP?" = yes (in the DHCP logs you can see that, the interface (lagg0) goes down, but it never goes down)

on the other hand there is no traffic on it, I tried now, there is not even PING towards MikroTik, but it seems completely "alive"

"Both links good in ifconfig -vvv lagg0 ?"

Does it still do that if you do not have any custom loader variables or sysctls set?

just the usual, these:

and
strangely on the other side, MikroTik (CSS610-8G-2S+IN) does not display LAGG

the LAGG as you can see, are on igb3-4
maybe I have to remove everything from loader.conf.local and so examine this way...(?)

what's going on in my head:

-does the group is affected, if I configure the parent intefaces separately?
(but this is not logical, unless I configure the lagg0 interface only and not the parents

like:

loader.conf.local igb3-4 (disable EEE / FC) (parents)
and / or sys. tunables...?

instead:
dev.lagg0..........etc?

The final question is why the 10G NIC is behaving properly...(?)

stephenw10

Mmm, that lagg does not look good.
I expect to see something more like:

[2.4.5-RELEASE][admin@7100.stevew.lan]/root: ifconfig -vvv lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
	ether 00:e0:ed:86:a6:8c
	inet6 fe80::2e0:edff:fe86:a68c%lagg0 prefixlen 64 scopeid 0x15
	inet 172.21.16.206 netmask 0xffffff00 broadcast 172.21.16.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=10<LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 0
	lag id: [(8000,00-E0-ED-86-A6-8C,02B2,0000,0000),
		 (0001,60-9C-9F-54-14-F2,561F,0000,0000)]
	laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0001),
		 (0001,60-9C-9F-54-14-F2,561F,0001,0041)]
	laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-E0-ED-86-A6-8C,02B2,8000,0002),
		 (0001,60-9C-9F-54-14-F2,561F,0001,0043)]

You have 0 active ports. Those that are shown are defaulted.

Something is mis-matched there.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Something is mis-matched there.

since I saw somewhere that the Silicom, a Netgate partner or vica versa....

I was hoping you had come across a similar Silicom issue...
setting up a LAGG is not a big deal, so the weird behavior is likely to be sought deeper... (NIC FW or I dont know)

so I do remove everything (loader.conf..... sys tunables) which is related to the NIC and try to set up the LAGG again

BTW:
have you ever seen the LEDs flashes when you save the "sys tunables" setting?
this is the strangest fact of all

+++edit:
yes what still conspicuous:

and you saw that weird thing is that DHCPOFFER is present in the DHCP log, ergo there must be some traffic on the interface

stephenw10

Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1

It looks like there are probably no lacpdus arriving from the switch for some reason.

Silicom bought ADI who designed a number of our systems. I've never really used their stand alone NICs though.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Try enabling lacp debugging and see if anything is happening:

I don't do it today, a few days and I'll be back.
Somewhere here is a Cisco SG350X-24 in the lab, I will try it with this switch too...
Thank you for your guidance so far...

@stephenw10 "Silicom bought ADI who designed a number of our systems."

I love the Silicom stuffs and I have never had a problem with it before...
I didn't know about "marriage", - ADI

Have a nice Christmas

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1

What seems certain is that the Silicom i350-F4 is causing the LAGG problem.... (igb...)

because with an Intel X710-D4 (4 ports 10Gig SFP) everything works perfectly

sysctl net.link.lagg.lacp.debug=1

ifconfig -vvv lagg0

DHCP on the LAGG interface is also problem-free with this NIC
the problem now is that we don't need 10Gig in this pfSense box

stephenw10

Do you see any incoming traffic in the lacp debug messages when using the igb NICs?

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Do you see any incoming traffic

Yes the DHCP req. it arrives but the answer to it doesn't go out, due to....

In the meantime, I talked about this with Silicom support...
OP ROM is disabled at the factory on these cards, so Cisco CIMC does not recognize NIC MAC addresses...

SILICOM:
"Hi,
This product is shipped without the PXE boot room enabled and it is not programmed on it."

it is likely that the special behavior is caused by this, so this is not a pfSense problem (Cisco + Silicom w/o OP ROM)
if I don't use the LAGG setting on the ports (interfaces), everything behaves as expected

now it seems to me that it only affects the LAGG settings, don't ask why - I have not seen such a thing

The solution will be a dual rate SFP module loaded with Intel code, so we will have 1 / 10G at our disposal
https://www.fs.com/de-en/products/36431.html

stephenw10

Hmm, the only time I've seen anything similar to this was on the very early 7100 ix ports. They worked in every way but not as part of an LACP LAGG. It's been a while since we saw it but if I recall they would see traffic coming in but seemingly never sent anything out.
Completely different NIC type though.

Steve

DaddyGo

@stephenw10 said in LACP LAGG in Silicom NICs:

Completely different NIC type though.

Yep, I think I'll wait a bit and test again under 2.5.(?)
although, if I read it correctly (somewhere), only "ixl" got a brand new driver under FB12