LACP LAGG in Silicom NICs
-
@stephenw10 said in VPN tunnel is connected but packets are not getting transferred in pfsense 2.4.5, please suggest some solution.:
Steve
Hi Steve,
Sorry to ask you here...(-:
how much are you in the picture with LACP (LAG on pfSense) stuff and Silicom NICs?I need to your help on this theme (if it is possible), not in pfSense, but I trust you: https://www.silicom-usa.com/pr/server-adapters/networking-adapters/gigabit-ethernet-networking-server-adapters/pe2g4sfpi35l-server-adapter/
btw:
as PM is not preferred by anyone,
sorry to ask you here -
That should work. I know of no reason why it wouldn't unless you have one of the NICs with LAN bypass that require a special driver.
Do you have that NIC? Is it failing to connect?
Steve
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Do you have that NIC? Is it failing to connect?
Sorry for my late reply, now I get access to this hardware again...
-so this will be a pfSense which is designed for 10Gig and some direction 1GigThe base is a Cisco UCS C220 M3, with the following network options:
- 2 ports LOM I350
- PE2G4SFPI35L Server Adapter with I350AM4 (SFP 4x1Gig)
- PE310G4I71L Server Adapter with Intel XL710BM1 (SFP 4x10Gig)
All interfaces are perfectly detected by the OS (that's a total of ten).
the problem is with the 4x1Gig card (PE2G4SFPI35L) when setting up 2 ports as LAGG with LACP...the configuration is almost complete, but I faced the following under the interface tests:
step1: a test client (Intel i211) connected to the LAGG interface, the DHCP server on interface (LAGG0) detects a DHCP request, but returns an error that it is not possible to send 300 bytes of data on this interface...
something like this:
Notes:
-
configured for 10Gig NIC with the same setting LAGG / LACP - 2 ports), DHCP works without problems
-
strange short crashes, when adding 4x1Gig card settings in "System Tunables", such as EEE disable, FC disable
in this case the two LEDs on the current channel of the NIC flashes and the GUI is inaccessible for about 6 to 8 minutes
(this happens per interface, igb2 - 5)
there is no need to restart, after a while the GUI will recover and the LEDs will stop flashing
if I disassemble the LAGG all the interfaces here work perfectly...
-
Hmm, that is odd.
Does the LACP continue to show as UP?
Both links good in
ifconfig -vvv lagg0
?Does it still do that if you do not have any custom loader variables or sysctls set?
Steve
-
THX :)
"Does the LACP continue to show as UP?" = yes (in the DHCP logs you can see that, the interface (lagg0) goes down, but it never goes down)
- on the other hand there is no traffic on it, I tried now, there is not even PING towards MikroTik, but it seems completely "alive"
"Both links good in ifconfig -vvv lagg0 ?"
Does it still do that if you do not have any custom loader variables or sysctls set?
just the usual, these:
and
strangely on the other side, MikroTik (CSS610-8G-2S+IN) does not display LAGGthe LAGG as you can see, are on igb3-4
maybe I have to remove everything from loader.conf.local and so examine this way...(?)what's going on in my head:
-does the group is affected, if I configure the parent intefaces separately?
(but this is not logical, unless I configure the lagg0 interface only and not the parentslike:
loader.conf.local igb3-4 (disable EEE / FC) (parents)
and / or sys. tunables...?instead:
dev.lagg0..........etc?The final question is why the 10G NIC is behaving properly...(?)
-
Mmm, that lagg does not look good.
I expect to see something more like:[2.4.5-RELEASE][admin@7100.stevew.lan]/root: ifconfig -vvv lagg0 lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO> ether 00:e0:ed:86:a6:8c inet6 fe80::2e0:edff:fe86:a68c%lagg0 prefixlen 64 scopeid 0x15 inet 172.21.16.206 netmask 0xffffff00 broadcast 172.21.16.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 lagg options: flags=10<LACP_STRICT> flowid_shift: 16 lagg statistics: active ports: 2 flapping: 0 lag id: [(8000,00-E0-ED-86-A6-8C,02B2,0000,0000), (0001,60-9C-9F-54-14-F2,561F,0000,0000)] laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> [(8000,00-E0-ED-86-A6-8C,02B2,8000,0001), (0001,60-9C-9F-54-14-F2,561F,0001,0041)] laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> [(8000,00-E0-ED-86-A6-8C,02B2,8000,0002), (0001,60-9C-9F-54-14-F2,561F,0001,0043)]
You have 0 active ports. Those that are shown are defaulted.
Something is mis-matched there.
Steve
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Something is mis-matched there.
since I saw somewhere that the Silicom, a Netgate partner or vica versa....
I was hoping you had come across a similar Silicom issue...
setting up a LAGG is not a big deal, so the weird behavior is likely to be sought deeper... (NIC FW or I dont know)so I do remove everything (loader.conf..... sys tunables) which is related to the NIC and try to set up the LAGG again
BTW:
have you ever seen the LEDs flashes when you save the "sys tunables" setting?
this is the strangest fact of all+++edit:
yes what still conspicuous:and you saw that weird thing is that DHCPOFFER is present in the DHCP log, ergo there must be some traffic on the interface
-
Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1
It looks like there are probably no lacpdus arriving from the switch for some reason.
Silicom bought ADI who designed a number of our systems. I've never really used their stand alone NICs though.
Steve
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Try enabling lacp debugging and see if anything is happening:
I don't do it today, a few days and I'll be back.
Somewhere here is a Cisco SG350X-24 in the lab, I will try it with this switch too...
Thank you for your guidance so far...@stephenw10 "Silicom bought ADI who designed a number of our systems."
I love the Silicom stuffs and I have never had a problem with it before...
I didn't know about "marriage", - ADIHave a nice Christmas
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Try enabling lacp debugging and see if anything is happening:
sysctl net.link.lagg.lacp.debug=1What seems certain is that the Silicom i350-F4 is causing the LAGG problem.... (igb...)
because with an Intel X710-D4 (4 ports 10Gig SFP) everything works perfectly
sysctl net.link.lagg.lacp.debug=1
ifconfig -vvv lagg0
DHCP on the LAGG interface is also problem-free with this NIC
the problem now is that we don't need 10Gig in this pfSense box -
Do you see any incoming traffic in the lacp debug messages when using the igb NICs?
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Do you see any incoming traffic
Yes the DHCP req. it arrives but the answer to it doesn't go out, due to....
In the meantime, I talked about this with Silicom support...
OP ROM is disabled at the factory on these cards, so Cisco CIMC does not recognize NIC MAC addresses...SILICOM:
"Hi,
This product is shipped without the PXE boot room enabled and it is not programmed on it."it is likely that the special behavior is caused by this, so this is not a pfSense problem (Cisco + Silicom w/o OP ROM)
if I don't use the LAGG setting on the ports (interfaces), everything behaves as expectednow it seems to me that it only affects the LAGG settings, don't ask why - I have not seen such a thing
The solution will be a dual rate SFP module loaded with Intel code, so we will have 1 / 10G at our disposal
https://www.fs.com/de-en/products/36431.html -
Hmm, the only time I've seen anything similar to this was on the very early 7100 ix ports. They worked in every way but not as part of an LACP LAGG. It's been a while since we saw it but if I recall they would see traffic coming in but seemingly never sent anything out.
Completely different NIC type though.Steve
-
@stephenw10 said in LACP LAGG in Silicom NICs:
Completely different NIC type though.
Yep, I think I'll wait a bit and test again under 2.5.(?)
although, if I read it correctly (somewhere), only "ixl" got a brand new driver under FB12