LAN with quad NIC configured with LAGG-LACP and VLANs

jumpace07

This is interesting.. I have 2 NIC's in a LAGG group with 2.1.5 and things work fine, upgrade to 2.2 and things go pear shaped. I just posted my own post regarding issues and this might be the same as nothing seems to work over the link. I didn't suspect the lagg, but something else but after this it very well could be the lagg not functioning properly as all my vlans are carried by them.

You might be onto something.

CrossEye

I posted a bug report for the issue.

https://redmine.pfsense.org/issues/4280

If more information is needed, I can definitely provide it.

However I definitely think this is an issue now, tested on two separate boxes now and both of them experienced the same issue with a fresh install and a very basic config on them.

Take care all,

stephenw10

What NICs are you using? Do you have any hardware off-loading enabled at all? Nothing in the logs? What does ifconfig report?

Steve

CrossEye

@stephenw10:

What NICs are you using? Do you have any hardware off-loading enabled at all? Nothing in the logs? What does ifconfig report?

Steve

Hey Steve,

I use two Dell Optiplex 3010 i5 @ 3.10GHz with 16GB of RAM and a Intel quad NIC.

I also just tested the same scenario with a Dell Optiplex 9010 and two additional single PCIe Intel NICs. The result was the same as before, as soon as I configured the LAGG, all traffic stopped passing.

I have no hardware off-loading and checking the logs reveals nothing. Also ifconfig shows the carrier signal as active and the interface as up.

As far as I can tell, It's like pfSense thinks the interface is working correctly.

mir

This setting: Hardware Checksum Offloading (system->advanced->network)
should actually be checked to prevent hardware offloading

mir

Maybe you could try to check this setting:
ARP Handling Suppress ARP messages
This option will suppress ARP log messages when multiple interfaces reside on the same broadcast domain

CrossEye

Ok so I created a VM with a fresh install of 2.2 and did the following,

System > Advanced > Networking
Checked "Disable hardware checksum offload"
Rebooted

Created a LAGG with the protocol set to LACP.
Created a new interface with the LAGG.
Created a new rule allowing any protocol on the new LAGG interface.

Started a Tcpdump on the interface and starting pinging, curling HTTP and HTTPS (to try and generate traffic) and nothing showed up on the Tcpdump.

Checked the routes and everything looks good. Even tried unchecking "Disable hardware checksum offload" and rebooting again and trying the above test… still did not work.

I am not at a loss as to why this is happening, anyone else have any ideas?

Thanks again all,

Gabriel

stephenw10

Hmm. If this was some bug in FreeBSD 10.1 I'd expect to see far more mailing/forum traffic about it. Also I'm sure someone must have tested a LAGG when 2.2 was in Beta/RC (though I haven't personally).
VM is kind of a special case, hardware off-loading doesn't really apply. Exactly what NICs are you using, which Intel cards?

Steve

stephenw10

I only see this:
https://lists.freebsd.org/pipermail/freebsd-net/2014-February/037756.html
Do you have the sysctl:
net.link.lagg.0.lacp.lacp_strict_mode: 1

Change it to 0 to go back to FreeBSD 9 behaviour. You may have to cycle the LAGG or wait a while as it says in the post.

Steve

CrossEye

@stephenw10:

Hmm. If this was some bug in FreeBSD 10.1 I'd expect to see far more mailing/forum traffic about it. Also I'm sure someone must have tested a LAGG when 2.2 was in Beta/RC (though I haven't personally).
VM is kind of a special case, hardware off-loading doesn't really apply. Exactly what NICs are you using, which Intel cards?

Steve

I couldn't agree more Steve but you are more than welcome to run the same test and verify my results. I quite possibility could be misconfiguring it but I have done this setup multiple times now with great results until pfSense 2.2.

The NICs I use are the quad Intel Pro/1000 I340-T4,
http://ark.intel.com/products/49186/Intel-Ethernet-Server-Adapter-I340-T4

I never did any 2.2 beta testing, so I cannot speak if it worked in beta versions but I know that all of the 2.1.x stable builds work great.

Please if anyone can shed light on this issue, I am all ears.

Thanks again,

Gabriel

CrossEye

@stephenw10:

I only see this:
https://lists.freebsd.org/pipermail/freebsd-net/2014-February/037756.html
Do you have the sysctl:
net.link.lagg.0.lacp.lacp_strict_mode: 1

Change it to 0 to go back to FreeBSD 9 behaviour. You may have to cycle the LAGG or wait a while as it says in the post.

Steve

Great find Steve and I will run the test on the VM now and later tonight repeat the test on the physical hardware.

Gabriel

jumpace07

Late last night I also built a fresh 2.2 VM and rebuilt everything manually, a totally fresh setup so nothing coming across from a dodgy upgrade or anything.

I am going to be testing tonight with my LACP LAGG connection to my switch which is currently working perfectly with 2.1.5.

Will post results as well.

cmb

The root issue is what I added to the 2.2 upgrade guide here.

Either enable active LACP on your switch, or disable strict mode as noted there. We'll change the default back to its previous setting for 2.2.1.
https://redmine.pfsense.org/issues/4308

Steve - thanks for saving me some time searching by finding that first. :)