LACP LAGG Poor Performance & Errors

VioletDragon

Hi folks,

I have a problem ive got a 4Gig LACP LAGG interface however the performance is bad until i enable Hardware Large Receive Offloading then i can get Gig Speeds across the VLANs 110mbps when disabled i only get around 55mbps but issue is when enabled and disabled i find the internet drops out then comes back on although WAN stays connected, another issue i have are Errors in and out, when i have Hardware Large Receive Offloading enable errors in and out stacks up but when disable the errors don't count up as bad.

Hardware im using, Intel S1200BTLRM, 4GB RAM, Xeon 1220L v2. CPU is not maxed out. Using onboard and PCI-E HP NC360T for 4 Port Lagg. Cisco SG300-28. Virtualization Server is also the same hardware with 32GB of RAM and Xeon 1220 V2, Storage Server is also the same hardware Xeon 1220 v2, 32GB and loaded with disks. All Servers have 4Gig LAGG.

Thanks.

Jack.

stephenw10

I assume you mean 110MBps/55Mbps (Mega Bytes per second) here?

How are you testing? Do you see that same throttling on any type of test?

Where are the errors you're seeing exactly?

Steve

VioletDragon

Correct yes. This is on transfers using NFS.

Errors in and out on the interface.

Screenshot from 2020-07-07 13-26-51.png

stephenw10

Try running netstat -i at the CLI to see errors on each interface in the lagg. You may have a bad link/NIC in there.

That error count in not significant though. I doubt it's contributing to the issue.

Steve

VioletDragon

@stephenw10 said in LACP LAGG Poor Performance & Errors:

netstat -i

Here is the output of netstat. looks like the issue is on em1?

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
em0    1500 <Link#1>      00:1b:78:0        53084679     0     0 37124563     0     0
em1    1500 <Link#2>      00:1b:78          62595631   851     0 160879503    0     0
em2    1500 <Link#3>      00:1b:780         80781899     0     0 46500177     0     0
em3    1500 <Link#4>      00:1b:78:         56655267     0     0  7814671     0     0
enc0*  1536 <Link#5>      enc0                     0     0     0        0     0     0
lo0   16384 <Link#6>      lo0                2976933     0     0  2976933     0     0
lo0       - localhost     localhost                0     -     -        0     -     -
lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        0     -     -
lo0       - your-net      localhost          2735632     -     -  2963262     -     -
pfsyn  1500 <Link#7>      pfsync0                  0     0     0        0     0     0
pflog 33160 <Link#8>      pflog0                   0     0     0    53615     0     0
lagg0  1500 <Link#9>      00:1b:78:59:7c:60 253117476  851     0 252318914  122     0
lagg0     - fe80::%lagg0/ fe80::21b:78ff:fe        0     -     -        2     -     -
lagg0     - 192.168.1.0/2 pfSense              20633     -     -    94884     -     -
lagg0     - 10.10.10.1/32 10.10.10.1            1291     -     -        0     -     -
lagg0  1500 <Link#10>     00:1b:78:59:7c:60    45783     0     0    62087     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.50.0/ 192.168.50.254        2529     -     -     2514     -     -
lagg0  1500 <Link#11>     00:1b:78:59:7c:60     3379     0     0     5851     6     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.60.0/ 192.168.60.254         312     -     -      151     -     -
lagg0  1500 <Link#12>     00:1b:78:59:7c:60 60392807     0     0 11903596     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.70.0/ 192.168.70.254        2695     -     -     8225     -     -
lagg0  1500 <Link#13>     00:1b:78:59:7c:60  1401008     0     0   865801    10     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        2     -     -
lagg0     - 192.168.80.0/ 192.168.80.254        8359     -     -    42261     -     -
lagg0  1500 <Link#14>     00:1b:78:59:7c:60 79678949     0     0 88751734     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.90.0/ 192.168.90.254       11033     -     -    10272     -     -
lagg0  1500 <Link#15>     00:1b:78:59:7c:60       16     0     0       16     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.100.0 192.168.100.254          8     -     -        8     -     -
lagg0  1500 <Link#16>     00:1b:78:59:7c:60   407946     0     0  1063255     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.101.0 192.168.101.254       2495     -     -   106581     -     -
lagg0  1500 <Link#17>     00:1b:78:59:7c:60        0     0     0        0     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        2     -     -
lagg0     - 192.168.102.0 192.168.102.254          0     -     -        0     -     -
lagg0  1500 <Link#18>     00:1b:78:59:7c:60      190     0     0  3130195     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.103.0 192.168.103.254          0     -     -        0     -     -
lagg0  1500 <Link#19>     00:1b:78:59:7c:60      158     0     0        0     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
lagg0     - 192.168.104.0 192.168.104.254          0     -     -        0     -     -
lagg0  1500 <Link#20>     00:1b:78:59:7c:60  3190559     0     0  3192721     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        0     -     -
lagg0     - 192.168.105.0 192.168.105.254         77     -     -       77     -     -
lagg0  1500 <Link#21>     00:1b:78:59:7c:60        0     0     0        0     8     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        2     -     -
lagg0     - 192.168.106.0 192.168.106.254          0     -     -        0     -     -
lagg0  1500 <Link#22>     00:1b:78:59:7c:60 32540809     0     0 31556923     7     0
lagg0     - fe80::%lagg0. fe80::21b:78ff:fe        0     -     -        1     -     -
pppoe  1492 <Link#23>     pppoe0            32538634     0     0 31554988     0     0
pppoe     - fe80::%pppoe0 fe80::21b:78ff:fe        0     -     -        0     -     -
pppoe     - 81.15 mail.violetdragon 10930806     -     -        1     -     -
ovpns  1500 <Link#24>     ovpns1            10704750     0     0 19707822     0     0
ovpns     - fe80::%ovpns1 fe80::21b:78ff:fe        0     -     -        0     -     -
ovpns     - 192.168.200.0 192.168.200.1       226058     -     -     4051     -     -

stephenw10

Yes, possibly. Try disconnecting that or removing it from the lagg.

851 is not a huge number though.

VioletDragon

Errors seems to go when i remove the 4th Port, if i only assign 3 Ports it is fine i have tried different Network Cards and still the same problem. Ive tried a HP 364T on different hardware and tried different cabling still the same problem.

FreeNAS and XCP-ng has 0 errors and have the same hardware.

stephenw10

Hmm, odd.

Do you see the expected throughput when there are no errors?

VioletDragon

Nope. i am limited to only 50 to 60mbps until i enable Hardware Large Receive Offloading then i can get way over 100mbps without any issues except errors then just stacks up and internet drops out when that is enabled.

stephenw10

I assume you are enabling LRO globally. It is actually added/removed from all the interfaces involved here?

What does ifcofig -a show?

Steve

VioletDragon

Here you go.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:60
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:61
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b0
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0 prefixlen 64 scopeid 0x9
	inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
	inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.50 prefixlen 64 scopeid 0xa
	inet 192.168.50.254 netmask 0xffffff00 broadcast 192.168.50.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 50 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.60 prefixlen 64 scopeid 0xb
	inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 60 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.70 prefixlen 64 scopeid 0xc
	inet 192.168.70.254 netmask 0xffffff00 broadcast 192.168.70.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 70 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.80 prefixlen 64 scopeid 0xd
	inet 192.168.80.254 netmask 0xffffff00 broadcast 192.168.80.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 80 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.90 prefixlen 64 scopeid 0xe
	inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 90 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.100 prefixlen 64 scopeid 0xf
	inet 192.168.100.254 netmask 0xffffff00 broadcast 192.168.100.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 100 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.101 prefixlen 64 scopeid 0x10
	inet 192.168.101.254 netmask 0xffffff00 broadcast 192.168.101.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 101 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.102: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.102 prefixlen 64 scopeid 0x11
	inet 192.168.102.254 netmask 0xffffff00 broadcast 192.168.102.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 102 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.103 prefixlen 64 scopeid 0x12
	inet 192.168.103.254 netmask 0xffffff00 broadcast 192.168.103.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 103 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.104: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.104 prefixlen 64 scopeid 0x13
	inet 192.168.104.254 netmask 0xffffff00 broadcast 192.168.104.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 104 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.105: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.105 prefixlen 64 scopeid 0x14
	inet 192.168.105.254 netmask 0xffffff00 broadcast 192.168.105.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 105 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.106: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.106 prefixlen 64 scopeid 0x15
	inet 192.168.106.254 netmask 0xffffff00 broadcast 192.168.106.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 106 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.40 prefixlen 64 scopeid 0x16
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 40 vlanpcp: 0 parent interface: lagg0
	groups: vlan
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
	inet6 fe80::21b:78ff:fe59:7c60%pppoe0 prefixlen 64 scopeid 0x17
	inet 81.150.180.216 --> 81.139.208.1 netmask 0xffffffff
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::21b:78ff:fe59:7c60%ovpns1 prefixlen 64 scopeid 0x18
	inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffff00
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun openvpn
	Opened by PID 81392

stephenw10

Yeah so no LRO on any em interface or on lagg0 or any VLAN on it.

So what happens when you enable it? Which is pretty much never required normally.

How does it fail?

Steve

VioletDragon

@stephenw10 if I enable it. I get great speeds 110mbps but after so long the Internet drops out then comes back up. WAN isn't disconnecting tho but with it disabled the Internet seems fine. When it is enabled the errors are Pilling up. What do you suggest?

VioletDragon

The Errors have always happened I tried different hardware such as motherboards, cpus, ram etc and network cards but always had issues with error in and out on the Lagg.

stephenw10

Hmm, well I would first try to find why you lose the WAN. What actually happens at that point?

Is it only WAN you lose?

Steve

VioletDragon

No. Everything on the network. Can't connect to webmail on my Mail Server which is on the same network when it happens. WAN is in a Vlan on the Lagg interface.

stephenw10

OK that sounds more like it's exhausting something. mbufs maybe. That would likely be shown on the console.

It certainly should log something.

VioletDragon

I set the mbufs to 10000 as recommended on the Tuning guide. So you think I should enable LRO then look at the logs?

It is just strange when LRO is enabled the errors are just counting up.

stephenw10

mbufs should be set to 1000000 unless you have low available RAM.

https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#mbuf-nmbclusters

Steve

VioletDragon

That is what I've done. Machine has 4gigs of RAM but only 35% is used.