LACP LAGG Poor Performance & Errors
-
Hi folks,
I have a problem ive got a 4Gig LACP LAGG interface however the performance is bad until i enable Hardware Large Receive Offloading then i can get Gig Speeds across the VLANs 110mbps when disabled i only get around 55mbps but issue is when enabled and disabled i find the internet drops out then comes back on although WAN stays connected, another issue i have are Errors in and out, when i have Hardware Large Receive Offloading enable errors in and out stacks up but when disable the errors don't count up as bad.
Hardware im using, Intel S1200BTLRM, 4GB RAM, Xeon 1220L v2. CPU is not maxed out. Using onboard and PCI-E HP NC360T for 4 Port Lagg. Cisco SG300-28. Virtualization Server is also the same hardware with 32GB of RAM and Xeon 1220 V2, Storage Server is also the same hardware Xeon 1220 v2, 32GB and loaded with disks. All Servers have 4Gig LAGG.
Thanks.
Jack.
-
I assume you mean 110MBps/55Mbps (Mega Bytes per second) here?
How are you testing? Do you see that same throttling on any type of test?
Where are the errors you're seeing exactly?
Steve
-
Correct yes. This is on transfers using NFS.
Errors in and out on the interface.
-
Try running
netstat -i
at the CLI to see errors on each interface in the lagg. You may have a bad link/NIC in there.That error count in not significant though. I doubt it's contributing to the issue.
Steve
-
@stephenw10 said in LACP LAGG Poor Performance & Errors:
netstat -i
Here is the output of netstat. looks like the issue is on em1?
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll em0 1500 <Link#1> 00:1b:78:0 53084679 0 0 37124563 0 0 em1 1500 <Link#2> 00:1b:78 62595631 851 0 160879503 0 0 em2 1500 <Link#3> 00:1b:780 80781899 0 0 46500177 0 0 em3 1500 <Link#4> 00:1b:78: 56655267 0 0 7814671 0 0 enc0* 1536 <Link#5> enc0 0 0 0 0 0 0 lo0 16384 <Link#6> lo0 2976933 0 0 2976933 0 0 lo0 - localhost localhost 0 - - 0 - - lo0 - fe80::%lo0/64 fe80::1%lo0 0 - - 0 - - lo0 - your-net localhost 2735632 - - 2963262 - - pfsyn 1500 <Link#7> pfsync0 0 0 0 0 0 0 pflog 33160 <Link#8> pflog0 0 0 0 53615 0 0 lagg0 1500 <Link#9> 00:1b:78:59:7c:60 253117476 851 0 252318914 122 0 lagg0 - fe80::%lagg0/ fe80::21b:78ff:fe 0 - - 2 - - lagg0 - 192.168.1.0/2 pfSense 20633 - - 94884 - - lagg0 - 10.10.10.1/32 10.10.10.1 1291 - - 0 - - lagg0 1500 <Link#10> 00:1b:78:59:7c:60 45783 0 0 62087 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.50.0/ 192.168.50.254 2529 - - 2514 - - lagg0 1500 <Link#11> 00:1b:78:59:7c:60 3379 0 0 5851 6 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.60.0/ 192.168.60.254 312 - - 151 - - lagg0 1500 <Link#12> 00:1b:78:59:7c:60 60392807 0 0 11903596 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.70.0/ 192.168.70.254 2695 - - 8225 - - lagg0 1500 <Link#13> 00:1b:78:59:7c:60 1401008 0 0 865801 10 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 2 - - lagg0 - 192.168.80.0/ 192.168.80.254 8359 - - 42261 - - lagg0 1500 <Link#14> 00:1b:78:59:7c:60 79678949 0 0 88751734 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.90.0/ 192.168.90.254 11033 - - 10272 - - lagg0 1500 <Link#15> 00:1b:78:59:7c:60 16 0 0 16 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.100.0 192.168.100.254 8 - - 8 - - lagg0 1500 <Link#16> 00:1b:78:59:7c:60 407946 0 0 1063255 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.101.0 192.168.101.254 2495 - - 106581 - - lagg0 1500 <Link#17> 00:1b:78:59:7c:60 0 0 0 0 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 2 - - lagg0 - 192.168.102.0 192.168.102.254 0 - - 0 - - lagg0 1500 <Link#18> 00:1b:78:59:7c:60 190 0 0 3130195 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.103.0 192.168.103.254 0 - - 0 - - lagg0 1500 <Link#19> 00:1b:78:59:7c:60 158 0 0 0 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - lagg0 - 192.168.104.0 192.168.104.254 0 - - 0 - - lagg0 1500 <Link#20> 00:1b:78:59:7c:60 3190559 0 0 3192721 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 0 - - lagg0 - 192.168.105.0 192.168.105.254 77 - - 77 - - lagg0 1500 <Link#21> 00:1b:78:59:7c:60 0 0 0 0 8 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 2 - - lagg0 - 192.168.106.0 192.168.106.254 0 - - 0 - - lagg0 1500 <Link#22> 00:1b:78:59:7c:60 32540809 0 0 31556923 7 0 lagg0 - fe80::%lagg0. fe80::21b:78ff:fe 0 - - 1 - - pppoe 1492 <Link#23> pppoe0 32538634 0 0 31554988 0 0 pppoe - fe80::%pppoe0 fe80::21b:78ff:fe 0 - - 0 - - pppoe - 81.15 mail.violetdragon 10930806 - - 1 - - ovpns 1500 <Link#24> ovpns1 10704750 0 0 19707822 0 0 ovpns - fe80::%ovpns1 fe80::21b:78ff:fe 0 - - 0 - - ovpns - 192.168.200.0 192.168.200.1 226058 - - 4051 - -
-
Yes, possibly. Try disconnecting that or removing it from the lagg.
851 is not a huge number though.
-
Errors seems to go when i remove the 4th Port, if i only assign 3 Ports it is fine i have tried different Network Cards and still the same problem. Ive tried a HP 364T on different hardware and tried different cabling still the same problem.
FreeNAS and XCP-ng has 0 errors and have the same hardware.
-
Hmm, odd.
Do you see the expected throughput when there are no errors?
-
Nope. i am limited to only 50 to 60mbps until i enable Hardware Large Receive Offloading then i can get way over 100mbps without any issues except errors then just stacks up and internet drops out when that is enabled.
-
I assume you are enabling LRO globally. It is actually added/removed from all the interfaces involved here?
What does
ifcofig -a
show?Steve
-
Here you go.
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:78:59:7c:60 hwaddr 00:1b:78:59:7c:60 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:78:59:7c:60 hwaddr 00:1b:78:59:7c:61 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:78:59:7c:60 hwaddr 00:1e:67:a1:8f:b1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:78:59:7c:60 hwaddr 00:1e:67:a1:8f:b0 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active enc0: flags=0<> metric 0 mtu 1536 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: enc lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync pflog0: flags=100<PROMISC> metric 0 mtu 33160 groups: pflog lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0 prefixlen 64 scopeid 0x9 inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255 inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.50 prefixlen 64 scopeid 0xa inet 192.168.50.254 netmask 0xffffff00 broadcast 192.168.50.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 50 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.60 prefixlen 64 scopeid 0xb inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 60 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.70 prefixlen 64 scopeid 0xc inet 192.168.70.254 netmask 0xffffff00 broadcast 192.168.70.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 70 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.80 prefixlen 64 scopeid 0xd inet 192.168.80.254 netmask 0xffffff00 broadcast 192.168.80.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 80 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.90 prefixlen 64 scopeid 0xe inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 90 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.100 prefixlen 64 scopeid 0xf inet 192.168.100.254 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 100 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.101 prefixlen 64 scopeid 0x10 inet 192.168.101.254 netmask 0xffffff00 broadcast 192.168.101.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 101 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.102: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.102 prefixlen 64 scopeid 0x11 inet 192.168.102.254 netmask 0xffffff00 broadcast 192.168.102.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 102 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.103 prefixlen 64 scopeid 0x12 inet 192.168.103.254 netmask 0xffffff00 broadcast 192.168.103.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 103 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.104: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.104 prefixlen 64 scopeid 0x13 inet 192.168.104.254 netmask 0xffffff00 broadcast 192.168.104.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 104 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.105: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.105 prefixlen 64 scopeid 0x14 inet 192.168.105.254 netmask 0xffffff00 broadcast 192.168.105.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 105 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.106: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.106 prefixlen 64 scopeid 0x15 inet 192.168.106.254 netmask 0xffffff00 broadcast 192.168.106.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 106 vlanpcp: 0 parent interface: lagg0 groups: vlan lagg0.40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 00:1b:78:59:7c:60 inet6 fe80::21b:78ff:fe59:7c60%lagg0.40 prefixlen 64 scopeid 0x16 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 40 vlanpcp: 0 parent interface: lagg0 groups: vlan pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492 inet6 fe80::21b:78ff:fe59:7c60%pppoe0 prefixlen 64 scopeid 0x17 inet 81.150.180.216 --> 81.139.208.1 netmask 0xffffffff nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet6 fe80::21b:78ff:fe59:7c60%ovpns1 prefixlen 64 scopeid 0x18 inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffff00 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: tun openvpn Opened by PID 81392
-
Yeah so no LRO on any em interface or on lagg0 or any VLAN on it.
So what happens when you enable it? Which is pretty much never required normally.
How does it fail?
Steve
-
@stephenw10 if I enable it. I get great speeds 110mbps but after so long the Internet drops out then comes back up. WAN isn't disconnecting tho but with it disabled the Internet seems fine. When it is enabled the errors are Pilling up. What do you suggest?
-
The Errors have always happened I tried different hardware such as motherboards, cpus, ram etc and network cards but always had issues with error in and out on the Lagg.
-
Hmm, well I would first try to find why you lose the WAN. What actually happens at that point?
Is it only WAN you lose?
Steve
-
No. Everything on the network. Can't connect to webmail on my Mail Server which is on the same network when it happens. WAN is in a Vlan on the Lagg interface.
-
OK that sounds more like it's exhausting something. mbufs maybe. That would likely be shown on the console.
It certainly should log something.
-
I set the mbufs to 10000 as recommended on the Tuning guide. So you think I should enable LRO then look at the logs?
It is just strange when LRO is enabled the errors are just counting up.
-
mbufs should be set to 1000000 unless you have low available RAM.
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#mbuf-nmbclusters
Steve
-
That is what I've done. Machine has 4gigs of RAM but only 35% is used.