LACP LAGG Poor Performance & Errors

VioletDragon

Errors seems to go when i remove the 4th Port, if i only assign 3 Ports it is fine i have tried different Network Cards and still the same problem. Ive tried a HP 364T on different hardware and tried different cabling still the same problem.

FreeNAS and XCP-ng has 0 errors and have the same hardware.

stephenw10

Hmm, odd.

Do you see the expected throughput when there are no errors?

VioletDragon

Nope. i am limited to only 50 to 60mbps until i enable Hardware Large Receive Offloading then i can get way over 100mbps without any issues except errors then just stacks up and internet drops out when that is enabled.

stephenw10

I assume you are enabling LRO globally. It is actually added/removed from all the interfaces involved here?

What does ifcofig -a show?

Steve

VioletDragon

Here you go.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:60
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:61
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b0
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0 prefixlen 64 scopeid 0x9
	inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
	inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.50 prefixlen 64 scopeid 0xa
	inet 192.168.50.254 netmask 0xffffff00 broadcast 192.168.50.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 50 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.60 prefixlen 64 scopeid 0xb
	inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 60 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.70 prefixlen 64 scopeid 0xc
	inet 192.168.70.254 netmask 0xffffff00 broadcast 192.168.70.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 70 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.80 prefixlen 64 scopeid 0xd
	inet 192.168.80.254 netmask 0xffffff00 broadcast 192.168.80.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 80 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.90 prefixlen 64 scopeid 0xe
	inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 90 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.100 prefixlen 64 scopeid 0xf
	inet 192.168.100.254 netmask 0xffffff00 broadcast 192.168.100.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 100 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.101 prefixlen 64 scopeid 0x10
	inet 192.168.101.254 netmask 0xffffff00 broadcast 192.168.101.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 101 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.102: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.102 prefixlen 64 scopeid 0x11
	inet 192.168.102.254 netmask 0xffffff00 broadcast 192.168.102.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 102 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.103 prefixlen 64 scopeid 0x12
	inet 192.168.103.254 netmask 0xffffff00 broadcast 192.168.103.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 103 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.104: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.104 prefixlen 64 scopeid 0x13
	inet 192.168.104.254 netmask 0xffffff00 broadcast 192.168.104.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 104 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.105: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.105 prefixlen 64 scopeid 0x14
	inet 192.168.105.254 netmask 0xffffff00 broadcast 192.168.105.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 105 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.106: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.106 prefixlen 64 scopeid 0x15
	inet 192.168.106.254 netmask 0xffffff00 broadcast 192.168.106.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 106 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.40 prefixlen 64 scopeid 0x16
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 40 vlanpcp: 0 parent interface: lagg0
	groups: vlan
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
	inet6 fe80::21b:78ff:fe59:7c60%pppoe0 prefixlen 64 scopeid 0x17
	inet 81.150.180.216 --> 81.139.208.1 netmask 0xffffffff
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::21b:78ff:fe59:7c60%ovpns1 prefixlen 64 scopeid 0x18
	inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffff00
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun openvpn
	Opened by PID 81392

stephenw10

Yeah so no LRO on any em interface or on lagg0 or any VLAN on it.

So what happens when you enable it? Which is pretty much never required normally.

How does it fail?

Steve

VioletDragon

@stephenw10 if I enable it. I get great speeds 110mbps but after so long the Internet drops out then comes back up. WAN isn't disconnecting tho but with it disabled the Internet seems fine. When it is enabled the errors are Pilling up. What do you suggest?

VioletDragon

The Errors have always happened I tried different hardware such as motherboards, cpus, ram etc and network cards but always had issues with error in and out on the Lagg.

stephenw10

Hmm, well I would first try to find why you lose the WAN. What actually happens at that point?

Is it only WAN you lose?

Steve

VioletDragon

No. Everything on the network. Can't connect to webmail on my Mail Server which is on the same network when it happens. WAN is in a Vlan on the Lagg interface.

stephenw10

OK that sounds more like it's exhausting something. mbufs maybe. That would likely be shown on the console.

It certainly should log something.

VioletDragon

I set the mbufs to 10000 as recommended on the Tuning guide. So you think I should enable LRO then look at the logs?

It is just strange when LRO is enabled the errors are just counting up.

stephenw10

mbufs should be set to 1000000 unless you have low available RAM.

https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#mbuf-nmbclusters

Steve

VioletDragon

That is what I've done. Machine has 4gigs of RAM but only 35% is used.

stephenw10

And all NICs are still failing when LRO is enabled?

You should try to find out what's actually failing when that happens.

VioletDragon

I know it's not a hardware issue as I have tried different motherboards, with Cpus and network cards with different memory I have tried moving the Lagg ports on the switch to other ports and different cabling still the same issue.

Cool_Corona

Can you manually change the interface settings to full duplex in the GUI??

I would like to see if that makes a difference.

Run ifconfig -a afterwards

VioletDragon

i only have these options.

stephenw10

You would need to set that on the parent interfaces. And that's tricky because you can't assign them when they're in a lagg. You could only do it using ifconfig directly.

So you don't see any errors if you remove any one of the four links?

If you have 3 links in the lagg and enable LRO does it still stop responding?

Steve

VioletDragon

I get no errors with only 3 Ports assigned to the LAGG. The issue with everything isnt a problem with LRO because it still happens when it is disabled. Something is seriously broken here i did update to the latest version of pfSense which is when this problem started happening with drop outs. When the drop outs happen VLANs are not reachable and the internet is not reachable everything drops out for a min or so then comes back on for a few hours and then drops out again but none of the links or WAN is dropping out.