LACP LAGG Poor Performance & Errors

stephenw10

I assume you are enabling LRO globally. It is actually added/removed from all the interfaces involved here?

What does ifcofig -a show?

Steve

VioletDragon

Here you go.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:60
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1b:78:59:7c:61
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	hwaddr 00:1e:67:a1:8f:b0
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0 prefixlen 64 scopeid 0x9
	inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
	inet 10.10.10.1 netmask 0xffffffff broadcast 10.10.10.1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto lacp lagghash l2,l3,l4
	laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.50 prefixlen 64 scopeid 0xa
	inet 192.168.50.254 netmask 0xffffff00 broadcast 192.168.50.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 50 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.60 prefixlen 64 scopeid 0xb
	inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 60 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.70 prefixlen 64 scopeid 0xc
	inet 192.168.70.254 netmask 0xffffff00 broadcast 192.168.70.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 70 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.80 prefixlen 64 scopeid 0xd
	inet 192.168.80.254 netmask 0xffffff00 broadcast 192.168.80.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 80 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.90 prefixlen 64 scopeid 0xe
	inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 90 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.100 prefixlen 64 scopeid 0xf
	inet 192.168.100.254 netmask 0xffffff00 broadcast 192.168.100.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 100 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.101 prefixlen 64 scopeid 0x10
	inet 192.168.101.254 netmask 0xffffff00 broadcast 192.168.101.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 101 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.102: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.102 prefixlen 64 scopeid 0x11
	inet 192.168.102.254 netmask 0xffffff00 broadcast 192.168.102.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 102 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.103 prefixlen 64 scopeid 0x12
	inet 192.168.103.254 netmask 0xffffff00 broadcast 192.168.103.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 103 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.104: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.104 prefixlen 64 scopeid 0x13
	inet 192.168.104.254 netmask 0xffffff00 broadcast 192.168.104.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 104 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.105: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.105 prefixlen 64 scopeid 0x14
	inet 192.168.105.254 netmask 0xffffff00 broadcast 192.168.105.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 105 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.106: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.106 prefixlen 64 scopeid 0x15
	inet 192.168.106.254 netmask 0xffffff00 broadcast 192.168.106.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 106 vlanpcp: 0 parent interface: lagg0
	groups: vlan
lagg0.40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:1b:78:59:7c:60
	inet6 fe80::21b:78ff:fe59:7c60%lagg0.40 prefixlen 64 scopeid 0x16
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 40 vlanpcp: 0 parent interface: lagg0
	groups: vlan
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
	inet6 fe80::21b:78ff:fe59:7c60%pppoe0 prefixlen 64 scopeid 0x17
	inet 81.150.180.216 --> 81.139.208.1 netmask 0xffffffff
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::21b:78ff:fe59:7c60%ovpns1 prefixlen 64 scopeid 0x18
	inet 192.168.200.1 --> 192.168.200.2 netmask 0xffffff00
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun openvpn
	Opened by PID 81392

stephenw10

Yeah so no LRO on any em interface or on lagg0 or any VLAN on it.

So what happens when you enable it? Which is pretty much never required normally.

How does it fail?

Steve

VioletDragon

@stephenw10 if I enable it. I get great speeds 110mbps but after so long the Internet drops out then comes back up. WAN isn't disconnecting tho but with it disabled the Internet seems fine. When it is enabled the errors are Pilling up. What do you suggest?

VioletDragon

The Errors have always happened I tried different hardware such as motherboards, cpus, ram etc and network cards but always had issues with error in and out on the Lagg.

stephenw10

Hmm, well I would first try to find why you lose the WAN. What actually happens at that point?

Is it only WAN you lose?

Steve

VioletDragon

No. Everything on the network. Can't connect to webmail on my Mail Server which is on the same network when it happens. WAN is in a Vlan on the Lagg interface.

stephenw10

OK that sounds more like it's exhausting something. mbufs maybe. That would likely be shown on the console.

It certainly should log something.

VioletDragon

I set the mbufs to 10000 as recommended on the Tuning guide. So you think I should enable LRO then look at the logs?

It is just strange when LRO is enabled the errors are just counting up.

stephenw10

mbufs should be set to 1000000 unless you have low available RAM.

https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#mbuf-nmbclusters

Steve

VioletDragon

That is what I've done. Machine has 4gigs of RAM but only 35% is used.

stephenw10

And all NICs are still failing when LRO is enabled?

You should try to find out what's actually failing when that happens.

VioletDragon

I know it's not a hardware issue as I have tried different motherboards, with Cpus and network cards with different memory I have tried moving the Lagg ports on the switch to other ports and different cabling still the same issue.

Cool_Corona

Can you manually change the interface settings to full duplex in the GUI??

I would like to see if that makes a difference.

Run ifconfig -a afterwards

VioletDragon

i only have these options.

stephenw10

You would need to set that on the parent interfaces. And that's tricky because you can't assign them when they're in a lagg. You could only do it using ifconfig directly.

So you don't see any errors if you remove any one of the four links?

If you have 3 links in the lagg and enable LRO does it still stop responding?

Steve

VioletDragon

I get no errors with only 3 Ports assigned to the LAGG. The issue with everything isnt a problem with LRO because it still happens when it is disabled. Something is seriously broken here i did update to the latest version of pfSense which is when this problem started happening with drop outs. When the drop outs happen VLANs are not reachable and the internet is not reachable everything drops out for a min or so then comes back on for a few hours and then drops out again but none of the links or WAN is dropping out.

Derelict

This post is deleted!

VioletDragon

Not sure why you deleted your message. But the network cards i have here are the HP NC360T all Genuine the Quad NICs are the HP 364T and genuine. The onboard Ports are on the Intel Server Board.

Update,

Whether LRO is disabled or not the firewall becomes unresponsive on all its interfaces, LAGG is not accepting any packets. Cannot ping the Firewall or neither the VLANs although VLANs still work providing the Server and Client is on the same VLAN cross VLANs do not work, SplitDNS does not work so cant connect to Web Servers or Mail Servers. Local Domains do not work. This is a firewall problem can access the Servers on the same Subnet via its IP Address. This problem has started happening since updating to 2.4.5 i have also tried a reinstall and still the same issue, different hardware from network cards, boards, memory, Power Supplies etc but still the same problem.

When drop outs occurs it is for a minute then comes back online. Strange thing is that if your on a skype call the call does not drop, if your playing a live tv stream from tvheadend that continues to play. Problem is on WIFI as well so not a client problem

stephenw10

I'm guessing that Derelict misread the output packets as idrops because the columns were all borked. I went back and tidied up the netstat output.

If existing connections stay functioning but you cannot opem new ones that sounds more like exhausting the state table. Does it show anywhere near the limit?
I would expect that too to be logged though. What exactly is shown in the system log when it fails and then comes back?

Steve