LAGG and VPNs

[stephenw10]

pfSense uses slow timeout by default for LACP so i might just be that. Check the Status > Interfaces page or run ifconfig and see what the status of the lagg links are.

For example:

[23.09-DEVELOPMENT][admin@3100.stevew.lan]/root: ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAGG0
	options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 00:08:a2:0c:0b:ba
	hwaddr 00:00:00:00:00:00
	inet 192.168.221.10 netmask 0xffffff00 broadcast 192.168.221.255
	inet6 fe80::208:a2ff:fe0c:bba%lagg0 prefixlen 64 scopeid 0xd
	laggproto lacp lagghash l2,l3,l4
	lagg options:
		flags=14<USE_NUMA,LACP_STRICT>
		flowid_shift: 16
	lagg statistics:
		active ports: 2
		flapping: 0
	lag id: [(8000,00-08-A2-0C-0B-BA,01AB,0000,0000),
		 (8000,00-90-0B-76-8E-55,018B,0000,0000)]
	laggport: mvneta0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-08-A2-0C-0B-BA,01AB,8000,0001),
		 (8000,00-90-0B-76-8E-55,018B,8000,0006)]
	laggport: mvneta2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
		[(8000,00-08-A2-0C-0B-BA,01AB,8000,0008),
		 (8000,00-90-0B-76-8E-55,018B,8000,0005)]
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[stevencavanagh]

Think I changed to slow at one point to try it but will try again.

I will also run the ifconfig and see what I get.

Cheers
Steve

[stevencavanagh]

Forgot to mention that the LAGG link show green on the status page until I plug the patch cables in at which point they remain green for a short whilst then fail but no traffic goes through.

[stephenw10]

The link LEDs on the ports remain lit though?

[stevencavanagh]

The LAGG port link lights are and have always been off

[stevencavanagh]

The switch also indicates the LAG is down

[stephenw10]

Ok, that implies the switch is shutting down the ports which in turn implies it's seeing a flood/loop to trigger that.

[stevencavanagh]

so, i currently have:

[stevencavanagh]

Not sure where the loop could be!

The switch has 4 connections associated with pfsense:-

The WAN link - igb0.101
The 2 LAG links - igb2 & igb3
Additional link for this laptop - igb1

[stevencavanagh]

log for LAGG interface:-

+

[stephenw10]

Ok, so I'd guess there's some misconfiguration on the switch for the VLANs that are separating those 4 links and it's creating a loop.

[stevencavanagh]

Quite possibly but I cannot find it.

It is now back working with the draytek router without issue and all i have done is change the short timeout back to long, switch back to static from LACP on the switch and swap the WAN cable from modem to router! Spoke with draytek this morning and they couldn’t see an issue with the setup either.

Unless the loop is being created via the connections to the pfsense box somehow?

[stephenw10]

Unlikely unless you have and bridged NICs in pfSense.

Does the Draytek router use the same 4 links to the switch?

Does the LAGG come up correctly if only those links are connected between pfSense and the switch?

[stevencavanagh]

The router has 2 links to the switch (LAG) and there are 4 links from this switch to another but that is all working. Yes, the draytek router is using the same links.

Never seen the LAGG come up yet to the pfsense either with just those links or others.

Is it possible to let me know what the pfsense settings should be for a LAGG with VPNs and I can try and make the switch match, although there aren't that many options really on the draytek switch to get wrong to be fair.

[stephenw10]

@stevencavanagh said in LAGG and VPNs:

The switch has 4 connections associated with pfsense:-

The WAN link - igb0.101
The 2 LAG links - igb2 & igb3
Additional link for this laptop - igb1

What are the 4 connections then? That looks like 4 NICs in pfSense that are connected to the switch no?

Any VPNs you might have setup wouldn't have any effect here this is a layer 2 or even 1 issue.

I assume you are able to see a link between pfSense and the switch if there isn't a lagg in play?

[stevencavanagh]

the first connection is from the pfsense to the modem (igb0.101)
the LAGG (2 connections igb2 & igb3) from pfsense to the switch
the last is igb1 which is a connection from the pfsense to the laptop

Yes, if the LAGG is removed and a single cable put direct to the switch from the pfsense (obviously different port from the LAGG) then I get a connection, although at 100MB not 1GB as it should! No idea why though.

For info - the pfsense box is a DELL PC (i5) with 2 twin port NICs, giving a total of 5 ports if you include the motherboard one.

[stephenw10]

Ah, Ok only two links to the switch. Hard to see how that could be a loop then. The fact it only links at 100M is not a great sign! Is it set to fixed speed in the switch maybe?

[stevencavanagh]

Nope, changed the patch cable, changed the port and forced it 1GB where it stopped working completely so put back to auto and it auto negotiates at 100M

[stephenw10]

Hmm, with igb NICs too. About the best supported hardware there is...

If you force them to 100M does it link?

[stevencavanagh]

To clarify, the 2 cables for the LAG show as 1G on the switch, it is the single cable to another port (when the LAG is disconnected) that defaults to 100M but it works.