HA interface assignment best practices
After a recent bad experience with PA products I was asked to setup a HA pfSense cluster for evaluation purposes. I picked up two identical 1U servers with E3 Xeon's, 2 onboard NICs and 4 port add-in NIC for this purpose. I have been reading through the documentation and forums without finding any clear guideline or best practices guide for optimal interface assignment for this type of setup.
My current plan is to use a pair of stacked switches and connect each server to both of them using one interface from onboard and add-in NIC forming 4 interface strong lagg between each server and the switch stack where I would separate lan/wan traffic with VLANs for the sake of simplicity for the duration of this evaluation.
This would leave two interfaces free on the add-in NIC on each server. I was planing on using these for the XMLRPC and pfsync traffic between the two servers but I am a bit unsure of what would be the best way to go about it. Turn them into a lagg and use shared? or separated? VLAN for the synchronization traffic.
I guess using only one interface for the synchronization traffic and finding some kind of solution to connecting wan to the other remaining interface would be an option as well if you guys think that separating wan from lan traffic with only VLAN's is a terrible idea even in evaluation environment.
As far as infosec policy's, expected load and other use case related questions go I am not too worried about them at this stage. I just want to create something functional that doesn't cause anyone to faint when they see it and could be easily transferred to newer hardware if we ever decide to move forward with this project.
You can certainly use LACP and VLANs to do LAN and WAN in the lab.
Many people (me included) do not like mixing inside and outside traffic on one switch/stack. Many people (me included) do it anyway. I have not seen a recent, credible case of VLAN hopping with the exception maybe of TP-Link's VLAN1 nonsense. Even less of a reason to be concerned in the lab.
But in your case, I would probably do a lag for the outside and a lag for the inside, with two interfaces each even if they are to the same stack, and one of the add-on ports for SYNC.