2 pfSense VMs - Gateways monitoring ~40% Packet Loss
I've been using pfSense for a while and I've always used only 1 instance for production and tried a few times to use 2 instances as a lab environment.
In the LAB environment one thing always puzzled me: once I get both pfSesne1 and pfSense2 boxes up and HA setup, both boxes show Gateway monitoring with around 40% packet loss.
The Hypervisor is a Linux CentOS box, with Link Aggregation + bridge + vlan. All works fine.
I have 2 "fake" uplinks ( 2 Router appliances connected to the same WAN link, providing 2 VLANs to simulate 2 WAN connections ). I've always setup WAN0 to monitor 184.108.40.206 and WAN1 to monitor 220.127.116.11 on both boxes:
pfSense1: WAN0 = 10.10.10.2 / WAN1 = 10.10.11.2
pfSense2: WAN0 = 10.10.10.3 / WAN1 = 10.10.11.3
WAN0 CARP = 10.10.10.1
WAN1 CARP = 10.10.11.1
redundancy/HA works fine.
I'll call this "BAD CONFIG" scenario.
After a bit of tricking with some configs, I decided to leave box2 to monitor the router appliances IPs: 10.10.10.254 and 10.10.11.254.
All of a sudden everything worked fine! Gateways packet loss is at 0%, GREAT!
I'll call this "GOOD CONFIG" scenario.
So, i tried setting pfSense1 WAN1 monitor ip to 18.104.22.168 ( the same used for WAN0 monitoring ), and pfSense error poped up:
The following input errors were detected: The monitor IP address "22.214.171.124" is already in use. A different monitor IP must be chosen.
It's also worth mentioning that under "GOOD CONFIG", pinging 126.96.36.199 and 188.8.131.52 works fine. When using the "BAD CONFIG", I get the ~40% packet loss results.
So... I'd like to gently ask for directions here:
What am I supposed to read to understand this behavior ?
In this scenario of 2 boxes with 2 WAN interfaces each, should I use 4 different external IPs for monitoring ?