2 pfSense VMs - Gateways monitoring ~40% Packet Loss

msilveira

Hi.
I've been using pfSense for a while and I've always used only 1 instance for production and tried a few times to use 2 instances as a lab environment.
In the LAB environment one thing always puzzled me: once I get both pfSesne1 and pfSense2 boxes up and HA setup, both boxes show Gateway monitoring with around 40% packet loss.

The Hypervisor is a Linux CentOS box, with Link Aggregation + bridge + vlan. All works fine.
I have 2 "fake" uplinks ( 2 Router appliances connected to the same WAN link, providing 2 VLANs to simulate 2 WAN connections ). I've always setup WAN0 to monitor 8.8.8.8 and WAN1 to monitor 8.8.4.4 on both boxes:
pfSense1: WAN0 = 10.10.10.2 / WAN1 = 10.10.11.2
pfSense2: WAN0 = 10.10.10.3 / WAN1 = 10.10.11.3
WAN0 CARP = 10.10.10.1
WAN1 CARP = 10.10.11.1

redundancy/HA works fine.

I'll call this "BAD CONFIG" scenario.

After a bit of tricking with some configs, I decided to leave box2 to monitor the router appliances IPs: 10.10.10.254 and 10.10.11.254.

All of a sudden everything worked fine! Gateways packet loss is at 0%, GREAT!
I'll call this "GOOD CONFIG" scenario.

So, i tried setting pfSense1 WAN1 monitor ip to 8.8.8.8 ( the same used for WAN0 monitoring ), and pfSense error poped up:

The following input errors were detected:

    The monitor IP address "8.8.8.8" is already in use. A different monitor IP must be chosen.

It's also worth mentioning that under "GOOD CONFIG", pinging 8.8.8.8 and 8.8.4.4 works fine. When using the "BAD CONFIG", I get the ~40% packet loss results.

So... I'd like to gently ask for directions here:
What happens?
What am I supposed to read to understand this behavior ?

In this scenario of 2 boxes with 2 WAN interfaces each, should I use 4 different external IPs for monitoring ?

Thanks!