try this,

after the system comes back up, and pfsense master fails back then goes down, try to reboot your modem.
It's possible it has something to do with the modem and its arp table.

Just reporting back my success.

I successfully brought up a temporary node to the same version as my other node. Moved the slave VM over and tested the fail over. Both way worked.

I later had a look at the event logs and I saw the incompatibility of the integration software on VM on my host.

All these trouble and it was because the VM on the node didn't have the right version of integration software….

I hope this can help others too... If you are running pfSense on a VM, make sure you check the integration software and have the correct version installed. Sometimes when you migrate back and forth, you lose track on the software version and it may not be compatible with the host's version!

Thank you.

Thanks for your tips! Thats almost exactly the way I do it right now because of this strange behavior.

Hindsight is 20/20  :-[

I'm taking a short outage on the weekend to update the primary properly. Lesson learned.

I'm not quite sure I followed but I think we have a similar setup in our data center.  Our WAN IP is in a /29 along with its gateway (a data center router).  A /25 is routed to our WAN IP.  pfSense's LAN IP is in the /25 (x.x.x.1) so is the gateway for the "LAN's" public IP addresses.

If you want a second device in the "outside" /29 you need to set it up in parallel with your pfSense not behind it.  A router won't pass "WAN subnet" traffic back through into the LAN since that's not where it is supposed to go.

Alright, I have a new issue now that I have used the tcp_outgoing_address command to specify my VIP for all outgoing HTTP traffic.

Nothing in my setup has changed except for enabling the clamAV engine in squid.  Since doing so, pages load slowly or not at all.

If I remove the tcp_outgoing_address command from my custom options, the problem goes away.

Files from eicar.com are caught by clamAV and there is no impact to performance.

As soon as I re-enter the tcp_outgoing_address into my squid custom options everything goes in the crapper.

Any ideas anyone?

Limiters and pfsync still won't work properly together.
You may disable state sync.
https://forum.pfsense.org/index.php?topic=108815.0

Good morning and once again thank you for your effort.

I set /24 because I thought that it has to be the same mask as the mask of the WAN IP. I also tried /32 though…

What do you mean with "private subnet"? 10.0.0.0/8 172.16.0.0/12 etc.? That's not the case...

I hope I find the time today logging/monitoring.

Is it possible that other NAT rules somehow interfere?

Hi all,

It's work!
Thank

Its now working!

Forgot to add 51.148.46.xx/29 to the Cisco router and set the interface. (WHAT A NOOB)

On pfSense all what is needed is to add the IP's to "Virtual IP Addresses" and set them up on "Firewall: NAT: 1:1"

I deleted Gateway51 from the gateway list as its not needed

Found the issue, PEBKAC. The LAN interfaces had inconsistent IPv6 settings (one was set to DHCP6 and the other to None). After setting them both to None the CARP failover works as expected.

Hi,

it's work, set rule for firewall, thank.

On your LAN side, if you have, say, a LAN and DMZ, you need rules to pass from LAN to DMZ without a gateway set. Under that, you can have a rule from LAN to any with a gateway set for whatever Multi-WAN scenario you setup (LB, failover, etc).

Hi,

I managed to resolve the issue for our case in the end.

The two servers we're using as our pfSense boxes are Dell PowerEdge R210II servers, each came loaded with 2 on board Gigabit Ethernet ports (one being used as the WAN interface and the other for the LAN interface).

In the first instance I had setup the pfSync to use the LAN interface, which I'm led to believe is a big no no, so I then set up a separate VLAN for the pfSync to use, but as this was still using the physical adaptor shared by the LAN interface, it made no difference.

In the end I bought and fitted an additional PCIe Gigabit Ethernet card in each of the servers, set up a VLAN to use the new physical adaptor (not being used by anything else) and set the pfSync to use the new VLAN and since then I have seen no issues with the sync slowing down or the Backup box becoming unresponsive whilst adding users.

I have now put the new pair into production and we've seen no problems.

Thanks everyone for their help and suggestions.

Hopefully this will help somebody else encountering similar issues.

Cheers,
Jan

Hi,

I managed to resolve the issue for our case.

The two servers we're using as our pfSense boxes are Dell PowerEdge R210II servers, each came loaded with 2 on board Gigabit Ethernet ports (one being used as the WAN interface and the other for the LAN interface).

In the first instance I had setup the pfSync to use the LAN interface, which I'm led to believe is a big no no, so I then set up a separate VLAN for the pfSync to use, but as this was still using the physical adaptor shared by the LAN interface, it made no difference.

In the end I bought and fitted an additional PCIe Gigabit Ethernet card in each of the servers, set up a VLAN to use the new physical adaptor (not being used by anything else) and set the pfSync to use the new VLAN and since then I have seen no issues with the sync slowing down or the Backup box becoming unresponsive.

Hope this helps.

Cheers,
Jan

The firewall sends out traffic from the interface MAC. It can receive traffic using the CARP MAC.

It won't satisfy all of the requirements for this ISP if it requires both.