Dual WAN failover gateway group do not work

[ChrisT]

Hi all, I have deployed a pfsense infra as shown below:

In this deployment, I am using two pfsense appliances in HA and I work with CARP VIP addresses. I have two ISPs with one router each and I am using these routers as gateways to the corresponding WAN interface. I have only one WAN physical interface, in which I created two sub-interfaces, one for VLAN 400 (ISP1) and one for VLAN 401 (ISP2).

I also created one failover Gateway Group (failoverGWgroup) which contains the above mentioned gateways.

In the physical interface cxl1 in each pfsense I have created sub-interfaces for my internal network. I have several VLANs which are using the .254 as their Default Gateway that happens to be the CARP VIP for each VLAN interface.

I want to configure my WAN failover and I think that I have configured my system correctly, though I cannot make it work I think I miss something in the DNS configuration.

Example:
I have set up my TEST_VLAN 10.55.22.0/24. pfsense1 has an IP 10.55.22.253 and pfsense2 has an IP address 10.55.22.252. Both appliances use as CARP the 10.55.22.254

The WAN interface for the first provider in pfsense1 has an IP address 1.1.1.2 and the WAN interface for the first provider in pfsense2 has an IP address 1.1.1.3. Both are using the CARP 1.1.1.1

Then I wanted to configure my TEST_VLAN for WAN failover. So, what I did, was to create a firewall rule in the corresponding interface that allows everything from the TEST_VLAN network to any and in the advanced settings I set as default gateway the above mentioned Gateway Group.

I did the same for another VLAN, the TEST2_VLAN

The strange thing is that VLAN2 has internet connection and I can access internet without issues. I am going out with a public IP of the ISP1. The TEST_VLAN though, right after I set the gateway group as its default gateway, does not going to any website, although I can ping the google DNS.

And besides the above, whenever I try to set the ISP gateway as "down", although I see that the ISP2 router is now the default gateway, no internet access (not even ping the google DNS) exists.

Any ideas? Is there any official tutorial for this?

Thank you all in advance!

[Rico]

Tutorials HA:
https://www.netgate.com/resources/videos/high-availability-on-pfsense-24.html
https://www.netgate.com/resources/videos/pfsense-high-availability.html
https://www.netgate.com/resources/videos/pfsense-high-availability-part-2.html

Multi WAN:
https://www.netgate.com/resources/videos/multi-wan-on-pfsense-23.html

-Rico

[ChrisT]

@rico Thanks Rico. The tutorial for the multi WAN has been watched in detail but didn't help a lot in the problem that I have.

Also, I was reading the pfsense book and although it has a date of February 2019, there are things from previous versions, like the Default Gateway Switching. I believe, Default Gateway Switching has been replaced with the Default Gateway area under System/Routing/Gateways?

[Rico]

pfSense 2.4.4 Release Notes: https://www.netgate.com/blog/pfsense-2-4-4-release-now-available.html
"Default Gateway Group: The default gateway may now be configured using a Gateway Group setup for failover, which replaces Default Gateway Switching."

-Rico

[ChrisT]

Currently all the problems that I have are because of a misconfigurated appliance. Our case is a kind of special, because we need to work side by side with our old firewall and this is causing some troubles. For example, the public IP address that I was trying to use , was still used by the old firewall. This I noticed it when I went to Diagnostics/ARP Table and I found out that the IP address that I wanted to use is still in use.