@Orbettino said in Cannot get failover WAN to work:

@dwr953topfsense Hello, have you made any progress? I've the exact same problem here on my setup. Failover works but fallback isn't.

Yes, I did. All works OK now.

Currently we have added static route to each location to access other site through single vendor if that vendor connectivity breaks we have manually shift to other vendor
Both vendor provided MPLS through cloud if site1 location mpls fails for vendor1 but it working on site2 due to this I have to manually shift to other vendor2 even if that site MPLS link is working
I tried OSPF free but it is not showing any neighbors router

I realized that I do not need to add 192.168.0.x since my WAN interface is 192.168.0.1 and /32 was incorrect too. I have removed that. I can see the route in the table but still the ping to google.com or 8.8.8.8 or 192.168.0.1 from a VM(192.168.1.100) connected to pfsense is very random. how can I troubleshoot that?

edit: do I have to reboot each time I save anything? that seems to do the trick

So it was actually an issue in the router behind the RG replacement. I have dual WAN connections set up in it and had failover rules in place for outbound traffic. I decided to create a rule for traffic to the WAN NET to use the default route instead of the failover policy route. Immediately after applying the changes connections are being maintained and there are no more entries appearing in the RG replacement pfSense logs.

Adding these notes in case of the 1 in a million chance someone else encounters the same issue.

You something like this

https://forum.netgate.com/topic/124545/monitoring-pfsense-using-nagios-and-ssh

Or a bash script like this .....

if curl -s --head --request GET https://example.com | grep "200 OK" > /dev/null; then echo "mysite.com is UP" else echo "mysite.com is DOWN" fi

Working solution: I was finally able to get what I wanted, by manipulating the strongswan config manually, and restart IPSec.

PFSense already has a bypass LAN setting, that can be checked and unchecked in it's IPSec-config, so my solution is just to edit the list of networks that have status as "Shunted".

Can be done via the GUI: Diagnostics -> Command Prompt -> Paste Command -> Execute

(Using sed to replace the relevant lines in the strongswan-config-file and restart ipsec)

sed -i '' -e 's#192.168.250.0/23,fdd0:192:168:250::/64#192.168.250.0/23,[DMZ-IPv4-NET]/29,fdd0:192:168:250::/64,[DMZ-IPv6-NET]/64#g' /var/etc/ipsec/ipsec.conf ipsec restart

If one runs ipsec statusall, then all necessary networks (both LAN-to-LAN, and DMZ-to-LAN) will be listed under "Shunted Connections".

OK, I have figured this out, so I am sharing here in the hopes it helps others. I was doing a few significant things wrong.

A much-simplified Example:

New Site Network:

Netgate/pfSense firewall/router 192.168.1.0/24 LAN Actually VLAN'd, half dozen subnets, etc.

Old Site Network

Cisco Firepower firewall/router 172.31.0.0/16 No VLANs, subnets, etc.

I needed to make the old network play nice with the new one until resources could be fully migrated over time.

I tried defining an interface on the new network that used an IP address from the old network, setting up routing and rules between the two, etc. That ended badly.

Maybe I had it backwards: I tried defining an interface on the old network that used an IP address from the new network, setting up routing and rules between the two, etc. That ended badly.

I tried setting up a new simple, dedicated subnet solely for the purpose of interconnecting the two routers to manage transferring data between the two networks, static routes, etc. That did work.

I call it a transport network, but I bet you networking guys who know what you are doing actually already have a name for it (I'd be curious what that is).

Where making the new network at one geographic site talk to the old network at a different site is concerned, I discovered that adding P2s on the IPSEC S2S tunnels was the trick (and not setting static routes, which I had tried).

Problem solved. Thanks.

Forgot to mention, I also then edited the LAN rule to allow all on lan to all using Wan_Group gateway.

Your not going to be able to subnet it out if its directly connected and your bridging it.

Why is /24 too many for a 1:1? Not like you have to setup each on on its own, you just do a 1:1 for the whole /24

Your x.x.163.0/24 would just map to say 192.168.163/24 where .1 is .1 and .2 is .2 and so on..

The correct solution for using a /24 would be for the /24 to be routed to you..

Duh...Ah geez I wasn't thinking. Thanks again.

@CvH said in Force client to use 2nd gateway:

I do it as soon as possible, tx !

as soon as possible was today 🙈 and it worked
tx a lot !

172.1.1/24 - dude come on!!

NetRange: 172.0.0.0 - 172.15.255.255
Organization: AT&T Corp. (AC-3280)

Don't use address space that is not yours.. Use valid rfc space there 172.16.1/24 would be fine..

So did you turn off NAT, if not to get to stuff behind pfsense you would have to port forward.. If you want to route and firewall only, then make sure you turn off nat.

And yes devices sitting on your 192.168 wan network would need a host route to tell them how to get to the 172 and 10 networks.. If your clients on your wan are talking to some other router as their default, so yeah they would need route to get to behind pfsense.. If you try and route them off your default router your going to run into asymmetrical problem.

The correct solution here is to have your downstream router (pfsense) connected to your upstream via a transit network.