Weirdness with non-local gateway…

ipodgorny

Long story short…

Remote device (tried another pfSense, cisco router, computer..) is reachable (ping, ssh, https..) until I configure gateway for that device with:

Use non-local gateway through interface specific route.

Than it becomes unreachable… example:

Everything OK until I create a gateway for example on pfSense 1 with IP 172.16.2.2 using "non-local gateway through interface specific route" option.

pfSense box shows the gateway as online and pingable, but Laptop can no longer ping 172.16.2.2, but can ping all the other addresses including the ones behind that gateway...

Firewall rules are "Pass" everything in all directions on all interfaces, no NAT whatsoever is configured.

You may ask what am I trying to accomplish is to set up separate gateway groups from primary to remote sites (there are 5 of them).
If that gateway goes down meaning MPLS is down (main site or the other end) than VPN gateway is used.

Any ideas on another way to accomplish this kind of thing are welcome:)

cmb

You can't do that. "Non-local" means non-local at layer 3, it must be local at layer 2. That's not for the circumstance you show there. You have to route to your next hop router, which has a local IP, use it.

johnpoz

"If that gateway goes down meaning MPLS is down (main site or the other end) than VPN gateway is used."

So your pfsense box has internet access other than the mpls? If so then sure you could use that to get to the other pfsense box. Where is that path in your drawing, it before the cisco router, after?

Take a look here on how you work with multiwan connections, see the failover section.

ipodgorny

@johnpoz:

"If that gateway goes down meaning MPLS is down (main site or the other end) than VPN gateway is used."

So your pfsense box has internet access other than the mpls? If so then sure you could use that to get to the other pfsense box. Where is that path in your drawing, it before the cisco router, after?

Take a look here on how you work with multiwan connections, see the failover section.

Thank you for your answer,

Yes there is another interface with Internet access on pfSense box. I don't really care at all about the Internet access though.
Primary concern is site to site connectivity. Let me paraphrase my question a bit.

Is there a way to track "something" through a gateway whether "something" is UP or DOWN and if it's down use another gateway / route / interface?
There are 5 "somethings" that I need to track through one gateway, and make routing decision for each one depending on its status.
Using different IP in gateway tracking not going to work as it can only do 1. That is why I hoped to set up a separate gateway for each location I need to track.

Somewhat like tracking in Cisco, which bumps route metric, swaps route, or does many different actions when "false" is returned.

ipodgorny

@cmb:

You can't do that. "Non-local" means non-local at layer 3, it must be local at layer 2. That's not for the circumstance you show there. You have to route to your next hop router, which has a local IP, use it.

Thanks, that clears it up. I thought that Non-Local was both layer 2 and 3, now it all makes sense.

cmb

The gateways' monitor IPs are basically the equivalent of IOS's tracking. If your backup path is going to be a VPN, you can use OpenVPN for that, which will give you an additional gateway.

@ipodgorny:

Thanks, that clears it up. I thought that Non-Local was both layer 2 and 3, now it all makes sense.

That's just the nature of IP routing, you can't tell a system to use a gateway that isn't on the same layer 2 network (or otherwise directly-connected, like a VPN).