@toehl001

https://forum.netgate.com/topic/60600/gratuitous-arp-from-virtual-ips/17

there are more info here
https://forum.netgate.com/topic/84269/multi-wan-gateway-failover-not-switching-back-to-tier-1-gw-after-back-online/86

thanks @viragomann works here!

If your not worried about them talking to each other, since you have any any rules - what is the point of multiple segments in the first place? ;)

Fixed it! Yes, I did assign a gateway, however I was confused when I assigned it, because I thought it was the gateway IP of my interface, not the gateway of the internet service IP. Such a silly mistake! Spent all night on it trying to figure it out.

That's the joke

Pretty much comes down to usable subnet addresses -3 so a /29 on the interface has 3 addresses usable for HA services, a /28 has 11, /27 27, etc.

I was able to find a solution to this problem with the help of Netgate admin expert (@stephenw10) advice, the trick was to force Unbound to use "localhost" for outgoing queries. Unbound, when it is configured to operate via localhost outbound gateway would use only the gateway currently chosen by the system (i.e. dictated by the failover rules). This allowed me to completely eliminate unwanted unbound DNS/DNSSEC traffic via an expensive SIM-based failover link.

See more details in the following thread:

https://forum.netgate.com/topic/150176/php-shell-has-gatewaystatus-but-why-does-it-report-all-gateways-as-status-none

that is still not a route.. But sure if device answers (your modem) for an IP on your network then it would show in the mac address table.

Here I tried pinging a bunch of different addresses in my /23 and you can see them now in my arp table, with the mac address of my modem.

arptablemodem.jpg

If you had done some sort of scan of /22 then yeah you would of see mac address of your cable modem for all of the IPs.

It's similar, but different, since I am talking about the phase 2 selectors, not phase 1 counterparts.
The point with phase 2 selectors on VTI is, that they should be ignored for routing. pfSense seems not to support defining routes just via a particular interface, but relies on the remote gateway IP that is derived from the phase 2 network. Consequently, the adjacent 0.0.0.0/0 "network" is parsed as the default route in my case. At least that's my theory.

@johnpoz Sorry to bump this thread but I have related question. Does this downstream router need to be on its own network, or can it just stay on a VLAN different from the clients I need to route?

Let's say I already have a SERVICES VLAN, none of the hosts on this VLAN will be routed via this Wireguard gateway. Would placing this downstream router on this VLAN solve the asymmetric routing issue you explained?

You could send some of that /27 across OpenVPN to the other site if the /27 is routed to you.

If the interface is a /27 that's going to be much more difficult.

I just figured it out. I wasn't setting the PVID of the switch ports correctly. Once I set the PVID of the untagged ports to the same VLAN ID as what I wanted the packets entering those ports tagged as, as I was able to connect to the cameras.

Yet again the need to be explicit in your instructions proves itself.