After i try to verify one by one. Now i saw a problem and solve it . I assign a wrong get way on router. Really happy and Thank you for all your help.

Why did you start another thread on this? If your routing is correct, then yes firewalls could be an issue. Also policy routing could be problem.. If your using say a vpn on pfsense, and sending traffic out some vpn interface before you allow it to go to the mik to get to the 172.16 network.

You would have to put a rule above your policy route to be able to allow 192.168 to ping 172.16

I'll try that, thanks.

Use the HAProxy package in pfsense itself.

Here's some walkthroughs on setup:
https://blog.devita.co/pfsense-to-proxy-traffic-for-websites-using-pfsense/
https://www.thawes.com/2018/01/configuring-pfsense-haproxy-http-https/
http://nathandarnell.com/haproxy-in-pfsense-as-a-reverse-proxy

Here's the documentation:
https://docs.netgate.com/pfsense/en/latest/packages/haproxy-package.html

Here's the cache/proxy forum topic here with lots of posts:
https://forum.netgate.com/category/52/cache-proxy

Jeff

Hi Thanks yeah I had done all of that and it wasn't working.. However in the Open VPN Server advanced configuration I did add a push route for 10.190.36.0 255.255.255.0 and now I am able to communicate with resources on the Worthing LAN.
So I am guessing this was the missing link...

yup there is that option but obviusly i can't try it

@sho1sho1sho1 Please show Diagnostics / Routes

Just remember doing this can be a bit harder on the resources of the box. Especially if you have allot of LAN to LAN traffic..

Add another MX record to point to your isp b WAN address. That way mail should be able to reach your mail server no matter which gateway is up.

It didn't help. Any traffic initiated from external sources via that tunnel ends up being responded via WAN interface. Any ideas?

You can't hairpin NAT like that with FreeBSD/pf. They won't be able to reach each other through the WAN address.

With manual port forward rules, you can enable NAT reflection which adds more rules behind the scenes to cover that scenario. That is not possible with UPnP.

@KOM okay when I get the opportunity I will post that, I think I'm going to leave this pfsense VM a live just because I want to track down the real issue. Yeah I've seen a lot more users having issues with hyperv I agree.

I still don't know why the above firewall rules didn't work, but what did work was:

flipping the rule order so that devices which are only allowed through the Tier 1 WAN gateway is the last rule enable Sys > Adv > Misc "Flush all states when a gateway goes down" was also needed for certain devices (i.e. VoIP phones) allowed through either gateway

@Derelict thanks for your answer. I'm not sure I understood the problem. After reading the article I think the problem is because I created groups for load balacing + failover while load balancing manage the failover itself: Gateways that are load balanced will automatically failover between each other. So I have to get rid of my groups + rules about failover.
And for default gateway IPv4? I put automatic?

Well, i figured it out.

I was doing the logical thing by adding the remote network to each side (Site B) and to the OpenVPN service (hosted on Site A). And that wasn't working.

So I started messing around with the openvpn firewall. Turns out that you need an additional explicit route on the mobile client server config.

Source: openvpn mobile client subnet (192.168.1.0/24 in this example)
Destination: any

Now the traffic routes. I'm sure that is documented somewhere but i couldn't come up with the right search phrase. I only figured it out with lucky guesses.

Now those lucky bastards on OpenVPN Client can see the network resources on Site B. (And much more since this is a mesh setup.)

Based on my understanding (which might be wrong), set your Gateway Group as the default Gateway then you shouldn't need to modify the firewall rules.

In my case, I need to modify the firewall rules to block all but high priority devices from using the backup (Tier 2) WAN. But, if thats not a concern of yours then I think you should be fine.