up! every help is appreciated

Perhaps to clarify the situation just a little bit: There is a destination subnet that needs to be routed trough a specific gateway, this cannot change until certain legacy applications are completely phased out. The newer applications need to connect to hosts in the same destination subnet but use a new gateway. The only way we can distinguish the old and new application traffic is by their source addresses. This is why we looked at a firewall rule to redirect packets with a specific source and destination to a gateway that is not the normal gateway for that destination subnet. This works fine when the first packet comes from the side where the source based routing firewall rule is in effect but is entirely broken when the other side initiates the connection. Then the first return packet is sent over the default gateway instead. (As per the routing config rather than the firewall rule.) I'm also entirely open to alternative solutions to the problem, but it feels intuitively feels like it should work both ways. But as explained earlier in this thread, apparently states throw a spanner in the works.

Just put peer-to-peer from Office 2 and 3 to Office 1. If there is not much traffic between office 2 and 3, then let it just route via Office 1, otherwise make a 3rd peer-to-per. If the other locations really are 1-user things, then you could have a "road warrior" style server in Office1 for them to connect to from OpenVPN client on their PC. But those other locations will need some internet connecting device, so sometimes it is just as easy to put a basic pfSense there and let it be the internet gateway and have a peer-to-peer link back to Office1. Post questions when you get stuck - plenty of people here that are happy to help.

From what I've read it's preferable to manually configure the DNS servers and specify the gateway, under System -> General Setup, and unchecking "Allow DNS servers…" This is how I've configured it in my multi-wan scenario, two vdsl connections with different ISPs. This ensures that queries are routed out the correct gateway mitigating issues you describe. Also, it looks like your client device is appending the domain suffix to your query so you should add a full stop after it, e.g. host google.com.

Yep, it's been covered and documented: https://doc.pfsense.org/index.php/Multi-WAN https://doc.pfsense.org/index.php/What_is_policy_routing And some more here: https://doc.pfsense.org/index.php/Category:Multi-WAN

Still on bge NICs? Just ran through a couple tests there, one on igb and one on vmx in ESX and both update the default route's MTU as well. default            172.27.44.1        UGS        121  9000      vmx0

I got it working. Thanks anyway.

It works when I add a quick floating rule. Maybe I should read more about this, just seems kind of strange?

We experienced a dramatic slow down on network throughput after migrating from esxi 4.1 to 5.1.  This wasn't just internet traffic… it was all vms. I advise some throughput tests from vm to vm, physical client to vm and then physical client to pfsense vm, vm to pfsense vm. We've migrated our pfsense vma to a physical one and haven't witnessed any throughput issues.

Thanks Phil.

With a single-threaded download it can only go over 1 link. If using a download manager that starts multiple downloads on parts of the file, then you benefit from a gateway group with multiple gateways at Tier1. The total download speed can be the total of all links. But if the downlaod swamps a link so much that the link ping times get really high, then the gateway monitoring might think the gateway is down - which is not really true. As you have found, the easy way to fix that is to increase the latency threshold. Another way is to do traffic shaping on the interface/s and give ICMP/ping priority. Or a bit of both.

I agree it is an "odd way" of doing it. I guess that's why I'm having such a hard time figuring it out. For very stupid reasons I have to do it this roundabout way… I have tried explaining how much easyer it would have been having pfsense do it all  but I failed to convince.  :-\  can't really say much more then that.  :-X Both of the routers have AP connected to them (sense you where wondering) I'm going to try using NAT on pfsense, I feel doing anything advanced router 1 (also know as boring verizon router) will not be supported. In fact the closest thing you can do is port forwarding, an advance feature for home routers when it came out.  :-\ Thanks, I'll let you both know how it ends up

Could be the OpenVPN protocol or device mode of any relevance?

Can anyone please help?

Up please, the failover is stil not working.  :(