Connection timeouts when using non-default gateway

[horribol82]

So, first of all, here is my configuration:

ISP1_GW -> Default
ISP2_GW -> WiFi and Failover
Bind DNS Server on pfsense
Squid Proxy on pfsense (not transparent, not in use by wifi)

My problem is when setting the gateway on WiFi to use ANY Gateway, I start seeing lots of timeouts when browsing websites.
I've tried setting both ISP1 and ISP2 as default gateway and they both work fine. But if I set a manual gateway on a rule, then I have lots of timeouts. Even when using the same gateway that is used by default.

My current problematic rule:

Tests:
I've set up a virtual machine with Ubuntu and used that for testing. When using a device on the WPUB network, loading sites can take forever because half of the connections timeout. I can even notice the timeouts by trying to telnet ports 80 and 443 of multiple websites, where it might instantly connect 2 o 3 times in a row before having to wait 15-30 seconds to connect.

• DNS: I've tried using pfsense and external DNS servers. Never seemed to have any problem resolving names.
• Different gateways: As I mentioned before, both connections work fine by themselves when set as default

Does anyone know what might be wrong here? I've tried the same configuration in multiple settings and it works fine, the only major difference is this the only pfsense I have currently using Squid.