Routing between different subnets.

Jikolo

Hello!
I have a little stupid question, but i'm really stucked.
We have two pfsense in different location.
In first location we have LAN net 10.200.1.0/24 with gateway 10.200.1.1.
In second location we have LAN net 10.200.40.0/24 with gateway 10.200.40.3.

All time we used OpenVPN static connect and all worked fine, but recently we got dedicated canal between two location.
I'm try to setting routing, like this:
Static route to 10.200.40.0/24 through GW 10.200.1.1.
Static route to 10.200.1.0/24 through GW 10.200.40.3.

Okay, it work, but... not good.
We have a traceroute like this:
From 10.200.1.13 => 10.200.1.1 => 10.200.40.51- looks good, but the network was unstable - especially with TCP-connections.
And iperf(for speed test) show like this:

[ ID] Interval > Transfer     Bandwidth 
[  4]   0.00-1.00   sec   256 KBytes  2.09 > Mbits/sec 
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec 
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec 
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
etc.

After i add on local computer 10.200.1.30 route like:

route add 10.200.40.0 255.255.255.0 10.200.40.3

Everything began to work perfectly, iperf show this(with the same settings):

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  66.5 MBytes   558 Mbits/sec
[  4]   1.00-2.00   sec  68.5 MBytes   574 Mbits/sec
[  4]   2.00-3.00   sec  68.0 MBytes   571 Mbits/sec
[  4]   3.00-4.00   sec  69.6 MBytes   584 Mbits/sec
[  4]   4.00-5.00   sec  46.0 MBytes   386 Mbits/sec
[  4]   5.00-5.24   sec  16.2 MBytes   562 Mbits/sec

And now i'm stucked - how i can push all traffic from 1.0 for 40.0 subnet through remote GW, not for each machine? And vice versa - from 40.0 to 1.0:)
I can't add GW to interface LAN - i get "The gateway address 10.200.40.3 does not lie within one of the chosen interface's subnets.".

Please, help!
Thank you in advance!

jimp

How are the two networks connected now? You can't send traffic through a gateway in another subnet like that. You need some kind of transit network. For example, if it's a dedicated circuit, you'd have that plugged into an additional NIC (or VLAN) on both pfSense firewalls, and then you'd have some other unrelated subnet to talk between them there. Then you use the address in that subnet as a gateway to reach the other.

If you have your LANs plugged together so they're all in the same Layer 2/flat network that is going to be a huge mess.