Another Multi-Wan Multi-Lan help needed

[routergman]

I have way over-complicated things on my network.  But trying to solve a very basic problem and after hours of googling and tinkering I feel like I am no closer to a solution.

Everything is in a ESXi 5.5 box with vswitches.  My esxi box has 4 network cards which are all in use

I have 2 pfsense VMs with VMs attached to them and one separate physical fileserver attached.  I can't get the VMs from one router to see the fileserver on the other router.

Simple diagram

pfsense VM1 (11.0.0.1) –---> WAN1
  |--------------------> Fileserver (Hardware) (11.0.0.201)

pfsense VM2 (13.0.0.1) -----> WAN2
  |--------------------> Windows VM (13.0.0.101)

What is the easiest way to get 13.0.0.101 to connect to 11.0.0.201?

I don't want to combine the networks; VM1 and VM2 are currently working perfectly on their own so don't want to merge them into a single pfsense (which would be least complicated)

Thanks!

[tim.mcmanus]

There probably isn't a route between both networks, and they are both probably NATed (the routers).

You need to either turn off NAT on the routers or create a route between them via an SSL tunnel or another VPN tunnel.

[routergman]

As I drew the pic there is no route.  But I can create a route.  This is what I had been trying to do, but couldn't get it to work.  Any tips or tutorials?

I definitely don't wan the packets to go out through the WANs.  That would be very very slow.

Yes both routers are NATed

[routergman]

I tried creating a new 3rd vswitch.  Adding an interface to VM1 and VM2 both attached to the new switch.  Gave each interface a static IP (10.0.1.1 and 10.0.1.2).  On each pfsense router I then created a gateway.  Then I added a static route to each pointing to 13.0.0.0/24 10.0.1.1 and 11.0.0.0/24 10.0.1.2

This did not work.  tracert shows packets going out to the WAN not the other router.

I clearly don't get how static routes work.  I really expected the above to work.

I also tried adding fierwall rules instead of static routes, with the other network as the destination and the gateway as the new gateways defined above.  Also didn't work.  Still tried to go out to the WAN via default gateway.

Help!  driving me crazy.

[tim.mcmanus]

Create a VPN from one to the other.  That's a very easy way to do it, and the traffic will be encrypted.

[routergman]

@tim.mcmanus:

Create a VPN from one to the other.  That's a very easy way to do it, and the traffic will be encrypted.

That would go over the WAN right?  One of the WANs is very very slow so that wouldn't work.

Plus setting up a VPN shouldn't be easier than adding a route should it?

[tim.mcmanus]

I know you don't want to merge pfSense instances into one, but that's honestly the easiest way to do it.  Then you can have WAN1, WAN2, LAN1, LAN2, and do the routing that way.

Im a big fan of making things as simple as possible because when you need to make changes, it doesn't become rocket science.

You can add a route inbetween the boxes, but that would require connecting them together.  I suggested a VPN, because it actually is easier.  Otherwise, you need to create another virtual switch, add that switch as an interface to each pfSense VM, create routes and policies between those two boxes, and maintain it.  If something goes wonky, there's a lot in there that you'll need to troubleshoot.

I've attached an example of what I have on my firewall.  You'll notice that I have LAN going to LAN2, and that's because I have two LANs on my pfSense box and route all traffic between them with that rule.  However, I have one server on my LAN that I want to route out a specific internet connection (WAN2), so I specified the gateway that machine should use.  That's how I tell it to route traffic to those destinations.


[routergman]

@tim.mcmanus:

I know you don't want to merge pfSense instances into one, but that's honestly the easiest way to do it.  Then you can have WAN1, WAN2, LAN1, LAN2, and do the routing that way.

Thanks for the screenshot.

I tried again adding a vSwitch and interfaces to both boxes and still couldn't get the rules to pass packets between them.  It should have worked but it didn't.  I looked at the routing table and it looked like the rule was too far down in the table and never got triggered, which would explain why the packets went out to the internet.

In the end I mostly did as you suggested.    I added the VMs that were attached to the second rounter into the first router so they can connect.  Then I added an interface to the second router as wan2 on the first router.  Then I added a rule to the first router to push traffic to the second router and out through WAN2.  A little kludgy but it works.  The DHCP is the only issue with the setup now, but I just used static IPs and it works ok.  Not as isolated as previously but its working now.

Thx again for the help.