Outbound connectivity working but not inbound - multiple subnets and routers

[grahamrb]

I'm trying to set up a network as pictured below. Specifically I have an existing 10.0.0.0/24 network (which is attached to a non-pfsense router) and I would like to add a separate network (192.168.10.0/24) behind a pfsense router within my VMWare ESXi Server.

I've installed pfsense on a VM and set up two vSwitches, one which connects to the rest of the network and the other which connects to the 192.168.10.0 subnet.

• I've attached the WAN interface to the main vSwitch and the LAN interface to the 192.168.10.0
• I've also set a static IP on the WAN interface and set up DHCP on the LAN interface
• I've configured a gateway on the WAN interface to the gateway on the 10.0.0.0 router
• I've added a firewall rule so I can remotely manage the pfsense from the WAN interface
• On the router attached to the 10.0.0.0 network I've created a static route to the static IP address attached to the pfsense WAN interface
• I've configured a zone on the 10.0.0.0 router for blah.com to resolve through the pfsense (not sure if I've done this correctly to be honest)

Now, what's working:

• from within the 192 network I can access the the internet
• from within the 192 network I can access computers on the 10 network
• from within the 192 network I can resolve external DNS queries
• from the 10 network I can remotely administer the pfsense

What's not working:

• I cannot access any machines within the 192 network
• I cannot resolve DNS entries which are configured on the pfsense from a machine on the 10 network

Any suggestions would be appreciated. I'm really not sure if I'm missing some firewall setting, have incorrectly configured something with NAS (which from what I understand shouldn't be required), missed something else or completely configured this incorrectly.

PS. For anyone wondering why this would every be wanted - I want to be able to pick up the pfsense and VMs in the 192 network and drop it into a different ESXi server and have it work (ie, create a self-contained network).

[johnpoz]

what picture?  I stopped reading after you said there was a picture and then no attachment or link.

So you created a route on this 10 router… So your not natting on pfsense?
"On the router attached to the 10.0.0.0 network I've created a static route to the static IP address attached to the pfsense WAN interface"

If your not natting you have to create rules on pfsense wan to allow the traffic you want to allow through.  If your natting then you would have to create port forwards on what you want to allow through.  If your natting there would be no reason for a route on the 10 router.  since its only ever going to send traffic to the pfsense wan IP in the 10 network if you want to get behind pfsense.

Either way natting or routing your clients behind pfsense also have to allow the traffic on any firewalls they would be running.

[grahamrb]

weird, it's showing up for me, I'll attach it to this post

(EDIT: Also attached to the original post)

[johnpoz]

nice drawing btw!  Makes it very easy and clear to understand what your doing.  Except you have not stated if you turned off natting in pfsense or not.  and what are you rules to either port forward the traffic into machine behind pfsense.  Or allow it if not natting.

See my edit of my post.

[grahamrb]

LAN and WAN rules attached, I've tried with both NAT enabled and disabled, NAT is currently disabled.
With NAT enabled I can get internet access from within the 192 network, with NAT disabled I cannot.

Also note, I've said 10.0.0.0/24 in the original post/image (for simplicity), the network is actually 10.0.14.0/24 (which is reflected in the attached screenshots).


[johnpoz]

well without nat… Your 10 router would have to nat that 192 network to the public IP space if you want to allow internet access.

if your not natting that any any rule you have there should allow all access.. BUT!!!!  what I can see as a problem is you have asynchronous issue without nat.

Unless your doing host routing on the clients in the 10 network.  They talk to 10 router for access and then go to pfsense.  but then when traffic coming back pfsense says oh your going to 10.x network that is attached..

Any time your going to have a downstream router you really need to use a transit network so you don't have asynchronous routing.  If your going to have devices sit in that transit network they have to use host routing.

See attached edit of your drawing...  BTW - you kind of show 2 interfaces in the 10 network to your 10 router.  Are those the same network and only 1 interface or do you have multiple interfaces in the same network on that router?

edit: added example drawing with a transit network.

[grahamrb]

Thanks for the additional info and for taking the time to put together the drawing. I'm not sure I understand it entirely. Where do I configure the 172 network? Which device performs the gateway function for this network?

@johnpoz:

BTW - you kind of show 2 interfaces in the 10 network to your 10 router.  Are those the same network and only 1 interface or do you have multiple interfaces in the same network on that router?

Sorry about this, my diagram isn't correct. There is only one internal interface on the 10 router (other than the WAN interface on that router of course). The computers in the 10 subnet and the pfsense are effectively all connected to the same switch.

[johnpoz]

you would have to create this transit network either on another interface connected to your esxi host where you have your pfsense wan from your 10 router or with the use of a vlan.

" The computers in the 10 subnet and the pfsense are effectively all connected to the same switch."

You can not do it that way without issues with asynchronous routing..  Which is not good for anything ;)

You have 2 solutions here, either NAT at pfsense so the network behind pfsense is hidden and looks like everything is on the 10 when talking to 10 devices and when going out the internet through the 10 router.. Ie everything behind pfsense looks like it is pfsense wan IP in the 10 network.

Or if your not going to nat and have a downstream router, then you need to use a transit network or you would have to create host routing on all devices wanting to talk to stuff behind pfsense..  Ie a workstation on the 10 that wanted to talk to a 192 device behind pfsense would have to have a route to pfsense wan IP for taking to the 192 network.

Also when your not natting on pfsense your going to have to let your 10 router to also nat this 192 network to its public IP for internet.

The simple solution for you would be to just let pfsense nat.  For you to get to stuff behind pfsense then you would need to create port forwards on pfsense and if wanting to get to them from internet, you would have to create forwards on your 10 router to forward to pfsense wan IP.

The more robust solution would be not to nat and just use a transit network, and setup outbound natting on your 10 router for any networks you place behind your now downstream router pfsense.

If your goal is to be able to just plop pfsense on any esxi host that might be in different networks then nat is the way to go.