Subnets can't communicate

[marvosa]

Personally, I'd add a 2nd NIC to each box, then create a transit network between the two LAN interfaces and adjust your static routes accordingly.

[johnpoz]

^ exactly... if your going to do downstream router there needs to be a transit network.. NO HOSTS.. If your going to leverage a network with hosts on it, then you would need to be natting at the downstream router. Or you would need to put routes on every host in this network your using as transit. Or you will have asymmetrical routing issues.

[WisceBIat]

I've made some changes like adding extra interfaces. How would I go about setting static routes now?

[viragomann]

System > Routing

First go to the gateway tab and add a new gateway, select the proper interface, enter a name and the WAN IP of pfSense 2.
Then go to the Static Routes tab and add a route. Type in 192.168.2.0/24 into the Destination network box and select the gateway you've added before.

If the OPT1 IP of pfSense 1 is the default gateway on pfSense 2 there will be nothing to do here. However, in event ExpressVPN is the default GW you have also to add a static route to pfSense 2 in the same way with the respective other network and GW IP.

[WisceBIat]

@viragomann What would be the proper interface for the default gateway ?

[WisceBIat]

This still isn't working.

on pfSense 1 I created another default gateway, using OPT1 interface and giving it the IP of WAN attached to pfSense 2. Then a static route pointing to 192.168.2.0/24 and using the newly created default gateway.

Next I went onto pfSense 2 and created another default gateway on WAN, and gave it the IP of OPT1 attached to pfSense 1. Then a static route pointing to 192.168.1.0 using the newly created gateway.

[johnpoz]

Are you natting on pfsense2, to use pfsense as downstream router you have little use of natting. Especially if your connecting them via a transit network like you have shown.

BTW - why are you wanting to do this? As a learning experience? Why can you not just connect this 192.168.2 directly to pfsense1?

Also this is all running on virtual - your naming interfaces vmbrX

Is this the same VM host for both pfsense1 and 2? Oh just notice your bigger proxmox box.. so you have this all setup in virtual...

[WisceBIat]

pfSense 2 is doing NAT because I followed a guide to make all traffic go through ExpressVPN.
All this is done on a proxmox machine and I followed a guide to setup pfsense and expressvpn from their website

[johnpoz]

@WisceBIat said in Subnets can't communicate:

I followed a guide to setup pfsense and expressvpn from their website

Well that is problem 1 ;)

If you want expressvpn to be used by devices on your network, then just set it up on your edge pfsense.. And policy route whatever traffic or vlans you want to use that.. Is that the only reason your trying to setup a downstream pfsense?

[WisceBIat]

@johnpoz said in Subnets can't communicate:

@WisceBIat said in Subnets can't communicate:

I followed a guide to setup pfsense and expressvpn from their website

Well that is problem 1 ;)

If you want expressvpn to be used by devices on your network, then just set it up on your edge pfsense.. And policy route whatever traffic or vlans you want to use that.. Is that the only reason your trying to setup a downstream pfsense?

I was actually trying to nest, or chain pfSense VPNs together. The expressvpn part works perfectly. I just thought it would be easy to get the hosts on both lan's communicating, but so far it's been a 2 month ordeal that I can't figure out.

[johnpoz]

Well if your forcing traffic out some vpn on pfsense 2, how do you think it would ever talk to anything connected off pfsense1?

And if your natting traffic on pfsense 2, you would have to port forward on pfsense 2 if you want anything on pfsense to start a conversation with with something behind pfsense 2.

What I would suggest is turn off all the vpn stuff, and just setup your 2 pfsense to talk to each other via your transit network.. This should take all of about 2 minutes.

Once you have that working, then you can play with whatever you want to play with vpn access. But there is zero reason to be natting on that 2nd pfsense.. Since any traffic from it, be it vpn traffic or normal traffic would be natted at your edge pfsense.

Pfsense will auto create outbound nats for you for downstream networks once you create a route to them.. Unless you have turned off auto outbound nat for some reason - like I don't know following some shit vpn service guide ;) Sorry but have yet to see one written by anyone with clue one.. But yeah trust all your traffic with a service that doesn't know the basics of networking.. /rant on vpn service guides ;)

[WisceBIat]

@johnpoz said in Subnets can't communicate:

Well if your forcing traffic out some vpn on pfsense 2, how do you think it would ever talk to anything connected off pfsense1?

And if your natting traffic on pfsense 2, you would have to port forward on pfsense 2 if you want anything on pfsense to start a conversation with with something behind pfsense 2.

What I would suggest is turn off all the vpn stuff, and just setup your 2 pfsense to talk to each other via your transit network.. This should take all of about 2 minutes.

Once you have that working, then you can play with whatever you want to play with vpn access. But there is zero reason to be natting on that 2nd pfsense.. Since any traffic from it, be it vpn traffic or normal traffic would be natted at your edge pfsense.

Pfsense will auto create outbound nats for you for downstream networks once you create a route to them.. Unless you have turned off auto outbound nat for some reason - like I don't know following some shit vpn service guide ;) Sorry but have yet to see one written by anyone with clue one.. But yeah trust all your traffic with a service that doesn't know the basics of networking.. /rant on vpn service guides ;)

oddly enough, even with NAT disabled and expressvpn disconnected, I still can't get both subnets to communicate

[johnpoz]

Well your going to have to spell out what you have done if you want help what your missing or what you did wrong.

You have a transit network setup from your drawing..

So without any routing or gateways created can client on pfsense lan ping the IP of the pfsense 2 transit interface? 192.168.3.X from your drawing... vmbr4? .1 or .2? Use of a /30 where would make it clearer or that is a transit..

so

pfsense1 - 192.168.3.1 -- 192.168.3/30 --- 192.168.3.2 -- pfsense 2

Now client on 192.168.1.x should be able to ping 3.2, as long as you have allowed that on pfsense2 transit interface. Assuming pfs1 lan is default any any rules, and your not forcing traffic out some vpn on pfsense1

Once you can ping 3.2, then allow on your pfsense2 transit interface traffic that you want to get to 192.168.2/24

Setup a gateway on pfs1 that points to 3.2
Setup a route on pfs1 that points 192.168.2/24 to 3.2

If this transit network on pfs2 is wan, then its default gateway should point to 3.1 and that is all you would have to do. lan on both pfsense would be able to talk to each other.. And pfs2 lan should be able to get to the internet. Even with nat turned off on pfsense2

You will have to allow traffic on pfs1 transit interface for the downstream networks and the transit network if you want pfs2 to be able to get to the internet or pfs1 lan.

Again the auto outbound nat would create a outbound nat for your downstream network when you create a route to it.

[WisceBIat]

small changes made. Went from /24 to /30 in the transit network. Also added some IP information.

Right now PC 1 can ping 192.168.2.1
PC 2 can ping 192.168.1.2

PC 1 can't ping PC 2 and vice versa

[WisceBIat]

Anyone know what could be preventing the PCs from pinging each other? They can ping the interfaces of both pfSense boxes, so maybe it's a firewall issue or something on windows?

[akuma1x]

To pass ping traffic, you need specific rules - not TCP, not UDP, but it's specifically ICMP traffic. Over a transit network, like you've got setup, I'm pretty sure you still need the rules for the interfaces. Maybe one of the pros can comment on that for sure.

Also, since these are "different" subnets, and you said "Windows", you need to open the internal firewall rules on the computers to allow this traffic from the other subnets. Windows considers traffic from machines NOT on the same subnet to be hostile. I know in your situation they aren't hostile, but Windows is programmed that way.

Jeff

[johnpoz]

@WisceBIat said in Subnets can't communicate:

Anyone know what could be preventing the PCs from pinging each other?

The firewalls on the devices themselves.. If you can ping pfsense interface in the network ie your 2.1 and 1.2 address it screams your PCs firewalls.

Can tell you for sure that windows out of the box firewall does not allow pings from remote networks..

Simple way to validate that.. While your pinging pc2 from pc1, sniff on pfsense2 interface 2.1 - do you see it sending the pings on to pc2?

This is a very common mistake users make when they start to segment.. They forget about host firewalls and security software they are running, which might allow traffic of xyz type from the same network the device is on, but blocks when its not a local network.. Like what your doing pc1 is a different network than pc2.. Its not going to allow pinging unless you tell its firewall to allow it, or turn it off, etc.

[WisceBIat]

Ughh it was the firewalls on windows 10 the entire time!!! I had allowed ipv4 echo requests, but it looks like that wasn't enough because I just fully disabled it and now it works

[johnpoz]

Great now that you have connectivity working between your 2 networks via your transit network.

I assume pc2 has internet access as well?

Now you can start playing with vpn stuff if you so desire. You have disabled nat I take it on pfs2 I take it..

If you need help with the vpn stuff just ask.. happy to help.

I really do not see the point of pfs2 in your setup to be honest, other than a learning experience it serves no real purpose that I can see.

[WisceBIat]

@johnpoz

Internet is working as well! I will now try to re-connect the VPN and re-enable the NAT. Technically all I need is port 22/SSH open because that's the only way I'll be communicating between both LANs. I will see if I can just do port forwarding SSH and that will be good enough. Thanks for all your help so far