Two gateways, two subnets, one internet, subnet connectivity issue

Live4soccer7

I can ping everything on LAN1 (192.168.1.0), however when I try to ping something on LAN2 I get something like this:

92 bytes from dd-wrt (192.168.1.1): Destination Host Unreachable
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 5400 4fc9 0 0000 3f 01 a41b 192.168.1.115 192.168.5.1

92 bytes from dd-wrt (192.168.1.1): Redirect Host(New addr: 192.168.1.254)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 f9f5 0 0000 3f 01 f9ee 192.168.1.115 192.168.5.1

I'm not sure what the .254 address is.

Live4soccer7

I found where the .254 was coming from. DDWRT was holding on to an old route that I set up previously in this thread. A reboot and it let go. Makes most of what I've been doing irrelevant and must go back now. Waiting for DHCP on the other end so I can connect to a PC over VNC for testing. I have not idea why VNC works over this network, but I'm glad that it is working.

Live4soccer7

I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see. I have access from both directions and internet on both sides. One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

On the DDWRT Router (192.168.1.1) that is hooked to the modem, I have a static route setup. Operating mode is gateway

Lan Dest: 192.168.5.0
Subnet Mask: 255.255.255.0
Gateway: 192.168.1.112

Then on pfSense (192.168.5.1)
WAN of pfsense is essentially hooked to the LAN of DDWRT
WANIP: 192.168.1.112
Gateway: 192.168.1.1
LANIP: 192.168.5.1

Static Route WAN
Destination Network: 192.168.5.0
Gateway: 192.168.1.1

Firewall/NAT are still on essentiallly. I pretty much bypassed them with the local network rules I have set.

NAT --- Outbound
Hybrid Outbound NAT
I added the following rule
Interface WAN
Source 192.168.1.0/24
Source Port *
Destination 192.168.5.0
Destination Port: *
Nat Address: WAN Address
Nat Port *
Static Port NO

Basically on the firewall rules I added any/any to WAN and LAN.

I'm sure this looks a bit ridiculous to someone with much more networking knowledge and I bet there are some major flaws as well, but it works and it is still behind the head router's firewall.

Live4soccer7

Hopefully someone is still watching this thread.

I want to go back to the beginning and take a look at something fundamental from the big picture of the design.

Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
Network 2 router function as just a router and DHCP server for 192.168.1.1-149

Would this still allow each network to function by itself with the exception of internet and also not create redundancies in the wifi bridge traffic when devices on network 2 are communication between other devices on network 2?

I realize this thread has pretty much come full circle, but I have accomplished, mostly, what I aimed to do and have learned a lot in the process.

IMG_1154.JPG_thumb

marvosa

I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see.

Glad to hear it's working! The reason it's working is because traffic from LAN 2 is NAT'd to 192.168.1.112 as it exits the WAN interface and then routed to LAN 1's WAN interface where it's NAT'd again before hitting the internet. Since incoming traffic from LAN 2 is NAT'd to a subnet that's already in the routing table on LAN 1, it knows where to send the return traffic. Traffic sourced from LAN 1 is routed out the LAN interface and allowed thru LAN 2's firewall per the any/any rule.

Your outbound NAT rule on PFsense rule may or may not be helping the situation. Your NAT rule is translating all incoming LAN 1 traffic to the WAN IP of LAN 2. Since, LAN 2 already knows how to get to LAN 1's subnet, I doubt this rule is doing anything.

Unless there's a typo in your post about the static route on PFsense, the static route on PFsense isn't doing anything. Anything destined for 192.168.5.0/24 on LAN 2 would be handled by the switch and would never hit the firewall.

In short, it's working which is the most important thing, but because of the NATing you've lost all auditing capability.

I took some time to replicate your setup at home and got everything working as I was trying to explain it with one or two subtle differences.

My Setup
Modem -> PFsense_LAN1 -> Switch –-------->PFsense_LAN2 -> Switch -> Laptop 2 (192.168.5.2)
|
|-------> Laptop 1 (192.168.1.2)

Reproduced your subnets and connections on both sides
Statically assigned the LAN2 WAN interface with 192.168.1.254 and left the upstream gateway at "None"
Disabled the firewall and NAT on LAN2, so PFsense functions as a router only
Added a default route to 192.168.1.1 on the WAN interface of PFsense_LAN2 (System -> Routing -> Gateways) then add your gateway IP (192.168.1.1), then check the Default Gateway box
Added a static route on LAN1 for LAN2's LAN (destination 192.168.5.0/24 gateway 192.168.1.254). This step automatically adds an outbound NAT entry for 192.168.5.0/24 which allows LAN2 to get to the internet.

After the steps above, everything worked as expected:

Laptop 2 can access the internet thru LAN 1
Laptop 2 can ping Laptop 1 @ 192.168.1.2
Laptop 1 can ping Laptop 2 @ 192.168.5.2

If and when you want to try this, from what I can tell, the DDWRT side is set, you would only need to make some minor adjustments on the PFsense side to get things working.

Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

You have the right idea. There are routers at each site, so all traffic will stay local unless explicitly destined for the other side. Also, there are DHCP servers at each site, so if there's an issue with the wireless bridge, it won't affect the LAN on either side with the exception of internet access @ LAN 2.

Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

Yes, you went about it the right way, although you didn't join the subnets per se, but I know what you were trying to say :) You have separate sites directly connected by routers via a wireless bridge which is a perfectly sound design. The one caveat being ideally you want your firewall at the head end where the internet connection is, so you have more control over your network.

….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

Having 1 gateway (DDWRT) and PFsense acting as a router is exactly what you have. Even though you have the firewall enabled on PFsense, because of the any/any rule on the WAN it's essentially acting as a router. The other option would be to replace DDWRT with PFsense at the head end and connect the wireless bridge to the switch at LAN 2. While this would work, it's not necessarily better because it leaves LAN 2 completely dependent on LAN 1's infrastructure and all the broadcast traffic from both sites would be traversing the wireless bridge and overall network performance would suffer.

Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
Network 2 router function as just a router and DHCP server for 192.168.1.1-149

This wouldn't work, the subnets have to be unique at each site in a routed network. If for some reason you wanted the subnets to look similar at each site, you would have to do 192.168.1.0/25 for LAN 1 and 192.168.1.128/25 for LAN 2, but also understand that those are two different networks.

One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

This behavior is expected on windows machines. The windows firewall has ICMP echo reply disabled by default for traffic sourced outside of its local subnet. So, you either have to disable the windows firewall entirely or enable ICMP echo reply on every machine.

Live4soccer7

wow. I can't think you enough for the time you have invested in this. I surely hope this thread also helps out others as well.

I think I'll stick with the two subnets as it guarantees localization of the networks. I will try the tweaks you have mentioned as well. Ideally I'd like to have pfsense at the head of the network as well and I may do that in the future, but for now this will have to do.

My next project to tackle is to follow this tutorial on setting up OpenVPN with PIA: https://forum.pfsense.org/index.php?topic=76015.0
Do you foresee any issues integrating this with what's going on?

marvosa

My next project to tackle is to follow this tutorial on setting up OpenVPN with PIA: https://forum.pfsense.org/index.php?topic=76015.0
Do you foresee any issues integrating this with what's going on?

There are a couple things to consider if you are going to keep the current edge devices in place:

As currently connected, since PFsense is double NAT'd behind your edge router (DDWRT), you would first need to forward the OpenVPN listening port over to PFsense. Plus it may be necessary to add a static route on DDWRT for the OpenVPN tunnel network.
PFsense is only the gateway for LAN 2, so you'd only be able to route LAN 2's traffic thru the tunnel.

Live4soccer7

Yes, that is correct. I only want some of LAN2 traffic on OpenVPN.

If I can get it working on LAN2 with the pfSense Firewall and NAT disabled then I wouldn't need to forward to listening port over from DDWRT to pfSense, correct?

I don't really have any additional questions right now as I haven't started to dig in to the project yet. I hope to get to it in a few hours. I think the first step would be to get pfSense configured properly with the NAT/FIREWALL disabled and then move forward from there or I feel like I'll be compounding inefficiencies.

marvosa

If I can get it working on LAN2 with the pfSense Firewall and NAT disabled then I wouldn't need to forward to listening port over from DDWRT to pfSense, correct?

You're right. I was thinking about server connections, but a connection to PIA would be a client connection. Long night :)

If you do ever end up configuring a remote access server, then you would need to forward the listening port though.

Live4soccer7

Thanks, I'll be back as I progress through this. I'm going to clean up the connections I have right and then move on to the OpenVPN implementation.

Live4soccer7

I have the open VPN client setup on the router using this guide: https://www.privateinternetaccess.com/pages/client-support/pfsense

It worked great, however I'm still at the point where it would be best to disable NAT/Firewall and I'd like to only route specific IPs or an IP range through OpenVPN. If the pfsense box is setup as router only would I be using static routes somehow as the NAT/Firewall rules would no longer be functional in the "router only" state.

Live4soccer7

Looking more closely at everything, it almost seems like you HAVE to have NAT enabled on the pfsense machine in order to direct specific addresses through the VPN or not.