Two gateways, two subnets, one internet, subnet connectivity issue



  • I will apologize about my lack of knowledge in this topic and I'm sure you guys hear this all the time. Anyways, I've been using pfSense for a few years now and love it. I've mostly been using it for basic purposes, but really like the idea behind it. I have had zero issues with it. Anyways, on to the setup.

    I originally had two completely separate networks that are about 1/2 mile apart physically. I have lost internet at one location so I have joined them with a couple Ubiquiti nanostation M5s. I have internet access on both LANs now.

    LAN1
    ISP LAN
    WRT320N Router with DDWRT
    LAN IP: 192.168.1.1
    WAN IP: Provided by ISP

    LAN2
    pfSense Box Connected to WAN of LAN1 Router essentially through the wifi bridge using the Nanostation M5s (both M5s have IPs from 192.168.1.0 subnet).
    LAN IP: 192.168.5.1
    WAN IP: 192.168.1.112

    I can access each router from the other network, I have internet on both, and I can even run VNC to control a mac across the network too.
    I am NOT able to access my server from LAN1 (subnet 192.168.1.0) that is located on LAN2 (subnet 192.168.5.0)

    On the DDWRT router on LAN1 (subnet 192.168.1.0) I have setup up static routes as follows:
    Metric: 0
    Destination LAN: 192.168.5.0
    Subnet: 255.255.255.0
    Gateway: 192.168.1.112
    Interface: LAN & WLAN

    On the pfSense Router, I am not quite sure what to configure as it seems the firewall may be the part prohibiting the incoming connections. Remember, I can access the pfSense router from LAN1, but seems I can not get anything beyond the actual router (other clients) on the 192.168.5.0 subnet.

    I tried checking this option in pfSense: Bypass firewall rules for traffic on the same interface It didn't seem to do anything.
    I tried setting up WAN and LAN firewall rules with a source being 192.168.1.0 subnet and destination being 192.168.5.0 subnet. I know this may not make sense or perhaps I have it backwards, but I've been working on this for about a week and can't quite seem to get something right.
    I also have a static route in pfsense setup as well. Destination LAN: 192.168.1.0, Subnet: 255.255.255, Gateway: WAN DHCP 192.168.1.1
    I have also attempted to disable NAT as well and I didn't achieve anything AND I lost internet connectivity.

    I WANT to keep both networks in tact with their own gateways and subnets so that they will work independently if the wireless bridge is lost or has issues. Granted LAN2 won't have internet, but the actual network will still function internally.

    Any input on this?



  • bump



  • Why not just use an IPSEC tunnel between the two sites?



  • IPSEC connects the networks over internet, correct? If so, I'd like to keep the connection local.



  • You can use IPSEC over any connection medium, not just internet links.  In pfSense, you can select the interface for the local endpoint (which would be your wireless uplink interface).

    The reason I'm suggesting you use IPSEC vs the way you're trying to do it is it allows you to make easy routing changes on both sides.  For example, if you decide to add a VLAN on your local/remote site, all you'd have to do is specify another phase 2 entry on both sides to have it route between the two.

    https://doc.pfsense.org/index.php/Routing_internet_traffic_through_a_site-to-site_IPsec_tunnel



  • Thanks for the input. I'll have to do some reading on this.

    Will this work in conjunction with OpenVPN? I want to use OpenVPN with a VPN service for anonymity while online.



  • Gateway: 192.168.1.112

    Not sure what this is..  ^^

    If you are trying to reach the network behind your DDWRT router then you need to provide a path through that router.

    As the WAN of the DDWRT is on the LAN of the pfsense box there would be no reason that your local LAN traffic ever passed through your pfsense box.

    Will DDWRT run RIP?

    Another thing you could do is set up the second LAN on an interface on the pfSense box and connect it to the wireless bridge to a simple switch on the other side.  That would allow you to run the separate subnets yet control them easily from one box.

    Or just combine the two locations on the same LAN.



  • The WAN of the Pfsense box is connected to the LAN of the DDWRT through the wireless bridge. The IP that is assigned to the WAN on the pfSense box is 192.168.1.112

    I believe DDWRT will run RIP



  • Ah-  Had to read that about 4 times before I got it.

    WAN firewall rules?  Whatcha got?  Im looking for the rule that would allow the 192.168.1.x subnet access through the pfSense box to its LAN..



  • DDWRT box-  Same thing.  There is nothing that would flow to or though that box if your pfSense WAN was on its LAN. (except the built in switch of coarse)



  • If you take a step back, look at the design, and forget the fact that LAN2's WAN is in a reserved IP space, it becomes clear.  It's not working because you are essentially trying to access 192.168.5.0/24 over the internet, which you are not going to be able to do without a port forward and firewall entry.  You can remove your current static route on PFsense as reserved IP's are not routable over the internet, so that traffic will never egress a WAN interface.  You will also need to uncheck the "Block private networks and loopback addresses" option on your WAN interface @ LAN 2.

    As currently connected, you have two options:

    • Create port forwards for everything you want to connect to on LAN 2

    • Create a site-to-site tunnel between the two sites

    You have a design issue that needs to be addressed.  If you're connecting two sites via a direct connection, you want to connect the sites via LAN interfaces (not WAN).  So, if you were going to keep both edge devices where they are, I would:

    • Add a 3rd NIC to PFsense on LAN 2 and assign it a static IP in the 192.168.1.1/24 range (e.g. 192.168.1.254)

    • @ LAN 2, patch your wireless bridge to the 3rd NIC

    • @ LAN 2, add an any/any rule to the 3rd NIC interface

    • @ LAN 1, add an any/any rule to the LAN interface (you can refine it later if needed)

    • @ LAN 1, add a static route to 192.168.5.0/24 with a gateway of 192.168.1.254

    • @ LAN 2, PFsense already knows how to get to 192.168.1.0/24 because of the locally connected interface, so no static route needed here

    • And you're done.

    A design consideration, if you're keeping that DDWRT router, considering LAN 2 is routed thru LAN 1, ideally you want your firewall at the head end, so I would swap the edge devices.  Otherwise, replace that DDWRT router with PFsense, so both edge devices are PFsense.

    Another option would be to extend LAN 1's network over to LAN 2's location by simply plugging the wireless bridge into the switch @ LAN 2 and remove the edge device altogether.  If you go this route, the same design consideration applies, I would stick PFsense at the head end.



  • Yes, the pfSense WAN is connected to the LAN of the DDWRT

    For quick reference
    –---------------------
    DDWRT has LAN IP of: 192.168.1.1
    DDWRT has WAN assigned by ISP
    pfSense Box has WAN IP of 192.168.1.112
    pfSense Box has LAN IP of 192.168.5.1

    WAN Firewall Rule on pfSense box:
    Action: Pass
    Interface: WAN
    TCP/IP: IPv4
    Protocol: TCP
    Source: Set to Network - 192.168.1.0 /24
    Destination: Set to Network - 192.168.5.0 /24
    Destination Port: ANY

    Let me know what else to check. I'm completely stumped. I can get in to the pfsense config/admin from the 192.168.1.0 subnet, but can not access anything else behind the pfsense box.



  • I'd like to keep an edge router/gateway at each location to keep each network working independently if the wireless bridge fails. This way the networks can function by themselves with the exception of connectivity to the internet. Almost everything important is on LAN2 (pfsense side) and that is where work is done 99% of the time. I don't mind creating a firewall rule for the individual items in LAN1 that I want to have access to LAN2, that would be ok. The question here is would that use internet to create that connection or would it still function without internet connection? I'd like the networks to be able to talk if the internet is down or turned off.



  • This way the networks can function by themselves with the exception of connectivity to the internet.

    What do you mean by this?  Because hypothetically if you statically addressed everything on that remote site, your network would still function with just a switch and no router.  With your current setup, what does the router add in an outage?  DHCP?  Local DNS resolution?



  • I don't mind creating a firewall rule for the individual items in LAN1 that I want to have access to LAN2, that would be ok. The question here is would that use internet to create that connection or would it still function without internet connection? I'd like the networks to be able to talk if the internet is down or turned off.

    The two sites are connected via wireless bridges, so the internet is not involved in any communication between the two sites.



  • It adds DHCP.

    Example. LAN1 –---------wifi Bridge --------------LAN2--devices on lan 2

    If the wifi bridge goes down or there is an issue on the LAN1 hardware/software AND there is no DHCP on LAN2 then the entire LAN2 network will not function at all. I'd like to keep LAN2 function (except internet) regardless of what happens in LAN1.



  • A couple things. If I were to go with the route of connecting the two networks via LAN to LAN, could I simply reassign the WAN NIC to a LAN and not have a WAN port on the pfSense Machine and do what you have described?

    For now, I'd like to properly setup a port forward in pfSense to allow an IP from LAN1 to access LAN2. The IP I'd like to give access is 192.168.1.115. pfSense port forward options under firewall-nat-port forward

    Interface: WAN
    Protocol: TCP
    Source: What type of source? Wan IP, LanIP, Network etc…?
    Source Port Range: ANY
    Destination: LAN net: 192.168.5.0 /24 I think this is right, but let me know
    Destination Port Range: Type/Number???
    Redirect Target IP: ?
    Redirect Target Port: ?



  • If I were to go with the route of connecting the two networks via LAN to LAN, could I simply reassign the WAN NIC to a LAN and not have a WAN port on the pfSense Machine and do what you have described?

    Good question, I suppose if you remove the gateway from the "WAN" interface and just rename it… it would then become a LAN interface... sure.

    For now, I'd like to properly setup a port forward in pfSense to allow an IP from LAN1 to access LAN2. The IP I'd like to give access is 192.168.1.115. pfSense port forward options under firewall-nat-port forward

    Interface: WAN
    Protocol: TCP
    Source: What type of source? Wan IP, LanIP, Network etc…?
    Source Port Range: ANY
    Destination: LAN net: 192.168.5.0 /24 I think this is right, but let me know
    Destination Port Range: Type/Number???
    Redirect Target IP: ?
    Redirect Target Port: ?

    Port forwards are for redirecting external traffic to a specific internal resource on specific ports, so that's not going to work.  For example, if you wanted to only allow 192.168.1.115 access to a web server on 192.168.5.100, you would enter this:

    Interface: WAN
    Protocol: TCP
    Source: "Single host or alias" "192.168.1.115/32"
    Source Port Range: ANY
    Destination: "WAN address"
    Destination Port Range: 80
    Redirect Target IP: 192.168.5.100
    Redirect Target Port: 80

    If you want to allow anyone to the web server, you would change your source back to "any"

    Then, @ LAN 1, to access the web server @ LAN 2, you would enter the WAN address of LAN 2 in your browser…i.e. http://192.168.1.112 and the traffic will be redirected to 192.168.5.100 on port 80.



  • So the port forward doesn't really seem like the best way to go about it then.

    I'm not really familiar with the site-site tunnel at all or IPSEC.

    Seems like trying to change the WAN on the pfsense box to a LAN connection and attempt the following:

    You have a design issue that needs to be addressed.  If you're connecting two sites via a direct connection, you want to connect the sites via LAN interfaces (not WAN).  So, if you were going to keep both edge devices where they are, I would:
    Add a 3rd NIC to PFsense on LAN 2 and assign it a static IP in the 192.168.1.1/24 range (e.g. 192.168.1.254)
    @ LAN 2, patch your wireless bridge to the 3rd NIC
    @ LAN 2, add an any/any rule to the 3rd NIC interface
    @ LAN 1, add an any/any rule to the LAN interface (you can refine it later if needed)
    @ LAN 1, add a static route to 192.168.5.0/24 with a gateway of 192.168.1.254
    @ LAN 2, PFsense already knows how to get to 192.168.1.0/24 because of the locally connected interface, so no static route needed here
    And you're done.



  • Good question, I suppose if you remove the gateway from the "WAN" interface and just rename it… it would then become a LAN interface... sure.

    I'm not seeing where I can remove the gateway on the WAN interface.



  • So the port forward doesn't really seem like the best way to go about it then.

    Correct.  Because you have to create a port forward for every different connection you want to make

    I'm not really familiar with the site-site tunnel at all or IPSEC.

    I would actually use OpenVPN here, the setup is easier, but either way, there's no reason to add the encryption overhead if it isn't necessary.

    Seems like trying to change the WAN on the pfsense box to a LAN connection and attempt the following:

    From a design perspective, this is your best option, yes.  In an ideal world, you would configure a separate interface on DDWRT and create an isolated transit network, but that's another conversation and I'm not familiar with creating and assigning interfaces on DDWRT.

    I'm not seeing where I can remove the gateway on the WAN interface.

    In the "General Configuration" section, Change the IPv4 Configuration Type to "static" and then in the "Static IPv4 Configuration" section, leave the IPv4 Upstream gateway option as "None"



  • Thanks, reading over your post now. Someone posted up a picture of a hand written setup. I was looking over that when I refreshed and it was taken down.



  • @Live4soccer7:

    Thanks, reading over your post now. Someone posted up a picture of a hand written setup. I was looking over that when I refreshed and it was taken down.

    Yes, he had the right idea about creating a transit network on separate interfaces on both sides, but some of the networking was incorrect and it wouldn't have worked.



  • @ LAN 2, add an any/any rule to the 3rd NIC interface

    For this, you are referring to the firewall rules, correct? If I named newly resigned WAN to NIC3 since that's what you were referring to adding and to lessen confusion between the LANS then would the rule be like this:

    This rule is being created ON LAN2 (pfsense)

    Interface: NIC3
    TCP/IP: IPv4
    Protocol: TCP
    Source: any
    Destination: any
    Destination Port Range: Leave blank or?



  • I've created the static route on LAN1.

    Destination: 192.168.5.0
    Subnet MASK: 255.255.255.0
    Gateway: 192.168.1.254

    Then when I go to change the WAN to static and assign it an IP of 192.168.1.254, I get the following error in pfsense: This IPv4 address conflicts with a Static Route.

    edit: by the way, thank you very much for your assistance. It is greatly appreciated. Same goes to everyone else.



  • I kind of figured you might run into that error.  I'm thinking something like this might work:

    http://i.imgur.com/95ouWv4.jpg

    I accidentally deleted my post with it originally.



  • The local and remote site are backwards in respect to the hardware being used (pfsense vs ddwrt).

    Are you suggesting to still hook the wireless bridge up on the remote site to a WAN port or to a LAN port?



  • I have not used DDWRT in a long time so I'm unfamiliar with the options.  Do you have the ability to create an OPT type port on the DDWRT side?



  • I can unbridge one of the physical ports and I believe assign it a new IP.

    Can I load pfsense on this DDWRT router? or is there a better option. It is a WRT320N (V1) router.



  • Yea that's probably what you're looking for.

    I still think an IPSEC tunnel between the sites where the phase two protocol set to AH is your best option.  That way you avoid the NAT issue and port forwarding issues.

    I'm pretty sure DDWRT supports IPSEC but I couldn't tell you how to set it up.  Optimally, it'd be best if you could put a pfSense device on the remote site.  All you'd need is a spare PC with two NIC ports.



  • That'd be nice. I'm fresh out of NIC ports and extra PCs. I've utilized just about every piece of hardware I have owned over the last 15 years in the current networks on both sides.

    I am unfamiliar with IPSEC. I want to run OpenVPN with a VPN service on the pfsense side. Can I do the IPSEC and have the OpenVPN/VPN service simultaneously?



  • I don't see why you couldn't.  There's no overlapping ports.

    You'd just have to make sure the pfSense box is strong enough to do the encryption/decryption for both IPSEC and OpenVPN at the same time.  Though if you setup the IPSEC phase two entries with just AH, the CPU impact would be far less.



  • Seems like there should be a simpler solution than IPSEC for LAN communication. I'm just baffled, I figured connecting two subnets would have been much simpler.



  • It really is simple.  You don't have to use IPSEC, I was only suggesting it because I thought that would be the easiest solution in this case.

    In reality, the only difficult part of this is the DDWRT setup because I'm unfamiliar with it.

    If you could figure out how to not use the WAN port (to avoid NAT) on the DDWRT side, you'd just set it up like how I have in the picture and then add the rules to deny/allow the types of traffic you want pass between the two networks.

    Edit:

    This might be even better: http://www.dd-wrt.com/phpBB2/viewtopic.php?t=78029



  • Thanks, I'm reading the link now.

    The WAN port from the PFsense machine is connected to the LAN on the DDWRT. Just in case there was confusion. I do have a third router with DDRWT on it as well if that router could simply handle the routing between the two subnets, but it seems like that would be redundant.



  • @jamesonp:

    I kind of figured you might run into that error.  I'm thinking something like this might work:

    http://i.imgur.com/95ouWv4.jpg

    I accidentally deleted my post with it originally.

    jamesonp, the design is sound, but the interfaces on the transit network have to be on the same network.  i.e. the OPT interfaces would need to be 172.16.0.1/30 and 172.16.0.2/30 then adjust the static routes accordingly.

    Then when I go to change the WAN to static and assign it an IP of 192.168.1.254, I get the following error in pfsense: This IPv4 address conflicts with a Static Route.

    I'm guessing you forgot to remove your old route sending 192.168.1.0/24 to 192.168.1.1?  That would be the reason for the conflict.

    Also, after doing some more research, if you use my suggestion, the inter-connectivity between the sites would work, but we would still need to use policy routing to get LAN 2 to the internet.  Which is fine, but it's an extra step.  In your specific case, all we really needed to do from the beginning is disable outbound NAT (Firewall -> NAT -> Outbound -> check Disable Outbound NAT) which turns PFsense into a routing only firewall and put an any/any rule on the interface connected to the bridge.

    Another option is to disable both NAT and the firewall which turns PFsense into a routing only platform:
    System -> Advanced -> Firewall & NAT -> check Disable all packet filtering

    I apologize for missing these steps.  This could've been a much shorter thread :)



  • Thanks, I'll give it a shot today. Possibly this morning if I can find time before work. I'll post back.



  • I tried the following as it seemed like the simplest test.

    Another option is to disable both NAT and the firewall which turns PFsense into a routing only platform:
    System -> Advanced -> Firewall & NAT -> check Disable all packet filtering

    I have now deleted an firewall rules that I've created and same went for routing tables on the pfSense box. When I did the above, I lost internet connectivity AND access to the router on the other side of the wireless bridge. I still have a route on the DDWRT router to the pfsense gateway and 192.168.5.0 subnet. What am I missing as that seems like it should do the trick. I still had local access to the 192.168.5.0 clients though, that was good.



  • You now need a default route @ LAN 2 pointing back to LAN 1 (192.168.1.1).

    You also need to verify that the static route @ LAN 1 is pointed at the current IP configured @ LAN 2.  (i.e. if the LAN 2 interface is 192.168.1.254, then the route should be destination 192.168.5.0/24 gateway 192.168.1.254)



  • Ok, so I did this:

    System -> Advanced -> Firewall & NAT -> check Disable all packet filtering

    WAN was set to 192.168.1.254
    Default Gateway on wan is set to: 192.168.1.1

    Still nothing. I am on the 192.168.5.0 subnet doing all this and once I make those changes I can't access the internet. On LAN1 I have confirmed that I DO have the static route pointing at the WAN IP provided above.

    Is there something that needs to be set on LAN2 pfsense box to tell the LAN on that box to communicate with the WAN since all NAT/Filtering is disabled?