Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Two gateways, two subnets, one internet, subnet connectivity issue

    Scheduled Pinned Locked Moved General pfSense Questions
    66 Posts 4 Posters 13.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      Live4soccer7
      last edited by

      I've been using .112 instead of .254 :) My static routes on LAN1 match this.

      ![route table.png](/public/imported_attachments/1/route table.png)
      ![route table.png_thumb](/public/imported_attachments/1/route table.png_thumb)

      1 Reply Last reply Reply Quote 0
      • M
        marvosa
        last edited by

        Can you ping 192.168.1.1?  Can you ping 8.8.8.8?

        1 Reply Last reply Reply Quote 0
        • L
          Live4soccer7
          last edited by

          I doubt it, but I'll change the configuration back right now and let you know.

          1 Reply Last reply Reply Quote 0
          • L
            Live4soccer7
            last edited by

            Ping on both was 100% failure.

            1 Reply Last reply Reply Quote 0
            • M
              marvosa
              last edited by

              Can you post a network map of how things are currently connected?

              Also, what default gateway are your clients using on the PC's?

              What happens when you ping 192.168.1.1 from PFsense itself?

              1 Reply Last reply Reply Quote 0
              • L
                Live4soccer7
                last edited by

                What happens when you ping 192.168.1.1 from PFsense itself?

                I'm checking now.

                Please excuse my terrible drawing, but it is all there. Let me know if you need clarification.

                IMG_1153.JPG
                IMG_1153.JPG_thumb

                1 Reply Last reply Reply Quote 0
                • M
                  marvosa
                  last edited by

                  Where can I input commands in pfSense?

                  Either from the shell via ssh or by going to Diagnostics -> Ping

                  1 Reply Last reply Reply Quote 0
                  • L
                    Live4soccer7
                    last edited by

                    Ping Results directly from pfSense. Looks good. That means something is still prohibiting a connection between the LAN and WAN on the pfsense box as I'm unable to ping from a client connected to the pfSense Subnet (192.168.5.0)

                    PING 8.8.8.8 (8.8.8.8): 56 data bytes
                    64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=19.789 ms
                    64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=19.992 ms
                    64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=18.517 ms

                    –- 8.8.8.8 ping statistics ---
                    3 packets transmitted, 3 packets received, 0.0% packet loss
                    round-trip min/avg/max/stddev = 18.517/19.433/19.992/0.653 ms

                    PING 192.168.1.1 (192.168.1.1): 56 data bytes
                    64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=1.897 ms
                    64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=3.034 ms
                    64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=2.398 ms

                    --- 192.168.1.1 ping statistics ---
                    3 packets transmitted, 3 packets received, 0.0% packet loss
                    round-trip min/avg/max/stddev = 1.897/2.443/3.034/0.465 ms

                    1 Reply Last reply Reply Quote 0
                    • M
                      marvosa
                      last edited by

                      Run a tracert on your laptop to 8.8.8.8 and post the results.

                      1 Reply Last reply Reply Quote 0
                      • L
                        Live4soccer7
                        last edited by

                        Traceroute while I have internet access:

                        Traceroute has started…

                        traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
                        1  192.168.5.1 (192.168.5.1)  2.032 ms  1.566 ms  3.057 ms
                        2  192.168.1.1 (192.168.1.1)  4.491 ms  2.755 ms  3.445 ms
                        3  * * *
                        4  WANISP.ptr.us.xo.net (WANISP)  13.697 ms  11.825 ms  15.092 ms
                        5  ge-0-0-2-spk-cr1.cet.com (198.202.27.65)  14.796 ms  12.241 ms  12.642 ms
                        6  te-1-1-sea-cr2.cet.com (198.202.26.2)  18.589 ms  20.696 ms  20.326 ms
                        7  six.sea01.google.com (206.81.80.17)  20.489 ms  20.139 ms  19.410 ms
                        8  108.170.245.113 (108.170.245.113)  20.336 ms  20.854 ms  20.953 ms
                        9  209.85.250.19 (209.85.250.19)  20.182 ms  20.219 ms  21.039 ms
                        10  google-public-dns-a.google.com (8.8.8.8)  19.162 ms  20.637 ms  20.258 ms

                        Let me run one when I turn NAT/Firewall off

                        traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
                        1  192.168.5.1 (192.168.5.1)  1.249 ms  3.578 ms  1.490 ms
                        2  * * *
                        3  * * *
                        4  * * *
                        5  * * *
                        6  * * *
                        7  * * *
                        8  * * *
                        9  * * *
                        10  * * *
                        11  * * *
                        12  * * *
                        13  * * *

                        1 Reply Last reply Reply Quote 0
                        • M
                          marvosa
                          last edited by

                          So, that traceroute shows you hitting the LAN 2 internal interface, then routed to LAN 1 and out to the internet.

                          So, what do you mean when you say "traceroute while I have internet access"?

                          ahh… I think I know what you mean now, so once you turn off the firewall and NATing.... you can't get past the local adapter.  Which tells me there's an internal  routing issue.

                          1 Reply Last reply Reply Quote 0
                          • L
                            Live4soccer7
                            last edited by

                            Yes, that is correct. It hits the LAN interface on LAN2 and then dies on the spot.

                            1 Reply Last reply Reply Quote 0
                            • L
                              Live4soccer7
                              last edited by

                              Do I need a static route between the LAN interface and the WAN?

                              1 Reply Last reply Reply Quote 0
                              • L
                                Live4soccer7
                                last edited by

                                I can ping everything on LAN1 (192.168.1.0), however when I try to ping something on LAN2 I get something like this:

                                92 bytes from dd-wrt (192.168.1.1): Destination Host Unreachable
                                Vr HL TOS  Len  ID Flg  off TTL Pro  cks      Src      Dst
                                4  5  00 5400 4fc9  0 0000  3f  01 a41b 192.168.1.115  192.168.5.1

                                92 bytes from dd-wrt (192.168.1.1): Redirect Host(New addr: 192.168.1.254)
                                Vr HL TOS  Len  ID Flg  off TTL Pro  cks      Src      Dst
                                4  5  00 0054 f9f5  0 0000  3f  01 f9ee 192.168.1.115  192.168.5.1

                                I'm not sure what the .254 address is.

                                1 Reply Last reply Reply Quote 0
                                • L
                                  Live4soccer7
                                  last edited by

                                  I found where the .254 was coming from. DDWRT was holding on to an old route that I set up previously in this thread. A reboot and it let go. Makes most of what I've been doing irrelevant and must go back now. Waiting for DHCP on the other end so I can connect to a PC over VNC for testing. I have not idea why VNC works over this network, but I'm glad that it is working.

                                  1 Reply Last reply Reply Quote 0
                                  • L
                                    Live4soccer7
                                    last edited by

                                    I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see. I have access from both directions and internet on both sides. One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

                                    On the DDWRT Router (192.168.1.1) that is hooked to the modem, I have a static route setup. Operating mode is gateway

                                    Lan Dest: 192.168.5.0
                                    Subnet Mask: 255.255.255.0
                                    Gateway: 192.168.1.112

                                    Then on pfSense (192.168.5.1)
                                    WAN of pfsense is essentially hooked to the LAN of DDWRT
                                    WANIP: 192.168.1.112
                                    Gateway: 192.168.1.1
                                    LANIP: 192.168.5.1

                                    Static Route WAN
                                    Destination Network: 192.168.5.0
                                    Gateway: 192.168.1.1

                                    Firewall/NAT are still on essentiallly. I pretty much bypassed them with the local network rules I have set.

                                    NAT --- Outbound
                                    Hybrid Outbound NAT
                                    I added the following rule
                                    Interface WAN
                                    Source 192.168.1.0/24
                                    Source Port *
                                    Destination 192.168.5.0
                                    Destination Port: *
                                    Nat Address: WAN Address
                                    Nat Port *
                                    Static Port NO

                                    Basically on the firewall rules I added any/any to WAN and LAN.

                                    I'm sure this looks a bit ridiculous to someone with much more networking knowledge and I bet there are some major flaws as well, but it works and it is still behind the head router's firewall.

                                    1 Reply Last reply Reply Quote 0
                                    • L
                                      Live4soccer7
                                      last edited by

                                      Hopefully someone is still watching this thread.

                                      I want to go back to the beginning and take a look at something fundamental from the big picture of the design.

                                      Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

                                      Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

                                      ….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

                                      Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
                                      Network 2 router function as just a router and DHCP server for 192.168.1.1-149

                                      Would this still allow each network to function by itself with the exception of internet and also not create redundancies in the wifi bridge traffic when devices on network 2 are communication between other devices on network 2?

                                      I realize this thread has pretty much come full circle, but I have accomplished, mostly, what I aimed to do and have learned a lot in the process.

                                      IMG_1154.JPG
                                      IMG_1154.JPG_thumb

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        marvosa
                                        last edited by

                                        I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see.

                                        Glad to hear it's working!  The reason it's working is because traffic from LAN 2 is NAT'd to 192.168.1.112 as it exits the WAN interface and then routed to LAN 1's WAN interface where it's NAT'd again before hitting the internet.  Since incoming traffic from LAN 2 is NAT'd to a subnet that's already in the routing table on LAN 1, it knows where to send the return traffic.  Traffic sourced from LAN 1 is routed out the LAN interface and allowed thru LAN 2's firewall per the any/any rule.

                                        Your outbound NAT rule on PFsense rule may or may not be helping the situation.  Your NAT rule is translating all incoming LAN 1 traffic to the WAN IP of LAN 2.  Since, LAN 2 already knows how to get to LAN 1's subnet, I doubt this rule is doing anything.

                                        Unless there's a typo in your post about the static route on PFsense, the static route on PFsense isn't doing anything.  Anything destined for 192.168.5.0/24 on LAN 2 would be handled by the switch and would never hit the firewall.

                                        In short, it's working which is the most important thing, but because of the NATing you've lost all auditing capability.

                                        I took some time to replicate your setup at home and got everything working as I was trying to explain it with one or two subtle differences.

                                        My Setup
                                        Modem -> PFsense_LAN1 -> Switch –-------->PFsense_LAN2 -> Switch -> Laptop 2 (192.168.5.2)
                                                                                        |
                                                                                        |-------> Laptop 1 (192.168.1.2)

                                        • Reproduced your subnets and connections on both sides

                                        • Statically assigned the LAN2 WAN interface with 192.168.1.254 and left the upstream gateway at "None"

                                        • Disabled the firewall and NAT on LAN2, so PFsense functions as a router only

                                        • Added a default route to 192.168.1.1 on the WAN interface of PFsense_LAN2 (System -> Routing -> Gateways) then add your gateway IP (192.168.1.1), then check the Default Gateway box

                                        • Added a static route on LAN1 for LAN2's LAN (destination 192.168.5.0/24 gateway 192.168.1.254).  This step automatically adds an outbound NAT entry for 192.168.5.0/24 which allows LAN2 to get to the internet.

                                        After the steps above, everything worked as expected:

                                        • Laptop 2 can access the internet thru LAN 1

                                        • Laptop 2 can ping Laptop 1 @ 192.168.1.2

                                        • Laptop 1 can ping Laptop 2 @ 192.168.5.2

                                        If and when you want to try this, from what I can tell, the DDWRT side is set, you would only need to make some minor adjustments on the PFsense side to get things working.

                                        Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

                                        You have the right idea.  There are routers at each site, so all traffic will stay local unless explicitly destined for the other side.  Also, there are DHCP servers at each site, so if there's an issue with the wireless bridge, it won't affect the LAN on either side with the exception of internet access @ LAN 2.

                                        Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

                                        Yes, you went about it the right way, although you didn't join the subnets per se, but I know what you were trying to say :)  You have separate sites directly connected by routers via a wireless bridge which is a perfectly sound design.  The one caveat being ideally you want your firewall at the head end where the internet connection is, so you have more control over your network.

                                        ….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

                                        Having 1 gateway (DDWRT) and PFsense acting as a router is exactly what you have.  Even though you have the firewall enabled on PFsense, because of the any/any rule on the WAN it's essentially acting as a router.  The other option would be to replace DDWRT with PFsense at the head end and connect the wireless bridge to the switch at LAN 2.  While this would work, it's not necessarily better because it leaves LAN 2 completely dependent on LAN 1's infrastructure and all the broadcast traffic from both sites would be traversing the wireless bridge and overall network performance would suffer.

                                        Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
                                        Network 2 router function as just a router and DHCP server for 192.168.1.1-149

                                        This wouldn't work, the subnets have to be unique at each site in a routed network.  If for some reason you wanted the subnets to look similar at each site, you would have to do 192.168.1.0/25 for LAN 1 and 192.168.1.128/25 for LAN 2, but also understand that those are two different networks.

                                        One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

                                        This behavior is expected on windows machines.  The windows firewall has ICMP echo reply disabled by default for traffic sourced outside of its local subnet.  So, you either have to disable the windows firewall entirely or enable ICMP echo reply on every machine.

                                        1 Reply Last reply Reply Quote 0
                                        • L
                                          Live4soccer7
                                          last edited by

                                          wow. I can't think you enough for the time you have invested in this. I surely hope this thread also helps out others as well.

                                          I think I'll stick with the two subnets as it guarantees localization of the networks. I will try the tweaks you have mentioned as well. Ideally I'd like to have pfsense at the head of the network as well and I may do that in the future, but for now this will have to do.

                                          My next project to tackle is to follow this tutorial on setting up OpenVPN with PIA: https://forum.pfsense.org/index.php?topic=76015.0
                                          Do you foresee any issues integrating this with what's going on?

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            marvosa
                                            last edited by

                                            My next project to tackle is to follow this tutorial on setting up OpenVPN with PIA: https://forum.pfsense.org/index.php?topic=76015.0
                                            Do you foresee any issues integrating this with what's going on?

                                            There are a couple things to consider if you are going to keep the current edge devices in place:

                                            • As currently connected, since PFsense is double NAT'd behind your edge router (DDWRT), you would first need to forward the OpenVPN listening port over to PFsense. Plus it may be necessary to add a static route on DDWRT for the OpenVPN tunnel network.

                                            • PFsense is only the gateway for LAN 2, so you'd only be able to route LAN 2's traffic thru the tunnel.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.