• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Multiple Sites Routing with Site to Site and Road Warrior

Scheduled Pinned Locked Moved OpenVPN
22 Posts 5 Posters 4.9k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • B
    BEB Consulting
    last edited by May 23, 2017, 6:48 PM

    Ok, I am sure this has been asked 1000s of time, however I am still not able to wrap my head around it to get it to actually work.

    I have 3 SITES with PF, SPH, PVG and RGB.  SPH is acting as the server for OpenVPN for both PVG and RGB, plus the server for RoadWarrior. They all run on separate ports as separate instances on SPH Site.

    The Site to Sites are PRE-SHARED KEY setups. Roadwarrior is Certificate based.

    I been having two issues, that are sort of related. They both are routing issues.

    The site to Sites map out like this:
    SPH 10.0.160.0/19 –-> PVG 10.0.32.0/19 = WORKS

    SPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = WORKS

    PVG 10.0.32.0/19 ---> SPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = FAILS

    RGB 10.0.96.0/19 ---> SPH 10.0.160.0.19 ---> PVG 10.0.32.0/19 = FAILS

    I am not sure if I need to do iroutes (client side) or push routes (server side) to get  the two paths that fail to work OR do I just need to add them to the REMOTE NETWORKS on SPH or do they go on RGB Client and PVG Client OR do I need to do all three (iroute, push routes and REMOTE NETWORK entries) to make the routing to work.

    Boiled down I need to get all the site to sites to see everything that is behind the PF FWs over the VPN.

    Then once I can get that working, I need to get RoadWarrior to see all 3 sites as well, right now it only sees SPH

    The RoadWarrior maps out like this:

    RR --> SPH 10.0.160.0/19 = WORKS

    RR --> PVG 10.0.32.0/19 = FAILS

    RR --> RGB 10.0.96.0/19 FAILS

    SPH is OpenVPN Server, SPH as 2 Site to Site Server instances one for PVG and one for RGB.

    PVG and RGB are OpenVPN Clients to SPH as the HUB VPN

    SPH is also the server for RoadWarrior.

    Boiling this down, I need RR clients to be able to touch everything in all three sites. Again I am not sure if I need to push routes (RR Server side) or iroute (RR Client Overrides) or add the SITES to the REMOTE NETWORKS on the RR Client or RR Server or do I need to do all three to make routing work.

    I am trying to avoid putting OpenVPN Servers at PVG and RGB, I'd like to keep them just clients for site to site. I am also trying to avoid putting RR Servers at PVG and RGB as well.

    ANY guidance is greatly appreciated. As I have tried several of the suggestions when searching Muilti-Site to Site Routing, only to have traffic completely stop and end up rolling back the configuration to what works again.

    1 Reply Last reply Reply Quote 1
    • B
      BEB Consulting
      last edited by May 26, 2017, 3:53 PM

      Anyone? Anything?

      1 Reply Last reply Reply Quote 0
      • V
        viragomann
        last edited by May 26, 2017, 5:31 PM

        You're running 2 separate site-to-site instances, as I understand, right?

        So there are no client specific overrides needed.
        Just go the site-to-site clients settings and add each remote network you want to access to "Remote networks" box (comma-separated, in CIDR notation).
        On PVG client enter: <sph-lan>,<rgb-lan>, <rr-tunnel network="">On RGB client: <sph-lan>,<pvg-lan>, <rr-tunnel network="">Then go to the road-warrior server settings and add all networks the clients should be able to reach to the "Local networks" box: <sph-lan>,<rgb-lan>, <pvg-lan>Check that the firewall rules permit the desired access.</pvg-lan></rgb-lan></sph-lan></rr-tunnel></pvg-lan></sph-lan></rr-tunnel></rgb-lan></sph-lan>

        1 Reply Last reply Reply Quote 0
        • B
          BEB Consulting
          last edited by May 31, 2017, 9:50 PM May 31, 2017, 9:46 PM

          You're running 2 separate site-to-site instances, as I understand, right?
          yes, there are 3 OpenVPN Server instances actually, 2 are site to site, and one is RR all three are on SPH.

          RGB and PVG are Site to Site Clients to SPH.

          I have added the routes as perscribed:
          On PVG client enter: <sph-lan>,<rgb-lan>, <rr-tunnel network="">On RGB client: <sph-lan>,<pvg-lan>, <rr-tunnel network="">Then go to the road-warrior server settings and add all networks the clients should be able to reach to the "Local networks" box: <sph-lan>,<rgb-lan>, <pvg-lan>The ROADWARRIOR routes perfectly now. –Thank you very much!

          However the site to sites still do not route right....

          SPH --> RGB = WORKS
          SPH --> PVG = WORKS

          RGB ---> SPH --> PVG = FAILS
          PVG ---> SPH --> RGB = FAILS

          RGB ---> SPH = WORKS
          PVG ---> SPH = WORKS

          The firwall rule on SPH, RGB and PVG allows any traffic from all to all on the OpenVPN interface at each end point.

          But I am not able to hit anything on PVG from RGB going through SPH. I am also not able to go the other way, not able to hit anything on RGB from PVG going through SPH.

          I am able to hit anything in PVG from SPH, as well as being able to hit anything in RGB from SPH, so that is working.

          Do I need static route on RGB pointing to PVG with SPH as gateway, as well as a static route on PVG pointing to RGB with SPH as gateway?

          Not sure what else I am missing.....all looks like it should work.</pvg-lan></rgb-lan></sph-lan></rr-tunnel></pvg-lan></sph-lan></rr-tunnel></rgb-lan></sph-lan>

          1 Reply Last reply Reply Quote 0
          • V
            viragomann
            last edited by Jun 1, 2017, 6:03 PM

            Is there an outbound NAT rule defined on openVPN interface of RGB or PVG?

            1 Reply Last reply Reply Quote 0
            • B
              BEB Consulting
              last edited by Jun 1, 2017, 10:59 PM Jun 1, 2017, 10:53 PM

              There is one single rule on all 3 sites OpenVPN interface under Rules, to allow any OpenVPN traffic to pass in either direction in and out.

              See screenshot attached….

              There is PURE NAT enabled with auto outbound rule creation.

              see other screenshot...

              Pfsense-OpenVPN-Rule.PNG
              Pfsense-OpenVPN-Rule.PNG_thumb
              Pfsense-OpenVPN-Rule.PNG
              Pfsense-OpenVPN-Rule.PNG_thumb
              Pfsense-OpenVPN-NAT.PNG
              Pfsense-OpenVPN-NAT.PNG_thumb

              1 Reply Last reply Reply Quote 0
              • V
                viragomann
                last edited by Jun 1, 2017, 11:14 PM

                Okay, your outbound NAT is set in automatic mode, but are there rules for the OpenVPN interface listed at the bottom? For instance, if you assign a gateway to a VPN connection, pfSense may generate an outbound NAT rule for that interface.

                If not, maybe it helps to assign an interface to each of the site-to-site servers on SPH. Sometimes packets get miss-routed without this.

                At least I would go to troubleshooting with packet capture. On SPH take captures on the different OpenVPN interfaces (you've assigned above) to see if the packets are routed correctly and have their proper IP addresses.

                1 Reply Last reply Reply Quote 0
                • M
                  marvosa
                  last edited by Jun 4, 2017, 2:38 AM

                  Post the server1.conf and server2.conf from SPH and the client1.conf from each client site.

                  After that, I would add any/any rules to both the LAN interface and the OpenVPN interface on all sites until basic IP connectivity is established.  I would then disable the software firewall on the endpoints you are testing from.

                  1 Reply Last reply Reply Quote 0
                  • B
                    BEB Consulting
                    last edited by Jun 5, 2017, 9:33 PM

                    using FIND there is no server1.conf, server2.conf or client1.conf in any directory.

                    When I add in interface, for OpenVPN, then none of the routing works, I am not able to ping ANYTHING but SPH's LAN interface, as soon as I remove the OpenVPN interface,what routing that was works before comes back.

                    SPH sees traffic from PVG and RGB, it just is not routing from PVH through SPH to RGB or vice versa.

                    There is NO software firewalls at any ENDPOINT. Only via Pfsense.

                    LAN Rules are ALL all between OpenVPN and LAN, block out outbound on WAN, except for ports that are Forwarded to a LAN host.

                    1 Reply Last reply Reply Quote 0
                    • M
                      marvosa
                      last edited by Jun 6, 2017, 3:57 AM

                      The OpenVPN config files are in "/var/etc/openvpn".  What syntax did you use for the find command?  Because it works for me:

                      [2.3.4-RELEASE][admin@example.home]/: find / -name "server*.conf"
                      /var/etc/openvpn/server1.conf
                      /var/etc/openvpn/server3.conf
                      /var/etc/openvpn/server2.conf
                      /var/etc/openvpn/server4.conf
                      /var/etc/openvpn/server6.conf
                      /var/etc/openvpn/server8.conf

                      You can also use the GUI:

                      Diagnostics -> Edit File -> Browse to "/var/etc/openvpn"

                      LAN Rules are ALL all between OpenVPN and LAN, block out outbound on WAN, except for ports that are Forwarded to a LAN host.

                      I don't want to make assumptions on what I think you're saying here, so it would be better to post screen shots of the rules on each interface.

                      1 Reply Last reply Reply Quote 0
                      • B
                        BEB Consulting
                        last edited by Jun 10, 2017, 3:12 PM Jun 10, 2017, 3:09 PM

                        I don't want to make assumptions on what I think you're saying here, so it would be better to post screen shots of the rules on each interface.

                        The LAN rules are the defaults when pfsense is installed, no changes, but I can post here with the conf files.

                        OpenVPN Rule Screenshot posted already…..above first screenshot, this rule is the same OpenVPN rule on ALL sites.

                        The OpenVPN config files are in "/var/etc/openvpn".  What syntax did you use for the find command?  Because it works for me:

                        used find "server.conf"

                        I'll try your command and post the files......

                        --Thanks and stand by....might be a day or two....just got a few other fires on other projects I need to stomp out first.....like Employee Mid-year reviews!  :P

                        1 Reply Last reply Reply Quote 0
                        • D
                          Derelict LAYER 8 Netgate
                          last edited by Jun 10, 2017, 6:25 PM

                          If I were you I would switch the site-to-site networks to SSL/TLS so you can just push the necessary routes to the site-to-site clients as your needs change.

                          But:

                          SPH Site-to-Site server to PVG:

                          IPv4 Remote Networks: 10.0.32.0/19

                          SPH Site-to-Site server to RGB:

                          IPv4 Remote Networks: 10.0.96.0/19

                          SPH Remote Access Server:

                          IPv4 Local Networks: 10.0.160.0/19, 10.0.96.0/19, 10.0.32.0/19 (OR Redirect Gateway set, which will, of course, also send this traffic through the tunnel)

                          PVG Client to SPH:

                          IPv4 Remote Networks: 10.0.160.0/19, 10.0.96.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19

                          RGB Client to SPH:

                          IPv4 Remote Networks: 10.0.160.0/19, 10.0.32.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • B
                            BEB Consulting
                            last edited by Jun 10, 2017, 8:28 PM

                            I have tried SSL/TLS it was SO SLOW that the connections within the VPN would constantly drop.

                            As per requested the conf files….

                            PVG-client1.conf.txt
                            RBG-client1.conf.txt
                            SPH-server1.conf.txt
                            SPH-server3.conf.txt

                            1 Reply Last reply Reply Quote 1
                            • B
                              BEB Consulting
                              last edited by Jun 10, 2017, 8:29 PM

                              Here are the WAN rules on SPH

                              ![WAN Rules.PNG](/public/imported_attachments/1/WAN Rules.PNG)
                              ![WAN Rules.PNG_thumb](/public/imported_attachments/1/WAN Rules.PNG_thumb)

                              1 Reply Last reply Reply Quote 0
                              • B
                                BEB Consulting
                                last edited by Jun 10, 2017, 8:30 PM

                                Here are the LAN rules, which are the SAME on all sites.

                                The openVPN rules are on another post previously.

                                ![LAN Rules on ALL sites.PNG](/public/imported_attachments/1/LAN Rules on ALL sites.PNG)
                                ![LAN Rules on ALL sites.PNG_thumb](/public/imported_attachments/1/LAN Rules on ALL sites.PNG_thumb)

                                1 Reply Last reply Reply Quote 0
                                • B
                                  BEB Consulting
                                  last edited by Jun 10, 2017, 8:37 PM

                                  @Derelict:

                                  If I were you I would switch the site-to-site networks to SSL/TLS so you can just push the necessary routes to the site-to-site clients as your needs change.

                                  But:

                                  SPH Site-to-Site server to PVG:

                                  IPv4 Remote Networks: 10.0.32.0/19

                                  SPH Site-to-Site server to RGB:

                                  IPv4 Remote Networks: 10.0.96.0/19

                                  SPH Remote Access Server:

                                  IPv4 Local Networks: 10.0.160.0/19, 10.0.96.0/19, 10.0.32.0/19 (OR Redirect Gateway set, which will, of course, also send this traffic through the tunnel)

                                  PVG Client to SPH:

                                  IPv4 Remote Networks: 10.0.160.0/19, 10.0.96.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19

                                  RGB Client to SPH:

                                  IPv4 Remote Networks: 10.0.160.0/19, 10.0.32.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19

                                  So you mean add

                                  SPH Site-to-Site server to PVG:

                                  IPv4 Remote Networks: 10.0.96.0/19

                                  SPH Site-to-Site server to RGB:

                                  IPv4 Remote Networks: 10.0.32.0/19

                                  As the way you specify:

                                  SPH Site-to-Site server to PVG:

                                  IPv4 Remote Networks: 10.0.32.0/19

                                  SPH Site-to-Site server to RGB:

                                  IPv4 Remote Networks: 10.0.96.0/19

                                  These are the site PVG and RGB subnets that are LOCAL already to the sites listed. Are you meaning add the other sites non routing sites to REMOTE NETWORKS on their mated sites?

                                  I've attached the SPH to PVG Server screenshot….and it already has 10.0.32.0/19 on REMOTE NETWORKS, so do you mean add 10.0.96.0/19 as well?

                                  ![SPH Server to PVG.PNG](/public/imported_attachments/1/SPH Server to PVG.PNG)
                                  ![SPH Server to PVG.PNG_thumb](/public/imported_attachments/1/SPH Server to PVG.PNG_thumb)

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    Derelict LAYER 8 Netgate
                                    last edited by Jun 10, 2017, 9:20 PM Jun 10, 2017, 8:58 PM

                                    I explained what Remote networks need to be where. I cannot tell exactly what needs to be added to what is there because I do not know what is already there.

                                    Adding something to IPv4 Remote Networks for a shared-key OpenVPN server or client creates a kernel ROUTE into that OpenVPN instance for those networks. Those routes are visible in Diagnostics > Routes.

                                    For traffic to be directed into OpenVPN there has to be a ROUTE telling pfSense to send it there.

                                    For traffic to be allowed IN there has to be a RULE passing it (or an already-existing state allowing the traffic, which would include reply traffic).

                                    I've attached the SPH to PVG Server screenshot….and it already has 10.0.32.0/19 on REMOTE NETWORKS, so do you mean add 10.0.96.0/19 as well?

                                    If you want RGB to send traffic to SPH for delivery to PVG then RGB needs the PVG networks added as remote networks. Why would you want to route RGB networks to PVG? That makes no sense.

                                    PVG 10.0.32.0/19 –-> SPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = FAILS

                                    That tells you exactly what you need:

                                    PVG needs 10.0.160.0/19 and 10.0.96.0/19 as IPv4 Remote Networks

                                    RGB 10.0.96.0/19 –-> SPH 10.0.160.0/19 ---> PVG 10.0.32.0/19 = FAILS

                                    And RGB needs 10.0.160.0/19 and 10.0.32.0/19 as IPv4 Remote Networks

                                    It is just interfaces, rules, and routes. Not really different from anything else.

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      BEB Consulting
                                      last edited by Jun 11, 2017, 2:58 PM

                                      If you want RGB to send traffic to SPH for delivery to PVG then RGB needs the PVG networks added as remote networks. Why would you want to route RGB networks to PVG? That makes no sense.

                                      We have application servers in all three sites that need to talk to each other. I need routing to all three sites. I am trying to avoid having VPNs from PVG to RGB, first off because PVG have dynamic IP from the ISP so setting them up as servers is almost impossible. Secondly only SPH and RGB have a static IP assigned to them so those are the only ones that could possible be OpenVPN Servers.

                                      We also have clients at all three sites that need to talk to servers in all three sites as well.

                                      This is why we need routing between PVG, SPH and RGB to work.

                                      Prior to creating this thread I tried, adding the remote sites to REMOTE NETWORKS, however the VPN went down, and would not come back up….just kept trying to reconnect but never would. I will try adding them again and see what happens. I thought I was missing something but it appears maybe not.....I maybe didn't wait long enough.....for the VPN to establish. I will attempt it again and see it if it works.

                                      1 Reply Last reply Reply Quote 0
                                      • johnpozJ
                                        johnpoz LAYER 8 Global Moderator
                                        last edited by Jun 11, 2017, 3:12 PM

                                        "first off because PVG have dynamic IP from the ISP so setting them up as servers is almost impossible."

                                        I really don't buy this… So your saying ISP changes the IPs on those how often?  I have dynamic IP from my isp - and its been the same for well over 2 years.  As long as you renew your lease and your equipment is not offline for extended periods with how dhcp normally works your IP should rarely change.  Are you saying your ISP forces your IP to change - if so how often?

                                        And why can you not just use some sort of dynamic dns to allow have the same fqdn point to whatever your IP might be, etc..

                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                        If you get confused: Listen to the Music Play
                                        Please don't Chat/PM me for help, unless mod related
                                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          Derelict LAYER 8 Netgate
                                          last edited by Jun 11, 2017, 5:07 PM

                                          SSL/TLS vs. Shared Key performs generally the same. It is a matter of authentication, not transport.

                                          The real issue is whether the server transport itself is UDP (preferred) or TCP (can be pretty poor).

                                          You don't seem to be grasping something. Not sure how else to explain it.

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          20 out of 22
                                          • First post
                                            20/22
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received