Multiple Sites Routing with Site to Site and Road Warrior
-
Ok, I am sure this has been asked 1000s of time, however I am still not able to wrap my head around it to get it to actually work.
I have 3 SITES with PF, SPH, PVG and RGB. SPH is acting as the server for OpenVPN for both PVG and RGB, plus the server for RoadWarrior. They all run on separate ports as separate instances on SPH Site.
The Site to Sites are PRE-SHARED KEY setups. Roadwarrior is Certificate based.
I been having two issues, that are sort of related. They both are routing issues.
The site to Sites map out like this:
SPH 10.0.160.0/19 –-> PVG 10.0.32.0/19 = WORKSSPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = WORKS
PVG 10.0.32.0/19 ---> SPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = FAILS
RGB 10.0.96.0/19 ---> SPH 10.0.160.0.19 ---> PVG 10.0.32.0/19 = FAILS
I am not sure if I need to do iroutes (client side) or push routes (server side) to get the two paths that fail to work OR do I just need to add them to the REMOTE NETWORKS on SPH or do they go on RGB Client and PVG Client OR do I need to do all three (iroute, push routes and REMOTE NETWORK entries) to make the routing to work.
Boiled down I need to get all the site to sites to see everything that is behind the PF FWs over the VPN.
Then once I can get that working, I need to get RoadWarrior to see all 3 sites as well, right now it only sees SPH
The RoadWarrior maps out like this:
RR --> SPH 10.0.160.0/19 = WORKS
RR --> PVG 10.0.32.0/19 = FAILS
RR --> RGB 10.0.96.0/19 FAILS
SPH is OpenVPN Server, SPH as 2 Site to Site Server instances one for PVG and one for RGB.
PVG and RGB are OpenVPN Clients to SPH as the HUB VPN
SPH is also the server for RoadWarrior.
Boiling this down, I need RR clients to be able to touch everything in all three sites. Again I am not sure if I need to push routes (RR Server side) or iroute (RR Client Overrides) or add the SITES to the REMOTE NETWORKS on the RR Client or RR Server or do I need to do all three to make routing work.
I am trying to avoid putting OpenVPN Servers at PVG and RGB, I'd like to keep them just clients for site to site. I am also trying to avoid putting RR Servers at PVG and RGB as well.
ANY guidance is greatly appreciated. As I have tried several of the suggestions when searching Muilti-Site to Site Routing, only to have traffic completely stop and end up rolling back the configuration to what works again.
-
Anyone? Anything?
-
You're running 2 separate site-to-site instances, as I understand, right?
So there are no client specific overrides needed.
Just go the site-to-site clients settings and add each remote network you want to access to "Remote networks" box (comma-separated, in CIDR notation).
On PVG client enter: <sph-lan>,<rgb-lan>, <rr-tunnel network="">On RGB client: <sph-lan>,<pvg-lan>, <rr-tunnel network="">Then go to the road-warrior server settings and add all networks the clients should be able to reach to the "Local networks" box: <sph-lan>,<rgb-lan>, <pvg-lan>Check that the firewall rules permit the desired access.</pvg-lan></rgb-lan></sph-lan></rr-tunnel></pvg-lan></sph-lan></rr-tunnel></rgb-lan></sph-lan> -
You're running 2 separate site-to-site instances, as I understand, right?
yes, there are 3 OpenVPN Server instances actually, 2 are site to site, and one is RR all three are on SPH.RGB and PVG are Site to Site Clients to SPH.
I have added the routes as perscribed:
On PVG client enter: <sph-lan>,<rgb-lan>, <rr-tunnel network="">On RGB client: <sph-lan>,<pvg-lan>, <rr-tunnel network="">Then go to the road-warrior server settings and add all networks the clients should be able to reach to the "Local networks" box: <sph-lan>,<rgb-lan>, <pvg-lan>The ROADWARRIOR routes perfectly now. –Thank you very much!However the site to sites still do not route right....
SPH --> RGB = WORKS
SPH --> PVG = WORKSRGB ---> SPH --> PVG = FAILS
PVG ---> SPH --> RGB = FAILSRGB ---> SPH = WORKS
PVG ---> SPH = WORKSThe firwall rule on SPH, RGB and PVG allows any traffic from all to all on the OpenVPN interface at each end point.
But I am not able to hit anything on PVG from RGB going through SPH. I am also not able to go the other way, not able to hit anything on RGB from PVG going through SPH.
I am able to hit anything in PVG from SPH, as well as being able to hit anything in RGB from SPH, so that is working.
Do I need static route on RGB pointing to PVG with SPH as gateway, as well as a static route on PVG pointing to RGB with SPH as gateway?
Not sure what else I am missing.....all looks like it should work.</pvg-lan></rgb-lan></sph-lan></rr-tunnel></pvg-lan></sph-lan></rr-tunnel></rgb-lan></sph-lan>
-
Is there an outbound NAT rule defined on openVPN interface of RGB or PVG?
-
There is one single rule on all 3 sites OpenVPN interface under Rules, to allow any OpenVPN traffic to pass in either direction in and out.
See screenshot attached….
There is PURE NAT enabled with auto outbound rule creation.
see other screenshot...
-
Okay, your outbound NAT is set in automatic mode, but are there rules for the OpenVPN interface listed at the bottom? For instance, if you assign a gateway to a VPN connection, pfSense may generate an outbound NAT rule for that interface.
If not, maybe it helps to assign an interface to each of the site-to-site servers on SPH. Sometimes packets get miss-routed without this.
At least I would go to troubleshooting with packet capture. On SPH take captures on the different OpenVPN interfaces (you've assigned above) to see if the packets are routed correctly and have their proper IP addresses.
-
Post the server1.conf and server2.conf from SPH and the client1.conf from each client site.
After that, I would add any/any rules to both the LAN interface and the OpenVPN interface on all sites until basic IP connectivity is established. I would then disable the software firewall on the endpoints you are testing from.
-
using FIND there is no server1.conf, server2.conf or client1.conf in any directory.
When I add in interface, for OpenVPN, then none of the routing works, I am not able to ping ANYTHING but SPH's LAN interface, as soon as I remove the OpenVPN interface,what routing that was works before comes back.
SPH sees traffic from PVG and RGB, it just is not routing from PVH through SPH to RGB or vice versa.
There is NO software firewalls at any ENDPOINT. Only via Pfsense.
LAN Rules are ALL all between OpenVPN and LAN, block out outbound on WAN, except for ports that are Forwarded to a LAN host.
-
The OpenVPN config files are in "/var/etc/openvpn". What syntax did you use for the find command? Because it works for me:
[2.3.4-RELEASE][admin@example.home]/: find / -name "server*.conf"
/var/etc/openvpn/server1.conf
/var/etc/openvpn/server3.conf
/var/etc/openvpn/server2.conf
/var/etc/openvpn/server4.conf
/var/etc/openvpn/server6.conf
/var/etc/openvpn/server8.confYou can also use the GUI:
Diagnostics -> Edit File -> Browse to "/var/etc/openvpn"
LAN Rules are ALL all between OpenVPN and LAN, block out outbound on WAN, except for ports that are Forwarded to a LAN host.
I don't want to make assumptions on what I think you're saying here, so it would be better to post screen shots of the rules on each interface.
-
I don't want to make assumptions on what I think you're saying here, so it would be better to post screen shots of the rules on each interface.
The LAN rules are the defaults when pfsense is installed, no changes, but I can post here with the conf files.
OpenVPN Rule Screenshot posted already…..above first screenshot, this rule is the same OpenVPN rule on ALL sites.
The OpenVPN config files are in "/var/etc/openvpn". What syntax did you use for the find command? Because it works for me:
used find "server.conf"
I'll try your command and post the files......
--Thanks and stand by....might be a day or two....just got a few other fires on other projects I need to stomp out first.....like Employee Mid-year reviews! :P
-
If I were you I would switch the site-to-site networks to SSL/TLS so you can just push the necessary routes to the site-to-site clients as your needs change.
But:
SPH Site-to-Site server to PVG:
IPv4 Remote Networks: 10.0.32.0/19
SPH Site-to-Site server to RGB:
IPv4 Remote Networks: 10.0.96.0/19
SPH Remote Access Server:
IPv4 Local Networks: 10.0.160.0/19, 10.0.96.0/19, 10.0.32.0/19 (OR Redirect Gateway set, which will, of course, also send this traffic through the tunnel)
PVG Client to SPH:
IPv4 Remote Networks: 10.0.160.0/19, 10.0.96.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19
RGB Client to SPH:
IPv4 Remote Networks: 10.0.160.0/19, 10.0.32.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19
-
I have tried SSL/TLS it was SO SLOW that the connections within the VPN would constantly drop.
As per requested the conf files….
PVG-client1.conf.txt
RBG-client1.conf.txt
SPH-server1.conf.txt
SPH-server3.conf.txt -
Here are the WAN rules on SPH
 -
Here are the LAN rules, which are the SAME on all sites.
The openVPN rules are on another post previously.
 -
If I were you I would switch the site-to-site networks to SSL/TLS so you can just push the necessary routes to the site-to-site clients as your needs change.
But:
SPH Site-to-Site server to PVG:
IPv4 Remote Networks: 10.0.32.0/19
SPH Site-to-Site server to RGB:
IPv4 Remote Networks: 10.0.96.0/19
SPH Remote Access Server:
IPv4 Local Networks: 10.0.160.0/19, 10.0.96.0/19, 10.0.32.0/19 (OR Redirect Gateway set, which will, of course, also send this traffic through the tunnel)
PVG Client to SPH:
IPv4 Remote Networks: 10.0.160.0/19, 10.0.96.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19
RGB Client to SPH:
IPv4 Remote Networks: 10.0.160.0/19, 10.0.32.0/19, SPH Remote Access Tunnel Network (as yet unspecified in thread) unless included in 10.0.160.0/19
So you mean add
SPH Site-to-Site server to PVG:
IPv4 Remote Networks: 10.0.96.0/19
SPH Site-to-Site server to RGB:
IPv4 Remote Networks: 10.0.32.0/19
As the way you specify:
SPH Site-to-Site server to PVG:
IPv4 Remote Networks: 10.0.32.0/19
SPH Site-to-Site server to RGB:
IPv4 Remote Networks: 10.0.96.0/19
These are the site PVG and RGB subnets that are LOCAL already to the sites listed. Are you meaning add the other sites non routing sites to REMOTE NETWORKS on their mated sites?
I've attached the SPH to PVG Server screenshot….and it already has 10.0.32.0/19 on REMOTE NETWORKS, so do you mean add 10.0.96.0/19 as well?
 -
I explained what Remote networks need to be where. I cannot tell exactly what needs to be added to what is there because I do not know what is already there.
Adding something to IPv4 Remote Networks for a shared-key OpenVPN server or client creates a kernel ROUTE into that OpenVPN instance for those networks. Those routes are visible in Diagnostics > Routes.
For traffic to be directed into OpenVPN there has to be a ROUTE telling pfSense to send it there.
For traffic to be allowed IN there has to be a RULE passing it (or an already-existing state allowing the traffic, which would include reply traffic).
I've attached the SPH to PVG Server screenshot….and it already has 10.0.32.0/19 on REMOTE NETWORKS, so do you mean add 10.0.96.0/19 as well?
If you want RGB to send traffic to SPH for delivery to PVG then RGB needs the PVG networks added as remote networks. Why would you want to route RGB networks to PVG? That makes no sense.
PVG 10.0.32.0/19 –-> SPH 10.0.160.0/19 ---> RGB 10.0.96.0/19 = FAILS
That tells you exactly what you need:
PVG needs 10.0.160.0/19 and 10.0.96.0/19 as IPv4 Remote Networks
RGB 10.0.96.0/19 –-> SPH 10.0.160.0/19 ---> PVG 10.0.32.0/19 = FAILS
And RGB needs 10.0.160.0/19 and 10.0.32.0/19 as IPv4 Remote Networks
It is just interfaces, rules, and routes. Not really different from anything else.
-
If you want RGB to send traffic to SPH for delivery to PVG then RGB needs the PVG networks added as remote networks. Why would you want to route RGB networks to PVG? That makes no sense.
We have application servers in all three sites that need to talk to each other. I need routing to all three sites. I am trying to avoid having VPNs from PVG to RGB, first off because PVG have dynamic IP from the ISP so setting them up as servers is almost impossible. Secondly only SPH and RGB have a static IP assigned to them so those are the only ones that could possible be OpenVPN Servers.
We also have clients at all three sites that need to talk to servers in all three sites as well.
This is why we need routing between PVG, SPH and RGB to work.
Prior to creating this thread I tried, adding the remote sites to REMOTE NETWORKS, however the VPN went down, and would not come back up….just kept trying to reconnect but never would. I will try adding them again and see what happens. I thought I was missing something but it appears maybe not.....I maybe didn't wait long enough.....for the VPN to establish. I will attempt it again and see it if it works.
-
"first off because PVG have dynamic IP from the ISP so setting them up as servers is almost impossible."
I really don't buy this… So your saying ISP changes the IPs on those how often? I have dynamic IP from my isp - and its been the same for well over 2 years. As long as you renew your lease and your equipment is not offline for extended periods with how dhcp normally works your IP should rarely change. Are you saying your ISP forces your IP to change - if so how often?
And why can you not just use some sort of dynamic dns to allow have the same fqdn point to whatever your IP might be, etc..
-
SSL/TLS vs. Shared Key performs generally the same. It is a matter of authentication, not transport.
The real issue is whether the server transport itself is UDP (preferred) or TCP (can be pretty poor).
You don't seem to be grasping something. Not sure how else to explain it.