IPsec routing problem
-
I have two sites with pfsense - siteA and siteB there siteA is the main site. siteA is in Sweden and siteB in Thailand.
siteA has 192.168.12.xxx and siteB 192.168.11.xxx. They are connected with IPsec and it is working perfect with one exception.I need to connect to a Nextcloud server at siteB through siteA IPsec connection. The ISP in Thailand blocks ports and refuse to change that so I must go through siteA. I prefer to have the server local at siteB as I spend most of the year there. I try to explain the route to my Nextcloud server:
So, me out having a cup of coffee somewhere and want to check on my Nextcloud server.
I enter the domain name for my Nextcloud server and it goes to siteA.
pfsense at siteA has a NAT rule (portforward) that points to the ip address of my Nextcloud server. That ip address is at siteB.
And here the story ends - timeout.If I have the Nextcloud server at siteA it works but having it at siteB I get timeout so I must do something wrong with the routing in pfsense at siteA but I can't figure out what.
All other traffic between 192.168.11.xx and 192.168.12.xx works so no routing problem there.
Any ideas?
-
@seanr22a said in IPsec routing problem:
siteA has 192.168.12.xxx and siteB 192.168.11.xxx. They are connected with IPsec and it is working perfect with one exception.
I guess, it's a policy-based IPSec connection?
pfsense at siteA has a NAT rule (portforward) that points to the ip address of my Nextcloud server. That ip address is at siteB
This would only work with policy-based IPSec if site B directs its whole upstream traffic to A, which might not be desired, I assume.
So your only one option to make this work is to change the VPN. You can use IPSec VTI, OpenVPN or Wireguard.
-
@viragomann said in IPsec routing problem:
I guess, it's a policy-based IPSec connection?
I'm not so experienced with this so I can't say yes or no to that.
I have a basic IPsec tunnel setup between the two sites following Netgate docs.
siteA:
siteB:
-
@seanr22a
Yes, tunnel mode IPSec is a traditional policy-based connection.
The mode can be selected in the phase 2 settings.However, if you want to switch over to VTI (routed IPSec) you have to do this on both sites.
-
I set this up around 17 years ago and have just made small changes over the years so I need to refresh my knowledge and have to read up on VTI. Maybe Netgate have something in their docs.
Thank you for pointing me to what I need to do
-
VTI up and running. The old Phase1/Phase2 deleted.
All traffic between the two sites is working as before but the portforward to the Nextcloud server on the 192.168.11.xx network is still not working so it's like before. It must be something special needed when portforwarding over IPsec.
2am now, time to sleep :)
-
@seanr22a I donāt think the tunnel needs to be VTi to make this work. I have two sites as well using tunnel based IPsec, and I have no problem having only one device on site B use internet access through site A (and thus enable working portforwards).
The trick is you need to enable split phase twoās in your phase one setting, and then you need to have an additonal phase2 for your server on Site B that tells it to reach internet through the tunnel.
Phase 1 split setting:
Phase 2 for Site B - Notice ID 3, thats the device that reaches Internet through Site A.
You need to have to āoppositeā phase 2 in site A configured as well:
The all you need it the required firewall rules to allow the special device to reach internet over the IPsec interface, and your port forward and such. Remember to also modify the outbound NAT at site A so it properly NATās the special device on site B before forwarding traffic to the internet (NAT using the WAN address).
EDIT: In my little setup it is actually 8 IPās (a /29) that I tell to reach the internet through the IPsec tunnel.
-
@keyser
Thank you for your suggestions! I will give it a try next time I try to get this working.I was wrong about my VTI working. I deleted the old phase 1/phase 2 but didn't realize the active connection was still alive despite the delete. When I cleared the active connection, I lost the connection between the sites. Luckily, I have a backup backdoor so I don't need to take a 10-hour flight to fix this
The new VTI connection is up phase 1 and phase 2 but I can't get any traffic going so I have done something wrong but I'm not skilled enough to find out what. I followed this one https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/routed-vti.html but it take for granted you have the knowledge about this already.
I gave this up for now and set everything back as it was from the beginning. I installed an Apache proxy at siteA that send all request to the Nextcloud server at siteB so now it's working but I don't like getting one more point of failure so I will soon give VTI a try again.
-
@seanr22a said in IPsec routing problem:
All traffic between the two sites is working as before
So you have already add static routes on either site?
but the portforward to the Nextcloud server on the 192.168.11.xx network is still not working so it's like before. It must be something special needed when portforwarding over IPsec.
Exactly.
At site B you have to go to the IPSec Advanced Settings and set the " IPsec Filter Mode" to "Filter IPSec and Transport on assigned interfaces".
Then you need to assign an interface to the IPSec instance and move over the firewall rules, which are allowing incoming traffic from A to this new interface.
You have to ensure, that there is no firewall pass rule on the default IPSec tab matching the forwarded traffic.This is necessary to enable reply-to. It's only working on interface rules, but not on IPSec, which is an interface group for all IPSec connections.
Group rule have precedence over interface rules.
If you have multiple IPSec connection, best practice would be to assign an interface to each and define your rules there.@keyser said in IPsec routing problem:
I donāt think the tunnel needs to be VTi to make this work. I have two sites as well using tunnel based IPsec, and I have no problem having only one device on site B use internet access through site A (and thus enable working portforwards).
Yes, that's an option as I mentioned above already.
@seanr22a said in IPsec routing problem:
I installed an Apache proxy at siteA that send all request to the Nextcloud server at siteB so now it's working
A proxy at A will work as well, since it masquerades the source IP in forwarded traffic.
-
@seanr22a Be advised that enabling VTI mode and changing the policy to "filter IPsec and Transport on assigned interfaces" effectively disables the option to also use Mobile user IPsec VPN as that requires tunneled IPsec.
Just a little heads up in-case you are using Mobile User VPN on any of the two firewalls. -
@seanr22a I see a few ways around this one...
First of all, how do you access NextCloud today, what port(s) do you use? And do you use a reverse proxy?
One way around ISP port blocking is to change the Listen port in the application (NextCloud) and use something they don't block. Clearly they don't block everything, since you can set up an IPSec connection.
Another solution that should work with your existing VPN is something similar to what I'm doing. I'm also running NextCloud and I use Nginx Proxy Manager to access it over HTTPS.
In Nginx I have simply set up my NextCloud server as one of the Hosts pointing to it's internal IP. It doesn't matter if that server is running on the same network as Nginx, a VLAN, or at my remote location access over Wireguard VPN, Nginx doesn't care. So with your existing IPSec connection, assuming you can ping your NextCloud server from your LAN in Sweden it will work.[EDIT] I see now you already did this, with Apache Proxy, which is pretty much the same solution. Although my understanding is that Nginx is much more efficient for this type of task.
I guess it would be more efficient if you set up Nginx in Thailand instead, so you don't get the "tromboning effect", when you are in Thailand. But then, again, you need to make sure you can use ports that your ISP doesn't block. I guess one challenge might be port 80 which is used for certificate authorization...
BTW, have you tried using Wireguard instead of IPSec? It's a bit more like IPSec VTI in the setup, and is way more efficient. Although, the value of changing depends on what bandwidth you have at each end, and what HW your are running.
-
@seanr22a Also if you are worried about the Proxy being a point of failure, you can always run two (or more) instances on separate machines. Then use something like keepalived to maintain that HA setup on case one of them fails.
