IPsec routing problem
-
@seanr22a said in IPsec routing problem:
siteA has 192.168.12.xxx and siteB 192.168.11.xxx. They are connected with IPsec and it is working perfect with one exception.
I guess, it's a policy-based IPSec connection?
pfsense at siteA has a NAT rule (portforward) that points to the ip address of my Nextcloud server. That ip address is at siteB
This would only work with policy-based IPSec if site B directs its whole upstream traffic to A, which might not be desired, I assume.
So your only one option to make this work is to change the VPN. You can use IPSec VTI, OpenVPN or Wireguard.
-
@viragomann said in IPsec routing problem:
I guess, it's a policy-based IPSec connection?
I'm not so experienced with this so I can't say yes or no to that.
I have a basic IPsec tunnel setup between the two sites following Netgate docs.
siteA:
siteB:
-
@seanr22a
Yes, tunnel mode IPSec is a traditional policy-based connection.
The mode can be selected in the phase 2 settings.However, if you want to switch over to VTI (routed IPSec) you have to do this on both sites.
-
I set this up around 17 years ago and have just made small changes over the years so I need to refresh my knowledge and have to read up on VTI. Maybe Netgate have something in their docs.
Thank you for pointing me to what I need to do
-
VTI up and running. The old Phase1/Phase2 deleted.
All traffic between the two sites is working as before but the portforward to the Nextcloud server on the 192.168.11.xx network is still not working so it's like before. It must be something special needed when portforwarding over IPsec.
2am now, time to sleep :)
-
@seanr22a I don’t think the tunnel needs to be VTi to make this work. I have two sites as well using tunnel based IPsec, and I have no problem having only one device on site B use internet access through site A (and thus enable working portforwards).
The trick is you need to enable split phase two’s in your phase one setting, and then you need to have an additonal phase2 for your server on Site B that tells it to reach internet through the tunnel.
Phase 1 split setting:
Phase 2 for Site B - Notice ID 3, thats the device that reaches Internet through Site A.
You need to have to “opposite” phase 2 in site A configured as well:
The all you need it the required firewall rules to allow the special device to reach internet over the IPsec interface, and your port forward and such. Remember to also modify the outbound NAT at site A so it properly NAT’s the special device on site B before forwarding traffic to the internet (NAT using the WAN address).
EDIT: In my little setup it is actually 8 IP’s (a /29) that I tell to reach the internet through the IPsec tunnel.
-
@keyser
Thank you for your suggestions! I will give it a try next time I try to get this working.I was wrong about my VTI working. I deleted the old phase 1/phase 2 but didn't realize the active connection was still alive despite the delete. When I cleared the active connection, I lost the connection between the sites. Luckily, I have a backup backdoor so I don't need to take a 10-hour flight to fix this
The new VTI connection is up phase 1 and phase 2 but I can't get any traffic going so I have done something wrong but I'm not skilled enough to find out what. I followed this one https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/routed-vti.html but it take for granted you have the knowledge about this already.
I gave this up for now and set everything back as it was from the beginning. I installed an Apache proxy at siteA that send all request to the Nextcloud server at siteB so now it's working but I don't like getting one more point of failure so I will soon give VTI a try again.
-
@seanr22a said in IPsec routing problem:
All traffic between the two sites is working as before
So you have already add static routes on either site?
but the portforward to the Nextcloud server on the 192.168.11.xx network is still not working so it's like before. It must be something special needed when portforwarding over IPsec.
Exactly.
At site B you have to go to the IPSec Advanced Settings and set the " IPsec Filter Mode" to "Filter IPSec and Transport on assigned interfaces".
Then you need to assign an interface to the IPSec instance and move over the firewall rules, which are allowing incoming traffic from A to this new interface.
You have to ensure, that there is no firewall pass rule on the default IPSec tab matching the forwarded traffic.This is necessary to enable reply-to. It's only working on interface rules, but not on IPSec, which is an interface group for all IPSec connections.
Group rule have precedence over interface rules.
If you have multiple IPSec connection, best practice would be to assign an interface to each and define your rules there.@keyser said in IPsec routing problem:
I don’t think the tunnel needs to be VTi to make this work. I have two sites as well using tunnel based IPsec, and I have no problem having only one device on site B use internet access through site A (and thus enable working portforwards).
Yes, that's an option as I mentioned above already.
@seanr22a said in IPsec routing problem:
I installed an Apache proxy at siteA that send all request to the Nextcloud server at siteB so now it's working
A proxy at A will work as well, since it masquerades the source IP in forwarded traffic.
-
@seanr22a Be advised that enabling VTI mode and changing the policy to "filter IPsec and Transport on assigned interfaces" effectively disables the option to also use Mobile user IPsec VPN as that requires tunneled IPsec.
Just a little heads up in-case you are using Mobile User VPN on any of the two firewalls. -
@seanr22a I see a few ways around this one...
First of all, how do you access NextCloud today, what port(s) do you use? And do you use a reverse proxy?
One way around ISP port blocking is to change the Listen port in the application (NextCloud) and use something they don't block. Clearly they don't block everything, since you can set up an IPSec connection.
Another solution that should work with your existing VPN is something similar to what I'm doing. I'm also running NextCloud and I use Nginx Proxy Manager to access it over HTTPS.
In Nginx I have simply set up my NextCloud server as one of the Hosts pointing to it's internal IP. It doesn't matter if that server is running on the same network as Nginx, a VLAN, or at my remote location access over Wireguard VPN, Nginx doesn't care. So with your existing IPSec connection, assuming you can ping your NextCloud server from your LAN in Sweden it will work.[EDIT] I see now you already did this, with Apache Proxy, which is pretty much the same solution. Although my understanding is that Nginx is much more efficient for this type of task.
I guess it would be more efficient if you set up Nginx in Thailand instead, so you don't get the "tromboning effect", when you are in Thailand. But then, again, you need to make sure you can use ports that your ISP doesn't block. I guess one challenge might be port 80 which is used for certificate authorization...
BTW, have you tried using Wireguard instead of IPSec? It's a bit more like IPSec VTI in the setup, and is way more efficient. Although, the value of changing depends on what bandwidth you have at each end, and what HW your are running.
-
@seanr22a Also if you are worried about the Proxy being a point of failure, you can always run two (or more) instances on separate machines. Then use something like keepalived to maintain that HA setup on case one of them fails.
-
I have three web sites including Nextcloud on the server at siteB and they are behind Cloudflare CDN (free version). I use an Apache reverse proxy at siteA now to get around the port blocking issue (Sending the traffic over the IPsec to the server at siteB). The ping time is around 230ms and I get around 10Mb up and 45Mb down from siteB to siteA. I spend most of my time in Thailand so the speed I get here is most important.
It had been interesting to learn how to use VTI but I have started to realize it was a bad idea from the beginning. The proxy setup work well but it's slow so I think renting a small VPS in Thailand and have the proxy running there + IPsec so it's local the speed should at least increase 10-fold.
The whole thing from the beginning was port blocking the ISP is doing in Thailand. There is one more ISP in my area but they use CGNAT so it's not an option.
I've seen that many users say nginx is faster and use less resources but in my very small setup I really don't think it matters. And I'm lazy because I would need to learn nginx as well
Thank you everyone for ideas. It got me to see what I need to do
-
@seanr22a said in IPsec routing problem:
I have three web sites including Nextcloud on the server at siteB and they are behind Cloudflare CDN (free version). I use an Apache reverse proxy at siteA now to get around the port blocking issue (Sending the traffic over the IPsec to the server at siteB). The ping time is around 230ms and I get around 10Mb up and 45Mb down from siteB to siteA. I spend most of my time in Thailand so the speed I get here is most important.
I get the Proxy setup, that's what I use to access my NextCloud server, as well as my Homeassistant and some other stuff. I just happen to use Nginx.
But I'm not sure I understand how Cloudflare CDN fits into this setup that you have?
If you host your server at your home in Thailand, and you access it via Sweden using some DynDNS service to find your Swedish IP, then you go directly via the VPN to site B. Where does Cloudflare come into play?And I'm curious, which ISP is it, and which ports do they block? And what ports don't they block?
I've seen that many users say nginx is faster and use less resources but in my very small setup I really don't think it matters.
I agree, probably wouldn't make a noticeable difference if you changed. If you are curious however, and use docker, it's actually super simple to set up and has a very intuitive UI...
BUT, what could potentially improve performance quite a bit is if you change VPN to Wireguard. Depends on what HW you run pfsense on of course, but on smaller machines I can see a real difference even at moderate speeds.
I have a site with pfsense running on a tiny PC Engines APU2 and I can saturate the 250 Mbit connection to that site over Wireguard. But on an IPSec connection I can perhaps get 80-90 Mbit when testing with e.g. iperf or openspeedtest.
