no internet return traffic to ipsec tunnel?
-
Just to start, yes, I have searched this forum and google this issue to death. Not sure what else to do to get this to work, but my goal is to have an IPSec Tunnel from my home router to a pfSense box on a remote server and to have the subnet that is configured on home router to send traffic out the remote pfSense box's internet. To sum it up, below attaching image to show how I currently have it setup.
The IPSec Tunnel between 100.200.10.1 (home site) and 100.100.10.1 (remote site) is up and good.
I have a routes setup so luxtest0 can ping luxtest1 and luxtest2.
Also luxtest1 and luxtest2 can ping luxtest0 and pfsense (10.3.96.3)
luxtest1 and luxtest2 are both isolated from the internet on 100.200.10.1 so both box currently show no internet connection.
luxtest1 and luxtest2 only have access to the 192.168.5.x IPSec Tunnel.
On luxtest2, I have nginx setup and luxtest0 can curl the small website on luxtest2.
On pfSense, I have HAProxy setup that even NAT's port 8080 from 100.100.10.1 (wan - pfsense) to luxtest2 port 80. which is accessible via -> http://100.100.10.1:8080 (website will come up and work.
Now for the problem / issue. While I can do all above items, when I try any sort of internet connect from luxtest1 or luxtest2 to the internet via the remote site (100.100.10.1), I see outbound traffic on the pfSense box, but I get nothing back. The pings time out and curl errors out.
My outbound traffic for example looks like -> (from luxtest2) ping 8.8.8.8 on pfsense (status/system logs/firewall/normal view) Source (100.100.10.1) Destination (8.8.8.8) ICMP (green - pass), then the (luxtest2) ping times out.
-
@randallc79 -- adding a highlighted visual as well...
Traffic works : Green, Blue & Orange
Traffic fails: red (goes out, but no return)
-
@randallc79
This will only work, if you push the whole upstream traffic from the router to pfSense by setting the remote network in its phase 2 to 0.0.0.0/0 or even with a routed IPSec setup.If that is not desired or not possible use OpenVPN.
-
Yup, is this policy based or route based IPSec?
Given that you see states on pfSense when testlux2 tries to ping 8.8.8.8 though the tunnel musy be carrying that traffic.
What states do you actually see?You should see a state on ipsec (or the assigned vti interface if it's route based).
And a state on WAN with NAT from 192.168.5.X to 100.100.10.1.Both states should show traffic in both directions on the packet counters.
Steve
-
@viragomann said in no internet return traffic to ipsec tunnel?:
@randallc79
This will only work, if you push the whole upstream traffic from the router to pfSense by setting the remote network in its phase 2 to 0.0.0.0/0 or even with a routed IPSec setup.If that is not desired or not possible use OpenVPN.
Yes, on the remote (100.200.10.1) router there is a default route for 0.0.0.0/0 to goto (ipsec tunnel) 10.3.96.3
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Yup, is this policy based or route based IPSec?
Given that you see states on pfSense when testlux2 tries to ping 8.8.8.8 though the tunnel musy be carrying that traffic.
What states do you actually see?You should see a state on ipsec (or the assigned vti interface if it's route based).
And a state on WAN with NAT from 192.168.5.X to 100.100.10.1.Both states should show traffic in both directions on the packet counters.
Steve
I did a simple ping from 192.168.5.103 (luxtest2) -> 8.8.8.8 --- this is what the state looked like. (pasted below... looks horrible so attaching pic as well)
States
Inferface Protocol Source (Orignal Source) -> Destination (Orignal Destination) State Packets Bytes
WAN udp 100.200.10.1:500 -> 100.100.10.1:500 MULTIPLE:MULTIPLE 416.907 K / 416.903 K 49.48 MiB / 49.48 MiB
lo0 icmp 10.3.96.3:43769 -> 10.3.96.3:43769 0:0 693.444 K / 693.444 K 19.18 MiB / 19.18 MiB
WAN tcp 100.200.10.1:51837 -> 100.100.10.1:8443 ESTABLISHED:ESTABLISHED 1.826 K / 8.698 K 128 KiB / 10.93 MiB
WAN esp 100.200.10.1 -> 100.100.10.1 NO_TRAFFIC:SINGLE 3 / 0 420 B / 0 B
IPsec icmp 192.168.5.103:20690 -> 8.8.8.8:20690 0:0 3 / 342 252 B / 28 KiB
WAN icmp 100.100.10.1:24400 (192.168.5.103:20690) -> 8.8.8.8:24400 0:0 6 / 3 588 B / 252 B
LAN icmp 8.8.8.8:20690 -> 192.168.5.103:20690 0:0 342 / 0 28 KiB / 0 B
-
Ok, once again. is this policy based or route based IPSec?
Knowing that is absolutely fundamental to making this work.However your states make it look like it's route based with a missing route.
IPsec icmp 192.168.5.103:20690 -> 8.8.8.8:20690 0:0 3 / 342 252 B / 28 KiB WAN icmp 100.100.10.1:24400 (192.168.5.103:20690) -> 8.8.8.8:24400 0:0 6 / 3 588 B / 252 B LAN icmp 8.8.8.8:20690 -> 192.168.5.103:20690 0:0 342 / 0 28 KiB / 0 BYou have pings coming in on the ipsec interface (342) but only 3 replies.
What looks like 6 pings leave the WAN correctly NAT'd but there are only 3 replies.
Most of that traffic seem to be on LAN which shouldn't be there at all. And it looks like all replies from 8.8.8.8 which I don't coming in from anywhere else...
Were the states cleared or allowed to timeout before the test was run?
Also there is:
lo0 icmp 10.3.96.3:43769 -> 10.3.96.3:43769 0:0 693.444 K / 693.444 K 19.18 MiB / 19.18 MiBThat is the LAN IP address pinging itself, which should definitely not be happening. It implies you may have added the LAN IP as a gateway on it's own interface.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Ok, once again. is this policy based or route based IPSec?
It's Route based IPSec.
Knowing that is absolutely fundamental to making this work.
However your states make it look like it's route based with a missing route.
IPsec icmp 192.168.5.103:20690 -> 8.8.8.8:20690 0:0 3 / 342 252 B / 28 KiB
WAN icmp 100.100.10.1:24400 (192.168.5.103:20690) -> 8.8.8.8:24400 0:0 6 / 3 588 B / 252 B
LAN icmp 8.8.8.8:20690 -> 192.168.5.103:20690 0:0 342 / 0 28 KiB / 0 BVery possible, on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN).
You have pings coming in on the ipsec interface (342) but only 3 replies.
What looks like 6 pings leave the WAN correctly NAT'd but there are only 3 replies.
Most of that traffic seem to be on LAN which shouldn't be there at all. And it looks like all replies from 8.8.8.8 which I don't coming in from anywhere else...
Were the states cleared or allowed to timeout before the test was run?Yes, I cleared them before doing the ping -c 3 8.8.8.8 on 192.168.5.103
Also there is:
lo0 icmp 10.3.96.3:43769 -> 10.3.96.3:43769 0:0 693.444 K / 693.444 K 19.18 MiB / 19.18 MiBThat is the LAN IP address pinging itself, which should definitely not be happening. It implies you may have added the LAN IP as a gateway on it's own interface.
I checked my pfSense config -> System/Routing/Gateways and I have two entries...
[Name] -- [Default] -- [Interface] -- [Gateway] -- [Monitor IP] -- [Description]
WANGW -- Default (IPv4) -- WAN -- 100.100.10.1 -- (empty) -- WAN Gateway
LAN -- (empty) -- LAN -- 10.3.96.3 -- 10.3.96.3 -- (empty) -- (empty)under -> Static Routes I have...
[Network] -- [Gateway] -- [Interface] -- [Description]
192.168.5.0/24 -- LAN - 10.3.96.3 -- LAN -- (empty) -
Just for testing, In Static Routes, I disabled that (192.168.5.0/24 - LAN 10.3.96.3 LAN) Route and in Gateways disabled the (LAN, LAN, 10.3.96.3, 10.3.96.3) gateway as well.
Then did pings from 192.168.5.103 --> 8.8.8.8
and this is what I got on the states -->IPsec icmp 192.168.5.103:18673 -> 8.8.8.8:18673 0:0 3 / 3 252 B / 252 B
WAN icmp 100.100.10.1:37645 (192.168.5.103:18673) -> 8.8.8.8:37645 0:0 3 / 3 252 B / 252 B
ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999msStill getting nothing back to the test box... But the lo0 - 10.3.96.3 icmp is gone now.
-
Yeah you should remove the LAN gateway in pfSense unless you are routing traffic to some other subnet you have attached there. And if so the gateway would be whatever internal router there not the LAN IP itself. Also make sure there is no gateway defined on LAN in Interfaces > LAN. That would cause automatic outbound NAT to trigger on all traffic leaving LAN. Not what you want.
@randallc79 said in no internet return traffic to ipsec tunnel?:
on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN)
Why would you have a default route via the LAN? I expect it to be via the WAN.
That does explain why you had the ping traffic incorrectly leaving LAN though.If you are using route based IPSec (VTI) you need to assign the VTI interface and you will that have a VTI gateway. The static route for 192.168.5.0/24 needs to be via that gateway so pfSense sends traffic for it back across the tunnel.
It's surprising you're able to reach anything across the tunnel without that.
See: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/routed-vti.html#ipsec-interface-assignment
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Yeah you should remove the LAN gateway in pfSense unless you are routing traffic to some other subnet you have attached there. And if so the gateway would be whatever internal router there not the LAN IP itself. Also make sure there is no gateway defined on LAN in Interfaces > LAN. That would cause automatic outbound NAT to trigger on all traffic leaving LAN. Not what you want.
yes, I originally disabled them. Now I have completely removed them.
@randallc79 said in no internet return traffic to ipsec tunnel?:
on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN)
Why would you have a default route via the LAN? I expect it to be via the WAN.
That does explain why you had the ping traffic incorrectly leaving LAN though.I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
If you are using route based IPSec (VTI) you need to assign the VTI interface and you will that have a VTI gateway. The static route for 192.168.5.0/24 needs to be via that gateway so pfSense sends traffic for it back across the tunnel.
also went in an added a VTI (P2) on that IPSec tunnel, going to do more reading on it, because now all traffic that was working is now dead. I can't even ping either direction.
It's surprising you're able to reach anything across the tunnel without that.
Yea, not sure either after reading about VTI, but yet now that I have one setup. I'm completely dead in water both directions, so pretty sure I broke something in setting that up.
-
Hmm, are both these sites pfSense?
@randallc79 said in no internet return traffic to ipsec tunnel?:
I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
Are you sure the tunnel is configured as route based? Because without VTI gateways no traffic will be routed across the tunnel. It seems like you may have routed the traffic via the WAN.
Or the tunnel is actually policy based and carrying the LAN traffic between the sites.Either way that route change is wrong. The only thing that should be using the 100.100.10.1 address at the 10.200.10.1 router is the IPSec P1 config. Everything else should be inside the tunnel.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Hmm, are both these sites pfSense?
Only 100.100.10.1 is pfSense. 100.200.10.1 is a paloalto.
@randallc79 said in no internet return traffic to ipsec tunnel?:
I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
Are you sure the tunnel is configured as route based? Because without VTI gateways no traffic will be routed across the tunnel. It seems like you may have routed the traffic via the WAN.
Or the tunnel is actually policy based and carrying the LAN traffic between the sites.I believe so, even asked my brother (he's a network admin, he just doesn't know pfsense) who checked my PA and he said it all looked correct for a route based IPSec tunnel.
Either way that route change is wrong. The only thing that should be using the 100.100.10.1 address at the 10.200.10.1 router is the IPSec P1 config. Everything else should be inside the tunnel.
I double checked that and it looks like that P1 is only one with 100.100.10 - 10.200.10.1 ... also same on the other end.
-
Ok, so you will need to assign the VTI interface, add a gateway and add static routes for PA LAN subnet via that gateway in pfSense. The gateway will be the PA side of the VTI transit subnet.
You will need to stop/start the IPSec tunnel after doing that to see the traffic.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Ok, so you will need to assign the VTI interface, add a gateway and add static routes for PA LAN subnet via that gateway in pfSense. The gateway will be the PA side of the VTI transit subnet.
You will need to stop/start the IPSec tunnel after doing that to see the traffic.
Steve
So would I have to delete the already created P2? (1 tunnel LAN 192.168.5.0/24 ESP)
Only ask, because I've tried multiple way tonight to enable that VTI and each time I break the 'tunnel'. even the PA shows disconnected, so now I'm wondering if this is even possible.
I've been able to get traffic between the two networks to flow. I've been able to get traffic from the PA side to go out the pfSense side. I'm just stuck on how to get it back to the machine that sent it (ie pings etc).
-
You previously said you were using route based IPSec (VTI) and if that is the case you just need to assign it.
If that isn't what you have and your actually using policy based (tunnel mode) then you need toi make sure the policy matches the traffic you are trying to pass.Route based will probably work better for you here though so, yes, if you don't have a VTI P2 I would remove what you have now and add one. The PA has to be setup to match that though.
Can we see a screenshot of your current working config?
It wasn't working before because the default route was via the LAN. Now that has been removed try re-running the ping test and check the states again.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
You previously said you were using route based IPSec (VTI) and if that is the case you just need to assign it.
If that isn't what you have and your actually using policy based (tunnel mode) then you need toi make sure the policy matches the traffic you are trying to pass.Route based will probably work better for you here though so, yes, if you don't have a VTI P2 I would remove what you have now and add one. The PA has to be setup to match that though.
Can we see a screenshot of your current working config?
It wasn't working before because the default route was via the LAN. Now that has been removed try re-running the ping test and check the states again.
Steve
-
Ok, so that's policy based IPSec (tunnel mode) at the pfSense end.
I'm not familiar enough with PA to know if that screen confirms route vase there. It does appear to have tunnel interfaces which implies it might.The P2 policy you have configured there is only carrying traffic between the LAN subnet(10.3.93.X) and 192.168.5.0/24. Which means it isn't carrying traffic between 192.168.5.102 and 8.8.8.8 for example.
I would confirm the PA is using route based IPSec and then switch pfSense to match. That way you can route whatever traffic you want across the tunnel.
Otherwise you have to do this: https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-route-internet-traffic.htmlAnd that can be inconvenient because it often over-matches and pulls traffic over the tunnel you do not want to be.
Steve
