no internet return traffic to ipsec tunnel?
-
Ok, once again. is this policy based or route based IPSec?
Knowing that is absolutely fundamental to making this work.However your states make it look like it's route based with a missing route.
IPsec icmp 192.168.5.103:20690 -> 8.8.8.8:20690 0:0 3 / 342 252 B / 28 KiB WAN icmp 100.100.10.1:24400 (192.168.5.103:20690) -> 8.8.8.8:24400 0:0 6 / 3 588 B / 252 B LAN icmp 8.8.8.8:20690 -> 192.168.5.103:20690 0:0 342 / 0 28 KiB / 0 BYou have pings coming in on the ipsec interface (342) but only 3 replies.
What looks like 6 pings leave the WAN correctly NAT'd but there are only 3 replies.
Most of that traffic seem to be on LAN which shouldn't be there at all. And it looks like all replies from 8.8.8.8 which I don't coming in from anywhere else...
Were the states cleared or allowed to timeout before the test was run?
Also there is:
lo0 icmp 10.3.96.3:43769 -> 10.3.96.3:43769 0:0 693.444 K / 693.444 K 19.18 MiB / 19.18 MiBThat is the LAN IP address pinging itself, which should definitely not be happening. It implies you may have added the LAN IP as a gateway on it's own interface.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Ok, once again. is this policy based or route based IPSec?
It's Route based IPSec.
Knowing that is absolutely fundamental to making this work.
However your states make it look like it's route based with a missing route.
IPsec icmp 192.168.5.103:20690 -> 8.8.8.8:20690 0:0 3 / 342 252 B / 28 KiB
WAN icmp 100.100.10.1:24400 (192.168.5.103:20690) -> 8.8.8.8:24400 0:0 6 / 3 588 B / 252 B
LAN icmp 8.8.8.8:20690 -> 192.168.5.103:20690 0:0 342 / 0 28 KiB / 0 BVery possible, on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN).
You have pings coming in on the ipsec interface (342) but only 3 replies.
What looks like 6 pings leave the WAN correctly NAT'd but there are only 3 replies.
Most of that traffic seem to be on LAN which shouldn't be there at all. And it looks like all replies from 8.8.8.8 which I don't coming in from anywhere else...
Were the states cleared or allowed to timeout before the test was run?Yes, I cleared them before doing the ping -c 3 8.8.8.8 on 192.168.5.103
Also there is:
lo0 icmp 10.3.96.3:43769 -> 10.3.96.3:43769 0:0 693.444 K / 693.444 K 19.18 MiB / 19.18 MiBThat is the LAN IP address pinging itself, which should definitely not be happening. It implies you may have added the LAN IP as a gateway on it's own interface.
I checked my pfSense config -> System/Routing/Gateways and I have two entries...
[Name] -- [Default] -- [Interface] -- [Gateway] -- [Monitor IP] -- [Description]
WANGW -- Default (IPv4) -- WAN -- 100.100.10.1 -- (empty) -- WAN Gateway
LAN -- (empty) -- LAN -- 10.3.96.3 -- 10.3.96.3 -- (empty) -- (empty)under -> Static Routes I have...
[Network] -- [Gateway] -- [Interface] -- [Description]
192.168.5.0/24 -- LAN - 10.3.96.3 -- LAN -- (empty) -
Just for testing, In Static Routes, I disabled that (192.168.5.0/24 - LAN 10.3.96.3 LAN) Route and in Gateways disabled the (LAN, LAN, 10.3.96.3, 10.3.96.3) gateway as well.
Then did pings from 192.168.5.103 --> 8.8.8.8
and this is what I got on the states -->IPsec icmp 192.168.5.103:18673 -> 8.8.8.8:18673 0:0 3 / 3 252 B / 252 B
WAN icmp 100.100.10.1:37645 (192.168.5.103:18673) -> 8.8.8.8:37645 0:0 3 / 3 252 B / 252 B
ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999msStill getting nothing back to the test box... But the lo0 - 10.3.96.3 icmp is gone now.
-
Yeah you should remove the LAN gateway in pfSense unless you are routing traffic to some other subnet you have attached there. And if so the gateway would be whatever internal router there not the LAN IP itself. Also make sure there is no gateway defined on LAN in Interfaces > LAN. That would cause automatic outbound NAT to trigger on all traffic leaving LAN. Not what you want.
@randallc79 said in no internet return traffic to ipsec tunnel?:
on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN)
Why would you have a default route via the LAN? I expect it to be via the WAN.
That does explain why you had the ping traffic incorrectly leaving LAN though.If you are using route based IPSec (VTI) you need to assign the VTI interface and you will that have a VTI gateway. The static route for 192.168.5.0/24 needs to be via that gateway so pfSense sends traffic for it back across the tunnel.
It's surprising you're able to reach anything across the tunnel without that.
See: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/routed-vti.html#ipsec-interface-assignment
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Yeah you should remove the LAN gateway in pfSense unless you are routing traffic to some other subnet you have attached there. And if so the gateway would be whatever internal router there not the LAN IP itself. Also make sure there is no gateway defined on LAN in Interfaces > LAN. That would cause automatic outbound NAT to trigger on all traffic leaving LAN. Not what you want.
yes, I originally disabled them. Now I have completely removed them.
@randallc79 said in no internet return traffic to ipsec tunnel?:
on the remote side 192.168.5.103 should be routed (looking for 8.8.8.8) through the default which is setup as 0.0.0.0/0 --> 10.3.96.3 (which is the pfSense LAN)
Why would you have a default route via the LAN? I expect it to be via the WAN.
That does explain why you had the ping traffic incorrectly leaving LAN though.I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
If you are using route based IPSec (VTI) you need to assign the VTI interface and you will that have a VTI gateway. The static route for 192.168.5.0/24 needs to be via that gateway so pfSense sends traffic for it back across the tunnel.
also went in an added a VTI (P2) on that IPSec tunnel, going to do more reading on it, because now all traffic that was working is now dead. I can't even ping either direction.
It's surprising you're able to reach anything across the tunnel without that.
Yea, not sure either after reading about VTI, but yet now that I have one setup. I'm completely dead in water both directions, so pretty sure I broke something in setting that up.
-
Hmm, are both these sites pfSense?
@randallc79 said in no internet return traffic to ipsec tunnel?:
I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
Are you sure the tunnel is configured as route based? Because without VTI gateways no traffic will be routed across the tunnel. It seems like you may have routed the traffic via the WAN.
Or the tunnel is actually policy based and carrying the LAN traffic between the sites.Either way that route change is wrong. The only thing that should be using the 100.100.10.1 address at the 10.200.10.1 router is the IPSec P1 config. Everything else should be inside the tunnel.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Hmm, are both these sites pfSense?
Only 100.100.10.1 is pfSense. 100.200.10.1 is a paloalto.
@randallc79 said in no internet return traffic to ipsec tunnel?:
I went into the (remote 100.200.10.1) router and switched that from pointing to 10.3.96.3, to now pointing to 100.100.10.1
Are you sure the tunnel is configured as route based? Because without VTI gateways no traffic will be routed across the tunnel. It seems like you may have routed the traffic via the WAN.
Or the tunnel is actually policy based and carrying the LAN traffic between the sites.I believe so, even asked my brother (he's a network admin, he just doesn't know pfsense) who checked my PA and he said it all looked correct for a route based IPSec tunnel.
Either way that route change is wrong. The only thing that should be using the 100.100.10.1 address at the 10.200.10.1 router is the IPSec P1 config. Everything else should be inside the tunnel.
I double checked that and it looks like that P1 is only one with 100.100.10 - 10.200.10.1 ... also same on the other end.
-
Ok, so you will need to assign the VTI interface, add a gateway and add static routes for PA LAN subnet via that gateway in pfSense. The gateway will be the PA side of the VTI transit subnet.
You will need to stop/start the IPSec tunnel after doing that to see the traffic.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
Ok, so you will need to assign the VTI interface, add a gateway and add static routes for PA LAN subnet via that gateway in pfSense. The gateway will be the PA side of the VTI transit subnet.
You will need to stop/start the IPSec tunnel after doing that to see the traffic.
Steve
So would I have to delete the already created P2? (1 tunnel LAN 192.168.5.0/24 ESP)
Only ask, because I've tried multiple way tonight to enable that VTI and each time I break the 'tunnel'. even the PA shows disconnected, so now I'm wondering if this is even possible.
I've been able to get traffic between the two networks to flow. I've been able to get traffic from the PA side to go out the pfSense side. I'm just stuck on how to get it back to the machine that sent it (ie pings etc).
-
You previously said you were using route based IPSec (VTI) and if that is the case you just need to assign it.
If that isn't what you have and your actually using policy based (tunnel mode) then you need toi make sure the policy matches the traffic you are trying to pass.Route based will probably work better for you here though so, yes, if you don't have a VTI P2 I would remove what you have now and add one. The PA has to be setup to match that though.
Can we see a screenshot of your current working config?
It wasn't working before because the default route was via the LAN. Now that has been removed try re-running the ping test and check the states again.
Steve
-
@stephenw10 said in no internet return traffic to ipsec tunnel?:
You previously said you were using route based IPSec (VTI) and if that is the case you just need to assign it.
If that isn't what you have and your actually using policy based (tunnel mode) then you need toi make sure the policy matches the traffic you are trying to pass.Route based will probably work better for you here though so, yes, if you don't have a VTI P2 I would remove what you have now and add one. The PA has to be setup to match that though.
Can we see a screenshot of your current working config?
It wasn't working before because the default route was via the LAN. Now that has been removed try re-running the ping test and check the states again.
Steve
-
Ok, so that's policy based IPSec (tunnel mode) at the pfSense end.
I'm not familiar enough with PA to know if that screen confirms route vase there. It does appear to have tunnel interfaces which implies it might.The P2 policy you have configured there is only carrying traffic between the LAN subnet(10.3.93.X) and 192.168.5.0/24. Which means it isn't carrying traffic between 192.168.5.102 and 8.8.8.8 for example.
I would confirm the PA is using route based IPSec and then switch pfSense to match. That way you can route whatever traffic you want across the tunnel.
Otherwise you have to do this: https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-route-internet-traffic.htmlAnd that can be inconvenient because it often over-matches and pulls traffic over the tunnel you do not want to be.
Steve
