[IPSEC site-to-site] Subnets Connectivity
-
Did that. No luck. Replies are returning to ipsec2000.
Feels like pfsense can not route them back to 172.19.5.219 or operation reverse to NAT doesn't take place.
-
@nomatter
And if you try to do so ?
What will be the result ? -
PING 192.168.179.117 (192.168.179.117) from 172.19.5.254: 56 data bytes --- 192.168.179.117 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss
It's ok only when Source address is VTI
PING 192.168.179.117 (192.168.179.117) from 172.16.17.30: 56 data bytes 64 bytes from 192.168.179.117: icmp_seq=0 ttl=252 time=280.394 ms 64 bytes from 192.168.179.117: icmp_seq=1 ttl=252 time=280.279 ms 64 bytes from 192.168.179.117: icmp_seq=2 ttl=252 time=280.492 ms --- 192.168.179.117 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 280.279/280.388/280.492/0.087 ms
-
@nomatter Your initial tcpdump of ipsec2000 doesn't reveal what the actual IP if, suggest running with
-n
option.
On the surface this looks like a routing problem.
Check the routing table on the C3945 and make sure it has routes to 172.19.1.0/24 and 172.19.5.0/24 via the tunnel. -
Tried to tcpdump with -n option same results:
tcpdump -n -i ipsec2000 | grep 192.168.179.117 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes 18:02:04.472813 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 105, length 64 18:02:04.752956 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 105, length 64 18:02:05.473363 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 106, length 64 18:02:05.753784 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 106, length 64 18:02:06.472929 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 107, length 64 18:02:06.753029 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 107, length 64 18:02:07.472973 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 108, length 64 18:02:07.753102 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 108, length 64 18:02:08.473044 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 109, length 64 ^C66 packets captured 66 packets received by filter 0 packets dropped by kernel
I'll ask guys from other side to add routes. I'm sure they don't have them. But will need to wait until Monday at least.
Can I do something relevant on my side while waiting? -
@nomatter Thanks, that dump is showing the VTI interface pinging the destination host.
The VTI on the subnet /30 shared between both devices, so it is normal that the C3945 knows how to get back to it, but if there are no routes, the C3945 has no way to know that it has to send traffic for 172.19.1.0/24 and 172.19.5.0/24 to 172.16.17.30. -
@awebster
Ah, yes. I missed that on initial dump there is pfsense.localdomain instead of address. You're right.
I asked other side to add 2 routes 172.19.5.0/24 and 172.19.1.0/24 using 172.16.17.30 as Gateway. Will wait.
I'll return to this thread after their reply.
Thank you guys for now -
@awebster
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(Is there any way to work around setting routes on their side?
Actually, I think routes on their side is needed if traffic originates out there.
For example to ping google.com we don't need google to setup routes.
There must be some misconfiguration on my side.
-
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(
Is there any way to work around setting routes on their side?@nomatter, one possible way to work around an overlapping subnet problem is to use 1:1 NAT, but before you do that, ask the other side what subnets that you could present to them that won't cause overlap, the you can setup the pfSense to NAT 172.19.1.x and 172.19.5.x into something that both sides can agree on.
For example 172.19.1.0/24 <=> 10.19.1.0/24 and 172.19.5.0/24 <=> 10.19.5.0/24
On their side they would need to put routes for 10.19.1.0/24 and 10.19.5.0/24
When you ping their system with their address (no change), it looks to them like it is coming from a 10.19.x.x address
When they ping your system, they use 10.19.x.x address and it looks to you like it is coming from their address (no change).Alternatively if you don't want to use 1:1 NAT, depending on how much work is involved, you could rebuild the AWS setup using different subnets.
-
SO if I understand correctly at first 172.19.5.0/24 is translated to 10.19.5.0/24 and then outbound NAT is taking place translating 10.19.5.0/24 to 172.16.17.30 which is 'allowed' to communicate with remote side?
-
@nomatter Not quite... but you do bring up another way that this could be accomplished, albeit in a limited fashion.
If you don't need the ability to reach the AWS instances from the remote side, you can hide NAT everything behind the 172.16.17.30 address, but this will only allow one-way initiated connections, unless you port-forward certain ports inside AWS. Functional, but limited; I wouldn't recommend it.Here's an example of what happens with 1:1 NAT based on your original diagram...
Lets look at what happens when Instance 172.19.5.219 pings instance 192.168.179.119- Ping packet has SRC=172.19.5.219 and DST=192.168.179.119
- Packet arrives at pfSense on 172.19.5.254 and gets NATted, so now looks like this: SRC=10.19.5.219 DST=192.168.179.119.
- The routing table of pfSense says to reach 192.168.179.119 you need to send the packet to 172.16.17.29 via the VTI interface of the IPSec tunnel.
- Packet crosses the IPSec tunnel and arrives at C3945 on its tunnel interface.
- Router looks up packet DST=192.168.179.119, which is an attached network and forwards it to 192.168.179.119.
- The instance 192.168.179.119 receives the ping request, it is from 10.19.5.219, and replies to the packet with ping reply. The reply packet has SRC=192.168.179.119 and DST=10.19.5.219 - so far so good, but we're only half way there.
- Packet arrives at the C3945 router and DST=10.19.5.219 address is looked up in the C3945 routing table which says to reach 10.19.5.0/24 you need to forward it to 172.16.17.30 via the tunnel interface.
- Packet is received by pfSense on 172.16.17.30 and DST address 10.19.5.219 is NATted to 172.19.5.219. The packet looks like this: SRC=192.168.179.119 and DST=172.19.5.219
- Packet is sent out interface 172.19.5.254 to host 172.19.5.219 which sees ping reply packet with SRC=192.168.179.119.
The above routing and NAT does not take into consideration the firewall aspects of this scenario. If the remote side has firewalls, it needs to be aware that the AWS instances are behind 10.19.x.x, not 172.19.x.x.
Similarly, the pfSense needs to be aware that the AWS instances will be accessed using 10.19.x.x addresses.
Maybe someone with more in-depth knowledge of pfSense internals can chime in on whether inbound firewall rules on IPSEC interface are pre-NAT or post-NAT. Its easy enough to test it out with some logging to see which one works properly. -
Thank you for such detailed explanation!
Seems like adding NAT 1:1 is making things more complicated than they should be.
I'll setup VPC with another CIDR block instead.But I can not understand is it really necessary for them to have route to my private subnet 172.19.5.0/24 if stick to Routed Phase 2 mode.
I looked to ICMP replies captured at VTI interface in wireshark and they have dest addr 172.16.17.30Internet Protocol Version 4, Src: 192.168.179.118, Dst: 172.16.17.30
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 84
Identification: 0x305c (12380)
Flags: 0x4000, Don't fragment
Time to live: 252
Protocol: ICMP (1)
Header checksum: 0x12a3 [validation disabled]
[Header checksum status: Unverified]
Source: 192.168.179.118
Destination: 172.16.17.30Shouldn't they have dest address 172.19.5.219?
Looks like from their perspective everything is ok. C3945 receives ICMP requests from 172.16.17.30 and sends replies...
Maybe pfsense doesn't handle those replies properly not redirecting them to original source?Also it's strange for me why I can not ping 172.16.17.30 from 172.19.5.219 which is one of the router interfaces...
-
@nomatter
Freebsd does not always work correctly on a virtual machine. In particular, IP packet checksums are not always correctly calculated by network card drivers . Try to enable this option
/System/Advanced/Networking -
@nomatter said in [IPSEC site-to-site] Subnets Connectivity:
Shouldn't they have dest address 172.19.5.219?
Only if that is where the ping initiated from, but I suspect that there might be some confusion from the configuration of the VPN.
In the IPSEC Phase 2, it is possible to configure it to ping a remote destination to keep the tunnel up. In this case it will ping from 172.16.17.30 to that remote destination, which will be replying, I suspect this is what you are actually seeing. -
Enabled but nothing significant happens.
There was misconfiguration on AWS VPC Routing table layer that's why I couldn't ping 172.16.17.30 which is pfsense's VTI.
Now I'm able to ping 172.16.17.30 from 172.19.5.219Should I be able to ping 172.16.17.29 which is remote tunnel endpoint? 172.16.17.28/30 is directly attached to pfsense so if i understand correctly in normal circumstances I should be able to ping 172.16.17.29 from any host in 172.19.5.0/24
For now I can not.
Is it normal? -
@awebster
I don't have checkmark on "automatically ping host" in Phase 2 settings but nevertheless there is ICMP traffic between 172.16.17.30 and 172.16.17.29 all the time and it's not related to my ICMP requests from host in private subnet.tcpdump -nni ipsec2000 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes >>>07:41:53.453287 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 16, length 64 07:41:53.668029 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13119, length 8 07:41:53.668077 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54233, length 8 >>> 07:41:53.732895 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 16, length 64] 07:41:53.947746 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13119, length 8 07:41:53.947762 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54233, length 8 07:41:54.209324 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54234, length 8 07:41:54.209372 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13120, length 8 >>> 07:41:54.453365 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 17, length 64 07:41:54.488891 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54234, length 8 07:41:54.488906 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13120, length 8 07:41:54.724422 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13121, length 8 07:41:54.724473 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54235, length 8 >>> 07:41:54.733203 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 17, length 64 07:41:55.004501 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13121, length 8 07:41:55.004517 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54235, length 8 07:41:55.265786 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54236, length 8
-
Should I be able to ping 172.16.17.29 which is remote tunnel endpoint?
@nomatter, you will not be able to ping 172.16.17.29 from 172.19.5.0/24 subnet because the C3945 doesn't know how to get back to 172.19.5.0.
-
So I if understand you correctly, you 100% sure that issue is caused by missing routes on the other side?
And there is no possibility that I missed something in configuration on my side? -
@nomatter 100%
-
@awebster
So other side refused to set routes for my private network :) They say that's my private network should be NATed to virtual tunnel endpoint. And it's working like that for all other parties.I've set lab environment with ipsec tunnel between 2 pfSenses.
Of course we need routes on both sides with opposite virtual IPs and endpoints. That's confirmed.
Everything is working between 2 pfsenses until...Turned that there is another issue:
When trying to apply Outbound NAT on VTI interface connectivity between hosts in remote private networks is lost.
And I have exactly the same situation which was discussed previously in this topic:
ICMP replies are getting back to VTI interface but don't going further to source host.Are there any issues or limitations with outbound NAT to VTI in pfsense?