[IPSEC site-to-site] Subnets Connectivity
-
@nomatter Not quite... but you do bring up another way that this could be accomplished, albeit in a limited fashion.
If you don't need the ability to reach the AWS instances from the remote side, you can hide NAT everything behind the 172.16.17.30 address, but this will only allow one-way initiated connections, unless you port-forward certain ports inside AWS. Functional, but limited; I wouldn't recommend it.Here's an example of what happens with 1:1 NAT based on your original diagram...
Lets look at what happens when Instance 172.19.5.219 pings instance 192.168.179.119- Ping packet has SRC=172.19.5.219 and DST=192.168.179.119
- Packet arrives at pfSense on 172.19.5.254 and gets NATted, so now looks like this: SRC=10.19.5.219 DST=192.168.179.119.
- The routing table of pfSense says to reach 192.168.179.119 you need to send the packet to 172.16.17.29 via the VTI interface of the IPSec tunnel.
- Packet crosses the IPSec tunnel and arrives at C3945 on its tunnel interface.
- Router looks up packet DST=192.168.179.119, which is an attached network and forwards it to 192.168.179.119.
- The instance 192.168.179.119 receives the ping request, it is from 10.19.5.219, and replies to the packet with ping reply. The reply packet has SRC=192.168.179.119 and DST=10.19.5.219 - so far so good, but we're only half way there.
- Packet arrives at the C3945 router and DST=10.19.5.219 address is looked up in the C3945 routing table which says to reach 10.19.5.0/24 you need to forward it to 172.16.17.30 via the tunnel interface.
- Packet is received by pfSense on 172.16.17.30 and DST address 10.19.5.219 is NATted to 172.19.5.219. The packet looks like this: SRC=192.168.179.119 and DST=172.19.5.219
- Packet is sent out interface 172.19.5.254 to host 172.19.5.219 which sees ping reply packet with SRC=192.168.179.119.
The above routing and NAT does not take into consideration the firewall aspects of this scenario. If the remote side has firewalls, it needs to be aware that the AWS instances are behind 10.19.x.x, not 172.19.x.x.
Similarly, the pfSense needs to be aware that the AWS instances will be accessed using 10.19.x.x addresses.
Maybe someone with more in-depth knowledge of pfSense internals can chime in on whether inbound firewall rules on IPSEC interface are pre-NAT or post-NAT. Its easy enough to test it out with some logging to see which one works properly. -
Thank you for such detailed explanation!
Seems like adding NAT 1:1 is making things more complicated than they should be.
I'll setup VPC with another CIDR block instead.But I can not understand is it really necessary for them to have route to my private subnet 172.19.5.0/24 if stick to Routed Phase 2 mode.
I looked to ICMP replies captured at VTI interface in wireshark and they have dest addr 172.16.17.30Internet Protocol Version 4, Src: 192.168.179.118, Dst: 172.16.17.30
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 84
Identification: 0x305c (12380)
Flags: 0x4000, Don't fragment
Time to live: 252
Protocol: ICMP (1)
Header checksum: 0x12a3 [validation disabled]
[Header checksum status: Unverified]
Source: 192.168.179.118
Destination: 172.16.17.30Shouldn't they have dest address 172.19.5.219?
Looks like from their perspective everything is ok. C3945 receives ICMP requests from 172.16.17.30 and sends replies...
Maybe pfsense doesn't handle those replies properly not redirecting them to original source?Also it's strange for me why I can not ping 172.16.17.30 from 172.19.5.219 which is one of the router interfaces...
-
@nomatter
Freebsd does not always work correctly on a virtual machine. In particular, IP packet checksums are not always correctly calculated by network card drivers . Try to enable this option
/System/Advanced/Networking -
@nomatter said in [IPSEC site-to-site] Subnets Connectivity:
Shouldn't they have dest address 172.19.5.219?
Only if that is where the ping initiated from, but I suspect that there might be some confusion from the configuration of the VPN.
In the IPSEC Phase 2, it is possible to configure it to ping a remote destination to keep the tunnel up. In this case it will ping from 172.16.17.30 to that remote destination, which will be replying, I suspect this is what you are actually seeing. -
Enabled but nothing significant happens.
There was misconfiguration on AWS VPC Routing table layer that's why I couldn't ping 172.16.17.30 which is pfsense's VTI.
Now I'm able to ping 172.16.17.30 from 172.19.5.219Should I be able to ping 172.16.17.29 which is remote tunnel endpoint? 172.16.17.28/30 is directly attached to pfsense so if i understand correctly in normal circumstances I should be able to ping 172.16.17.29 from any host in 172.19.5.0/24
For now I can not.
Is it normal? -
@awebster
I don't have checkmark on "automatically ping host" in Phase 2 settings but nevertheless there is ICMP traffic between 172.16.17.30 and 172.16.17.29 all the time and it's not related to my ICMP requests from host in private subnet.tcpdump -nni ipsec2000 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes >>>07:41:53.453287 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 16, length 64 07:41:53.668029 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13119, length 8 07:41:53.668077 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54233, length 8 >>> 07:41:53.732895 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 16, length 64] 07:41:53.947746 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13119, length 8 07:41:53.947762 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54233, length 8 07:41:54.209324 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54234, length 8 07:41:54.209372 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13120, length 8 >>> 07:41:54.453365 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 17, length 64 07:41:54.488891 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54234, length 8 07:41:54.488906 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13120, length 8 07:41:54.724422 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13121, length 8 07:41:54.724473 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54235, length 8 >>> 07:41:54.733203 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 17, length 64 07:41:55.004501 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13121, length 8 07:41:55.004517 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54235, length 8 07:41:55.265786 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54236, length 8
-
Should I be able to ping 172.16.17.29 which is remote tunnel endpoint?
@nomatter, you will not be able to ping 172.16.17.29 from 172.19.5.0/24 subnet because the C3945 doesn't know how to get back to 172.19.5.0.
-
So I if understand you correctly, you 100% sure that issue is caused by missing routes on the other side?
And there is no possibility that I missed something in configuration on my side? -
@nomatter 100%
-
@awebster
So other side refused to set routes for my private network :) They say that's my private network should be NATed to virtual tunnel endpoint. And it's working like that for all other parties.I've set lab environment with ipsec tunnel between 2 pfSenses.
Of course we need routes on both sides with opposite virtual IPs and endpoints. That's confirmed.
Everything is working between 2 pfsenses until...Turned that there is another issue:
When trying to apply Outbound NAT on VTI interface connectivity between hosts in remote private networks is lost.
And I have exactly the same situation which was discussed previously in this topic:
ICMP replies are getting back to VTI interface but don't going further to source host.Are there any issues or limitations with outbound NAT to VTI in pfsense?
-
Summary:
- For unknown reason src-nat doesn't work properly with VTI. Observed behavior is described above.
- Routed VTI can be used and everything is ok without using NAT to tunnel interface. Setup is simple. Netgate even has video on this.
- My issue was solved using software (infrastructure is in AWS) from another vendor.
-
@nomatter Thanks for the followup.
I've not experimented with VTI source NAT before, and I'm surprised to see that it doesn't in fact work.