[IPSEC site-to-site] Subnets Connectivity
-
Recreated IpSec - same results.
Here are Firewall Rules
#System aliases loopback = "{ lo0 }" WAN = "{ xn0 }" IPSEC = "{ ipsec2000 }" LAN = "{ xn1 }" IPsec = "{ enc0 }" OpenVPN = "{ openvpn }" # Outbound NAT rules (manual) nat on $IPSEC inet from 172.19.5.0/24 to any port 500 -> 172.16.17.30/32 static-port nat on $IPSEC inet from 172.19.5.0/24 to any -> 172.16.17.30/32 port 1024:65535 nat on $WAN inet from 127.0.0.0/8 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - localhost to WAN nat on $WAN inet from 127.0.0.0/8 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - localhost to WAN nat on $WAN inet6 from ::1/128 to any port 500 -> (xn0) static-port # Auto created rule for ISAKMP - localhost to WAN nat on $WAN inet6 from ::1/128 to any -> (xn0) port 1024:65535 # Auto created rule - localhost to WAN nat on $WAN inet from 172.25.53.0/24 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - IPsec client to WAN nat on $WAN inet from 172.25.53.0/24 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - IPsec client to WAN nat on $WAN inet from 172.16.17.29 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - IPsec VTI: SX site-to-site to WAN nat on $WAN inet from 172.16.17.29 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - IPsec VTI: SX site-to-site to WAN # Outbound NAT rules (automatic) # Subnets to NAT table <tonatsubnets> { 127.0.0.0/8 ::1/128 172.19.5.0/24 172.25.53.0/24 172.16.17.29 } nat on $WAN inet from <tonatsubnets> to any port 500 -> 172.19.1.148/32 static-port nat on $WAN inet6 from <tonatsubnets> to any port 500 -> (xn0) static-port nat on $WAN inet from <tonatsubnets> to any -> 172.19.1.148/32 port 1024:65535 nat on $WAN inet6 from <tonatsubnets> to any -> (xn0) port 1024:65535 anchor "userrules/*" pass in quick on $IPsec inet proto icmp from any to any tracker 1574329796 keep state label "USER_RULE" pass in quick on $IPsec inet proto tcp from any to any tracker 1574273174 flags S/SA keep state label "USER_RULE" pass in quick on $IPsec inet from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" pass in quick on $IPsec inet6 from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto udp from any to any port 4500 tracker 1565015851 keep state label "USER_RULE" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto udp from any to any port 500 tracker 1565015834 keep state label "USER_RULE" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto esp from any to any tracker 1565015776 keep state label "USER_RULE: IPsec" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto icmp from any to 172.19.1.148 tracker 1565007607 keep state label "USER_RULE: Default ICMP rule" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 22 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default SSH rule _replace_src_with_mgmtnet_" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 443 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default HTTPS rule _replace_src_with_mgmtnet_" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 80 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default HTTP rule _replace_src_with_mgmtnet_" pass in log quick on $LAN inet from 172.16.17.28/30 to 172.19.5.0/24 tracker 1574333773 keep state label "USER_RULE" pass in log quick on $LAN inet from 172.19.5.0/24 to any tracker 1574331909 keep state label "USER_RULE" # VPN Rules pass out route-to ( xn0 172.19.1.1 ) proto udp from (self) to [REMOTE_IP] port = 500 tracker 1000106361 keep state label "IPsec: SX site-to-site - outbound isakmp" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto udp from [REMOTE_IP] to (self) port = 500 tracker 1000106362 keep state label "IPsec: SX site-to-site - inbound isakmp" pass out route-to ( xn0 172.19.1.1 ) proto udp from (self) to [REMOTE_IP] port = 4500 tracker 1000106363 keep state label "IPsec: SX site-to-site - outbound nat-t" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto udp from [REMOTE_IP] to (self) port = 4500 tracker 1000106364 keep state label "IPsec: SX site-to-site - inbound nat-t" pass out route-to ( xn0 172.19.1.1 ) proto esp from (self) to [REMOTE_IP] tracker 1000106365 keep state label "IPsec: SX site-to-site - outbound esp proto" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto esp from [REMOTE_IP] to (self) tracker 1000106366 keep state label "IPsec: SX site-to-site - inbound esp proto"Hope I showed all relevant rules.
-
@nomatter said in
And if to refuse NAT OUTBOUND for a network 172.19.5.0 / 24 on the IPSEC interface and to add a static route on CISCO for this network ?
Do you have firewall rules that are not quite necessary
pass in log quick on $LAN inet from 172.16.17.28/30 to 172.19.5.0/24 tracker 1574333773 keep state label "USER_RULE"
pass in log quick on $LAN inet from 172.19.5.0/24 to any tracker 1574331909 keep state label "USER_RULE"pass in quick on $IPsec inet proto icmp from any to any tracker 1574329796 keep state label "USER_RULE"
pass in quick on $IPsec inet proto tcp from any to any tracker 1574273174 flags S/SA keep state label "USER_RULE"
pass in quick on $IPsec inet from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" -
Yes, I know they are not neccessary. I've added them to track packets in mid of troubleshooting and forget to remove.
Unfortunately I have no access to the remote side. I can just ask their network guy to add it.
Without NAT there are no ICMP replies even to ipsec2000. They recommended to setup NAT so I believe they don't have any config for my private network.
Actually I thought if ICMP replies are coming to pfsense than there is some misconfiguration on my side.
-
Try to move the NAT OUTBOUND to manual mode
-
Did that. No luck. Replies are returning to ipsec2000.
Feels like pfsense can not route them back to 172.19.5.219 or operation reverse to NAT doesn't take place.
-
@nomatter
And if you try to do so ?
What will be the result ?
-
PING 192.168.179.117 (192.168.179.117) from 172.19.5.254: 56 data bytes --- 192.168.179.117 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossIt's ok only when Source address is VTI
PING 192.168.179.117 (192.168.179.117) from 172.16.17.30: 56 data bytes 64 bytes from 192.168.179.117: icmp_seq=0 ttl=252 time=280.394 ms 64 bytes from 192.168.179.117: icmp_seq=1 ttl=252 time=280.279 ms 64 bytes from 192.168.179.117: icmp_seq=2 ttl=252 time=280.492 ms --- 192.168.179.117 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 280.279/280.388/280.492/0.087 ms -
@nomatter Your initial tcpdump of ipsec2000 doesn't reveal what the actual IP if, suggest running with
-noption.
On the surface this looks like a routing problem.
Check the routing table on the C3945 and make sure it has routes to 172.19.1.0/24 and 172.19.5.0/24 via the tunnel. -
Tried to tcpdump with -n option same results:
tcpdump -n -i ipsec2000 | grep 192.168.179.117 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes 18:02:04.472813 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 105, length 64 18:02:04.752956 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 105, length 64 18:02:05.473363 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 106, length 64 18:02:05.753784 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 106, length 64 18:02:06.472929 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 107, length 64 18:02:06.753029 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 107, length 64 18:02:07.472973 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 108, length 64 18:02:07.753102 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 108, length 64 18:02:08.473044 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 109, length 64 ^C66 packets captured 66 packets received by filter 0 packets dropped by kernelI'll ask guys from other side to add routes. I'm sure they don't have them. But will need to wait until Monday at least.
Can I do something relevant on my side while waiting? -
@nomatter Thanks, that dump is showing the VTI interface pinging the destination host.
The VTI on the subnet /30 shared between both devices, so it is normal that the C3945 knows how to get back to it, but if there are no routes, the C3945 has no way to know that it has to send traffic for 172.19.1.0/24 and 172.19.5.0/24 to 172.16.17.30. -
@awebster
Ah, yes. I missed that on initial dump there is pfsense.localdomain instead of address. You're right.
I asked other side to add 2 routes 172.19.5.0/24 and 172.19.1.0/24 using 172.16.17.30 as Gateway. Will wait.
I'll return to this thread after their reply.
Thank you guys for now -
@awebster
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(Is there any way to work around setting routes on their side?
Actually, I think routes on their side is needed if traffic originates out there.
For example to ping google.com we don't need google to setup routes.
There must be some misconfiguration on my side.
-
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(
Is there any way to work around setting routes on their side?@nomatter, one possible way to work around an overlapping subnet problem is to use 1:1 NAT, but before you do that, ask the other side what subnets that you could present to them that won't cause overlap, the you can setup the pfSense to NAT 172.19.1.x and 172.19.5.x into something that both sides can agree on.
For example 172.19.1.0/24 <=> 10.19.1.0/24 and 172.19.5.0/24 <=> 10.19.5.0/24
On their side they would need to put routes for 10.19.1.0/24 and 10.19.5.0/24
When you ping their system with their address (no change), it looks to them like it is coming from a 10.19.x.x address
When they ping your system, they use 10.19.x.x address and it looks to you like it is coming from their address (no change).Alternatively if you don't want to use 1:1 NAT, depending on how much work is involved, you could rebuild the AWS setup using different subnets.
-
SO if I understand correctly at first 172.19.5.0/24 is translated to 10.19.5.0/24 and then outbound NAT is taking place translating 10.19.5.0/24 to 172.16.17.30 which is 'allowed' to communicate with remote side?
-
@nomatter Not quite... but you do bring up another way that this could be accomplished, albeit in a limited fashion.
If you don't need the ability to reach the AWS instances from the remote side, you can hide NAT everything behind the 172.16.17.30 address, but this will only allow one-way initiated connections, unless you port-forward certain ports inside AWS. Functional, but limited; I wouldn't recommend it.Here's an example of what happens with 1:1 NAT based on your original diagram...
Lets look at what happens when Instance 172.19.5.219 pings instance 192.168.179.119- Ping packet has SRC=172.19.5.219 and DST=192.168.179.119
- Packet arrives at pfSense on 172.19.5.254 and gets NATted, so now looks like this: SRC=10.19.5.219 DST=192.168.179.119.
- The routing table of pfSense says to reach 192.168.179.119 you need to send the packet to 172.16.17.29 via the VTI interface of the IPSec tunnel.
- Packet crosses the IPSec tunnel and arrives at C3945 on its tunnel interface.
- Router looks up packet DST=192.168.179.119, which is an attached network and forwards it to 192.168.179.119.
- The instance 192.168.179.119 receives the ping request, it is from 10.19.5.219, and replies to the packet with ping reply. The reply packet has SRC=192.168.179.119 and DST=10.19.5.219 - so far so good, but we're only half way there.
- Packet arrives at the C3945 router and DST=10.19.5.219 address is looked up in the C3945 routing table which says to reach 10.19.5.0/24 you need to forward it to 172.16.17.30 via the tunnel interface.
- Packet is received by pfSense on 172.16.17.30 and DST address 10.19.5.219 is NATted to 172.19.5.219. The packet looks like this: SRC=192.168.179.119 and DST=172.19.5.219
- Packet is sent out interface 172.19.5.254 to host 172.19.5.219 which sees ping reply packet with SRC=192.168.179.119.
The above routing and NAT does not take into consideration the firewall aspects of this scenario. If the remote side has firewalls, it needs to be aware that the AWS instances are behind 10.19.x.x, not 172.19.x.x.
Similarly, the pfSense needs to be aware that the AWS instances will be accessed using 10.19.x.x addresses.
Maybe someone with more in-depth knowledge of pfSense internals can chime in on whether inbound firewall rules on IPSEC interface are pre-NAT or post-NAT. Its easy enough to test it out with some logging to see which one works properly. -
Thank you for such detailed explanation!
Seems like adding NAT 1:1 is making things more complicated than they should be.
I'll setup VPC with another CIDR block instead.But I can not understand is it really necessary for them to have route to my private subnet 172.19.5.0/24 if stick to Routed Phase 2 mode.
I looked to ICMP replies captured at VTI interface in wireshark and they have dest addr 172.16.17.30Internet Protocol Version 4, Src: 192.168.179.118, Dst: 172.16.17.30
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 84
Identification: 0x305c (12380)
Flags: 0x4000, Don't fragment
Time to live: 252
Protocol: ICMP (1)
Header checksum: 0x12a3 [validation disabled]
[Header checksum status: Unverified]
Source: 192.168.179.118
Destination: 172.16.17.30Shouldn't they have dest address 172.19.5.219?
Looks like from their perspective everything is ok. C3945 receives ICMP requests from 172.16.17.30 and sends replies...
Maybe pfsense doesn't handle those replies properly not redirecting them to original source?Also it's strange for me why I can not ping 172.16.17.30 from 172.19.5.219 which is one of the router interfaces...
-
@nomatter
Freebsd does not always work correctly on a virtual machine. In particular, IP packet checksums are not always correctly calculated by network card drivers . Try to enable this option
/System/Advanced/Networking
-
@nomatter said in [IPSEC site-to-site] Subnets Connectivity:
Shouldn't they have dest address 172.19.5.219?
Only if that is where the ping initiated from, but I suspect that there might be some confusion from the configuration of the VPN.
In the IPSEC Phase 2, it is possible to configure it to ping a remote destination to keep the tunnel up. In this case it will ping from 172.16.17.30 to that remote destination, which will be replying, I suspect this is what you are actually seeing. -
Enabled but nothing significant happens.
There was misconfiguration on AWS VPC Routing table layer that's why I couldn't ping 172.16.17.30 which is pfsense's VTI.
Now I'm able to ping 172.16.17.30 from 172.19.5.219Should I be able to ping 172.16.17.29 which is remote tunnel endpoint? 172.16.17.28/30 is directly attached to pfsense so if i understand correctly in normal circumstances I should be able to ping 172.16.17.29 from any host in 172.19.5.0/24
For now I can not.
Is it normal? -
@awebster
I don't have checkmark on "automatically ping host" in Phase 2 settings but nevertheless there is ICMP traffic between 172.16.17.30 and 172.16.17.29 all the time and it's not related to my ICMP requests from host in private subnet.tcpdump -nni ipsec2000 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes >>>07:41:53.453287 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 16, length 64 07:41:53.668029 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13119, length 8 07:41:53.668077 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54233, length 8 >>> 07:41:53.732895 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 16, length 64] 07:41:53.947746 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13119, length 8 07:41:53.947762 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54233, length 8 07:41:54.209324 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54234, length 8 07:41:54.209372 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13120, length 8 >>> 07:41:54.453365 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 64007, seq 17, length 64 07:41:54.488891 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54234, length 8 07:41:54.488906 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13120, length 8 07:41:54.724422 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 5860, seq 13121, length 8 07:41:54.724473 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54235, length 8 >>> 07:41:54.733203 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 64007, seq 17, length 64 07:41:55.004501 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 5860, seq 13121, length 8 07:41:55.004517 IP 172.16.17.29 > 172.16.17.30: ICMP echo reply, id 48500, seq 54235, length 8 07:41:55.265786 IP 172.16.17.30 > 172.16.17.29: ICMP echo request, id 48500, seq 54236, length 8
