[IPSEC site-to-site] Subnets Connectivity
-
Hi,
I've configured IpSec Site-to-Site using PfSense from AWS Marketplace (on EC2 instance). But can not ping Remote Private Network from Local Private Network.
Details:
I have 2 elastic network interfaces on instance with pfSense. 1st is in public subnet and 2nd in private subnet (as it is described in pfsense docs)
I've Used Routed Phase2 Mode => ipsec2000 interface has been created => Static Route has been added
I've configured Outbound NAT for 172.19.5.0/24 to translate to 172.16.17.30 address.
IpSec is Up and I can ping remote tunnel endpoint and hosts in remote private network from pfsense itself:
PING 192.168.179.117 (192.168.179.117) from 172.16.17.30: 56 data bytes 64 bytes from 192.168.179.117: icmp_seq=0 ttl=252 time=280.156 ms 64 bytes from 192.168.179.117: icmp_seq=1 ttl=252 time=280.335 ms 64 bytes from 192.168.179.117: icmp_seq=2 ttl=252 time=280.528 ms --- 192.168.179.117 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 280.156/280.340/280.528/0.152 ms
Local Private Subnet - 172.19.5.0/24
xn1 interface address - 172.19.5.254
xn0 interface address - 172.19.1.148
ipsec2000 interface address - 172.16.17.30/30Remote Private Subnet - 192.168.179.0/24
Remote Tunnel Endpoint - 172.16.17.29/30When I ping 192.168.179.117 from 179.19.5.219 I can not get ICMP replies back.
But they are coming to VTI ipsec2000:[2.4.4-RELEASE][admin@pfSense.localdomain]/root: tcpdump -i ipsec2000 | grep 192.168.179.117 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes 10:35:05.341401 IP 192.168.179.117 > pfSense.localdomain: ICMP echo reply, id 29800, seq 31, length 64 10:35:06.061043 IP pfSense.localdomain > 192.168.179.117: ICMP echo request, id 29800, seq 32, length 64 10:35:06.341897 IP 192.168.179.117 > pfSense.localdomain: ICMP echo reply, id 29800, seq 32, length 64 10:35:07.061127 IP pfSense.localdomain > 192.168.179.117: ICMP echo request, id 29800, seq 33, length 64 10:35:07.341692 IP 192.168.179.117 > pfSense.localdomain: ICMP echo reply, id 29800, seq 33, length 64 10:35:08.061225 IP pfSense.localdomain > 192.168.179.117: ICMP echo request, id 29800, seq 34, length 64 10:35:08.341818 IP 192.168.179.117 > pfSense.localdomain: ICMP echo reply, id 29800, seq 34, length 64 10:35:09.061677 IP pfSense.localdomain > 192.168.179.117: ICMP echo request, id 29800, seq 35, length 64So the problem is ICMP replies don't get to Host which initially sends requests.
Please advise, what could be the cause of the issue and how can I resolve this to establish connectivity between private networks?
-
@nomatter
Hello
and if you run tcpdump on PFSense LAN interface ?
tcpdump -i xn1 icmp and host 192.168.179.117Is there any reply packets ?
-
@Konstanti
Hi,No, just requests:
[2.4.4-RELEASE][admin@pfSense.localdomain]/root: tcpdump -i xn1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on xn1, link-type EN10MB (Ethernet), capture size 262144 bytes 16:07:33.981401 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 14, length 64 16:07:34.981459 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 15, length 64 16:07:35.981537 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 16, length 64 16:07:36.981581 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 17, length 64 16:07:37.981629 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 18, length 64 16:07:38.981679 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 19, length 64 16:07:39.981731 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 20, length 64 16:07:40.981778 IP 172.19.5.219 > 192.168.179.117: ICMP echo request, id 3386, seq 21, length 64 ^C 8 packets captured 8 packets received by filter 0 packets dropped by kernel -
Are there any floating rules ?
If not, you can try to delete the tunnel from the PFSense side and create it again
-
@Konstanti
No, no Floating Rules.Do you mean delete all Ipsec Config and setup it again?
-
You can make a backup copy of the firewall settings, and then try to configure the tunnel again .
And show the rules on LAN (xn1) and IPSEC interfaces -
Recreated IpSec - same results.
Here are Firewall Rules
#System aliases loopback = "{ lo0 }" WAN = "{ xn0 }" IPSEC = "{ ipsec2000 }" LAN = "{ xn1 }" IPsec = "{ enc0 }" OpenVPN = "{ openvpn }" # Outbound NAT rules (manual) nat on $IPSEC inet from 172.19.5.0/24 to any port 500 -> 172.16.17.30/32 static-port nat on $IPSEC inet from 172.19.5.0/24 to any -> 172.16.17.30/32 port 1024:65535 nat on $WAN inet from 127.0.0.0/8 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - localhost to WAN nat on $WAN inet from 127.0.0.0/8 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - localhost to WAN nat on $WAN inet6 from ::1/128 to any port 500 -> (xn0) static-port # Auto created rule for ISAKMP - localhost to WAN nat on $WAN inet6 from ::1/128 to any -> (xn0) port 1024:65535 # Auto created rule - localhost to WAN nat on $WAN inet from 172.25.53.0/24 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - IPsec client to WAN nat on $WAN inet from 172.25.53.0/24 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - IPsec client to WAN nat on $WAN inet from 172.16.17.29 to any port 500 -> 172.19.1.148/32 static-port # Auto created rule for ISAKMP - IPsec VTI: SX site-to-site to WAN nat on $WAN inet from 172.16.17.29 to any -> 172.19.1.148/32 port 1024:65535 # Auto created rule - IPsec VTI: SX site-to-site to WAN # Outbound NAT rules (automatic) # Subnets to NAT table <tonatsubnets> { 127.0.0.0/8 ::1/128 172.19.5.0/24 172.25.53.0/24 172.16.17.29 } nat on $WAN inet from <tonatsubnets> to any port 500 -> 172.19.1.148/32 static-port nat on $WAN inet6 from <tonatsubnets> to any port 500 -> (xn0) static-port nat on $WAN inet from <tonatsubnets> to any -> 172.19.1.148/32 port 1024:65535 nat on $WAN inet6 from <tonatsubnets> to any -> (xn0) port 1024:65535 anchor "userrules/*" pass in quick on $IPsec inet proto icmp from any to any tracker 1574329796 keep state label "USER_RULE" pass in quick on $IPsec inet proto tcp from any to any tracker 1574273174 flags S/SA keep state label "USER_RULE" pass in quick on $IPsec inet from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" pass in quick on $IPsec inet6 from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto udp from any to any port 4500 tracker 1565015851 keep state label "USER_RULE" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto udp from any to any port 500 tracker 1565015834 keep state label "USER_RULE" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto esp from any to any tracker 1565015776 keep state label "USER_RULE: IPsec" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto icmp from any to 172.19.1.148 tracker 1565007607 keep state label "USER_RULE: Default ICMP rule" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 22 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default SSH rule _replace_src_with_mgmtnet_" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 443 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default HTTPS rule _replace_src_with_mgmtnet_" pass in quick on $WAN reply-to ( xn0 172.19.1.1 ) inet proto tcp from any to 172.19.1.148 port 80 tracker 1565007607 flags S/SA keep state label "USER_RULE: Default HTTP rule _replace_src_with_mgmtnet_" pass in log quick on $LAN inet from 172.16.17.28/30 to 172.19.5.0/24 tracker 1574333773 keep state label "USER_RULE" pass in log quick on $LAN inet from 172.19.5.0/24 to any tracker 1574331909 keep state label "USER_RULE" # VPN Rules pass out route-to ( xn0 172.19.1.1 ) proto udp from (self) to [REMOTE_IP] port = 500 tracker 1000106361 keep state label "IPsec: SX site-to-site - outbound isakmp" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto udp from [REMOTE_IP] to (self) port = 500 tracker 1000106362 keep state label "IPsec: SX site-to-site - inbound isakmp" pass out route-to ( xn0 172.19.1.1 ) proto udp from (self) to [REMOTE_IP] port = 4500 tracker 1000106363 keep state label "IPsec: SX site-to-site - outbound nat-t" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto udp from [REMOTE_IP] to (self) port = 4500 tracker 1000106364 keep state label "IPsec: SX site-to-site - inbound nat-t" pass out route-to ( xn0 172.19.1.1 ) proto esp from (self) to [REMOTE_IP] tracker 1000106365 keep state label "IPsec: SX site-to-site - outbound esp proto" pass in on $WAN reply-to ( xn0 172.19.1.1 ) proto esp from [REMOTE_IP] to (self) tracker 1000106366 keep state label "IPsec: SX site-to-site - inbound esp proto"Hope I showed all relevant rules.
-
@nomatter said in
And if to refuse NAT OUTBOUND for a network 172.19.5.0 / 24 on the IPSEC interface and to add a static route on CISCO for this network ?
Do you have firewall rules that are not quite necessary
pass in log quick on $LAN inet from 172.16.17.28/30 to 172.19.5.0/24 tracker 1574333773 keep state label "USER_RULE"
pass in log quick on $LAN inet from 172.19.5.0/24 to any tracker 1574331909 keep state label "USER_RULE"pass in quick on $IPsec inet proto icmp from any to any tracker 1574329796 keep state label "USER_RULE"
pass in quick on $IPsec inet proto tcp from any to any tracker 1574273174 flags S/SA keep state label "USER_RULE"
pass in quick on $IPsec inet from any to any tracker 1565007607 keep state label "USER_RULE: Default allow IPsec to any rule" -
Yes, I know they are not neccessary. I've added them to track packets in mid of troubleshooting and forget to remove.
Unfortunately I have no access to the remote side. I can just ask their network guy to add it.
Without NAT there are no ICMP replies even to ipsec2000. They recommended to setup NAT so I believe they don't have any config for my private network.
Actually I thought if ICMP replies are coming to pfsense than there is some misconfiguration on my side.
-
Try to move the NAT OUTBOUND to manual mode
-
Did that. No luck. Replies are returning to ipsec2000.
Feels like pfsense can not route them back to 172.19.5.219 or operation reverse to NAT doesn't take place.
-
@nomatter
And if you try to do so ?
What will be the result ?
-
PING 192.168.179.117 (192.168.179.117) from 172.19.5.254: 56 data bytes --- 192.168.179.117 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossIt's ok only when Source address is VTI
PING 192.168.179.117 (192.168.179.117) from 172.16.17.30: 56 data bytes 64 bytes from 192.168.179.117: icmp_seq=0 ttl=252 time=280.394 ms 64 bytes from 192.168.179.117: icmp_seq=1 ttl=252 time=280.279 ms 64 bytes from 192.168.179.117: icmp_seq=2 ttl=252 time=280.492 ms --- 192.168.179.117 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 280.279/280.388/280.492/0.087 ms -
@nomatter Your initial tcpdump of ipsec2000 doesn't reveal what the actual IP if, suggest running with
-noption.
On the surface this looks like a routing problem.
Check the routing table on the C3945 and make sure it has routes to 172.19.1.0/24 and 172.19.5.0/24 via the tunnel. -
Tried to tcpdump with -n option same results:
tcpdump -n -i ipsec2000 | grep 192.168.179.117 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec2000, link-type NULL (BSD loopback), capture size 262144 bytes 18:02:04.472813 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 105, length 64 18:02:04.752956 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 105, length 64 18:02:05.473363 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 106, length 64 18:02:05.753784 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 106, length 64 18:02:06.472929 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 107, length 64 18:02:06.753029 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 107, length 64 18:02:07.472973 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 108, length 64 18:02:07.753102 IP 192.168.179.117 > 172.16.17.30: ICMP echo reply, id 12413, seq 108, length 64 18:02:08.473044 IP 172.16.17.30 > 192.168.179.117: ICMP echo request, id 12413, seq 109, length 64 ^C66 packets captured 66 packets received by filter 0 packets dropped by kernelI'll ask guys from other side to add routes. I'm sure they don't have them. But will need to wait until Monday at least.
Can I do something relevant on my side while waiting? -
@nomatter Thanks, that dump is showing the VTI interface pinging the destination host.
The VTI on the subnet /30 shared between both devices, so it is normal that the C3945 knows how to get back to it, but if there are no routes, the C3945 has no way to know that it has to send traffic for 172.19.1.0/24 and 172.19.5.0/24 to 172.16.17.30. -
@awebster
Ah, yes. I missed that on initial dump there is pfsense.localdomain instead of address. You're right.
I asked other side to add 2 routes 172.19.5.0/24 and 172.19.1.0/24 using 172.16.17.30 as Gateway. Will wait.
I'll return to this thread after their reply.
Thank you guys for now -
@awebster
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(Is there any way to work around setting routes on their side?
Actually, I think routes on their side is needed if traffic originates out there.
For example to ping google.com we don't need google to setup routes.
There must be some misconfiguration on my side.
-
Turned out that other side is unable to set static routes to 172.19.X.X networks as there will be overlap with their local networks :(
Is there any way to work around setting routes on their side?@nomatter, one possible way to work around an overlapping subnet problem is to use 1:1 NAT, but before you do that, ask the other side what subnets that you could present to them that won't cause overlap, the you can setup the pfSense to NAT 172.19.1.x and 172.19.5.x into something that both sides can agree on.
For example 172.19.1.0/24 <=> 10.19.1.0/24 and 172.19.5.0/24 <=> 10.19.5.0/24
On their side they would need to put routes for 10.19.1.0/24 and 10.19.5.0/24
When you ping their system with their address (no change), it looks to them like it is coming from a 10.19.x.x address
When they ping your system, they use 10.19.x.x address and it looks to you like it is coming from their address (no change).Alternatively if you don't want to use 1:1 NAT, depending on how much work is involved, you could rebuild the AWS setup using different subnets.
-
SO if I understand correctly at first 172.19.5.0/24 is translated to 10.19.5.0/24 and then outbound NAT is taking place translating 10.19.5.0/24 to 172.16.17.30 which is 'allowed' to communicate with remote side?
