IPsec tunnel UP but unable to ping remote site

netsysadmin · Jan 3, 2014, 9:52 AM

Hello,

As the title says, I have an IPsec site-to-site VPN up (can be seen from menu Status -> IPsec), but am unable to ping hosts on either side.

I have the following setup:

LOCAL LAN <==> LOCAL pfSense <==> Cisco router <==> INTERNET <==> A router <==> REMOTE pfSense <==> REMOTE LAN

LOCAL LAN: 10.6.0.0/16
REMOTE LAN: 192.168.6.0/24

Local pfSense: 2.1
Remote pfSense: 2.0.3

From local host 10.6.0.7/16, I'm trying to ping remote host 192.168.6.105/24, and vice-versa.
I don't get any reply on both pings.

NB:
1. I have "Allow" rules on all tabs (LAN & IPsec) for these 2 subnets.
2. Local pfSense does not use NAT. There's no rule in "Advanced Outbound NAT" page.
3. Remote pfSense does NAT. I have a rule to NOT NAT traffic from 192.168.6.0/24 to 10.6.0.0/16.

I did the following 2 tests and made packet captures on both sides:

TEST #1: Ping from local host 10.6.0.7 to remote host 192.168.6.105
Results of packet capture on LAN & IPsec interfaces of both pfSense firewalls:
1. Local LAN: 4 ICMP requests from 10.6.0.7 to remote host 192.168.6.105, No ICMP reply
2. Local IPsec: 4 ICMP requests from 10.6.0.7 to remote host 192.168.6.105, No ICMP reply
3. Remote LAN: Nothing captured
4. Remote IPsec: Nothing captured

TEST #2: Ping from remote host 192.168.6.105 to local host 10.6.0.7
Results of packet capture on LAN & IPsec interfaces of both pfSense firewalls:
1. Remote LAN: 4 ICMP requests from host 192.168.6.105 to host 10.6.0.7, No ICMP reply
2. Remote IPsec: 4 ICMP requests from host 192.168.6.105 to host 10.6.0.7, No ICMP reply
3. Local LAN: 4 ICMP requests from remote host 192.168.6.105 to local host 10.6.0.7, 4 ICMP replies from local host 10.6.0.7 to remote host 192.168.6.105
4. Local IPsec: 4 ICMP requests from remote host 192.168.6.105 to local host 10.6.0.7, 4 ICMP replies from local host 10.6.0.7 to remote host 192.168.6.105

It seems the packets originating from the local LAN, after going into the IPsec tunnel, do not arrive at the remote LAN.
However, packets originating from the remote LAN, arrive at the local LAN without any issue. The reply packets from the local host do not reach the remote host.

Anyone has any idea what the cause of the problem could be?

Thanks in advance.

abidkhanhk · Jan 6, 2014, 3:44 AM

Hi,

I have almost similar issue,

i can RDP to the remote hosts, but cant ping or tracert,

normally this wouldnt bug me much but i have a number of PCs that are unable to connect to the Domain controller on the remote network.

IPsec rules on both pfboxes are pass on any to any.

any thoughts?

thanks

EDIT, my bad I wasn't paying attention to rule in IPsec, its tcp/udp, for ICMP u need a specific rule on both side.

netsysadmin · Jan 6, 2014, 2:02 PM

Hello,

For both of my pfSense boxes, the "Allow" LAN & IPsec interfaces rules have protocol set to "any".

Thanks.

netsysadmin · Jan 7, 2014, 12:26 PM

Any expert willing to help debug please?

Matthias · Jan 8, 2014, 3:00 PM

What kind of devices are you trying to ping? Windows devices? Anything else?

netsysadmin · Jan 8, 2014, 3:04 PM

Matthias, thank you for your reply.

The local test hosts are Windows & Mac.
The remote host is a Mac.

This is very very strange or there's something too obvious that I can see!

Currently, the IPsec VPN tunnel endpoints are the 2 pfSense firewalls.

I did the following 2 tests:
1. I moved the local IPsec tunnel endpoint to the local Cisco router and ran the same tests: similar results.
I configured an access list on the LAN interface of the router to log the test traffic originating from the local host: no test packet seems to reach the LAN interface of the router.
When I ping the local host from the remote host, the reply ICMP packets go through the local pfSense WAN interface towards the router, but no such packet is logged on the LAN interface of the router.

2. I configured another local pfSense box almost the same as the first one and set it up as the local tunnel endpoint: again similar results.
When I ping from the local host, the ICMP packets arrive on the local LAN interface of the 2nd pfSense box, enter the IPsec tunnel, but none seems to come out at the remote end.
When I ping the local host from the remote host, the reply ICMP packets arrive on the second local pfSense LAN interface, but again none seems to come out at the remote end.

Could the problem be with the remote pfSense?

Any help is appreciated.

Matthias · Jan 8, 2014, 3:09 PM

My first thought is that by default Windows firewall doesn't allow ICMP from remote subnets. But it looks like some sort of routing issue is happening. What happens if you ping the LAN interface on each network?

netsysadmin · Jan 8, 2014, 3:14 PM

Local host does not receive any reply when pinging LAN interface of remote pfSense.
Remote host does not receive any reply when pinging LAN interface of local pfSense.

netsysadmin · Jan 8, 2014, 3:19 PM

I forgot to mention one important thing: I have 3 other IPsec tunnels on the local pfSense with other remote devices and the others all work fine!

Matthias · Jan 8, 2014, 3:20 PM

Are any of the other remote devices a pfsense version lower than 2.1?

netsysadmin · Jan 8, 2014, 4:27 PM

There are 2 pfSense 2.0.3 boxes & 1 Cisco 1841 router.

NB: I even tried rebooting both pfSense boxes but that did not make any difference.

Swissnic · Jan 11, 2014, 10:17 AM

I have experienced a similar problem. I have 2 pfSense 2.1 boxes in different locations with static ips. I set up a simple IPsec and got it working. After 24 hours, the traffic flow dies, but the VPN shows it is UP. Can't ping or anything. Reboot both pfSence's and the VPN might work, or might not connect at all.

IPsec in 1.2.3 was rock solid! In 2.1 it seems VERY flaky! What happened???

newbieuser1234 · Jan 11, 2014, 12:12 PM

I am no expert, but under System/Advanced/Firewall NAT is "Disable Auto-added VPN rules" checked?

netsysadmin · Jan 13, 2014, 7:51 AM

Thank you for your replies.

@Swissnic: I remember that when I first set up the VPN, it was working fine. The next day, it wasn't! Rebooting has not solved the problem.
@newbieuser1234: On both pfSense boxes, the "Disable all auto-added VPN rules" option is NOT checked.

If there are other people with the same problem, maybe it is a bug?
But strangely, it is working absolutely fine for the other 3 IPsec VPNs!

newbieuser1234 · Jan 13, 2014, 1:13 PM

dumb question, but can you access the webconfigurator on the remote lan? Are you running snort or anything that could be blocking them. I added some aliases and whitelisted my ipsec tunnel addresses in snort. just a thought.

netsysadmin · Jan 13, 2014, 2:56 PM

At this point, no question is dumb.
No, I cannot access the webconfigurator via its remote local IP address.
But, I do have access to it via a public NATted IP address, i.e., I can make any change if required.

Snort is not installed on either pfSense.
Only ntop is installed on the remote one.
On the local one, HAVP (not enabled/running), iftop & mailreport are installed.

Thank you for trying to help.

newbieuser1234 · Jan 13, 2014, 3:07 PM

What do the ipsec logs say?

netsysadmin · Jan 14, 2014, 4:13 PM

Local IPsec log:

Jan 14 20:01:22 racoon: [Remote Site]: INFO: IPsec-SA established: ESP A.B.14.125[500]->X.Y.45.57[500] spi=13794811(0xd27dfb)
Jan 14 20:01:22 racoon: [Remote Site]: INFO: IPsec-SA established: ESP A.B.14.125[500]->X.Y.45.57[500] spi=256623936(0xf4bc540)
Jan 14 20:01:22 racoon: [Remote Site]: INFO: initiate new phase 2 negotiation: A.B.14.125[500]<=>X.Y.45.57[500]
Jan 14 20:01:22 racoon: [Remote Site]: INFO: ISAKMP-SA established A.B.14.125[500]-X.Y.45.57[500] spi:f9a7f6f8365b050a:7bc3360f027abf56
Jan 14 20:01:21 racoon: [Remote Site]: INFO: initiate new phase 1 negotiation: A.B.14.125[500]<=>X.Y.45.57[500]
Jan 14 20:01:21 racoon: [Remote Site]: INFO: IPsec-SA request for X.Y.45.57 queued due to no phase1 found.

Remote IPsec log:

Jan 14 20:01:22 racoon: [Local Site]: INFO: IPsec-SA established: ESP X.Y.45.57[500]-> A.B.14.125[500] spi=256623936(0xf4bc540)
Jan 14 20:01:22 racoon: [Local Site]: INFO: IPsec-SA established: ESP X.Y.45.57[500]-> A.B.14.125[500] spi=13794811(0xd27dfb)
Jan 14 20:01:22 racoon: [Local Site]: INFO: respond new phase 2 negotiation: X.Y.45.57[500]<=> A.B.14.125[500]
Jan 14 20:01:22 racoon: [Local Site]: [ A.B.14.125] INFO: received INITIAL-CONTACT
Jan 14 20:01:22 racoon: [Local Site]: INFO: ISAKMP-SA established X.Y.45.57[500]- A.B.14.125[500] spi:f9a7f6f8365b050a:7bc3360f027abf56
Jan 14 20:01:21 racoon: [Local Site]: INFO: respond new phase 1 negotiation: X.Y.45.57[500]<=> A.B.14.125[500]

netsysadmin · Jan 14, 2014, 5:02 PM

Below is an extract of the file /tmp/rules.debug:

# Outbound NAT rules
nat on $WAN_MAIN  proto udp from any to X.Y.45.56/29 port 500 -> A.B.14.125/32  static-port
nat on $WAN_MAIN  proto esp from any to X.Y.45.56/29 -> A.B.14.125/32 port 1024:65535

anchor "ipsec/*"
# Block all IPv6
block in log quick inet6 all label "Block all IPv6"
block out log quick inet6 all label "Block all IPv6"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log inet all label "Default deny rule IPv4"
block out log inet all label "Default deny rule IPv4"
block in log inet6 all label "Default deny rule IPv6"
block out log inet6 all label "Default deny rule IPv6"

# IPv6 ICMP is not auxilary, it is required for operation
# See man icmp6(4)
# 1    unreach         Destination unreachable
# 2    toobig          Packet too big
# 128  echoreq         Echo service request
# 129  echorep         Echo service reply
# 133  routersol       Router solicitation
# 134  routeradv       Router advertisement
# 135  neighbrsol      Neighbor solicitation
# 136  neighbradv      Neighbor advertisement
pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state

# Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state

# We use the mighty pf, we cannot be fooled.
block quick inet proto { tcp, udp } from any port = 0 to any
block quick inet proto { tcp, udp } from any to any port = 0
block quick inet6 proto { tcp, udp } from any port = 0 to any
block quick inet6 proto { tcp, udp } from any to any port = 0

# Snort package
block quick from <snort2c> to any label "Block snort2c hosts"
block quick from any to <snort2c> label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp

# SSH lockout
block in log quick proto tcp from <sshlockout> to any port 22 label "sshlockout"
# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout> to any port 443 label "webConfiguratorlockout"
block in quick from <virusprot> to any label "virusprot overload table"
# block bogon networks
# http://www.cymru.com/Documents/bogon-bn-nonagg.txt
# http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
block in log quick on $LAN_VLAN6 from <bogons> to any label "block bogon IPv4 networks from LAN_VLAN6"
antispoof for re1_vlan6
# block bogon networks
# http://www.cymru.com/Documents/bogon-bn-nonagg.txt
# http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
block in log quick on $WAN_MAIN from <bogons> to any label "block bogon IPv4 networks from WAN_MAIN"
antispoof for re2_vlan11
# block bogon networks
# http://www.cymru.com/Documents/bogon-bn-nonagg.txt
# http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

# loopback
pass in on $loopback inet all label "pass IPv4 loopback"
pass out on $loopback inet all label "pass IPv4 loopback"
pass in on $loopback inet6 all label "pass IPv6 loopback"
pass out on $loopback inet6 all label "pass IPv6 loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to ( re2_vlan11 10.0.0.1 ) from 10.0.0.253 to !10.0.0.0/16 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( re2_vlan11 10.0.0.1 ) from A.B.14.122 to !A.B.14.122/32 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( re2_vlan11 10.0.0.1 ) from A.B.14.125 to !A.B.14.125/32 keep state allow-opts label "let out anything from firewall host itself"
pass out on $IPsec all keep state label "IPsec internal host to host"

# User-defined rules follow

anchor "userrules/*"
pass  in  quick  on $IPsec inet from any to any keep state  label "USER_RULE"
# array key "wan" does not exist for "" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE"
# array key "lan" does not exist for "Default allow LAN to any rule" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN_VLAN6 inet from 10.6.0.253/16 to any keep state  label "USER_RULE"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet from $VPN_ALL_REMOTE_LANs to 10.6.0.253/16 keep state label "USER_RULE"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto esp  from $VPN_REMOTE_ENDPOINTS to A.B.14.122 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto esp  from $VPN_REMOTE_ENDPOINTS to A.B.14.125 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto udp  from $VPN_REMOTE_ENDPOINTS port 500 to A.B.14.122 port 500 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto udp  from $VPN_REMOTE_ENDPOINTS port 500 to A.B.14.125 port 500 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto udp  from $VPN_REMOTE_ENDPOINTS to A.B.14.122 port 500 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 ) inet proto udp  from $VPN_REMOTE_ENDPOINTS to A.B.14.125 port 500 keep state label "USER_RULE: Tunnel VPN IPSec"
pass  in  quick  on $WAN_MAIN reply-to ( re2_vlan11 10.0.0.1 )  proto tcp  from $NET_VPNs to 10.6.0.253/16 flags S/SA keep state label "USER_RULE"
# array key "opt5" does not exist for "" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE"
# array key "opt5" does not exist for "LAN 2" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE: LAN 2"
# array key "opt5" does not exist for "LAN 2" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE: LAN 2"
# array key "opt5" does not exist for "LAN 2" in array: {LAN_VLAN6 WAN_MAIN IPsec } label "USER_RULE: LAN 2"
# Could not locate interface for IPsec:  Remote Site</bogons></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c>

Maybe this can help find the problem.

Thank you

netsysadmin · Jan 14, 2014, 5:04 PM

The file /tmp/rules.debug contains some references to snort, but this package is not listed in the list of installed packages!
Is this normal?