Issue with IPSec VPN
I've got two sites, one on the East coast ("EAST") and the other in California ("WEST"). Both sites have pfsense 2.2.6 (nanobsd) installed. EAST runs off a 192.168.100.0/24 network, WEST runs 192.168.101.0/24. Both sites' WAN interfaces are connected to a cable modem which, I've been told by their respective ISPs, are operating in bridge mode. Both sites also have public IP ranges available for hosting systems available by external clients (WWW / FTP servers etc).
I recently set up an IPSec VPN between the two sites. Reasonably straight forward, or so I thought, until I see this issue:
Initiating a ping from EAST to WEST results in no packet loss, responses come back just fine. Furthermore, accessing any of the 'private' (ie, not publicly available) services that are hosted in WEST from EAST is just fine.
However, I can't access any of the EAST based services from WEST. From a WEST Windows workstation, when I ping an EAST host (192.168.100.20) I get "Reply from 192.168.100.3: Destination host unreachable". From research that I've done thus far, this means that my packet is actually getting to the EAST network, but there's no route back.
Furthermore, why is the IP address that I'm trying to ping, different from the one that I get the unreachable host from? I've checked, and I don't even have a host configured on 192.168.100.3, but 192.168.100.20 definitely exists on that network (it's pingable from another EAST workstation)
I believe I've checked everything that makes sense, routing tables etc but I can't seem to find the difference between the EAST config which is working compared to the WEST.
Just to re-iterate, I can ping just fine from EAST to WEST; and it looks like when I ping from WEST to EAST that the packet is reaching the host, but because the source is from the WEST, there appears to be no route back. This doesn't make too much sense to me, as I know there is a route to the WEST, as when a packet initiates from EAST, it gets there just fine.
I think first things first we need to confirm that the ICMP requests are actually reaching the destination node. Couple of things we can do to establish this. Download and install Wireshark on the EAST side node, 192.168.100.20 in this instance. Once installed, run Wireshark and add a filter for 'icmp'. This should show you what ICMP traffic (if any) is hitting the EAST side node. If not, then you can look at using the 'Packet Capture' feature within pfSense. You can access this by going to Diagnostics > Packet Capture and setting the relevant settings in here. If you can narrow down at which point this is failing, it makes troubleshooting far easier. Trace route may also help you with this scenario.
Not meaning to insult your intelligence here - but I assume you have added the relevant firewall rules on the EAST side under Firewall > Rules > IPsec?
Well - after some serious head scratching you'll NEVER guess what the issue was. Some joker (hilarious I think not) had added an EAST coast IP address to my WEST coast system. This not only explained why I wasn't getting responses back from the other side of the VPN, but also why the response was coming from an IP address that I hadn't even asked for. So all my logic into when a ping gets a response but with no destination was a little misguided.
For shiggles, I'll explain what happened in more detail :
On my WEST system (192.168.101.123) someone had added 192.168.100.3 to its IP stack. This obviously explained why, when attempting to ping the EAST coast system of 192.168.100.20 I get the (correct) response "Reply from 192.168.100.3: Destination host unreachable".