IPSEC tunnels up, won't pass traffic
-
Running in to a random issue across our deployments of pfSense.
For each of our sites, we have dual WAN and dual IPSEC tunnels to our datacenter for a hub and spoke configuration.
The datacenter device is a Palo Alto firewall.
The remote sites are all pfSense CE.For all the sites, we get VPN connected, routing works, policy based routing works - everything is pretty darn happy. However, we have a few sites that the IPSEC tunnel will randomly drop traffic.
The strange thing is that we can see the tunnel is up on both the Palo Alto and the pfSense device. We can review the logs and there is nothing that stands out as being problematic. We've reviewed the logs in the GUI and CLI and we can't find anything that is substantial to say why these devices aren't passing traffic.
We've adjusted MTU, set MSS, changed all the settings we can think of - then without any reason, they'll start passing traffic again.
Where should we start? What can we do to find the root cause of these issues?
So far, we've used wireshark to validate traffic is flowing in to the tunnel - which it will on both sides - but nothing comes out. That is, we see packet encap, but no packet decap.
Right now, I just need to know what would you do to start troubleshooting this issue?
Thank you for any input.