IPSEC Tunnel still says established, but stops passing traffic

SSI

We have a bit of an odd issue with our IPSEC tunnel. The issue is the tunnel connects just fine, and all traffic works as expected. Then randomly, and this can range from a few hours to multiple weeks before showing any issues, traffic just stops being passed altogether. The tunnel still says it's established, but disconnecting and reconnecting it fixes the issue immediately.

We believe we have tracked down where the issue is occurring by running 'ipsec statusall', but aren't sure why it's behaving in this manner. Let's say these are the IP's involved:
WAN IP: 210.50.50.51
pfsense Static IP: 150.30.30.41
Remote IP: 140.20.20.31

When we run ipsec statusall on the pfsense appliance, we see the following line:
Security Associations (1 up, 0 connecting):
con1000[16]: ESTABLISHED 2 hours ago, 150.30.30.41[150.30.30.41]...140.20.20.31[140.20.20.31]

That's what it looks like when everyone is working just fine. However, when the tunnel is failing to pass traffic, we notice it is instead using/seeing the WAN IP as the Local IP:
Security Associations (1 up, 0 connecting):
con1000[16]: ESTABLISHED 2 hours ago, 210.50.50.51[150.30.30.41]...140.20.20.31[140.20.20.31]

If we disconnect, and reconnect the tunnel, it changes back to 150.30.30.41[150.30.30.41] as expected.

Is there something we are missing that can prevent this behavior? In the IPSEC configuration, we have the 'My Identifier' field set to IP Address, and manually entered 150.30.30.41, but that doesn't seem to help. Please let us know if there are any other details we can provide to help pin this down.

This is on pfSense 2.4.3-RELEASE-p1, strongSwan 5.6.2, FreeBSD 11.1-RELEASE-p10, amd64

Derelict

@ssi said in IPSEC Tunnel still says established, but stops passing traffic:

WAN IP: 210.50.50.51
pfsense Static IP: 150.30.30.41

I don't understand that. What is on the WAN interface? How is that related to 150.30.30.41?

SSI

So we have Google Fiber and they do things a bit differently than we're used to. Our WAN interface is set to DHCP as required by Google, since our WAN IP is not static. However, since we are signed up for multiple static IPs, they route a subnet of IPs to our network. The WAN interface gets the 210.50.50.51 IP address, and 150.30.30.41 is one of the static IPs we have that we have assigned as the gateway interface on the pfsense appliance. Hopefully that makes some sense.

These are just fake IP's I'm using as an example btw.

Derelict

@ssi said in IPSEC Tunnel still says established, but stops passing traffic:

The WAN interface gets the 210.50.50.51 IP address, and 150.30.30.41 is one of the static IPs we have that we have assigned as the gateway interface on the pfsense appliance.

No. Still clear as mud. If they are routing a subnet to you, why is one of them configured as a gateway?

If I understand correctly I would:

Add 150.30.30.41 as a VIP on WAN
Tell the IPsec P1 to bind to that VIP instead of WAN.

SSI

Sorry - looking back I completely misspoke on the gateway interface issue.

Anyway, it turns out our colo provider was pointing to our WAN IP, and not the static IP. We just had them correct it, so we'll see if it's the final fix, but I'm guessing it likely is.

Derelict

Ah that might do it. Cool.