VTI to AWS becomes unstable after some time (weeks or months), requiring restart.

whosmatt

Hi all --

I've run across the same behavior now on a couple of different pfSense implementations and AWS tunnels to different regions / VPCs.

The setup is standard VTI to AWS, two tunnels up simultaneously and BGP via OpenBGPD (because FRR still seems not to play nice with VTI). After a fairly long period of time (weeks to months), the tunnels start to show heavy packet loss -- 10% to 40% or more via gateway monitoring (and real world observation) and seem only to recover if I fail over to the secondary firewall in a HA pair and/or reboot the appliance.

This is happening both on a couple of pfSense CE VMs we run on prem as well as an Azure VM. I'm still in the information gathering stage right now, as I just saw the behavior on a second implementation for the first time. Just wondering if this is known; I searched and didn't find anything that looked like this specific issue.

Crypto is: IKEv2, AES256,SHA256,DH14. VTI (P2) is AES256,SHA256,DH14.

Thanks,

M

jimp

I'm not aware of anything that matches that exactly. There was someone who posted not long ago that they observed what appeared to be a memory leak, but I don't know that they definitively correlated it to IPsec.

If it was something in IPsec settings, it would come up within hours/days after IKE/Child SA entries get renewed or rekeyed and so on. If it's going for weeks/months it's unlikely to be anything in your configuration.

Something like that might be solved by a new version of strongSwan or if it's an OS bug it may be fixed in a more recent version. If possible, give 2.4.5 snapshots a try and see if they behave better. 2.4.5 is very close to release, though, so if you wait a couple weeks you can just upgrade to the release and then keep an eye on it from there.

whosmatt

Thanks Jim. I'll just wait for the 2.4.5 release and see if anything changes. It's worth noting that I only see this with AWS tunnels -- I have other VTI tunnels on the same pfSense instances that don't misbehave at all.

whosmatt

So, I have an update.. this still happens on 2.4.5, multiple pfSense implementations. I discovered a charon process using a significant amount of CPU time and killing it fixed the problem (presumably until the next time it happens). This particular time it occurred after 3 weeks of uptime. I have logs if anyone is interested.

whosmatt

<30>May  3 13:05:41 charon: 11[KNL] <con1000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found

Messages like this repeated over and over at alarming frequency. They still show up when the tunnels are working well, but at much lower frequency.

alt text

Count dropped off a cliff when I killed the charon process.