Charon memory leak

cmb

This is part of what I'm looking into this week, hopefully can have some details to get reported upstream.

Any real world experiences and specifics on what you're using would be appreciated.

luma

Hi,

I have exactly the same problem on 2 VMWare instances.

Version : 2.2.4 (amd64)
Cryptographic Hardware : AESNI

I have a mix of IKEv1 and IKEv2 protocols. Mostly AES128.

I could share my obfuscated IPsec configuration if needed.

I have to stop and start IPsec service. Restart don't free memory…

Thanks cmb for your investigation

image.png_thumb

djamp42

I also have a mix of IKEv1 and IKEv2, i would say my end points are 75% pfsense and 25% cisco ASA. Remote endpoints are anything from 1.2.3 to 2.2.4. Every box i upgrade to 2.2.X i change the tunnel to IKEv2.

I'm only using PSK on all connections.
All my pfsense to pfsense tunnels are 3DES MD5 for both ph1 and ph2
All my pfsense to Cisco ASA tunnels are 3DES MD5 for ph1 and AES256 MD5 for ph2

I have about 100 tunnels on pfsense 2.2.3 AMD64

AMD Athlon II X4 640 Processor 4 CPUs: 1 package(s) x 4 core(s)
2GB Ram

Reboot or Restarting the IPSEC frees up memory for me. I have a copy of my latest crash report if you need one.

None of my end points seem to have this problem. They only have 1 to 3 tunnels each on them.

Thanks again cmb for looking at it!

djamp42

Even my end points with 1 VPN tunnel are having this problem. It just takes an really long time for it to run out of memory. Here is a box that has been up for 63 days.

root   33255   0.0 14.5 321168 298032  -  Is   15Jun15      7:33.03 /usr/local/libexec/ipsec/charon --use-syslog

Attach is the 3 month graph.

I've tried different ipsec settings but nothing seems to help. It seems charon is just broken.

3monthgraph.JPG_thumb

stemond

How can you solved it ?

Setting your tunnels from IKEv1 to IKEv2 ?

djamp42

I don't think you can. I've tried IKEv1 IKEv2 all sorts of different settings and Charon continues to eat memory.

luma

Any news on this problem ? Is Strongswan working well for anyone ? Or all >= 2.2.3 users affected ?

I would be happy to offer my help to find the culpit… Maybe we can open a Redmine ticket ?

Thanks !

MadBullet

Exactly the same issue for me in production with a low constant traffic. We need to restart the service every week.

I'm a bit afraid by this ticket from strongswan tracking :
https://wiki.strongswan.org/issues/964
Since I clearly do not have a high traffic on that pfSense node, it seems there IS a memory leak somewhere in charon… But in any case, they're talking about the v5.3, so if it's our issue, upgrade the pfsense dependency won't fix it.

I think we need a ticket, but where ? ... both places ?

Regards
Alex

djamp42

It is not that issue, see this thread also. https://forum.pfsense.org/index.php?topic=96187.0

CMB said he was going to look into it, but haven't heard anything back yet. i think it might be something with the FreeBSD port of strongswan because it doesn't seem like linux users are having this issue. Strongswan does have 5.3.3 coming out soon, but i don't see anything in release related to this.

Also from my testing this issue is in every 2.2.X release

mudshark79

Can confirm. Didin't check new threads before posting:

https://forum.pfsense.org/index.php?topic=98672.0

cmb

@MadBullet:

I'm a bit afraid by this ticket from strongswan tracking :
https://wiki.strongswan.org/issues/964

That's strictly related to their userland libipsec, which has no relevance to anything we use.

I confirmed the general issue.
https://redmine.pfsense.org/issues/5149
https://wiki.strongswan.org/issues/1106

luma

Hi,

May I ask you for news about this really anoying problem ?

Thanks and regards

djamp42

It's being worked currently. https://redmine.pfsense.org/issues/5149

cmb

@djamp42:

It's being worked currently. https://redmine.pfsense.org/issues/5149

There's an update on that ticket. Next snapshot run should resolve the serious leaks.