Normal charon memory usage?
-
I'm noticing on a pfsense 2.2.3 box with about 150 ipsec tunnels that this process gradual increases in inactive memory usage. I was just wonder if this is normal. I ask because the box is crashing every couple of days, and that's the only thing i noticed strange on it. When it does get to 100% the box doesn't crash right away, it takes about a day or two.
ps -aux -p 51263 USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 51263 0.0 83.7 2075220 1504496 - Ss Sun05AM 18:16.50 /usr/local/libexec/ipsec/charon --use-syslog -
I have the same problem with 2.2.3 (and 2.2.2). I have about 65 tunnels and have to disable IPSEC, then enable it, every couple of days to reset all the lost memory otherwise swap memory gets totally used up and the system crashes.
-
on 2.2.1
with 10 tunnels i obtain this
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 95377 0.0 83.5 5447544 1712484 - Is 7May15 222:45.45 /usr/local/libexec/ipsec/charon –use-syslogIs this normal ?
-
@All three above;
- You running 32 or 64 bit?
- How many RAM you have installed?
4 GB or 8 GB or perhaps more? - Is perhaps the CPU power to low?
Would a Intel Xeon E3-12xxv3 3,0 GHz, 3,4 GHz or 3,7 GHz do the job better?
I would suggest;
- Add more RAM to your firewalls
2 x 8 or 4 x 8 GB ECC RAM - try up tuning the NIC options step by step as described here Tuning and Troubleshooting Network Cards
especial the nmbclusters
Under System>Advanced, Networking, do you have TSO and LRO disabled? That's the default, and should be left that way. I have seen weird VPN issues where people have enabled one or both of those.
-
We have 2 pfSense boxes. Each runs version 2.2.3 and each has constant memory leaks. One box has 65+ tunnels and consumes 1 gig of RAM every 2 days. The other has 3 tunnels and 4 gigs of RAM and takes months to consume all that RAM. But RDD graphs show it is slowly eating it's way through the RAM.
The box with the 65 tunnels has been running for 5+ years. It started consuming RAM when it was updated to 2.2.2 (We went from 2.1.5 to 2.2.2).
Neither box had a problem with version 2.1.5, so I don't think it's a NIC issue.
-
We have 2 pfSense boxes. Each runs version 2.2.3 and each has constant memory leaks.
Perhaps a daemon leaks in the new version, could be.
One box has 65+ tunnels and consumes 1 gig of RAM every 2 days.
Ok for sure this is a huge amount of VPN tunnels, but can you not flush the logs from the
RAM after a while? Or are this all new build VPN keys that wasted your RAM?The other has 3 tunnels and 4 gigs of RAM and takes months to consume all that RAM.
Thats the exactly point I want to drive your attention on! Low amount of tunnels low or slow RAM usage
and many tunnels with fast and/or huge amount of RAM usage.The box with the 65 tunnels has been running for 5+ years.
Ok for sure I can understand your point now a little bit better, and I really don´t know how to say it
without making you perhaps angry, but this was not FreeBSD 10 or 10.1 as basis of your pfSense!
The last 5 years I really mean, right? And so perhaps since the change of 2.1.5 to 2.2.2 and up to
2.2.3 something was changing really, so that now your long time proofen setup is not running smooth
and liquid anymore and you might change something or plain adjust something like the mbufs size perhaps
that you will not running out of RAM in the future.Ok what I really want to say is the following:
In older times and loiwer versions of FreeBSD and also then on top pfSense, because it is based on FreeBSD
has had a let us call it, very old problem with the kernel buffer size, that comes from older days, allright?And now a newer version from FreeBSD, let us imagine ;-) is trying to solve this problem out
and let the FreeBSD users and of course also the pfSense users freeing some of this kernel memory
by setting up, under version 10.1, the mbufs size, to free the kernel memory, the memory usage will
be also there but not freezing the box, ok?And I want to trigger you now to this exactly point to try this out, ok? Not more but also not less!
I was getting out this by reading another forums thread I couldn´t find at the moment now, what
is really sad related to my very poor english language skills I must say! -
I am running
-
pfSense AMD64 2.2.3
-
AMD Athlon
II X4 640 Processor 4 CPUs: 1 package(s) x 4 core(s). -
2GB RAM
We actually had the same amount of tunnels running on a box with dual core and 1gb ram for long time, that was 2.1.X and with racoon we never saw this issue.
Attached is what the RRD graph looks like, when you see the spikes reset that's when it crashed. All the ones right in a row was us proactively rebooting it.
I went through all my IPSEC tunnels and nothing seems out of place. I was wondering if anyone else out there has 100+ Ipsec tunnels and not having this issue?
-
-
We actually had the same amount of tunnels running on a box with dual core and 1gb ram for long time, that was 2.1.X and with racoon we never saw this issue.
I imagine the same as above, related to the circumstance that the version jump from pfSense 2.1.x to version 2.2.2
was also a version jump from FreeBSD 8.3 to 10.0.I was wondering if anyone else out there has 100+ Ipsec tunnels and not having this issue?
And I am pretty sure they will be all, earlier or later, coming in the same trap as you.
Perhaps stronger and more powerful hardware and a pit of more RAM will do it also, without tuning the mbufs
size, but this is another story. -
Mbufs are running at 4 to 7% of the default count while RAM is still being consumed. Not sure how increasing the Mbuf max count will help.
Logs are set for silent. Clearing the log has no significant impact on RAM.
-
Not sure how increasing the Mbuf max count will help.
Then please read this article that would it perhaps explaining some how better.
Tuning FreeBSD to serve 100-200 thousands of connections -
There is a memory leak of some sort in strongswan under some condition(s).
djamp42: that's the worst I've seen, by far. Especially bad on a system with 2 GB RAM. Could you PM me a copy of your config from <ipsec>to</ipsec> ? Can copy/paste off of status.php which should trim out PSK and cert data which is unnecessary.
-
I have been changing all my pfsense - pfsense tunnels to IKEv2 as i upgrade them. I do have about 50 pfsense to cisco ASA tunnels that have to stay IKEv1 due to the issues with IKEv2. If someone has this working with a large amount of tunnels i would be more then happy to change my settings to see if it fixes it.
-
Also we average about 20Mb/s inbound and 50Mb/s output on the ipsec interface. It sounds like i could be running into this issue?
-
Also we average about 20Mb/s inbound and 50Mb/s output on the ipsec interface. It sounds like i could be running into this issue?
No, we don't use libipsec.
-
if you find a solution please post here
-
The most significant leaks are now fixed in 2.2.5.
-
Great Work!
-