pfsense 2.5.2 slowly leaking memory

netnerdy

version 2.5.2 had a major memleak because of the pcscd service. I have applied the fix for this, yet there is still something slowly leaking. you can see it in my memory chart:

Is someone looking into this?

Cool_Corona

This post is deleted!

Cool_Corona

This post is deleted!

Cool_Corona

@netnerdy

i dont see anything like that.

mr.rosh

@netnerdy
dats what I see. So what's your system RAM?

@cool_corona

netnerdy

I have two gigs of ram. What do you guys see when you do a monthly view with 1 hour resolution?

psp

Here is mine:

mr.rosh

@psp

Gertjan

@mr-rosh

Stats for one month.
At the begin of the month pfBlockerNG was on holiday ...
The "pcscd " was applied more then a month ago.

@netnerdy said in pfsense 2.5.2 slowly leaking memory:

still something ...

Tell us what you have that we don't, and you'll know what that "something" is.

mr.rosh

@gertjan
My deployment is a month old.. so it'll be not conslusive.

However, i'm on

Version 2.5.2-RELEASE (amd64)
built on Fri Jul 02 15:33:00 EDT 2021
FreeBSD 12.2-STABLE

Services are as below;

Gertjan

@mr-rosh said in pfsense 2.5.2 slowly leaking memory:

However, i'm on
Version 2.5.2-RELEASE (amd64)

We all are.
( if @netnerdy was mentioning another version, that answer would be "upgrade" )

Read "pcscd PC/SC Smart Card Daemon" to get rid :

so it won't auto start if your pfSense reboots.

stephenw10

Yes, the pcscd process is the most likely suspect. Start by disabling it:
https://redmine.pfsense.org/issues/11933#note-7

Steve

netnerdy

I think explaining the graph I posted in the beginning will help.

The big mountains you see there are from the time when I didn't have pcscd fix. I would reboot pfsense and pcscd would eventually eat away my free memory. Like I mentioned in my first post, I have the full pcscd fix, so it doesn't start on boot anymore. You can see the positive effect it had on memory. I haven't rebooted pfsense for 7 days now. But it seems something else is chipping away at the memory.

@psp you most likely have a mem leak problem as well. Not sure if it's same root cause.

My services are below.

I wonder if it's haproxy or vmware guest services. Perhaps it's wpa_supplicant (i do att bypass).

stephenw10

Is it actually still growing steadily? Hard to say from that graph.

Doesn't look like it's wired memory so what's using it? Check ps -auxwwd.

Also if it never actually exhausts it that may not be a problem. Unused RAM helps nothing.

Steve

netnerdy

@stephenw10

I would have expected a better response from Netgate: "If it never exhausts it that may not be a problem" is a frankly irresponsible way to categorize memory issues for a company that sells hardware that is supposed to run on devices with limited cpu and memory 24/7. Perhaps this kind of attitude allowed to slip a terrible memory leak like pcscd into production.

Also what is the expectation here? Am I supposed to wait until all my memory is exhausted so my router goes down and suffer the consequences? I would rather be proactive than to be reactive.

Sorry if this sounds too harsh, but please take this as constructive criticism as I really am not trying to troll you or Netgate.

mer

@netnerdy
I'm just a user of Netgate products but since the dawn of embedded devices there have been arguments around unused RAM.
One school is RAM should be 100% used the other is basically "you should always have 20% free RAM just in case".

If you have unused RAM you are not near capacity.
Is this device using ZFS or standard UFS? That makes a difference.

One also needs to understand the different types of memory: wired is basically kernel owned can't really be freed quickly. Cache and Free are self explanatory, Inactive is typically "was used but isn't now and should be freed next".

The position "if it never exhausts it may not be a problem" is technically accurate. Think garbage collection or lazy releasing of memory; so "it's allocated right now but in secs it's freed".

Cool_Corona

stephenw10

@netnerdy said in pfsense 2.5.2 slowly leaking memory:

Also what is the expectation here? Am I supposed to wait until all my memory is exhausted so my router goes down and suffer the consequences? I would rather be proactive than to be reactive.

I'm asking if it's actually a leak or just something using memory because your graph in the first screenshot looks to be leveling off at ~30% which generally wouldn't worry me.
The actual used memory value will vary wildly with how much RAM you have installed and what packages you're running. 30% of 64GB with no packages would be unexpected.

If it is a linear usage rise that characterises a memory leak then that's obviously a problem.
In which case check what is using the memory.

Steve

netnerdy

@stephenw10

Sorry if misunderstood, thx for the clarification. Unfortunately, I had to reboot my router for another reason so I'll need to wait a week to answer your question.

I started paying attention to mem usage just because of the pcscd leak. When I saw the second drop I was worried. I would be totally content with the explanation of "this kind of mem usage trend is by design", which seems to be case. I'll revive this if I see anything fishy. thx!

stephenw10

Mmm, hard to say without more data. If is is still leaking somewhere it's small. It could also be something that simply doesn't free memory unless it has to and since you still have >30% free it doesn't.
Keep monitoring it. Check the process list.

Steve