pcscd using large amounts of ram and swap space on pfsense 2.5.2 CE

tman904

Hi.

I'm running "pfsense 2.5.2 CE" on a "pc engines apu1d4" and recently its RAM usage has increased to 65% along with the swap partition filling to 100%.

I'm using QOS via the traffic shaping wizard and snort on 3 different interfaces on this install. Note I've ruled out the snort processes as a root cause because combined they are only using 768MB of RAM.

Upon connecting via ssh and running "top -o res -a" this showed pcscd using 2.3GB of RAM. As a work around I issued the "service pcscd onerestart" command and that seemed to free the allocated pages of ram and swap pcscd was using.

At the moment my RAM usage with QOS and snort running with the configuration as per above is 17% RAM used along with 0% swap being used.

I would like to figure out if this is an issue with FreeBSD 12.2 as a whole or just the pcscd process in isolation.

Thank you for your time and help with this issue.

SteveITS

@tman904 It's a known issue in pcscd for which there is a patch to disable it.

tman904

@steveits Thanks for the info Steve I appreciate it. It's good to see at the very least a work around is being implemented for this memory leak issue in pcscd.

tman904

Update:This will not survive a reboot.
For anyone else having this problem at least until this workaround is pushed out to pfsense widely.

I simply stopped and disabled pcscd as in my use case I didn't need to use smartcard readers if you do use smartcard readers don't run these commands.

Login to the shell via vga/serial console or ssh.
(As root or user with root privileges) service pcscd onestop
(As root or user with root privileges) service pcscd onedisable

Note pfsense does not use the standard rc.conf boot structure in FreeBSD. Because of that I'm not sure that this will survive a reboot or system upgrade. But it will prevent the system from running out of memory and crashing at least.

jimp

If you are using IPsec, doing that can be dangerous. You'd have to stop IPsec, then stop pcscd, then start pcscd, then start IPsec.

No need to use the CLI, there is a button to stop/start these services in the GUI, and you can use pfSsh.php playback svc stop <name> from the shell if you must.

The real workaround is already in several similar threads, and on https://redmine.pfsense.org/issues/11933#note-7, which is to apply the patch which makes pcscd optional and off by default.

SteveITS

@jimp said in pcscd using large amounts of ram and swap space on pfsense 2.5.2 CE:

patch which makes pcscd optional and off by default

If 21.09 is indeed being skipped, maybe Netgate should consider a point release with this? It seems like it would affect basically all installs.

tman904

@jimp Thank you jimp I will look into using those methods. I wasn't aware they were available. Now does pfSense 2.5.2 CE already have this patch applied or is there a patch update for 2.5.2 addressing this issue that I haven't installed?

Will the patch you linked be applied to the next release of pfSense CE then?

Also I'm not using IPsec VPNs but thank you for pointing that out.

SteveITS

@tman904 That's the patch I referenced above. :) No it's not in 2.5.2, and yes it should be in the next release as they generally include all fixes up until that point. It's actually listed in the 21.09 release notes, which would "pair" with 2.6, though it sounds like that version is getting skipped based on other forum threads.

tman904

@steveits Sorry about that Steve. :( I checked out the link for the redmine issue but for the life of me I can't find the patch for pcscd in order to apply it to my pfSense.

SteveITS

@tman904 The URL Jim linked is direct to the note with the patch commit ID (afcc0e9c97c1993ae6b95f886665fcb4375d26c7). Apply via System Patches. Or, in your case it sounds like you've already disabled it manually.

tman904

@steveits I did disable pcscd and it solves the issue as long as the system stays running after executing those commands. But as @jimp said I have to use the php commands in order to keep it disabled. As soon as I rebooted it starts again when using the traditional rc.conf boot commands.

I appreciate the guidance on using the patch I haven't patched a pfsense system before and had no clue where to start lol.

Update:
I've applied the pcscd patch and rebooted. Now it seems that pcscd is disabled properly! :)

One last question though. When I apply a patch to my system does that affect the ability to upgrade it through the normal builtin way or are there certain patches that could hinder that process I might apply in the future?

stephenw10

No. And in fact that patch is already included in newer versions. When you upgrade it will simply show as already applied and you can delete the patch from the System Patches package to prevent accidentally reverting it.

Steve

tman904

@stephenw10 Thank you Stephnew that clears up my confusion around updating and patching.

e-1-1

@jimp in my opinion, this issue warrants an advisory sent to users, and also a note in Known issues.

As an idea, I'd also love if advisories could be checked by internet facing boxes (those that can talk to the Netgate servers) and shown in the GUI and pushed via alerts to whatever is configured as alert system (Telegram for example).

lewis

This post is deleted!