23.01 wired memory climbs to 100%

[keyser]

There is an issue with 23.01 an wired memory usage/reporting.....
This is probably also the real issue behind the memory usage of the 3:00am cron job

I have been using the TFTP Package on a remote site for allowing me to upload a backup image of a Raspberry Pi to my pfSense's TFTP root.
With 23.01 the wired memory use on my SG-6100 climbs along with the size of the uploaded backup file until memory use hits 100%. Killing the upload, restarting TFTP or deleting the uploaded file does not release the memory again. Every monitoring system I have says 100% memory used - I have not tried "squeezing" the upload further to see if it starts swapping and slowing down because I have a lot of users on this system.

I'm pretty certain the issue is related to the ZFS ARC, because @jimp suggested I tried creating a turnable called vfs.zfs.arc_max and setting it 256Mb.
It has no immediate effect on the 100% wired memory usage, but after a loooong while memory will start to free up (when new data is written to the disk). If I do a "dd if=/dev/random of=./filex bs=2Gb count=1", then after creating the 2Gb file almost all the wired memory is released again, and it seems to fit very well with only 256MB being occupied as stated in the turnable.
After this, uploading with TFTP again does not increase wired memory usage above the 256Mb.

It might be that the system is given/freed enough memory from the ARC if memory is requested, but cache like that should not be reported as fixed/wired memory usage. It triggers all kinds of memory alarms on my monitoring systems.
I seem to have fixed the problem with the turnable, but perhaps there should be made some changes to this ARC behaviour....

[keyser]

@keyser I should mention that this site was on UFS until a clean 23.01 ZFS install, so I cannot say if a ZFS installed 22.05 would exibit the same behaviour :-)

[jimp]

"Wired" memory is memory which cannot be paged to disk and that includes caches and buffers, so it absolutely has to be reported the way it is. Wired doesn't mean it's permanently used or anything dangerous, just that when the system is looking to swap things under pressure it can't move those pages.

As you saw, the memory will be freed but not always immediately. It will happen over time (and sooner if certain other thresholds of memory usage are crossed). Though that doesn't always happen as fast as people want it to. If something large tries to start and allocate a bunch of RAM quickly then it may not get what it wants (or may swap a lot).

The best thing to do is to hardcode a limit on the ARC size as you have done (which apparently is now at vfs.zfs.arc.max though the older vfs.zfs.arc_max still works, they haven't removed it yet).

ARC is supposed to automatically be limited to the greater of "all but 1GB of RAM" or "half of RAM" but depending on packages installed and whatnot leaving only 1GB of RAM untouched is likely going to not be enough to run what people want, hence the need for manual tuning.

[keyser]

@jimp I will update to the new vfs.zfs.arc.max so it works going forward.

But just to clarify: The imposed limitation you describe is not there.
My SG-6100 hovers around 15% RAM usage (15% of 8Gib) after a reboot. When I do the TFTP upload without the ARC limiter, it goes to 100% Wired memory usage - all 8Gib is consumed.

[jimp]

@keyser said in 23.01 wired memory climbs to 100%:

@jimp I will update to the new vfs.zfs.arc.max so it works going forward.

But just to clarify: The imposed limitation you describe is not there.
My SG-6100 hovers around 15% RAM usage (15% of 8Gib) after a reboot. When I do the TFTP upload without the ARC limiter, it goes to 100% Wired memory usage - all 8Gib is consumed.

But how much of that is ARC vs other wired usage? Not all wired usage is ARC. If there is enough wired usage (>1GB) then the combined total of ARC and non-ARC wired usage could reach that high.

[keyser]

@jimp Well I can't tell now since it would require me to back of my edits and test again (not currently possible).

But I assume that the "suddenly" used remaining 6.8Gb memory is all dedicated to ARC. Nothing else happened during that test, and stopping the TFTP daemon afterwards changed nothing (So that did not use memory)

[jimp]

How much wired usage does the system show now? And how much ARC?

Stopping daemons is unlikely to have any affect on ARC since it's disk cache, it's not associated with any specific process. If a daemon wired some memory of its own it would free that, but mostly it's not typical daemon type memory usage that ends up being wired, but things like caches, buffers, the kernel itself, anything that consumes kernel memory, etc.

[keyser]

@jimp After implementing the 256MB arc.max size, the total memory usage is again back to 15-16% of 8Gib, and the ARC hovers persistently between 249-254MB. So i definitively works.

On another note: I have not disabled the cron jobs with the patch, so the memory usage caused by those cron jobs are definitively also arc related as I'm not seeing increased memory usage at 03:00am any more.

[Dobby_]

@keyser

Thanks for the reply! I got lower ram usage like before
and on top a lower cou and swap usage. (After applying
the patch and rebooting I mean)

CPU was on ~30 - 60%
ram was on nearly ~91 - 94%
swap was from ~50 - 100%

And on top I never have seen the message (swap pager: out of space)

All is fine now for me.