Note on 1100 memory usage with 23.01

BGSmith

A short note on memory usage with an 1100 after upgrading to 23.01.

I upgraded the 1100 yesterday (5 Mar 2023) when the web UI indicated an update was available. 23.01 beta was installed. Memory usage was at 85 or 88% and stayed there overnight.

Updated this morning to 23.01 (final) and after the upgrade reboot and restart memory usage was still at 88%. This was much higher that with the prior OS version.

After an hour I rebooted again. When the 1100 was back online memory usage was at 33% which is more like before.

Not sure if this is expected - the high usage or the drop after reboot - as the 1100 has been in use only 1 month more or less and is my first experience with pfsense. In any case, I thought I would drop this note.

thanks

SteveITS

@bgsmith First, it shouldn’t have installed a beta since 23.01 was released a few weeks ago. Was yours set to the current/stable update branch? If you’ve gone past 23.01 there’s not a way to downgrade without reinstalling.

There are a few long threads about ZFS memory usage. There is a patch to disable some unnecessary cron tasks (System Patches package, add patch, restart) that trigger a memory increase at 3:00 am. Discussion/patch:
https://forum.netgate.com/topic/177886/23-1-using-more-ram/72

BGSmith

@steveits said in Note on 1100 memory usage with 23.01:

https://forum.netgate.com/topic/177886/23-1-using-more-ram/72

Thanks @SteveITS. I think that there was a pause/delay on the release of 23.01 for the 1100 and one other model to correct some problems. Yesterday was the first day the web UI indicated the update was available. I have been checking daily since 23.01 was released. Not sure why beta was installed first as the general release was available soon after. Probably user (me) error!

This 1100 has a pretty generic configuration so far with no extra packages. I appreciate the link to the ZFS thread and will take a close look.

best regards

SteveITS

@bgsmith There was a hold. Now that I’m caffeinated, yeah an upgrade shouldn’t be offered unless they just allowed it again. Models purchased after April 2021, I think it was, aren’t affected.

Read through that thread. Overall ZFS ARC memory is supposed to be released as needed. It just looks like a big jump in usage especially on an 1100.

BGSmith

@bgsmith

Thanks again for the thread link. I duplicated the settings there by bingo600 to produce the following graph. The long section of elevated memory is the overnight usage of 23.01 beta, followed by the update to 23.01 final (still with elevated memory usage) and then the drop after reboot.

best regards

SteveITS

@bgsmith The patch should fix the 3am issue and the ZFS tuneable sets a memory limit.

stephenw10

If you were already on 23.01beta you would be able to see the update to 23.01rel because you would already have the dynamic pkg repo code. And your device will have already passed the EFI update issue.
Upgrades from 22.05 or earlier are still disabled while we test the new upgrade code.

Steve

BGSmith

@stephenw10 - thanks for the update. I am not 100% sure how I got the beta. I was following some guidelines that indicated selecting the development branch and then the stable branch to trigger an update. I suspect I somehow left the setting on development and triggered the upgrade to 20.01 beta.

In any case, I applied the patch that @SteveITS referenced yesterday and all seems to be functioning correctly. On the graph image below the first bump in 'wire' memory is 23,01beta overnight. The second, narrower bump is overnight with 23.01. I applied the patch and rebooted so last night did not see the 0300 bump.

best regards

jimp

This seems like a good time to try out my shiny new ZFS memory usage boilerplate message.

23.01 can appear to use more memory than expected if you are using ZFS for a few different reasons:

ZFS ARC (Adaptive Replacement Cache) usage is reported differently, though the behavior is similar to older versions #14011. ZFS will give up this cache memory as needed but not always fast enough for certain workloads.

To correct the memory reporting on graphs and the dashboard, install the System Patches package and then create an entry for 0d83ed084a987f3446a0cbdcf249fc5b8722726f to apply the fix.
The first boot after upgrading the ARC usage will be high from all the disk activity during the upgrade. This will gradually lower, or a reboot will make it go back to "normal".
FreeBSD default cron jobs are enabled when they shouldn't be #14016, and run at 3am causing RAM usage from ARC to jump overnight.

To correct this, install the System Patches package and then create an entry for ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 to apply the fix.
To set limits on ZFS ARC size to reduce the chance of contention for RAM, configure a tunable for vfs.zfs.arc.max to set a limit on how many bytes it can consume, e.g. 67108864 would set a 64MB limit. Also consider setting a "Free Target". Both are explained in detail on https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html.

A reboot is required to fully activate the changes from these patches. The ZFS ARC limit will take effect immediately but the wired memory may not be immediately freed, so a reboot is helpful for that as well.