Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Note on 1100 memory usage with 23.01

    Official Netgate® Hardware
    4
    9
    517
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      BGSmith last edited by

      A short note on memory usage with an 1100 after upgrading to 23.01.

      I upgraded the 1100 yesterday (5 Mar 2023) when the web UI indicated an update was available. 23.01 beta was installed. Memory usage was at 85 or 88% and stayed there overnight.

      Updated this morning to 23.01 (final) and after the upgrade reboot and restart memory usage was still at 88%. This was much higher that with the prior OS version.

      After an hour I rebooted again. When the 1100 was back online memory usage was at 33% which is more like before.

      Not sure if this is expected - the high usage or the drop after reboot - as the 1100 has been in use only 1 month more or less and is my first experience with pfsense. In any case, I thought I would drop this note.

      thanks

      S 1 Reply Last reply Reply Quote 0
      • S
        SteveITS @BGSmith last edited by SteveITS

        @bgsmith First, it shouldn’t have installed a beta since 23.01 was released a few weeks ago. Was yours set to the current/stable update branch? If you’ve gone past 23.01 there’s not a way to downgrade without reinstalling.

        There are a few long threads about ZFS memory usage. There is a patch to disable some unnecessary cron tasks (System Patches package, add patch, restart) that trigger a memory increase at 3:00 am. Discussion/patch:
        https://forum.netgate.com/topic/177886/23-1-using-more-ram/72

        Steve

        Only install packages for your version, or risk breaking it. If yours is older, select it in System/Update/Update Settings.
        When upgrading, let it finish; do not reboot early. Allow 10-15 minutes, or more depending on packages and device speed.

        B 1 Reply Last reply Reply Quote 0
        • B
          BGSmith @SteveITS last edited by

          @steveits said in Note on 1100 memory usage with 23.01:

          https://forum.netgate.com/topic/177886/23-1-using-more-ram/72

          Thanks @SteveITS. I think that there was a pause/delay on the release of 23.01 for the 1100 and one other model to correct some problems. Yesterday was the first day the web UI indicated the update was available. I have been checking daily since 23.01 was released. Not sure why beta was installed first as the general release was available soon after. Probably user (me) error!

          This 1100 has a pretty generic configuration so far with no extra packages. I appreciate the link to the ZFS thread and will take a close look.

          best regards

          S B 2 Replies Last reply Reply Quote 0
          • S
            SteveITS @BGSmith last edited by

            @bgsmith There was a hold. Now that I’m caffeinated, yeah an upgrade shouldn’t be offered unless they just allowed it again. Models purchased after April 2021, I think it was, aren’t affected.

            Read through that thread. Overall ZFS ARC memory is supposed to be released as needed. It just looks like a big jump in usage especially on an 1100.

            Steve

            Only install packages for your version, or risk breaking it. If yours is older, select it in System/Update/Update Settings.
            When upgrading, let it finish; do not reboot early. Allow 10-15 minutes, or more depending on packages and device speed.

            1 Reply Last reply Reply Quote 1
            • B
              BGSmith @BGSmith last edited by

              @bgsmith

              Thanks again for the thread link. I duplicated the settings there by bingo600 to produce the following graph. The long section of elevated memory is the overnight usage of 23.01 beta, followed by the update to 23.01 final (still with elevated memory usage) and then the drop after reboot.

              best regards

              pfsense-1100-memory-across-upgrad.png

              S 1 Reply Last reply Reply Quote 0
              • S
                SteveITS @BGSmith last edited by SteveITS

                @bgsmith The patch should fix the 3am issue and the ZFS tuneable sets a memory limit.

                Steve

                Only install packages for your version, or risk breaking it. If yours is older, select it in System/Update/Update Settings.
                When upgrading, let it finish; do not reboot early. Allow 10-15 minutes, or more depending on packages and device speed.

                1 Reply Last reply Reply Quote 2
                • stephenw10
                  stephenw10 Netgate Administrator last edited by

                  If you were already on 23.01beta you would be able to see the update to 23.01rel because you would already have the dynamic pkg repo code. And your device will have already passed the EFI update issue.
                  Upgrades from 22.05 or earlier are still disabled while we test the new upgrade code.

                  Steve

                  B 1 Reply Last reply Reply Quote 1
                  • Referenced by  S SteveITS 
                  • Referenced by  S SteveITS 
                  • B
                    BGSmith @stephenw10 last edited by BGSmith

                    @stephenw10 - thanks for the update. I am not 100% sure how I got the beta. I was following some guidelines that indicated selecting the development branch and then the stable branch to trigger an update. I suspect I somehow left the setting on development and triggered the upgrade to 20.01 beta.

                    In any case, I applied the patch that @SteveITS referenced yesterday and all seems to be functioning correctly. On the graph image below the first bump in 'wire' memory is 23,01beta overnight. The second, narrower bump is overnight with 23.01. I applied the patch and rebooted so last night did not see the 0300 bump.

                    pfsense-memory-trace2.png

                    best regards

                    1 Reply Last reply Reply Quote 1
                    • jimp
                      jimp Rebel Alliance Developer Netgate last edited by jimp

                      This seems like a good time to try out my shiny new ZFS memory usage boilerplate message.

                      23.01 can appear to use more memory than expected if you are using ZFS for a few different reasons:

                      • ZFS ARC (Adaptive Replacement Cache) usage is reported differently, though the behavior is similar to older versions #14011. ZFS will give up this cache memory as needed but not always fast enough for certain workloads.

                        To correct the memory reporting on graphs and the dashboard, install the System Patches package and then create an entry for 0d83ed084a987f3446a0cbdcf249fc5b8722726f to apply the fix.

                      • The first boot after upgrading the ARC usage will be high from all the disk activity during the upgrade. This will gradually lower, or a reboot will make it go back to "normal".

                      • FreeBSD default cron jobs are enabled when they shouldn't be #14016, and run at 3am causing RAM usage from ARC to jump overnight.

                        To correct this, install the System Patches package and then create an entry for ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 to apply the fix.

                      • To set limits on ZFS ARC size to reduce the chance of contention for RAM, configure a tunable for vfs.zfs.arc.max to set a limit on how many bytes it can consume, e.g. 67108864 would set a 64MB limit. Also consider setting a "Free Target". Both are explained in detail on https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html.

                      A reboot is required to fully activate the changes from these patches. The ZFS ARC limit will take effect immediately but the wired memory may not be immediately freed, so a reboot is helpful for that as well.

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 5
                      • Referenced by  B bigsy 
                      • Referenced by  B bigsy 
                      • Referenced by  B bigsy 
                      • Referenced by  S stephen.betts 
                      • First post
                        Last post