Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    1100 upgrade, 22.05->23.01, high mem usage

    Scheduled Pinned Locked Moved General pfSense Questions
    74 Posts 12 Posters 51.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DefenderLLCD
      DefenderLLC @jrey
      last edited by

      @jrey I totally agree, but this would also not be the first time that pfBlocker had a memory leak after an update and that package was definitely updated during the upgrade to 23.01. I'll know for sure tomorrow morning!

      1 Reply Last reply Reply Quote 0
      • beerguzzleB
        beerguzzle
        last edited by

        I just applied the pfblockerng update, going from 3.2.0.1 to 3.2.0.2. I did not reboot. The wired mem usage jumped from 55% to 63%.

        Netgate 1100 and Netgate 2100, latest pfsense+ version

        J 2 Replies Last reply Reply Quote 0
        • J
          jrey @beerguzzle
          last edited by

          @beerguzzle

          question will be, after the new version settles in, does that new memory come down?

          is it's not 3am where you are is it?

          The specific (and only) increase I have ever seen, is always and only at 3am, like clockwork.
          At that time it jumps up a good 20% and then stays there until the next system restart. Regardless of other activity on the system. Look at the graph I posted memory is a pretty flat line. If the memory usage were a heart rhythm I'd be worried it is so flat.

          Granted I also see some posts where others have memory usage with really wild fluctuations, they are usually running a lot more stuff on the box than I.

          In my case I don't think it has anything to do with Acme, or pfBlockerNG for the reasons and timing of the events i've previously reported.

          1 Reply Last reply Reply Quote 0
          • R
            rpsmith @keyser
            last edited by

            @keyser -- the main reason I went with netgate hardware was this firewall was going to be located 750 miles and 3 states away from me and I was led to believe netgate would be testing all new releases on their different hardware platforms prior to releasing new versions.

            That clearly doesn't appear to be the case as my SG-1100 is a "Plain Jane" as you could possibly find and there is a major flaw in 23.01 on their hardware!

            Roy...

            R 1 Reply Last reply Reply Quote 0
            • J
              jrey @beerguzzle
              last edited by

              @beerguzzle

              So here you go. in my case the issue is 100% cause by "periodic daily" that runs at 3:00am

              Screen Shot 2023-02-21 at 2.34.12 PM.png

              left of the red line is where it was after last night's spike, and represents the point at which I restarted the system now.
              Green line mark is showing the re-establishing of what I would consider the "normal base line" after a reboot and before the next 3:00am cycle
              Blue line is me "pretending" it is currently 3:00am and running "period daily" from an ssh session...

              there goes the memory.

              periodic daily is the default "periodic" provided by freebsd (I don't see any comments with to say that pfsense has modified it)
              the other /etc/rc.periodic that also normally runs at 3:00 seems to be the pfsense specific stuff. I didn't even run it in this tests be well the memory is already consumed !

              Sorry I may have captured and marked up the first screen capture too soon to show the flat line after periodic daily completed

              Screen Shot 2023-02-21 at 2.47.30 PM.png

              DefenderLLCD keyserK 2 Replies Last reply Reply Quote 0
              • DefenderLLCD
                DefenderLLC @jrey
                last edited by

                @jrey Same here on my 6100. That's the only cron job that kicks off at 3AM besides the hourly pfBlocker updates.

                1 Reply Last reply Reply Quote 0
                • keyserK
                  keyser Rebel Alliance @jrey
                  last edited by

                  @jrey Given that Cached memory goes "equally" down, could it be a performance optimization i FreeBSD 14 that evaluates the free memory available in the box and decides to increase wired memory use to increase performance instead of touching disk (and maybe/maybe not hitting cache memory) again and again?

                  Love the no fuss of using the official appliances :-)

                  J 2 Replies Last reply Reply Quote 0
                  • J
                    jrey @keyser
                    last edited by

                    @keyser

                    Very possible in my case - because the system is very flat lined with regards to memory. I was just providing the information for those that have a more scattered memory with huge swings. This "feature" would mask what is going on for some of them, because they are potentially seeing this huge swing on top of their systems "normal" up and downs.

                    I've said all along that my system is not stressed with regards to memory in the current configuration and I'm not concerned. I should have also said "likely normal" (my bad)

                    Thanks, JR

                    DefenderLLCD 1 Reply Last reply Reply Quote 0
                    • DefenderLLCD
                      DefenderLLC @jrey
                      last edited by DefenderLLC

                      @jrey said in 1100 upgrade, 22.05->23.01, high mem usage:

                      @keyser

                      Very possible in my case - because the system is very flat lined with regards to memory. I was just providing the information for those that have a more scattered memory with huge swings. This "feature" would mask what is going on for some of them, because they are potentially seeing this huge swing on top of their systems "normal" up and downs.

                      I've said all along that my system is not stressed with regards to memory in the current configuration and I'm not concerned. I should have also said "likely normal" (my bad)

                      Thanks, JR

                      it's not your device. My 6100 MAX is doing the EXACT same thing at the EXACT same time since upgrading to 23.01. This is what I see after a clean reboot and at 3AM:

                      IMG_1584.jpeg

                      1 Reply Last reply Reply Quote 0
                      • S SteveITS referenced this topic on
                      • J
                        jrey @keyser
                        last edited by jrey

                        @keyser said in 1100 upgrade, 22.05->23.01, high mem usage:

                        Given that Cached memory goes "equally" down

                        check again, is not cached memory going down, it is the free memory that goes down by an "equal" amount - cache is actually zero in my case.

                        Screen Shot 2023-02-22 at 7.30.47 AM.png

                        Still could be "optimization" as part of the "periodic daily", just not optimization of cache as suggested.

                        Thanks

                        Edit: I wonder if..

                        when I ran the periodic daily manually in my test it complained that mail was not available.

                        "By default, the output from running the periodic scripts is emailed to the root user on the local system"

                        this can't happen on the NetGate. So makes me think the memory might be the reports that it wants to mail, but can't. The reason it then never consumes more memory, is the reports themselves take the same "footprint" everyday.
                        There is a configuration setting in periodic's conf to tell the system not to mail these reports at all. Checking this now, if they are on, I'll turn them off and re-test or redirect them with conf option
                        email is likely a poor choice but they can go to a path as well
                        daily_output=”some_address@someplacevalid.com"
                        might be an option if this is in fact what is going on.

                        keyserK 1 Reply Last reply Reply Quote 0
                        • DefenderLLCD
                          DefenderLLC
                          last edited by DefenderLLC

                          Upgraded to pfBlockerNG 3.2.0_2 yesterday, disabled Suricata completely, and rebooted around 9:30 AM yesterday morning. The memory utilization doubled again at exactly 3AM. The only cron job running at 3AM is the daily periodical and hourly pfBlocker updates (which there wasn't any lists to update today at 3AM).

                          I've also attached my system log from 2AM to 6AM and there is nothing worthwhile in there to explain what is happening at 3AM besides the daily periodical.

                          98fc3a21-a003-4987-9005-a4cf7e76ba67-image.png

                          ddcb7f16-2a16-4264-8e0a-e1519023db97-image.png

                          J 1 Reply Last reply Reply Quote 0
                          • S SteveITS referenced this topic on
                          • S SteveITS referenced this topic on
                          • J
                            jrey @DefenderLLC
                            last edited by

                            @defenderllc

                            it is not pfB, my tasks for that don't run at 3am and another user on here doesn't even have it installed. So I think safe to rule that out.

                            I've been going back through "periodic daily" it is clearly something in there.
                            going on the theory that out of box is set to email the daily, weekly, monthly (and security) reports and but can't - this is the result of testing in that regard.

                            I've done the following test since my previous post earlier today.
                            create a file /etc/periodic.conf.local

                            in that file place these 6 lines

                            daily_output=/var/log/daily.log
                            daily_status_security_output=/var/log/daily_security.log
                            weekly_output=/var/log/weekly.log
                            weekly_status_security_output=/var/log/weekly_security.log
                            monthly_output=/var/log/monthly.log
                            monthly_status_security_output=/var/log/monthly_security.log

                            in the first test I only had the 3 lines for normal output(not security)
                            wanted to see if it was one or the other (ie normal or security logs that might in play..
                            then the second test (with security logs also redirected to files),

                            Screen Shot 2023-02-22 at 8.57.19 AM.png

                            graph shows only free and wire

                            making this change did have a minor impact on the footprint but not enough to worry about. but at least now I can look at these reports if needed, not a bad thing.

                            since the "periodic daily" just runs a bunch of scripts and each can be disable (and a lot are disabled by default) I now looking at the conf "enabled" ones to see if there is "one that is most likely to"..

                            DefenderLLCD 1 Reply Last reply Reply Quote 0
                            • DefenderLLCD
                              DefenderLLC @jrey
                              last edited by DefenderLLC

                              @jrey said in 1100 upgrade, 22.05->23.01, high mem usage:

                              @defenderllc

                              it is not pfB, my tasks for that don't run at 3am and another user on here doesn't even have it installed. So I think safe to rule that out.

                              I've been going back through "periodic daily" it is clearly something in there.
                              going on the theory that out of box is set to email the daily, weekly, monthly (and security) reports and but can't - this is the result of testing in that regard.

                              I've done the following test since my previous post earlier today.
                              create a file /etc/periodic.conf.local

                              in that file place these 6 lines

                              daily_output=/var/log/daily.log
                              daily_status_security_output=/var/log/daily_security.log
                              weekly_output=/var/log/weekly.log
                              weekly_status_security_output=/var/log/weekly_security.log
                              monthly_output=/var/log/monthly.log
                              monthly_status_security_output=/var/log/monthly_security.log

                              in the first test I only had the 3 lines for normal output(not security)
                              wanted to see if it was one or the other (ie normal or security logs that might in play..
                              then the second test (with security logs also redirected to files),

                              Screen Shot 2023-02-22 at 8.57.19 AM.png

                              graph shows only free and wire

                              making this change did have a minor impact on the footprint but not enough to worry about. but at least now I can look at these reports if needed, not a bad thing.

                              since the "periodic daily" just runs a bunch of scripts and each can be disable (and a lot are disabled by default) I now looking at the conf "enabled" ones to see if there is "one that is most likely to"..

                              Check out the newest entries in this thread. They are trying to figure out what is happening during the daily periodic...

                              https://forum.netgate.com/topic/177886/23-1-using-more-ram/56?_=1677011443983

                              J 1 Reply Last reply Reply Quote 0
                              • keyserK
                                keyser Rebel Alliance @jrey
                                last edited by

                                @jrey said in 1100 upgrade, 22.05->23.01, high mem usage:

                                @keyser said in 1100 upgrade, 22.05->23.01, high mem usage:

                                Given that Cached memory goes "equally" down

                                check again, is not cached memory going down, it is the free memory that goes down by an "equal" amount - cache is actually zero in my case.

                                Ooopps, my bad :-)

                                True that.

                                Love the no fuss of using the official appliances :-)

                                1 Reply Last reply Reply Quote 0
                                • R
                                  rpsmith @rpsmith
                                  last edited by

                                  @rpsmith -- and I forgot to mention it bricked my 1100 when I attempted to upgrade to 23.01!

                                  J S 2 Replies Last reply Reply Quote 0
                                  • J
                                    jrey @rpsmith
                                    last edited by

                                    @rpsmith

                                    Welp

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      SteveITS Galactic Empire @rpsmith
                                      last edited by

                                      @rpsmith That's probably the known issue which is why they halted upgrades on those two models. Per other threads, even after "knowing" what the problem was, Netgate has been unable to replicate it, at least as of a couple days ago.

                                      It's the hard way to learn the lesson but I always try to let updates simmer for a week or two, just in case something unforeseen happens. FWIW that's the first widespread problem like that I can recall in ~15 years of using pfSense for ourselves and our clients.

                                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                      Upvote 👍 helpful posts!

                                      R 1 Reply Last reply Reply Quote 0
                                      • J
                                        jrey @DefenderLLC
                                        last edited by

                                        @defenderllc

                                        Got it!
                                        Test restart system establish base line
                                        run the scripts as listed in order manually
                                        (I've listed them all, even though some of them are enabled but don't actually exist, which causes nothing to happen so meh)

                                        I've not included their full name, just the number (except for 480 because there are 2)

                                        100 not enabled
                                        110 not enabled
                                        120 yes - no impact
                                        130 enabled, does not exist
                                        140 enabled, does not exist
                                        150 enabled, does not exist
                                        200 yes - no impact
                                        210 yes - no impact
                                        211 yes - no impact
                                        222 not enabled
                                        223 not enabled
                                        300 not enabled
                                        310 yes - no impact
                                        400 yes - no impact
                                        401 not enabled
                                        404 not enabled
                                        406 not enabled
                                        409 not enabled
                                        410 not enabled
                                        420 yes - no impact
                                        430 yes - no impact
                                        440 enabled, does not exist

                                        450 yes - yes impact

                                        460 enabled - does not exist
                                        480.leapfile-ntpd enabled - does not exist
                                        480.status-ntpd - not enable, does not exist
                                        500 enabled, does not exist
                                        510 enabled, no impact
                                        800 not enabled
                                        999 there are no local scripts in the directory specified.

                                        Screen Shot 2023-02-22 at 10.59.49 AM.png

                                        Notice how after restarting the memory remained unchanged for some time until I ran script 450-status-security (it complete and wrote its own log file as expected, but also claimed the memory)
                                        most of these scripts don't actually do that much, but I still wait a little more than a minute to insure the graph reflected the results of each step.

                                        Next step - just to verify, I'll disable 450 in the conf and run the entire "periodic daily"

                                        okay I'm back - at 11:16 on this final graph after system had established flat line, I ran periodic daily (450 disabled)

                                        no lost memory!!

                                        Screen Shot 2023-02-22 at 11.21.04 AM.png

                                        JR

                                        keyserK S 2 Replies Last reply Reply Quote 1
                                        • R
                                          rpsmith @SteveITS
                                          last edited by rpsmith

                                          @steveits -- I normally do that as well but I made an exception in this case because I was upgrading a netgate box with a very "Plain Jane" (no packages added) config. However, I will definitely follow your advice in the future though!

                                          Roy...

                                          1 Reply Last reply Reply Quote 0
                                          • keyserK
                                            keyser Rebel Alliance @jrey
                                            last edited by

                                            @jrey and what does the 450 script do?

                                            Love the no fuss of using the official appliances :-)

                                            J 1 Reply Last reply Reply Quote 0
                                            • S SteveITS referenced this topic on
                                            • S SteveITS referenced this topic on
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.