Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    141 Posts 40 Posters 42.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ?
      A Former User
      last edited by

      Maybe avahi could cause problems, that I could understand.
      OpenVPN export isn't even called until you visit that page.

      1 Reply Last reply Reply Quote 0
      • G
        gusfersa
        last edited by gusfersa

        I upgrade six (6) pfsense production server at the same time from 2.4.4_p3, and I had problem with the conectivity. The ping time is very high above 7.000ms.

        I tried upgrade my pfsense server at home from 2.4.4_p3, but in this case I did a snapshot on vmware, and the problem is same. The ping time is very high and the navigation have a lot of problems.

        I restored the snapshot, and all return to normally

        At all server I have installed this packages:

        Open-VM-Tools
        openvpn-client-export
        squid
        snort
        zabbix-agent4

        I tried reinstall all packages, but the problem persist

        1 Reply Last reply Reply Quote 0
        • G
          Gektor
          last edited by Gektor

          Same troubles after upgrade from 2.4.4 to 2.4.5 on Hyper-V Windows Server 2019, 100% CPU usage (by pfctl process), long boot, and pfSense works with spikes and hangs.
          It seems that 2.4.5 not compatible with Hyper-V Windows Server 2019.
          Maybe it related:
          https://forum.netgate.com/topic/149595/2-4-5-a-20200110-1421-and-earlier-high-cpu-usage-from-pfctl/8

          1 Reply Last reply Reply Quote 0
          • T
            talaverde
            last edited by

            I'm having the same problem. I'm running 2.4.4.-p3 on Server 2016 with Hyper-V. I tried upgrading my 2nd CARP node to 2.4.5 yesterday, but it pegged the CPU and never became stable. I reverted that snapshot, shut it down and tried to upgrade my 1st CARP node, but the same problem. I've reverted both nodes to the snapshots.

            pfSense on Hyper-V has been rock solid up until now and all previous upgrades have been flawless.

            If I have time, I'll try installing a 2.4.5 VM from scratch to see if the problem occurs there too.

            1 Reply Last reply Reply Quote 0
            • G
              Gektor
              last edited by

              I have made clean reinstall system with catching config from updated system, first time boot was fast, then all packagers was restored (installed), after that system stuck at boot and lags after.
              Then i have found a source of problem — pfBlockerNG! When it's disabled, all works good, after enabling pfBlockerNG system lags totally.

              X 1 Reply Last reply Reply Quote 0
              • X
                xpxp2002 @Gektor
                last edited by

                @Gektor This is interesting. I had pfBlockerNG-devel installed on 2.4.4-p3. One of my earlier tests was to roll back to 2.4.4-p3, uninstall that package, then upgrade; and my system was still slow. Did you simply disable it, or uninstall the package?

                I will try this later today when I have an outage window.

                1 Reply Last reply Reply Quote 0
                • G
                  Gektor
                  last edited by Gektor

                  Mine is pfBlockerNG version 2.1.4_21, with this setting all works good:
                  7574e7a6-a678-4ee0-b6d6-5da00e69d698-изображение.png
                  Then i have disable all GeoIP lists, but enable DNSBL, and enable pfBlockerNG, and for now there is no problems with pfSense 2.4.5 on Hyper-V. System makes "crazy" when GeoIP lists is enabled in pfBlockerNG.
                  Have make post, maybe it will be helpful:
                  https://forum.netgate.com/topic/151726/pfblockerng-2-1-4_21-totally-lag-system-after-pfsense-upgrade-from-2-4-4-to-2-4-5

                  G 1 Reply Last reply Reply Quote 0
                  • G
                    gusfersa @Gektor
                    last edited by gusfersa

                    @Gektor I deleted all the installed packages:

                    Open-VM-Tools
                    openvpn-client-export
                    squid
                    snort
                    zabbix-agent4

                    and I disabled OpenVPN links unpriority; and the system conectivity was restored

                    G 1 Reply Last reply Reply Quote 0
                    • G
                      gusfersa @gusfersa
                      last edited by gusfersa

                      @gusfersa On another production server with the same installed packages, only I disabled OpenVPN link to an another pfsense server 2.4.5, and the system conectivity restored

                      1 Reply Last reply Reply Quote 0
                      • D
                        digitalgimpus
                        last edited by digitalgimpus

                        I've noticed something similar in terms of memory usage, but in my case cpu nice dropped in half and otherwise everything else seems status quo.

                        I'm not however noticing any latency outages or anything of that nature, but i've got plenty of free RAM so maybe that's the difference.

                        memory usage

                        T 1 Reply Last reply Reply Quote 0
                        • T
                          t41k2m3 @digitalgimpus
                          last edited by

                          @digitalgimpus said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                          I've noticed something similar in terms of memory usage, but in my case cpu nice dropped in half and otherwise everything else seems status quo.

                          I'm not however noticing any latency outages or anything of that nature, but i've got plenty of free RAM so maybe that's the difference.

                          memory usage

                          Same here, memory utilization spikes up from <20% before upgrade to 2.4.5 (w/all the same settings and packages) to 65-80% after upgrade.

                          Miscreant isolated to pfBlockerNG-devel (when uninstalled, memory use goes back to <20%) - running on netgate amd64 hardware, 8gb ram.

                          @BBcan177 any ideas on this, did this come up in the extensive testing done for 2.4.5? Any setting that could be tweaked (memory, feeds) or is this something that will require some coding/patching?

                          BBcan177B 1 Reply Last reply Reply Quote 0
                          • T
                            taz3146
                            last edited by

                            in quick testing here, it appears related to the pfblocker "maxmind GeoIP settings", either deleting the key or checking the box "disable maxmind csv database updates" makes the pfblocker pages respond near instantly again and gets rid of the long boot hang-time, which I'm assuming is breaking everything else and causing flapping in a loop as it keeps trying to reload it for high latency and other things!
                            I haven't tested further than that and cannot guarantee that's the only issue at hand, tested on minimal configured vm with nearly no traffic, but it slows it way down in many functions.

                            BBcan177B 1 Reply Last reply Reply Quote 0
                            • BBcan177B
                              BBcan177 Moderator @t41k2m3
                              last edited by

                              @t41k2m3
                              You are running on a physical machine and it looks like you are not experiencing any issues other than higher memory usage. That can be attributed to how many entries are in DNSBL, especially with TLD enabled. I assume it was the same as before but you didn't notice it. DNSBL in Unbound will create a pointer in memory for each domain and it can eat memory. Nothing I can do about that. The upcoming Unbound python integration will make a significant improvement in memory usage tho.

                              "Experience is something you don't get until just after you need it."

                              Website: http://pfBlockerNG.com
                              Twitter: @BBcan177  #pfBlockerNG
                              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                              T 1 Reply Last reply Reply Quote 0
                              • BBcan177B
                                BBcan177 Moderator @taz3146
                                last edited by

                                @taz3146
                                Are you in a virtualized environment as the others in this thread? There seems to be some issue with pfctl (which is used to create and update the IP aliases for the firewall rules) and with some virtualization software.
                                I have tested with VMware ESXi and can't reproduce these issues. Sent a message to the devs to see if the have any other guidance. Alternatively, setup a physical box with the same configuration and see if the problem exists without virtualization. Then we can attest narrow down the issue.
                                The deselection of settings in the IP tab should have no affect on anything. When you save that page it just writes settings to the config.xml and the nothing else. Probably you have something else happening in the background.
                                Would also suggest that everyone review the system.log and the pfblockerng.log for any other clues.

                                "Experience is something you don't get until just after you need it."

                                Website: http://pfBlockerNG.com
                                Twitter: @BBcan177  #pfBlockerNG
                                Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                ? 1 Reply Last reply Reply Quote 0
                                • ?
                                  A Former User @BBcan177
                                  last edited by

                                  @BBcan177 I agree. People blaming pfBlocker are missing the root cause of the problem, pfctl, not those apps/addons that feed it rules.

                                  1 Reply Last reply Reply Quote 0
                                  • ScottishTomS
                                    ScottishTom
                                    last edited by ScottishTom

                                    Just to add another data point : following upgrade to 2.4.5 from 2.4.4p3, I've noticed an increase in memory usage on a pfSense instance installed on a physical machine, but not any drastic increase in CPU usage. Memory usage jumped from ~7% to ~64% with no other changes bar the pfSense upgrade.

                                    Machine info : Intel J3160, 4GB DDR3, Dual Intel 82576EB NIC.

                                    Packages installed : openvpn-client-export, pfBlockerNG

                                    4c61e85d-a315-413e-a07e-28d0145b2e9b-image.png

                                    If any more info desired please just let me know.

                                    BBcan177B T 2 Replies Last reply Reply Quote 0
                                    • BBcan177B
                                      BBcan177 Moderator @ScottishTom
                                      last edited by

                                      @ScottishTom
                                      What version of the package? Would recommend devel and also try a reboot and see if that persists.
                                      Can also run these two commands to see what particular process is involved:

                                      ps auxwww 
                                      top -aSH
                                      

                                      "Experience is something you don't get until just after you need it."

                                      Website: http://pfBlockerNG.com
                                      Twitter: @BBcan177  #pfBlockerNG
                                      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                      ScottishTomS 1 Reply Last reply Reply Quote 0
                                      • T
                                        t41k2m3 @BBcan177
                                        last edited by

                                        @BBcan177 said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                                        @t41k2m3
                                        You are running on a physical machine and it looks like you are not experiencing any issues other than higher memory usage. That can be attributed to how many entries are in DNSBL, especially with TLD enabled. I assume it was the same as before but you didn't notice it. DNSBL in Unbound will create a pointer in memory for each domain and it can eat memory. Nothing I can do about that. The upcoming Unbound python integration will make a significant improvement in memory usage tho.

                                        @BBcan177 you are correct as to the summary of the situation, including same DNSBL entries, TLD on, only memory usage spikes (and not some of the other issues that seem to appear in virtualized environments). Not sure I'm following the theory of the case though. Meaning, given all else is equal (same pfS settings, same packages and their settings) but for the addition of pfS 2.4.5, it would reasonably follow (in fact proven by process of elimination) that some combination thereof (pfS 2.4.5 and pfB code/settings/others) begot a context writ large favoring these types of issues on different platforms. In fairness, there may be other contributing factors than pfB, though in this particular case, that is ostensibly not the case.

                                        So, question is what could/should/would we do about it? Re: unbound, the python integration is listed as a new feature/change (i.e. not upcoming, but present) and the Unbound 1.9.6 seems to be compiled with support for python. If that was/is intended to be the help/fix, not sure that it is performing quite as hoped. Recognizing this is brand new and may need some burnishing, wanted to get it on the radar screen for you and pfS devs. Thanks for all your efforts.

                                        BBcan177B 1 Reply Last reply Reply Quote 0
                                        • T
                                          t41k2m3 @ScottishTom
                                          last edited by

                                          @ScottishTom said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                                          Just to add another data point : following upgrade to 2.4.5 from 2.4.4p3, I've noticed an increase in memory usage on a pfSense instance installed on a physical machine, but not any drastic increase in CPU usage. Memory usage jumped from ~7% to ~64% with no other changes bar the pfSense upgrade.

                                          Machine info : Intel J3160, 4GB DDR3, Dual Intel 82576EB NIC.

                                          Packages installed : openvpn-client-export, pfBlockerNG

                                          4c61e85d-a315-413e-a07e-28d0145b2e9b-image.png

                                          If any more info desired please just let me know.

                                          This seems like virtually the same or similar setup and problem as previously described (with qualification that a process at fault was not yet identified/hypothesized).

                                          1 Reply Last reply Reply Quote 0
                                          • BBcan177B
                                            BBcan177 Moderator @t41k2m3
                                            last edited by BBcan177

                                            @t41k2m3
                                            I posted above two commands that you can use to find what is using memory. Report back with what you find. I haven't spent much time with the release of 2.4.5 as things have been hectic. I haven't checked if the version of Unbound has changed from 2.4.3/4. That might be a reason if something has changed in the Resolver code.
                                            In regards to the upcoming Unbound python integration, what you see in the Resolver settings will allow for a future release to integrate with the Resolver. It's just the plumbing and nothing else. There is no Python integration released yet.

                                            "Experience is something you don't get until just after you need it."

                                            Website: http://pfBlockerNG.com
                                            Twitter: @BBcan177  #pfBlockerNG
                                            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.