Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    141 Posts 40 Posters 41.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BBcan177B
      BBcan177 Moderator @t41k2m3
      last edited by

      @t41k2m3
      You are running on a physical machine and it looks like you are not experiencing any issues other than higher memory usage. That can be attributed to how many entries are in DNSBL, especially with TLD enabled. I assume it was the same as before but you didn't notice it. DNSBL in Unbound will create a pointer in memory for each domain and it can eat memory. Nothing I can do about that. The upcoming Unbound python integration will make a significant improvement in memory usage tho.

      "Experience is something you don't get until just after you need it."

      Website: http://pfBlockerNG.com
      Twitter: @BBcan177  #pfBlockerNG
      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

      T 1 Reply Last reply Reply Quote 0
      • BBcan177B
        BBcan177 Moderator @taz3146
        last edited by

        @taz3146
        Are you in a virtualized environment as the others in this thread? There seems to be some issue with pfctl (which is used to create and update the IP aliases for the firewall rules) and with some virtualization software.
        I have tested with VMware ESXi and can't reproduce these issues. Sent a message to the devs to see if the have any other guidance. Alternatively, setup a physical box with the same configuration and see if the problem exists without virtualization. Then we can attest narrow down the issue.
        The deselection of settings in the IP tab should have no affect on anything. When you save that page it just writes settings to the config.xml and the nothing else. Probably you have something else happening in the background.
        Would also suggest that everyone review the system.log and the pfblockerng.log for any other clues.

        "Experience is something you don't get until just after you need it."

        Website: http://pfBlockerNG.com
        Twitter: @BBcan177  #pfBlockerNG
        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

        ? 1 Reply Last reply Reply Quote 0
        • ?
          A Former User @BBcan177
          last edited by

          @BBcan177 I agree. People blaming pfBlocker are missing the root cause of the problem, pfctl, not those apps/addons that feed it rules.

          1 Reply Last reply Reply Quote 0
          • ScottishTomS
            ScottishTom
            last edited by ScottishTom

            Just to add another data point : following upgrade to 2.4.5 from 2.4.4p3, I've noticed an increase in memory usage on a pfSense instance installed on a physical machine, but not any drastic increase in CPU usage. Memory usage jumped from ~7% to ~64% with no other changes bar the pfSense upgrade.

            Machine info : Intel J3160, 4GB DDR3, Dual Intel 82576EB NIC.

            Packages installed : openvpn-client-export, pfBlockerNG

            4c61e85d-a315-413e-a07e-28d0145b2e9b-image.png

            If any more info desired please just let me know.

            BBcan177B T 2 Replies Last reply Reply Quote 0
            • BBcan177B
              BBcan177 Moderator @ScottishTom
              last edited by

              @ScottishTom
              What version of the package? Would recommend devel and also try a reboot and see if that persists.
              Can also run these two commands to see what particular process is involved:

              ps auxwww 
              top -aSH
              

              "Experience is something you don't get until just after you need it."

              Website: http://pfBlockerNG.com
              Twitter: @BBcan177  #pfBlockerNG
              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

              ScottishTomS 1 Reply Last reply Reply Quote 0
              • T
                t41k2m3 @BBcan177
                last edited by

                @BBcan177 said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                @t41k2m3
                You are running on a physical machine and it looks like you are not experiencing any issues other than higher memory usage. That can be attributed to how many entries are in DNSBL, especially with TLD enabled. I assume it was the same as before but you didn't notice it. DNSBL in Unbound will create a pointer in memory for each domain and it can eat memory. Nothing I can do about that. The upcoming Unbound python integration will make a significant improvement in memory usage tho.

                @BBcan177 you are correct as to the summary of the situation, including same DNSBL entries, TLD on, only memory usage spikes (and not some of the other issues that seem to appear in virtualized environments). Not sure I'm following the theory of the case though. Meaning, given all else is equal (same pfS settings, same packages and their settings) but for the addition of pfS 2.4.5, it would reasonably follow (in fact proven by process of elimination) that some combination thereof (pfS 2.4.5 and pfB code/settings/others) begot a context writ large favoring these types of issues on different platforms. In fairness, there may be other contributing factors than pfB, though in this particular case, that is ostensibly not the case.

                So, question is what could/should/would we do about it? Re: unbound, the python integration is listed as a new feature/change (i.e. not upcoming, but present) and the Unbound 1.9.6 seems to be compiled with support for python. If that was/is intended to be the help/fix, not sure that it is performing quite as hoped. Recognizing this is brand new and may need some burnishing, wanted to get it on the radar screen for you and pfS devs. Thanks for all your efforts.

                BBcan177B 1 Reply Last reply Reply Quote 0
                • T
                  t41k2m3 @ScottishTom
                  last edited by

                  @ScottishTom said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                  Just to add another data point : following upgrade to 2.4.5 from 2.4.4p3, I've noticed an increase in memory usage on a pfSense instance installed on a physical machine, but not any drastic increase in CPU usage. Memory usage jumped from ~7% to ~64% with no other changes bar the pfSense upgrade.

                  Machine info : Intel J3160, 4GB DDR3, Dual Intel 82576EB NIC.

                  Packages installed : openvpn-client-export, pfBlockerNG

                  4c61e85d-a315-413e-a07e-28d0145b2e9b-image.png

                  If any more info desired please just let me know.

                  This seems like virtually the same or similar setup and problem as previously described (with qualification that a process at fault was not yet identified/hypothesized).

                  1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator @t41k2m3
                    last edited by BBcan177

                    @t41k2m3
                    I posted above two commands that you can use to find what is using memory. Report back with what you find. I haven't spent much time with the release of 2.4.5 as things have been hectic. I haven't checked if the version of Unbound has changed from 2.4.3/4. That might be a reason if something has changed in the Resolver code.
                    In regards to the upcoming Unbound python integration, what you see in the Resolver settings will allow for a future release to integrate with the Resolver. It's just the plumbing and nothing else. There is no Python integration released yet.

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    1 Reply Last reply Reply Quote 0
                    • ScottishTomS
                      ScottishTom @BBcan177
                      last edited by ScottishTom

                      @BBcan177

                      Hi, thanks for the prompt reply.

                      Currently running version 2.1.4_21 as installed from pfSense's package manager.

                      Output of ps auxwww sorted by memory usage:

                      2d6a158e-ea39-4c98-b4f8-93c0ddf44ef4-image.png

                      Output of top -aSH sorted by size

                      37fd45f0-cf1f-4730-ae22-dfecdcb7d61a-image.png

                      Reboot does not appear to change anything, will just go try the devel version. Hope that helps.

                      BBcan177B 1 Reply Last reply Reply Quote 0
                      • BBcan177B
                        BBcan177 Moderator @ScottishTom
                        last edited by BBcan177

                        @ScottishTom
                        Would recommend to uninstall pfBlockerNG and install pfBlockerNG-devel. Then see how that goes. Ensure "Keep settings" is enabled. You will need to re-enter the MaxMind key into the IP tab. Also best to reboot to clear Unbound.

                        "Experience is something you don't get until just after you need it."

                        Website: http://pfBlockerNG.com
                        Twitter: @BBcan177  #pfBlockerNG
                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                        ScottishTomS 1 Reply Last reply Reply Quote 0
                        • ScottishTomS
                          ScottishTom @BBcan177
                          last edited by

                          @BBcan177

                          Have done as requested, now running pfBlockerNG-devel 2.2.5_29.

                          • uninstalled pfBlockerNG

                          • installed pfBlockerNG-devel

                          • rebooted

                          • force-updated DNSBL as it was complaining about being out of date.

                          • rebooted again

                          Still appear to be at ~64-65% memory usage

                          FWIW I'm not complaining at all personally, the software is working fine for me and I'm seeing packets being intercepted by the block lists. Just seems strange to have had an almost 10x memory usage increase.

                          Appreciate the blocker software and your work on it very much, really simplifies things :)

                          BBcan177B 1 Reply Last reply Reply Quote 0
                          • BBcan177B
                            BBcan177 Moderator @ScottishTom
                            last edited by

                            @ScottishTom
                            Thanks for reporting back. Will check it out tomorrow.

                            "Experience is something you don't get until just after you need it."

                            Website: http://pfBlockerNG.com
                            Twitter: @BBcan177  #pfBlockerNG
                            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                            T 1 Reply Last reply Reply Quote 1
                            • J
                              JohnGalt1717
                              last edited by JohnGalt1717

                              I'm getting the same thing with Hyper-V host, 2 deticated NICs all offloading turned off.

                              Interesting tidbit is that with me if I make any settings change and save it, it hanges for 20-30 seconds then pins the CPU, pings even over the lan to pfSense won't respond. Then it slowly comes back.

                              I have OpenVPN running but had no issues with 2.4.4. In the OpenVPN logs there is now an endless stream of: AEAD Decrypt error: bad packet ID (may be a replay): that weren't there before. The System time is correct.

                              AES-NI On or off doesn't make a difference (even after restart)

                              Internet is PPPoE if that makes a difference.

                              Download speed is normal. Upload speed is about half of what it should be compared to 2.4.4.

                              Note traffic shaper is enabled and I have some floating rules for Google Hangouts.

                              1 Reply Last reply Reply Quote 0
                              • ?
                                A Former User
                                last edited by A Former User

                                One thing to try would be to disable the Spectre/Meltdown mitigation, reboot, and see if that improves things.
                                I wonder if improvements have been made to it in FreeBSD 11.3 but those improvements maybe don't play well with virtualisation.

                                I'm not in a position to test at the moment, can anyone else give this idea a go?

                                S 1 Reply Last reply Reply Quote 0
                                • S
                                  snarfattack @A Former User
                                  last edited by

                                  @muppet I just disabled the meltdown mitigation, rebooted. Still have 100% CPU

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    ShadowBlade72
                                    last edited by

                                    I'm experiencing similar issues with 2.4.5. My CPU usage spikes to 100% anytime I access a website and causes an outage for about 20 seconds.

                                    Specs:
                                    Proxmox 6.1-8
                                    CPU: 8 vCPUs
                                    Memory: 8GB

                                    Packages:
                                    acme 0.6.5
                                    Avahi 2.1_1
                                    openvpn-client-export 1.4.20
                                    pfBlockerNG 2.1.4_21
                                    Service_Watchdog 1.8.6
                                    snort 3.2.9.10_2
                                    softflowd 1.2.6
                                    squid 0.4.44_15
                                    Status_Traffic_Totals 2.3.1

                                    1 Reply Last reply Reply Quote 1
                                    • S
                                      snarfattack
                                      last edited by

                                      I finally got my Hyper-V virtualized install back to mostly working by removing all IP alias lists that PFBlocker had created, except one that was critical for me. Once this was done, I only get 100% cpu for a second or two during the times pfctl runs. I can deal with that over the 5 minutes lockups that were occuring before.

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        t41k2m3 @BBcan177
                                        last edited by

                                        @BBcan177 said in Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5:

                                        @ScottishTom
                                        Thanks for reporting back. Will check it out tomorrow.

                                        @BBcan177 realizing this is going to sound somewhat odd... everything went back to normal on its own over night (typical state prior to the 2.4.5 upgrade - i.e. memory usage around 20%). None of the other issues - e.g. CPU spikes, latency, outage - manifested themselves on this setup. Also implemented your fix for Shalla and UT1 found here: https://forum.netgate.com/topic/151689/shallalist-and-ut1-lists-not-working-on-2-4-5-release-pfblockerng-devel-2-2-5_29
                                        Thanks for your help debugging.

                                        1 Reply Last reply Reply Quote 0
                                        • V
                                          ViniciusBr
                                          last edited by

                                          Hi,

                                          I reverted 3 upgrades already due to performance issues, all 3 uses pfblocker, 2 are virtualized with vmware and one with hyper-v (gen 1 vm).

                                          Look at the before/after graphs, impossible to upgrade to be honest:

                                          pfsense-01-cpu.JPG pfsense-02-cpu.JPG pfsense-01-mem.JPG pfsense-02-mem.JPG

                                          1 Reply Last reply Reply Quote 1
                                          • W
                                            WALK3R
                                            last edited by

                                            The new release is like how the RC was for me 100% CPU usage locks the system up so you can't do anything. I did a fresh install no packages less than 6 firewall rules and basic setup for routing and it started locking up today lasted about 2 days before I had to roll back to 2.4.4; only thing I changed was the subnet.

                                            I'm running proxmox with:

                                            CPU(s)

                                            24 x Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (2 Sockets)
                                            Kernel Version

                                            Linux 5.3.10-1-pve #1 SMP PVE 5.3.10-1 (Thu, 14 Nov 2019 10:43:13 +0100)
                                            PVE Manager Version

                                            pve-manager/6.1-3/37248ce6

                                            alt text
                                            alt text

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.