• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Increased Memory and CPU Spikes (causing latency/outage) with 2.4.5

Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
141 Posts 40 Posters 43.6k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • ?
    A Former User
    last edited by Mar 27, 2020, 1:31 AM

    Are you using KVM and/or virtio interfaces?

    X 1 Reply Last reply Mar 27, 2020, 1:33 AM Reply Quote 0
    • X
      xpxp2002 @A Former User
      last edited by Mar 27, 2020, 1:33 AM

      No. The pfSense instance is on Hyper-V (Windows Server 2019). The interfaces are synthetic NICs with VMQ. Been running this way for at least 4 or 5 months on 2.4.4 builds without ever seeing this kind of CPU usage.

      For what it’s worth, I killed the OpenVPN processes along with everything except “bare essentials” like dhcpd and unbound. Pfctl is still running away with all the CPU. The only way I can get it to stop is to roll back to my 2.4.4-p3 snapshot.

      1 Reply Last reply Reply Quote 0
      • ?
        A Former User
        last edited by Mar 27, 2020, 1:34 AM

        Interesting. Must be something to do with virtualisation then. Mine settles down after a while, does yours not? I take it a simple reboot doesn't fix the CPU problem?

        X 1 Reply Last reply Mar 27, 2020, 1:37 AM Reply Quote 0
        • X
          xpxp2002 @A Former User
          last edited by Mar 27, 2020, 1:37 AM

          I rebooted it twice so far. It hangs booting up from the point that it initializes the queues onward. Every step of the bootup after that takes far longer than it used to with 2.4.4-p3. The entire bootup process is over 2 minutes. With 2.4.4-p3, it’s about 40 seconds.

          And nope, it never settles down. I let it sit untouched for over 30 minutes at one point and pfctl was still churning away at 99%.

          S 1 Reply Last reply Mar 27, 2020, 1:42 AM Reply Quote 1
          • S
            swinn @xpxp2002
            last edited by Mar 27, 2020, 1:42 AM

            @xpxp2002 I'm tried upgrading this afternoon and saw the same thing. pfctl - 100%. I'm also using Hyper-V on Server 2019 and had to revert back to a snapshot. Been using pfSense under Hyper-V for years and this is the first time I've had to revert an upgrade.

            X 1 Reply Last reply Mar 27, 2020, 1:51 AM Reply Quote 0
            • X
              xpxp2002 @swinn
              last edited by Mar 27, 2020, 1:51 AM

              @swinn Perhaps there is something about these virtualized instances that is a problem, as @muppet suggested. Looking through the release notes, I don’t see anything specifically calling out virtualization that seems like it would cause this. Unless simply going to FreeBSD 11 is the issue.

              I’ve also run pfSense on Server 2016 for years without issues prior to this. Ran into an issue with Server 2019 and receive segment coalescing causing weird packet drop issues when I first went to 2019, but once I disabled RSC the issue went away. But this is the first time a pfSense upgrade didn’t go smoothly for me either.

              1 Reply Last reply Reply Quote 0
              • ?
                A Former User
                last edited by Mar 27, 2020, 1:55 AM

                We haven't had a big FreeBSD jump though, 11.2p10 to 11.3

                I can't find any release notes either that say something major/odd has changed with Virtualization in 11.3

                X 1 Reply Last reply Mar 27, 2020, 2:33 AM Reply Quote 0
                • X
                  xpxp2002 @A Former User
                  last edited by Mar 27, 2020, 2:33 AM

                  I’ve been using the snapshot to test individual settings and packages that seem like possible culprits. As I mentioned before, the bootup hangs on the first “configuring firewall...” so I’ve tried removing settings and packages that I expect would be initialized when the filter rule load is occurring, then performed the upgrade.

                  So far, I’ve ruled out queues/limiters, pfBlocker-NG-devel, and Service Watchdog.

                  1 Reply Last reply Reply Quote 0
                  • ?
                    A Former User
                    last edited by A Former User Mar 27, 2020, 2:41 AM Mar 27, 2020, 2:37 AM

                    The only packages I have installed are:

                    • Avahi
                    • OpenVPN-Export

                    Oh and I have fq_codel configured.

                    X 1 Reply Last reply Mar 27, 2020, 2:43 AM Reply Quote 0
                    • X
                      xpxp2002 @A Former User
                      last edited by Mar 27, 2020, 2:43 AM

                      @muppet I also have both of those. I’m out of time for testing but either one of those could be the culprit.

                      My first thought goes to Avahi trying to come up on an interface where it isn’t supported, but it could also be the OVPN export package struggling with the new version of OVPN.

                      1 Reply Last reply Reply Quote 0
                      • ?
                        A Former User
                        last edited by Mar 27, 2020, 2:43 AM

                        Maybe avahi could cause problems, that I could understand.
                        OpenVPN export isn't even called until you visit that page.

                        1 Reply Last reply Reply Quote 0
                        • G
                          gusfersa
                          last edited by gusfersa Mar 27, 2020, 6:47 AM Mar 27, 2020, 6:38 AM

                          I upgrade six (6) pfsense production server at the same time from 2.4.4_p3, and I had problem with the conectivity. The ping time is very high above 7.000ms.

                          I tried upgrade my pfsense server at home from 2.4.4_p3, but in this case I did a snapshot on vmware, and the problem is same. The ping time is very high and the navigation have a lot of problems.

                          I restored the snapshot, and all return to normally

                          At all server I have installed this packages:

                          Open-VM-Tools
                          openvpn-client-export
                          squid
                          snort
                          zabbix-agent4

                          I tried reinstall all packages, but the problem persist

                          1 Reply Last reply Reply Quote 0
                          • G
                            Gektor
                            last edited by Gektor Mar 27, 2020, 10:35 AM Mar 27, 2020, 10:15 AM

                            Same troubles after upgrade from 2.4.4 to 2.4.5 on Hyper-V Windows Server 2019, 100% CPU usage (by pfctl process), long boot, and pfSense works with spikes and hangs.
                            It seems that 2.4.5 not compatible with Hyper-V Windows Server 2019.
                            Maybe it related:
                            https://forum.netgate.com/topic/149595/2-4-5-a-20200110-1421-and-earlier-high-cpu-usage-from-pfctl/8

                            1 Reply Last reply Reply Quote 0
                            • T
                              talaverde
                              last edited by Mar 27, 2020, 11:26 AM

                              I'm having the same problem. I'm running 2.4.4.-p3 on Server 2016 with Hyper-V. I tried upgrading my 2nd CARP node to 2.4.5 yesterday, but it pegged the CPU and never became stable. I reverted that snapshot, shut it down and tried to upgrade my 1st CARP node, but the same problem. I've reverted both nodes to the snapshots.

                              pfSense on Hyper-V has been rock solid up until now and all previous upgrades have been flawless.

                              If I have time, I'll try installing a 2.4.5 VM from scratch to see if the problem occurs there too.

                              1 Reply Last reply Reply Quote 0
                              • G
                                Gektor
                                last edited by Mar 27, 2020, 1:19 PM

                                I have made clean reinstall system with catching config from updated system, first time boot was fast, then all packagers was restored (installed), after that system stuck at boot and lags after.
                                Then i have found a source of problem — pfBlockerNG! When it's disabled, all works good, after enabling pfBlockerNG system lags totally.

                                X 1 Reply Last reply Mar 27, 2020, 1:35 PM Reply Quote 0
                                • X
                                  xpxp2002 @Gektor
                                  last edited by Mar 27, 2020, 1:35 PM

                                  @Gektor This is interesting. I had pfBlockerNG-devel installed on 2.4.4-p3. One of my earlier tests was to roll back to 2.4.4-p3, uninstall that package, then upgrade; and my system was still slow. Did you simply disable it, or uninstall the package?

                                  I will try this later today when I have an outage window.

                                  1 Reply Last reply Reply Quote 0
                                  • G
                                    Gektor
                                    last edited by Gektor Mar 27, 2020, 2:03 PM Mar 27, 2020, 1:46 PM

                                    Mine is pfBlockerNG version 2.1.4_21, with this setting all works good:
                                    7574e7a6-a678-4ee0-b6d6-5da00e69d698-изображение.png
                                    Then i have disable all GeoIP lists, but enable DNSBL, and enable pfBlockerNG, and for now there is no problems with pfSense 2.4.5 on Hyper-V. System makes "crazy" when GeoIP lists is enabled in pfBlockerNG.
                                    Have make post, maybe it will be helpful:
                                    https://forum.netgate.com/topic/151726/pfblockerng-2-1-4_21-totally-lag-system-after-pfsense-upgrade-from-2-4-4-to-2-4-5

                                    G 1 Reply Last reply Mar 27, 2020, 2:10 PM Reply Quote 0
                                    • G
                                      gusfersa @Gektor
                                      last edited by gusfersa Mar 27, 2020, 2:12 PM Mar 27, 2020, 2:10 PM

                                      @Gektor I deleted all the installed packages:

                                      Open-VM-Tools
                                      openvpn-client-export
                                      squid
                                      snort
                                      zabbix-agent4

                                      and I disabled OpenVPN links unpriority; and the system conectivity was restored

                                      G 1 Reply Last reply Mar 27, 2020, 2:19 PM Reply Quote 0
                                      • G
                                        gusfersa @gusfersa
                                        last edited by gusfersa Mar 27, 2020, 2:19 PM Mar 27, 2020, 2:19 PM

                                        @gusfersa On another production server with the same installed packages, only I disabled OpenVPN link to an another pfsense server 2.4.5, and the system conectivity restored

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          digitalgimpus
                                          last edited by digitalgimpus Mar 27, 2020, 4:09 PM Mar 27, 2020, 4:07 PM

                                          I've noticed something similar in terms of memory usage, but in my case cpu nice dropped in half and otherwise everything else seems status quo.

                                          I'm not however noticing any latency outages or anything of that nature, but i've got plenty of free RAM so maybe that's the difference.

                                          memory usage

                                          T 1 Reply Last reply Mar 27, 2020, 8:55 PM Reply Quote 0
                                          12 out of 141
                                          • First post
                                            12/141
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received