Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr'

    General pfSense Questions
    5
    16
    910
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pirsig
      last edited by

      Hi, I recently upgraded/re-installed pfsense 22.01 on my SG-1100 a couple of days ago. Since then I have noticed consistently high CPU activity (50-100%).

      htop shows the culprit appears to be '/sbin/pfctl -vvsr'

      Based on previous threads I could find on similar issues I tried limiting state table to 65,000 to no effect.
      Attached is my /tmp/rules/debug file

      rules.debug.txt

      I also have the following packages installed:

      • acme
      • Avahi
      • haproxy
      • pfblockerNG-devel
      • snort

      I ran the offending command

      time pfctl -vvsr

      which gave a massive output:
      pfctl_output.txt
      ending:

      5.677u 4.668s 0:13.07 79.0%	194+179k 1+0io 0pf+0w

      I'm not sure what the issue is, can anyone help me?
      Thanks so much for your help!

      R BBcan177B 2 Replies Last reply Reply Quote 0
      • R
        rcoleman-netgate Netgate @pirsig
        last edited by

        @pirsig You have those packages installed, but what are running? pfblocker? snort? The two of those running together will likely make it choke, and snort is not a good idea for the eMMC that is in the 1100, either. In fact those and haproxy are fairly CPU intense.

        Ryan
        Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
        Requesting firmware for your Netgate device? https://go.netgate.com
        Switching: Mikrotik, Netgear, Extreme
        Wireless: Aruba, Ubiquiti

        S P 2 Replies Last reply Reply Quote 0
        • S
          SteveITS Rebel Alliance @rcoleman-netgate
          last edited by

          @rcoleman-netgate said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

          snort is not a good idea for the eMMC

          Is your comment based on the assumption of high logging volume? Just thinking about this thread on eMMC.

          @pirsig, the vv would be verbose output. Is this running all the time to generate the consistently high CPU? I haven't noticed consistent CPU usage on the 2100 I've upgraded, nor do I see that process (though I didn't watch for a long time). I did notice this regression though, in this thread.

          Another thing I've seen is if something is generating a lot of logs, multiple bzip instances run to compress them, and therefore Netgate recommends turning off log compression on slower CPUs. But, you'd see bzip in the process list if that was the issue.

          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
          Upvote 👍 helpful posts!

          R 1 Reply Last reply Reply Quote 0
          • P
            pirsig @rcoleman-netgate
            last edited by

            @rcoleman-netgate

            thanks for the information. I was running all the packages listed simultaneously. I uninstalled snort since I don't think I need to be running it.

            However it didn't seem to affect the CPU usage and the pfctl process is still running.
            @SteveITS - Yes the process is running continuously. And yes, I also have the same latency issue with the filter reload process.

            P 2 Replies Last reply Reply Quote 0
            • P
              pirsig @pirsig
              last edited by

              This post is deleted!
              1 Reply Last reply Reply Quote 0
              • P
                pirsig @pirsig
                last edited by

                @pirsig
                I have narrowed down the issue to pfblockerNG-devel. Uninstalling it removes the CPU load and pfctl process.

                S 1 Reply Last reply Reply Quote 0
                • S
                  SteveITS Rebel Alliance @pirsig
                  last edited by

                  @pirsig Interesting, were you using DNSBL? We have not used that at our clients, but from what I read here it can take a lot of memory and CPU to process large block lists. We use pfBlocker-devel for geoIP aliases and block feeds.

                  Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                  When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                  Upvote 👍 helpful posts!

                  P S 2 Replies Last reply Reply Quote 0
                  • P
                    pirsig @SteveITS
                    last edited by pirsig

                    @steveits

                    I am using DNSBL as well as several aliases for selective routing purposes. However it is not the DNSBL functionality since when I go to Status -> Services and stop the 'pfb_filter' service, leaving the 'pfb_dnsbl' service running the CPU usage drops immediately.

                    I should mention that I was previously running the pfsense 22.01 beta build with the same configuration (pfBlockerNG-devel + DNSBL, OpenVPN, torrents, snort, etc) with no issue, so I think it has to do with the new stable version.

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      SteveITS Rebel Alliance @pirsig
                      last edited by

                      @pirsig Hm, I'm not seeing that on three routers with 22.01 or 2.6. I'd guess it's related to the regression discussed above.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote 👍 helpful posts!

                      1 Reply Last reply Reply Quote 0
                      • S
                        serbus @SteveITS
                        last edited by serbus

                        @steveits said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                        from what I read here it can take a lot of memory and CPU to process large block lists

                        There is Wildcard Blocking (TLD) option on the Firewall > pfBlockerNG > DNSBL settings page that can really increase the load. Check the infoblock text for that option.

                        @pirsig said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                        /sbin/pfctl -vvsr

                        I did a really quick check and dont see this command being called from anywhere except the hidden status page (after logging in, browse to <pfsense url>/status.php). Not sure what would be calling it or why it would be pinning the CPU...

                        Edit: it looks like the pfblocker widget uses a variation of that call - "pfctl -vv -sr"...?

                        John

                        Lex parsimoniae

                        P 1 Reply Last reply Reply Quote 0
                        • R
                          rcoleman-netgate Netgate @SteveITS
                          last edited by

                          @steveits said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                          Is your comment based on the assumption of high logging volume? Just thinking about this thread on eMMC.

                          Yes. Most users that are doing that on an eMMC have stated they were unaware of the recommendation to use an SSD/HDD for that task (and since the 1100 lacks that ability) they had a configuration that destroyed their eMMC in weeks, or months.

                          Ryan
                          Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                          Requesting firmware for your Netgate device? https://go.netgate.com
                          Switching: Mikrotik, Netgear, Extreme
                          Wireless: Aruba, Ubiquiti

                          S 1 Reply Last reply Reply Quote 0
                          • S
                            SteveITS Rebel Alliance @rcoleman-netgate
                            last edited by

                            @rcoleman-netgate Interesting. I surely was unaware of that recommendation. Weeks to months seems absurdly low and makes we wonder about defaulting to ZFS per that thread and one or two other threads I've seen.

                            I'll pursue in that thread.

                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                            Upvote 👍 helpful posts!

                            R 1 Reply Last reply Reply Quote 0
                            • P
                              pirsig @serbus
                              last edited by

                              @serbus said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                              There is Wildcard Blocking (TLD) option on the Firewall > pfBlockerNG > DNSBL settings page that can really increase the load. Check the infoblock text for that option.

                              I checked the DNSBL settings and wildcard blocking is unchecked.

                              @serbus said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                              Edit: it looks like the pfblocker widget uses a variation of that call - "pfctl -vv -sr"...?

                              Thanks for the suggestion, I deleted the pfblocker widget from my home page, but the CPU is still pinned :/

                              However I did check the hidden status page as you suggested, and found a little more detail on the process, it seems to be called from the filterlog script?

                               0.0  4.4  62604 43828  -  S    10:10     0:19.21 |-- /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
32.2  0.3  11904  3192  -  R    11:39     0:03.80 | `-- /sbin/pfctl -vvsr
                              1 Reply Last reply Reply Quote 0
                              • R
                                rcoleman-netgate Netgate @SteveITS
                                last edited by

                                @steveits It comes from this published document:
                                https://www.netgate.com/supported-pfsense-plus-packages

                                The applications with the high logging or writes to the eMMC are noted as recommending an SSD or HDD.
                                There are, obviously, ways around that but they have their own caveats.

                                Ryan
                                Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                                Requesting firmware for your Netgate device? https://go.netgate.com
                                Switching: Mikrotik, Netgear, Extreme
                                Wireless: Aruba, Ubiquiti

                                1 Reply Last reply Reply Quote 0
                                • BBcan177B
                                  BBcan177 Moderator @pirsig
                                  last edited by

                                  @pirsig

                                  See here for a patch:

                                  https://www.reddit.com/r/pfBlockerNG/comments/sk9txi/ip_block_logging_not_working_pfsense_260rc/hvv99s1/?utm_source=reddit&utm_medium=web2x&context=3

                                  "Experience is something you don't get until just after you need it."

                                  Website: http://pfBlockerNG.com
                                  Twitter: @BBcan177  #pfBlockerNG
                                  Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                  P 1 Reply Last reply Reply Quote 2
                                  • P
                                    pirsig @BBcan177
                                    last edited by

                                    @bbcan177 said in updated to 22.01 - SG1100 high CPU usage '/sbin/pfctl -vvsr':

                                    https://www.reddit.com/r/pfBlockerNG/comments/sk9txi/ip_block_logging_not_working_pfsense_260rc/hvv99s1/?utm_source=reddit&utm_medium=web2x&context=3

                                    Installed the patch and it solved it! Thanks!

                                    1 Reply Last reply Reply Quote 0
                                    • P pirsig referenced this topic on
                                    • First post
                                      Last post