• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

*SOLVED* pfSense freezing for a second or two every 15 minutes

Scheduled Pinned Locked Moved General pfSense Questions
19 Posts 3 Posters 2.1k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • A
    awebster @stephenw10
    last edited by Aug 20, 2021, 1:54 PM

    @stephenw10 said in pfSense freezing for a second or two every 15 minutes:

    I mean it could be just a very large ruleset? If your config especially complex?

    The config is complex-ish. I am using pfBlocker so it does add some load with large tables. I'll try turning it off and see if it makes a difference and report back.

    Also I assume you do have some scheduled rules that are triggering this every 15m?

    No scheduled rules, but /etc/rc.filter_configure_sync runs every 15 minutes out of /etc/crontab

    This just started happening in 2.4.5p1? There was no significant change that created it?

    Yes, I can't pinpoint exactly when, but it appears to have started in the last 3-4 weeks. No changes other than what pfBlocker might be doing.

    When doing the filter reload as far as I can tell, it seems to get bogged down after the Setting up SCRUB information. Is there a way to get the reload status output on the CLI since the GUI refresh is affected by the packet delay problem.

    Generating ALTQ queues
    Loading filter rules
    Setting up logging information
    Setting up SCRUB information
    Processing down interface states
    Running plugins
    Running plugins (pf)
    Plugins completed.
    Done
    

    –A.

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by Aug 20, 2021, 2:13 PM

      This crontab entry:

      0,15,30,45	*	*	*	*	root	/etc/rc.filter_configure_sync
      

      Is only generated by having an active scheduled firewall rule AFAIK.

      pfBlocker adding something maybe?

      Steve

      A 1 Reply Last reply Aug 20, 2021, 3:15 PM Reply Quote 0
      • A
        awebster @stephenw10
        last edited by Aug 20, 2021, 3:15 PM

        @stephenw10 said in pfSense freezing for a second or two every 15 minutes:

        Is only generated by having an active scheduled firewall rule AFAIK.

        Yup, found a scheduled rule and removed it, so the cronjob is gone.
        While it doesn't resolve the underlying problem of packet delay during afilter reload, at least it isn't happening every 15 minutes, so the Zoom/Teams/Meet users are happy now πŸ˜ƒ
        Thanks @stephenw10 for helping narrow down the source of the problem. πŸ‘

        –A.

        K 1 Reply Last reply Aug 20, 2021, 5:29 PM Reply Quote 1
        • K
          keyser Rebel Alliance @awebster
          last edited by Aug 20, 2021, 5:29 PM

          @awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

          @stephenw10 said in pfSense freezing for a second or two every 15 minutes:

          Is only generated by having an active scheduled firewall rule AFAIK.

          Yup, found a scheduled rule and removed it, so the cronjob is gone.
          While it doesn't resolve the underlying problem of packet delay during afilter reload, at least it isn't happening every 15 minutes, so the Zoom/Teams/Meet users are happy now πŸ˜ƒ
          Thanks @stephenw10 for helping narrow down the source of the problem. πŸ‘

          My guess is you have pfBlockerNG setup with GEO blocking, and you have some enourmous blocking lists that causes a CPU spike whenever the list needs to be reloaded.

          One typical mistake with GEO blocking is that many people create a block rule for almost the entire world. That will cause a HUGE block list of addresses to load every time.

          The real way to solve that problem is to reverse the problem - create a allow rule from the few countries you wish to allow access from, and let the rest of the world hit the default deny rule in your firewall.

          This is how it’s done:
          1: Disable pfBlockerNG automatic rule generation
          2: Have pfBlocker generate a ALLOW ALIAS list with the few regions/countries needed.
          3: Use that list as source on your public facing services.

          Love the no fuss of using the official appliances :-)

          A 1 Reply Last reply Aug 20, 2021, 5:39 PM Reply Quote 0
          • A
            awebster @keyser
            last edited by Aug 20, 2021, 5:39 PM

            @keyser said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

            One typical mistake with GEO blocking is that many people create a block rule for almost the entire world. That will cause a HUGE block list of addresses to load every time.
            The real way to solve that problem is to reverse the problem - create a allow rule from the few countries you wish to allow access from, and let the rest of the world hit the default deny rule in your firewall.

            Agreed, and that is how I've configured it; my rule is to allow only North American IP blocks (v4 and v6) in toward the published services behind the firewall. Perhaps those lists have just gotten too big?!

            –A.

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by Aug 20, 2021, 8:15 PM

              The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.

              Steve

              A 1 Reply Last reply Aug 20, 2021, 9:19 PM Reply Quote 0
              • A
                awebster @stephenw10
                last edited by Aug 20, 2021, 9:19 PM

                @stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

                The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.

                Yup, I think the smoking gun might be pfB_NAmerica_v6.txt (only USA and Canada selected). If IPv6 list grows much more I can see that it'll be problematic for many people running pfBlocker.

                Oddly GeoIP hasn't updated since Aug 5th, but it appears to only update once a month.
                I ran /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dc to force it to update, and the next pfblocker cron run reduced the size of the resulting tables (see 17:00:00 pfblocker update log output below), namely this: Updating: pfB_NAmerica_v6 538 addresses added.97343 addresses deleted.

                Sadly, despite the updated DB, the filter reload still causes packet delay/loss. 😭

                I disabled any rules referencing pfB_*_v6, but the problem persists.
                Question: Does the fact that a rule is disabled still load the tables with pfctl ?

                Here is the output of the pfblocker log before and after GeoIP update:

                [ firehol3 ]
                        ( No remote timestamp/md5 unchanged )           Update not required
                [ adware ]
                                        Previous download failed.       Re-attempt download
                 UPDATE PROCESS START
                
                ===[  DNSBL Process  ]================================================
                
                [ EasyList ]             exists.
                [ adware ]               Downloading update .. 404 Not Found
                
                 [ DNSBL_Adware - adware ] Download FAIL [ 08/20/21 16:00:01 ]
                  Firewall and/or IDS are not blocking download.
                
                  Restoring previously downloaded file
                 .
                  ----------------------------------------------------------------------
                  Orig.    Unique     # Dups     # White    # Alexa    Final                
                  ----------------------------------------------------------------------
                  175      163        3          0          0          160                  
                  ----------------------------------------------------------------------
                
                [ DNSBL_IP ]             Updating aliastable... 
                  no changes.
                  Total IP count = 10
                
                ------------------------------------------
                Assembling database... completed
                Validating database... completed
                Reloading Unbound.... completed
                DNSBL update [ 2206 | PASSED  ]... completed [ 08/20/21 16:00:05 ]
                ------------------------------------------
                
                ===[  Continent Process  ]============================================
                
                [ pfB_Asia_v4 ]          exists.
                [ pfB_Asia_v6 ]          exists.
                [ pfB_Europe_v4 ]        exists.
                [ pfB_Europe_v6 ]        exists. [ 08/20/21 16:00:06 ]
                [ pfB_NAmerica_v4 ]      exists.
                [ pfB_NAmerica_v6 ]      exists. [ 08/20/21 16:00:09 ]
                [ pfB_Top_v4 ]           exists. [ 08/20/21 16:00:10 ]
                [ pfB_Top_v6 ]           exists.
                
                ===[  IPv4 Process  ]=================================================
                
                [ firehol3 ]             exists.
                [ shdrop ]               exists.
                
                ===[  IPv6 Process  ]=================================================
                
                [ shdrop6_v6 ]           exists.
                
                ===[  Aliastables / Rules  ]==========================================
                
                No changes to Firewall rules, skipping Filter Reload
                No Changes to Aliases, Skipping pfctl Update
                
                ===[  Kill States  ]==================================================
                
                 No matching states found
                ======================================================================
                
                ===[ FINAL Processing ]=====================================
                
                   [ Original IP count   ]  [ 986408 ]
                
                ===[ Permit List IP Counts ]=========================
                
                  716191 total
                  613306 /var/db/pfblockerng/permit/pfB_NAmerica_v6.txt
                   33299 /var/db/pfblockerng/permit/pfB_Europe_v4.txt
                   32977 /var/db/pfblockerng/permit/pfB_Europe_v6.txt
                   23984 /var/db/pfblockerng/permit/pfB_NAmerica_v4.txt
                    7102 /var/db/pfblockerng/permit/pfB_Asia_v4.txt
                    5523 /var/db/pfblockerng/permit/pfB_Asia_v6.txt
                
                ===[ Deny List IP Counts ]===========================
                
                   94563 total
                   62130 /var/db/pfblockerng/deny/pfB_Top_v4.txt
                   17782 /var/db/pfblockerng/deny/firehol3.txt
                   13609 /var/db/pfblockerng/deny/pfB_Top_v6.txt
                    1004 /var/db/pfblockerng/deny/shdrop.txt
                      38 /var/db/pfblockerng/deny/shdrop6_v6.txt
                
                ===[ DNSBL Domain/IP Counts ] ===================================
                
                    2216 total
                    2046 /var/db/pfblockerng/dnsbl/EasyList.txt
                     160 /var/db/pfblockerng/dnsbl/adware.txt
                      10 /var/db/pfblockerng/dnsbl/EasyList.ip
                       0 /var/db/pfblockerng/dnsbl/adware.fail
                
                ====================[ Last Updated List Summary ]==============
                
                Aug 5   03:00   pfB_Asia_v4
                Aug 5   03:00   pfB_Asia_v6
                Aug 5   03:00   pfB_Europe_v4
                Aug 5   03:00   pfB_Europe_v6
                Aug 5   03:00   pfB_NAmerica_v4
                Aug 5   03:02   pfB_NAmerica_v6
                Aug 5   03:02   pfB_Top_v4
                Aug 5   03:02   pfB_Top_v6
                Aug 17  00:05   shdrop
                Aug 20  00:00   shdrop6_v6
                Aug 20  07:00   firehol3
                
                IPv4 alias tables IP count
                -----------------------------
                145349
                
                IPv6 alias tables IP count
                -----------------------------
                665415
                
                Alias table IP Counts
                -----------------------------
                  810764 total
                  613306 /var/db/aliastables/pfB_NAmerica_v6.txt
                   62130 /var/db/aliastables/pfB_Top_v4.txt
                   33299 /var/db/aliastables/pfB_Europe_v4.txt
                   32977 /var/db/aliastables/pfB_Europe_v6.txt
                   23984 /var/db/aliastables/pfB_NAmerica_v4.txt
                   17782 /var/db/aliastables/pfB_firehol3.txt
                   13609 /var/db/aliastables/pfB_Top_v6.txt
                    7102 /var/db/aliastables/pfB_Asia_v4.txt
                    5523 /var/db/aliastables/pfB_Asia_v6.txt
                    1004 /var/db/aliastables/pfB_SpamhausDROP.txt
                      38 /var/db/aliastables/pfB_SpamhausIPv6DROP.txt
                      10 /var/db/aliastables/pfB_DNSBLIP.txt
                
                pfSense Table Stats
                -------------------
                table-entries hard limit  2000000
                Table Usage Count         924925
                
                 UPDATE PROCESS ENDED [ 08/20/21 16:00:12 ]
                CRON  PROCESS  START [ 08/20/21 17:00:00 ]
                [ firehol3 ]
                        ( No remote timestamp/md5 unchanged )           Update not required
                [ adware ]
                                        Previous download failed.       Re-attempt download
                 UPDATE PROCESS START
                
                ===[  DNSBL Process  ]================================================
                
                [ EasyList ]             exists.
                [ adware ]               Downloading update .. 404 Not Found
                
                 [ DNSBL_Adware - adware ] Download FAIL [ 08/20/21 17:00:01 ]
                  Firewall and/or IDS are not blocking download.
                
                  Restoring previously downloaded file
                 .
                  ----------------------------------------------------------------------
                  Orig.    Unique     # Dups     # White    # Alexa    Final                
                  ----------------------------------------------------------------------
                  175      163        3          0          0          160                  
                  ----------------------------------------------------------------------
                
                [ DNSBL_IP ]             Updating aliastable... 
                  no changes.
                  Total IP count = 10
                
                ------------------------------------------
                Assembling database... completed
                Validating database... completed
                Reloading Unbound.... completed
                DNSBL update [ 2206 | PASSED  ]... completed [ 08/20/21 17:00:05 ]
                ------------------------------------------
                
                ===[  Continent Process  ]============================================
                
                [ pfB_Asia_v4 ]          Changes found... Updating
                
                  Aggregation Stats:
                  ------------------
                  Original Final      
                  ------------------
                  7100     7099       
                  ------------------
                
                [ pfB_Asia_v6 ]          Changes found... Updating
                
                [ pfB_Europe_v4 ]        Changes found... Updating
                
                  Aggregation Stats:
                  ------------------
                  Original Final      
                  ------------------
                  92785    33496      
                  ------------------
                
                [ pfB_Europe_v6 ]        Changes found... Updating
                
                [ pfB_NAmerica_v4 ]      Changes found... Updating
                
                  Aggregation Stats:
                  ------------------
                  Original Final      
                  ------------------
                  134087   23023      
                  ------------------
                
                [ pfB_NAmerica_v6 ]      Changes found... Updating
                
                [ pfB_Top_v4 ]           Changes found... Updating
                
                  Aggregation Stats:
                  ------------------
                  Original Final      
                  ------------------
                  74426    62411      
                  ------------------
                
                [ pfB_Top_v6 ]           Changes found... Updating
                
                
                ===[  IPv4 Process  ]=================================================
                
                [ firehol3 ]             exists. [ 08/20/21 17:02:36 ]
                [ shdrop ]               exists.
                
                ===[  IPv6 Process  ]=================================================
                
                [ shdrop6_v6 ]           exists.
                
                ===[  Aliastables / Rules  ]==========================================
                
                No changes to Firewall rules, skipping Filter Reload
                
                 Updating: pfB_Asia_v4
                33 addresses added.36 addresses deleted.
                 Updating: pfB_Asia_v6
                107 addresses added.23 addresses deleted.
                 Updating: pfB_Europe_v4
                621 addresses added.424 addresses deleted.
                 Updating: pfB_Europe_v6
                3551 addresses added.2828 addresses deleted.
                 Updating: pfB_NAmerica_v4
                467 addresses added.1428 addresses deleted.
                 Updating: pfB_NAmerica_v6
                538 addresses added.97343 addresses deleted.
                 Updating: pfB_Top_v4
                1088 addresses added.807 addresses deleted.
                 Updating: pfB_Top_v6
                10751 addresses added.8019 addresses deleted.
                
                ===[  Kill States  ]==================================================
                
                 No matching states found
                ======================================================================
                
                ===[ FINAL Processing ]=====================================
                
                   [ Original IP count   ]  [ 896803 ]
                
                ===[ Permit List IP Counts ]=========================
                
                  616774 total
                  513839 /var/db/pfblockerng/permit/pfB_NAmerica_v6.txt
                   33710 /var/db/pfblockerng/permit/pfB_Europe_v6.txt
                   33496 /var/db/pfblockerng/permit/pfB_Europe_v4.txt
                   23023 /var/db/pfblockerng/permit/pfB_NAmerica_v4.txt
                    7099 /var/db/pfblockerng/permit/pfB_Asia_v4.txt
                    5607 /var/db/pfblockerng/permit/pfB_Asia_v6.txt
                
                ===[ Deny List IP Counts ]===========================
                
                   97576 total
                   62411 /var/db/pfblockerng/deny/pfB_Top_v4.txt
                   17782 /var/db/pfblockerng/deny/firehol3.txt
                   16341 /var/db/pfblockerng/deny/pfB_Top_v6.txt
                    1004 /var/db/pfblockerng/deny/shdrop.txt
                      38 /var/db/pfblockerng/deny/shdrop6_v6.txt
                
                ===[ DNSBL Domain/IP Counts ] ===================================
                
                    2216 total
                    2046 /var/db/pfblockerng/dnsbl/EasyList.txt
                     160 /var/db/pfblockerng/dnsbl/adware.txt
                      10 /var/db/pfblockerng/dnsbl/EasyList.ip
                       0 /var/db/pfblockerng/dnsbl/adware.fail
                
                ====================[ Last Updated List Summary ]==============
                
                Aug 17  00:05   shdrop
                Aug 20  00:00   shdrop6_v6
                Aug 20  07:00   firehol3
                Aug 20  17:00   pfB_Asia_v4
                Aug 20  17:00   pfB_Asia_v6
                Aug 20  17:00   pfB_Europe_v4
                Aug 20  17:00   pfB_Europe_v6
                Aug 20  17:00   pfB_NAmerica_v4
                Aug 20  17:02   pfB_NAmerica_v6
                Aug 20  17:02   pfB_Top_v4
                Aug 20  17:02   pfB_Top_v6
                
                IPv4 alias tables IP count
                -----------------------------
                144863
                
                IPv6 alias tables IP count
                -----------------------------
                569497
                
                Alias table IP Counts
                -----------------------------
                  714360 total
                  513839 /var/db/aliastables/pfB_NAmerica_v6.txt
                   62411 /var/db/aliastables/pfB_Top_v4.txt
                   33710 /var/db/aliastables/pfB_Europe_v6.txt
                   33496 /var/db/aliastables/pfB_Europe_v4.txt
                   23023 /var/db/aliastables/pfB_NAmerica_v4.txt
                   17782 /var/db/aliastables/pfB_firehol3.txt
                   16341 /var/db/aliastables/pfB_Top_v6.txt
                    7099 /var/db/aliastables/pfB_Asia_v4.txt
                    5607 /var/db/aliastables/pfB_Asia_v6.txt
                    1004 /var/db/aliastables/pfB_SpamhausDROP.txt
                      38 /var/db/aliastables/pfB_SpamhausIPv6DROP.txt
                      10 /var/db/aliastables/pfB_DNSBLIP.txt
                
                pfSense Table Stats
                -------------------
                table-entries hard limit  2000000
                Table Usage Count         831173
                
                 UPDATE PROCESS ENDED [ 08/20/21 17:02:43 ]
                

                –A.

                K S 2 Replies Last reply Aug 20, 2021, 9:47 PM Reply Quote 0
                • K
                  keyser Rebel Alliance @awebster
                  last edited by keyser Aug 20, 2021, 9:51 PM Aug 20, 2021, 9:47 PM

                  @awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

                  @stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

                  The v6 tables can get pretty huge. Check the pfBlocker logs after it updates to see the table sizes.

                  Yup, I think the smoking gun might be pfB_NAmerica_v6.txt (only USA and Canada selected). If IPv6 list grows much more I can see that it'll be problematic for many people running pfBlocker.

                  Oddly GeoIP hasn't updated since Aug 5th, but it appears to only update once a month.
                  I ran /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dc to force it to update, and the next pfblocker cron run reduced the size of the resulting tables (see 17:00:00 pfblocker update log output below), namely this: Updating: pfB_NAmerica_v6 538 addresses added.97343 addresses deleted.

                  Sadly, despite the updated DB, the filter reload still causes packet delay/loss. 😭

                  I disabled any rules referencing pfB_*_v6, but the problem persists.
                  Question: Does the fact that a rule is disabled still load the tables with pfctl ?

                  Seems likely that list is causing your problem. Some 600.000 entries is no small amount (and more than one third of the hard limit in pfBlockerNG).

                  There is the question of hardware as well. If you are trying to do this on fx. A SG-1100/SG-2100 or smaller, you have to remember that the single thread CPU core performance and memory bandwidth of that ARM processor, is less than half of fx. an Intel Atom based box. And the Atom is once again less than half as fast as a common 7th or 8th gen Intel Core CPU. Boxes with that caliber of CPU might not even register loading the same list.

                  EDIT: Just noticed you are running in a VM on a rather old Intel XEON. The virtualised single thread performance of that CPU is likely struggeling to reach a modern level Atom CPU core. So Hardware may well be part of the problem.

                  Love the no fuss of using the official appliances :-)

                  K 1 Reply Last reply Aug 20, 2021, 9:56 PM Reply Quote 0
                  • K
                    keyser Rebel Alliance @keyser
                    last edited by keyser Aug 20, 2021, 9:57 PM Aug 20, 2021, 9:56 PM

                    @keyser Yeah, just looked it up - Its way slower than a modern Atom, even before virtualisation.
                    So hardware is very likely pressing the issue.

                    Love the no fuss of using the official appliances :-)

                    1 Reply Last reply Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator @awebster
                      last edited by Aug 21, 2021, 12:22 AM

                      @awebster said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

                      I disabled any rules referencing pfB_*_v6, but the problem persists.
                      Question: Does the fact that a rule is disabled still load the tables with pfctl ?

                      No, I would not expect it to populate any tables that are not actually in use.

                      The other table that can be massive is bogons v6. But we have many users loading those on relatively low powered systems.

                      Steve

                      A 1 Reply Last reply Aug 31, 2021, 6:12 PM Reply Quote 0
                      • A
                        awebster @stephenw10
                        last edited by Aug 31, 2021, 6:12 PM

                        @stephenw10 said in *SOLVED* pfSense freezing for a second or two every 15 minutes:

                        The other table that can be massive is bogons v6.

                        Not too horrible at 124297 entries.

                        Some tables as pfctl seems them

                        -pa-r-- bogonsv6
                                Addresses:   124297
                                Cleared:     Fri Aug 20 01:07:38 2021
                                References:  [ Anchors: 0                  Rules: 1                  ]
                                Evaluations: [ NoMatch: 537822             Match: 0                  ]
                                In/Block:    [ Packets: 0                  Bytes: 0                  ]
                                In/Pass:     [ Packets: 0                  Bytes: 0                  ]
                                In/XPass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/Block:   [ Packets: 0                  Bytes: 0                  ]
                                Out/Pass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/XPass:   [ Packets: 0                  Bytes: 0                  ]
                        
                        -pa---- pfB_NAmerica_v6
                                Addresses:   505938
                                Cleared:     Fri Aug 20 01:07:38 2021
                                References:  [ Anchors: 0                  Rules: 0                  ]
                                Evaluations: [ NoMatch: 2                  Match: 12                 ]
                                In/Block:    [ Packets: 0                  Bytes: 0                  ]
                                In/Pass:     [ Packets: 0                  Bytes: 0                  ]
                                In/XPass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/Block:   [ Packets: 0                  Bytes: 0                  ]
                                Out/Pass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/XPass:   [ Packets: 0                  Bytes: 0                  ]
                        
                        -pa-r-- pfB_Top_v6
                                Addresses:   16341
                                Cleared:     Fri Aug 20 01:07:38 2021
                                References:  [ Anchors: 0                  Rules: 3                  ]
                                Evaluations: [ NoMatch: 51426              Match: 150                ]
                                In/Block:    [ Packets: 150                Bytes: 10800              ]
                                In/Pass:     [ Packets: 0                  Bytes: 0                  ]
                                In/XPass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/Block:   [ Packets: 0                  Bytes: 0                  ]
                                Out/Pass:    [ Packets: 0                  Bytes: 0                  ]
                                Out/XPass:   [ Packets: 0                  Bytes: 0                  ]
                        

                        –A.

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                          [[user:consent.lead]]
                          [[user:consent.not_received]]