Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Hard Crashing - Out of Memory

    General pfSense Questions
    5
    31
    936
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • X
      xxGBHxx last edited by

      Hi,

      I rearranged my rules today to sort out a few other problems I was having and also to clear out some of the 6 year old crud in my rulesets. Since then I've been getting the following errors first "randomly" and now every time I click on my OpenVPN tab in Rules.

      The firewall is a VM and it has 6GB ram allocated to it of which it's currently sitting at about 30% (so 2GB). I'm guessing I need to set a larger buffer somewhere but I've been through every forum post and setting on the firewall and nothing has made a difference. The error is:

      Crash report begins.  Anonymous machine information:
      
      amd64
      11.2-RELEASE-p6
      FreeBSD 11.2-RELEASE-p6 #3 518496b29ae(RELENG_2_4_4): Wed Dec 12 07:41:44 EST 2018     root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense
      
      Crash report details:
      
      PHP Errors:
      [13-Mar-2019 18:43:03 Europe/London] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 375427072 bytes) in /usr/local/www/firewall_rules.php on line 88
      [13-Mar-2019 18:43:03 Europe/London] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 375386112 bytes) in Unknown on line 0
      

      Anyone have any idea why it's doing this and/or what I need to change to allocate more RAM to whatever process is running out of memory?

      G

      1 Reply Last reply Reply Quote 0
      • KOM
        KOM last edited by

        My instance runs several packages and it works just fine with 2GB RAM total, of which 26% is used. You have some sort of problem.

        Which packages, if any, are you running?

        1 Reply Last reply Reply Quote 0
        • X
          xxGBHxx last edited by

          Thanks but nothing exciting.

          ACME which I've never used
          Lightsquid which is configured and working
          Open-VM-Tools
          pfBlockerNG set up with most of the blocklists (only about 20 lists and none of them more than half a million IP's)
          snort set up with a relatively small selection of rules (I'd say about half)
          squid set up as a transparent proxy
          System_Patches which I've never used

          But as I said it's not "out of memory" on the main status page, far from it. Just don't know where to look or what to tweak.

          G

          1 Reply Last reply Reply Quote 0
          • KOM
            KOM last edited by

            You have pretty much every heavy package installed. Squid can suck up amazing gobs of RAM all by itself. Squid is practically useless for caching these days, so unless you're using it for URL filtering with squidguard, I would remove it. If you definitely need squid, what do you have set for its Memory Cache Size?

            1 Reply Last reply Reply Quote 0
            • X
              xxGBHxx last edited by

              Yeh I have been considering removing Squid for that reason.

              That said, as I mentioned I have 6GB allocated to the FW and it's hovering between 30-40% according to the status page. I've not yet seen it go over 3GB used so it's barely touching the sides.

              I should mention this is just for my home system, so it's just 3 people hitting it on a 80Mbit broadband line so it's not exactly getting hammered. It also has shared use of a Intel I7 2600 processor which sits at about 45% utilization. I don't think the resources available to the VM are the issue.

              The Memory Cache Size is set to 1024 so well below the 50% maximum size. Unless these applications are using RAM that isn't reported on the Dashboard then it's no where near limiting.

              Thanks again for having a try though.

              G

              1 Reply Last reply Reply Quote 0
              • stephenw10
                stephenw10 Netgate Administrator last edited by

                @xxgbhxx said in Hard Crashing - Out of Memory:

                536870912

                It's exhausting the 512MB limit for php processes. It appears to be trying to allocate 375MB:

                	printf("<a href=\"diag_dump_states.php?ruleid=%s\" data-toggle=\"popover\" data-trigger=\"hover focus\" title=\"%s\" ",
                	    $rulesid, gettext("States details"));
                

                So when you are mousing over something in the ruleset? Do you have huge aliases? A very large number of states?

                Steve

                X 1 Reply Last reply Reply Quote 0
                • X
                  xxGBHxx last edited by

                  I don't have a large number of states - I've not noticed it goes over 1000 total for the entire firewall. I'm not sure what you'd call huge aliases but within Pf none of the aliases are over 20 IP's. In PfBlockerNG I've got a number (17) of blocklists defined and the largest one when I do a refresh is about 80,000 IP's so tiny on the scale of lists.

                  I've just gone and cycled through the interfaces it seems to be my OpenVPN interface that's causing the issue. Even when I just click on it to browse to it that's when it locks up and it's now giving me a 504 error. Thing is I can't get into the GUI to see what rules are there that might be causing the problems. How can I get to the rules and disable/delete to see if that solves the problem?

                  G

                  Gertjan 1 Reply Last reply Reply Quote 0
                  • X
                    xxGBHxx last edited by

                    Ok this time now that it's re-started I get

                    PHP errors
                    
                        PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 88, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 375427072 bytes) @ 2019-03-13 18:43:03
                        PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 491, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 532692992 bytes) @ 2019-03-13 20:31:43
                    
                    General
                    
                        Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN wizard' @ 2019-03-13 23:00:09
                        Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN wizard' @ 2019-03-13 23:00:46
                    

                    So it's clearly those last two lines. As I mentioned when I click on the tab it crashed the firewall so how do I get in there to delete the offending rule. The only range under "Internal_Subnets" is 172.17.10.0/24 so I don't understand why it's causing these problems.

                    Any ideas?

                    G

                    Gertjan 1 Reply Last reply Reply Quote 0
                    • X
                      xxGBHxx @stephenw10 last edited by

                      Hi,

                      @stephenw10 any ideas?

                      Thanks

                      Gary

                      1 Reply Last reply Reply Quote 0
                      • Gertjan
                        Gertjan @xxGBHxx last edited by

                        @xxgbhxx said in Hard Crashing - Out of Memory:

                        The only range under "Internal_Subnets" is 172.17.10.0/24 so I don't understand why it's causing these problems.

                        This particular alias is just the drip that overflows the bucket.

                        A radical test to check if you are on the right track : tell PHP it can use more memory. Do this by changing the php.ini file.

                        Edit : etc/rc.php_ini_setup - line 181 - add " memory_limit = 256M "

                        ....
                        ; File generated from /etc/rc.php_ini_setup
                        memory_limit = 256M
                        ...
                        

                        and restart PHP using console option 16.

                        Now test.

                        No "help me" PM's please. Use the forum.

                        X 1 Reply Last reply Reply Quote 0
                        • X
                          xxGBHxx @Gertjan last edited by

                          @gertjan said in Hard Crashing - Out of Memory:

                          etc/rc.php_ini_setup

                          Hi thanks for that.

                          So I tried 256M and it had no effect. I increased it to 1024M restarting PHP with option 16 both times and it had no effect.

                          PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 708, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 375398400 bytes) @ 2019-03-14 14:04:56
                          

                          It doesn't seem to increase the allowed memory size at all either way.

                          Any other ideas?

                          G

                          1 Reply Last reply Reply Quote 0
                          • Gertjan
                            Gertjan last edited by Gertjan

                            Place this 'code' in a file called whatever.php into the GUI webroot (/usr/local/www/) :

                            <?php
                            phpinfo();
                            ?>
                            

                            and open the file like
                            http://pfsense.yourlan.tld/whatever.php

                            Check the several occurrences of 'memory' : I can't find 512M (!!) or 536870912 bytes ....
                            "memory_limit" is the value that should have been changed. With 6 Gb you could even make that value bigger.

                            Btw :

                            @xxgbhxx said in Hard Crashing - Out of Memory:

                            (only about 20 lists and none of them more than half a million IP's)

                            Serious ??
                            You really ask unbound, the DNS server, to restart with lists close to half a million of IP's ??
                            Woooow.

                            No "help me" PM's please. Use the forum.

                            X 1 Reply Last reply Reply Quote 0
                            • X
                              xxGBHxx @Gertjan last edited by

                              @gertjan

                              memory_limit = 1024M for both the local and master value

                              Not sure what your reference to DNS means. The lists are all IP's so wouldn't go anywhere near DNS.

                              Thanks again though

                              G

                              1 Reply Last reply Reply Quote 0
                              • KOM
                                KOM last edited by

                                He might have meant pfBlocker, not Unbound.

                                1 Reply Last reply Reply Quote 0
                                • Gertjan
                                  Gertjan last edited by

                                  @kom said in Hard Crashing - Out of Memory:

                                  He might have meant pfBlocker, not Unbound.

                                  Or both.
                                  pfBlocker populate lists parsed by unbound.

                                  (I guess it's time that I replay with pfBlocker - but don't do what to block, or who ...)

                                  No "help me" PM's please. Use the forum.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10
                                    stephenw10 Netgate Administrator last edited by

                                    Or more likely that pfBlocker can add lists to both but DNSBL does not have to be used.

                                    I believe this is the only line that would need changing:
                                    https://github.com/pfsense/pfsense/blob/64d2dd619973fa986389de2b5db43b0043c5ddd6/src/etc/inc/config.inc#L43
                                    But I have not tried it and you should only do so as a test. It should not be trying to allocate that ammount of RAM to open the page.

                                    Steve

                                    X 1 Reply Last reply Reply Quote 0
                                    • X
                                      xxGBHxx last edited by

                                      @kom

                                      Ah

                                      pfBlocker handles lists many millions of IP's in size - well according to the 100's of posts I've read. Many people use much larger IP lists than I do.

                                      Either way it's not going anywhere near Unbound as the lists I use are all IP address so there is no resolution to be done unless I'm missing something obvious.

                                      Either way the firewall is no where near ram, disk or CPU limits.

                                      G

                                      1 Reply Last reply Reply Quote 0
                                      • X
                                        xxGBHxx @stephenw10 last edited by

                                        @stephenw10

                                        Do I need to do a firewall re-start for it to pick up the change?

                                        G

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10
                                          stephenw10 Netgate Administrator last edited by

                                          You would likely need to restart php and the webgui from the console menu at least. It might require a reboot.

                                          Steve

                                          X 1 Reply Last reply Reply Quote 0
                                          • X
                                            xxGBHxx @stephenw10 last edited by

                                            @stephenw10

                                            Increased that value to 2048 and restarted PHP/WEB. Also rebooted. Hasn't allowed me to get into that tab in the GUI as I still get a 504 error however it has now stopped giving me out of memory errors.

                                            One step forwards but I sense no nearer!

                                            Thanks again.

                                            Any other ideas?

                                            G

                                            1 Reply Last reply Reply Quote 0
                                            • stephenw10
                                              stephenw10 Netgate Administrator last edited by

                                              2048 is huge, is it exhausting the system RAM?

                                              It probably isn't because I would expect you might hit this before that:
                                              https://github.com/pfsense/pfsense/blob/RELENG_2_4_4/src/etc/rc.php_ini_setup#L271

                                              Check /tmp/php_errors.txt

                                              Check the main system log.

                                              I assume restarting php and the wbegui from the menu does not allow you back in?

                                              Steve

                                              X 1 Reply Last reply Reply Quote 0
                                              • X
                                                xxGBHxx @stephenw10 last edited by xxGBHxx

                                                @stephenw10

                                                I did it to ensure that exhausting memory wasn't the problem - appreciate it's way more than needed. I have plenty of system RAM (6GB) allocated to the VM so 2GB isn't anywhere near the limit for the system.

                                                I'm certain it's something that's corrupted with the OpenVPN rule I have as it's referencing a table that doesn't exist (assuming it's case sensitive)

                                                php_errors is a zero byte file.

                                                Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.

                                                Nothing I do allows me to click on the OpenVPN tab under "Rules". The rest of the GUI works fine and does not crash.

                                                I will have a fiddle.

                                                G

                                                Gertjan 1 Reply Last reply Reply Quote 0
                                                • Gertjan
                                                  Gertjan @xxGBHxx last edited by

                                                  @xxgbhxx said in Hard Crashing - Out of Memory:

                                                  Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.

                                                  What are you doing to check this ?

                                                  No "help me" PM's please. Use the forum.

                                                  X 1 Reply Last reply Reply Quote 0
                                                  • X
                                                    xxGBHxx @Gertjan last edited by

                                                    @gertjan @stephenw10

                                                    I'm just catting the file like I would any other log file so

                                                    cat /var/log/system.log
                                                    

                                                    I've renamed it to system.log.old and restarted. Log is now logging fine again.

                                                    The only error/issue I can see is

                                                    Mar 14 18:20:20 pfsense2 php-fpm[340]: /rc.filter_configure_sync: New alert found: Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN  wizard'
                                                    

                                                    Which is the same error I get if I click on the OpenVPN tab in the GUI (which then goes on to crash the FW)

                                                    I need some way of directly editing the rules for that tab to remove this reference (assuming that's where it is) so that the GUI stops crashing.

                                                    Apart from not being able to get into this part of the GUI the FW seems to be performing as I'd expect.

                                                    G

                                                    RonpfS 1 Reply Last reply Reply Quote 0
                                                    • RonpfS
                                                      RonpfS @xxGBHxx last edited by

                                                      @xxgbhxx said in Hard Crashing - Out of Memory:

                                                      I'm just catting the file like I would any other log file so
                                                      cat /var/log/system.log

                                                      Those files are circular files, the have to be read with

                                                       clog /var/log/system.log
                                                      

                                                      They are managed with Status / System Logs / Settings. The drawback is that resetting will reset all log files.

                                                      2.4.5-RELEASE-p1 (amd64)
                                                      Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                                      Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                                      1 Reply Last reply Reply Quote 1
                                                      • stephenw10
                                                        stephenw10 Netgate Administrator last edited by

                                                        Check the ruleset in /tmp/rules.debug.

                                                        If you made a change that that you think is causing this you can roll back to a previous config from the console menu.

                                                        Steve

                                                        X 1 Reply Last reply Reply Quote 0
                                                        • X
                                                          xxGBHxx @stephenw10 last edited by

                                                          @stephenw10

                                                          OK we have progress!

                                                          There are probably 500+ lines in there that say

                                                          pass  in log  quick  on $OpenVPN inet from $Internal_Subnets to any tracker 1424387567 keep state  label "USER_RULE: OpenVPN  wizard"
                                                          

                                                          That is clearly the issue.

                                                          I have NO idea where they've come from - I've not run the OpenVPN wizard as far as I can remember. That said of course I might have and not remembered though why would it have just randomly created 100's of entries in that ruleset?

                                                          If I did run it I have absolutely no way of knowing when. Is there absolutely no way I can just remove the offending lines from the OpenVPN rules from the command line?

                                                          G

                                                          1 Reply Last reply Reply Quote 0
                                                          • Gertjan
                                                            Gertjan @xxGBHxx last edited by Gertjan

                                                            Ok, then is becomes more clear.

                                                            @xxgbhxx said in Hard Crashing - Out of Memory:

                                                            I've just gone and cycled through the interfaces it seems to be my OpenVPN interface that's causing the issue. Even when I just click on it to browse to it that's when it locks up and it's now giving me a 504 error.

                                                            There is something with the rules on that interface. The GUI throws 504.
                                                            When rules are generated, PHP runs out of memory.

                                                            Export a backup of your config, and remove all firewall rules. Import back in again, rebuild your rules.

                                                            No "help me" PM's please. Use the forum.

                                                            X 1 Reply Last reply Reply Quote 0
                                                            • X
                                                              xxGBHxx @Gertjan last edited by

                                                              @gertjan

                                                              OK sorted.

                                                              Thank you very much.

                                                              When I exported the rules it created a 12MB 500,000+ line file. What was extraordinarily bizarre was that when I saved the file from the FW gui, the OS couldn't see the file so I couldn't open it to edit it. Even more bizarre was that if I went to save in the GUI, the FW save dialog could see the files but the OS and any other app couldn't.

                                                              In the end I opened the file "live" into Notepad++ but even then I couldn't actually edit the file. I had to save the file in Notepad++ and then re-open it to actually be able to edit it. First time I've ever had that.

                                                              Anyway, I deleted over 500,000 lines of config added by the OpenVPN Wizard and that took the file from 12MB and 550,000 lines to 61k and 2700 lines. Reloaded that back into the FW and with a bit more tidying up everything is working again and not crashing.

                                                              Thank you very much to everyone who chipped in. Was a "simple" fix I really should have come up with myself in the end but thanks for the help. If nothing else it's massively increased my knowledge.

                                                              G

                                                              1 Reply Last reply Reply Quote 0
                                                              • stephenw10
                                                                stephenw10 Netgate Administrator last edited by

                                                                Wow! Never seen anything even remotely like that. Weird.

                                                                I also noticed there are two aliases there Internal_SUbnets and Internal_Subnets. That seems suspicious, you might want to just check your config file now to see if both are still present.

                                                                Steve

                                                                X 1 Reply Last reply Reply Quote 0
                                                                • X
                                                                  xxGBHxx @stephenw10 last edited by

                                                                  @stephenw10

                                                                  Hi,

                                                                  I saw that too and checked for "SU" and "Su" and only "Internal_Subnets " exists so I have no idea where that came from either.

                                                                  I clearly must have accidentally clicked on on the Wizard at some point in the last few days and not noticed leading it to get very confused.

                                                                  It all seems good now though.

                                                                  Thanks again for your help

                                                                  G

                                                                  1 Reply Last reply Reply Quote 0
                                                                  • First post
                                                                    Last post