Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Memory Leak

    Scheduled Pinned Locked Moved General pfSense Questions
    35 Posts 8 Posters 9.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      Jason Crowley
      last edited by

      We are not running sipproxd or any other add-on package for that matter.  We have about 20 phones that use tftp to download configuration from an Internet-based SIP provider, so we use tftp-proxy a bit.  Other than that most of our traffic is SIP/RTP, and plain old Internet browsing (DNS, HTTP, HTTPS, etc.).

      It's not a very complicated setup.  We had some traffic shaping set up for the VOIP traffic, and we used the "Single Wan multi Lan" wizard to set it up.  VOIP traffic is on a different VLAN than the rest of the traffic, so that wizard seemed appropriate.  I have since disabled the traffic shaper in an attempt to narrow down the problem.

      1 Reply Last reply Reply Quote 0
      • K
        kejianshi
        last edited by

        Strange - My memory utilization has creeped up to over 80% now, but I suspect it won't go much higher and I think its pure squid usage.  What do your states look like?

        1 Reply Last reply Reply Quote 0
        • J
          Jason Crowley
          last edited by

          States almost never go above 2000.  They're pretty consistent too.  There aren't spikes in the state table size when the problem occurs.

          system_states_week.png
          system_states_week.png_thumb
          system_states_month.png
          system_states_month.png_thumb

          1 Reply Last reply Reply Quote 0
          • J
            Jason Crowley
            last edited by

            It's also interesting to note that when the firewall starts having issues, neither the processor usage nor the memory usage is above 50%.

            1 Reply Last reply Reply Quote 0
            • J
              Jason Crowley
              last edited by

              kejianshi,

              The dansguardian processes were in a post from timthetortoise who says he's not having issues.  I (Jason, the OP) am not running dansguardian or ClamAV, but I am having a memory leak issue.  My process lists is in the original post.

              Thanks for your help though!

              1 Reply Last reply Reply Quote 0
              • K
                kejianshi
                last edited by

                HAHA - Yea.  I saw that after I posted and realized my mistake and deleted it…  I was hoping no one saw it.  You caught me!

                Question.  Is this system upgraded or is it a fresh clean install?

                I've been told repeatedly that that shouldn't matter, but I'm not sure how true that is.

                1 Reply Last reply Reply Quote 0
                • J
                  Jason Crowley
                  last edited by

                  The system was upgraded to 2.1–not a clean install.  The problem existed both on 2.0.2 and continues on 2.1.

                  1 Reply Last reply Reply Quote 0
                  • K
                    kejianshi
                    last edited by

                    Is a new clean install out of the question?

                    1 Reply Last reply Reply Quote 0
                    • J
                      Jason Crowley
                      last edited by

                      At this point, nothing is out of the question.  Of course a new install is a pain, but we have to solve the problem.  The last thing we did was turn off the traffic shaper all together.  The folks behind this firewall don't seem to consume their Internet pipe (10Mb symmetrical), so the traffic shaper isn't that big of a deal.  Now I'm just waiting for the memory leak to start again.  Sometimes it takes a few days; others it takes a month or more.

                      We're in a virtualized environment, so I've considered setting up a separate gateway in a high-availability configuration.  It'd give me somewhere to move traffic when the problem occurs assuming the memory leak doesn't show up on both firewalls at the same time.  That's a few hours work and I have to make sure I have extra Internet IPs available, but it would reduce the risk of having to incur a middle-of-the-workday outage.

                      1 Reply Last reply Reply Quote 0
                      • K
                        kejianshi
                        last edited by

                        "traffic shaper" - Is that no longer running?

                        1 Reply Last reply Reply Quote 0
                        • J
                          Jason Crowley
                          last edited by

                          The traffic shaper is no longer running.

                          1 Reply Last reply Reply Quote 0
                          • K
                            kejianshi
                            last edited by

                            I'd wipe the drive and install clean.  Unless your rules are special in some way, I would not even do a restore of settings.  I'd redo them by hand to be sure you didn't import trouble.  Is this a 64 or 32 bit install?  Because I'd also prefer 32bit.

                            1 Reply Last reply Reply Quote 0
                            • J
                              Jason Crowley
                              last edited by

                              For your information, I think this problem has followed us through several iterations of pfSense from 1.2.3 (i think) on an ALIX board to 2.0.1 on a PC to 2.0.2 on a virtual server to 2.1 on the same server.  I do think the same XML config has been ported through all of those environments–both 32- and 64-bit.  I guess I'm in agreement that a scratch rebuild might be the next good thing to do.

                              It's an amd64 install today.  I'm curious, why do you prefer 32-bit?  Does anyone else have thoughts on 32-bit vs 64-bit?

                              1 Reply Last reply Reply Quote 0
                              • K
                                kejianshi
                                last edited by

                                Because with less than 4GB ram you don't NEED 64bit version and I've seen more issues with 64 than 32.  Some would say its my imagination, but I don't think so.  I think the 32bit version is just working better under VMs.  I've experienced that anyway.  Seems others have also, but thats more of something I've noticed than a scientific study of the issue.

                                1 Reply Last reply Reply Quote 0
                                • chpalmerC
                                  chpalmer
                                  last edited by

                                  I have a customer on a 64 bit version and it behaves stupendously. Not one issue!  :)

                                  Triggering snowflakes one by one..
                                  Intel(R) Core(TM) i5-4590T CPU @ 2.00GHz on an M400 WG box.

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    Jason Crowley
                                    last edited by

                                    We also have at least six other 64-bit installs that are working flawlessly.  A couple of them have fairly complicated configurations with ~10 interfaces, ~20 IPsec tunnels, 100s of firewall & NAT rules, traffic shaping, etc.  With all of that, they're still able to sustain 100Mb/s throughput easily.  I'm curious if there would be performance degradation moving from 64-bit to 32-bit.

                                    1 Reply Last reply Reply Quote 0
                                    • K
                                      kejianshi
                                      last edited by

                                      The problems I personally experienced were on 64bit version on ESXi 5.

                                      Same install, nothing else modified and it worked fine.  Tried several times.

                                      I couldn't solve it, so I just stayed with 32bit version.

                                      Sounds like your hypervisor isn't same, so probably depends.

                                      Also, the problems I had were mainly either with NTP or with nothing working well with 5 or more interfaces.

                                      There are just too many variables in hardware  and network to say anthing for sure for everybody, but as a rule, if I don't need 64bit, I use 32bit where I can.  As for it being a point of contention for this or that person, I say do whatever floats your boat and works for you.

                                      There won't be any performance losses.  64bit is just able to address more RAM.  Thats all.

                                      1 Reply Last reply Reply Quote 0
                                      • chpalmerC
                                        chpalmer
                                        last edited by

                                        Jason: Did you ever solve this?

                                        I ended up just re-installing the firmware over the original install via the firmware page and that seems to have solved it for me here.

                                        My other watchguard "customer" had his unit do the same thing starting last Wednesday and build up to over 30% before we caught it.  Restarted Apinger and it has been normal since…

                                        Mikes.jpg
                                        Mikes.jpg_thumb

                                        Triggering snowflakes one by one..
                                        Intel(R) Core(TM) i5-4590T CPU @ 2.00GHz on an M400 WG box.

                                        1 Reply Last reply Reply Quote 0
                                        • J
                                          Jason Crowley
                                          last edited by

                                          Since we've been through at least four firmware updates on this firewall and the problem still persists, I'm doubting a firmware reinstall will fix the problem.  I'm also pretty sure apinger is not the problem as I believe that would cause memory usage to spike in user space.  I'm seeing wired kernel memory balloon.

                                          My plan at this point is to build another near-identical firewall through the web gui, download the XML config from the new and old ones, and compare.  Maybe that will reveal some anomaly.  This XML has been pulled up through the 1.2 versions to 2.1.  Maybe something bad slipped through.

                                          1 Reply Last reply Reply Quote 0
                                          • K
                                            kejianshi
                                            last edited by

                                            My memory does exactly the same thing.  Looks like utilization climbs and climbs and then it hits about 90%, teeters there for a while and then will drop back down to about 30% all by its self, with no reset, reboot or anything and no ill effects and then the cycle starts again.  I don't worry about it.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.