Problem with Squid + HTTPS/SSL interception consuming all memory



  • Hi Guys,

    The pfSense of my work is reaching 99% of its memory consumption, at the beginning of the day the memory consumption is 5...6%, the consumption starts to increase during the day, until the half of the day is already at 45...50% and after a few minutes shoot 55...60...70...85...99% in a matter of seconds, if you leave the pfSense collapses and crashes.

    For this problem, the squid is the service that is consuming more memory. I have noticed with the tests that the villain is related with the option "HTTPS/SSL Interception: Enable SSL filtering" + "SSL/MITM Mode: Splice All", I leaved this option disabled for 2 days for testing and the memory consumption was stable between 5...10%.

    One way to avoid the crash is to click "Clear Disk Cache NOW" from the menu Services / Squid Proxy Server / Local Cache / Squid Hard Disk Cache Settings, with that memory consumption returns to its normal.

    I don't know what else to do, i tried several suggestions of cache settings and nothing solves it, also tried to do a new installation of pfSense and did not solve the problem.

    SETTINGS:

    pfSense 2.4.4-RELEASE (amd64)
    squid 0.4.44_6
    squidGuard 1.16.18_1
    Lightsquid 3.0.6_4

    CPU Type:
    Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
    Current: 3000 MHz, Max: 3001 MHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (inactive)

    Memory: 16GB
    Hard Disk: 1TB

    SQUID PROXY SERVER
    Transparent HTTP Proxy: Enable
    HTTPS/SSL Interception: Enable SSL filtering
    SSL/MITM Mode: Splice All

    LOCAL CACHE
    == Squid Cache General Settings ==
    Cache Replacement Policy: Heap LFUDA
    Low-Water Mark in %: 60
    High-Water Mark in %: 65

    == Squid Hard Disk Cache Settings ==
    Hard Disk Cache Size: 100
    Hard Disk Cache System: ufs
    Level 1 Directories: 16
    Minimum Object Size: 0
    Maximum Object Size: 4

    == Squid Memory Cache Settings ==
    Memory Cache Size: 64
    Maximum Object Size in RAM: 256
    Memory Replacement Policy: Heap GDSF

    Can anyone help me, please?



  • same issue, its solved?



  • Unfortunately no, we are thinking of replacing pfSense.
    An interim solution was to install the Cron package and set up to restart the Squid service 5 times during the day:

    /usr/local/etc/rc.d/squid.sh restart



  • i see, im try to disable access.log on webconfig and its helped for decreasing ram for this issue.



  • I disabled the log of Squid and Squidguard by Webconfig and in my case I noticed little difference in memory consumption.



  • I have the same problem as this problem and I have no chance to help



  • @atom1983 lets pray togther 😭



  • All..... The script came from user Remzej. I have it on a cron job to check every 5 minutes (we are a busy proxy environment)...

    */2 * * * * root /usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/monitor_memory_usage.php

    #!/usr/local/bin/php-cgi -f
    <?php
    /*

    • monitor_memory_usage.php

    • part of pfSense (https://www.pfsense.org)

    • Copyright (c) 2011-2015 Rubicon Communications, LLC (Netgate)

    • All rights reserved.

    • Licensed under the Apache License, Version 2.0 (the "License");

    • you may not use this file except in compliance with the License.

    • You may obtain a copy of the License at

    • http://www.apache.org/licenses/LICENSE-2.0

    • Unless required by applicable law or agreed to in writing, software

    • distributed under the License is distributed on an "AS IS" BASIS,

    • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

    • See the License for the specific language governing permissions and

    • limitations under the License.
      */
      require_once('config.inc');
      require_once('util.inc');
      require_once('squid.inc');
      global $config;

      // Monitor memory usage by remzej
      // Get SWAP usage funtion
      function swap_usage() {
      exec("/usr/sbin/swapinfo", $swap_info);
      $swap_used = "";
      foreach ($swap_info as $line) {
      if (preg_match('/(\d+)%$/', $line, $matches)) {
      $swap_used = $matches[1];
      break;
      }
      }
      return $swap_used;
      }

      // Get memory usage function
      function mem_usage() {
      $memory = "";
      exec("/sbin/sysctl -n vm.stats.vm.v_page_count vm.stats.vm.v_inactive_count " .
      "vm.stats.vm.v_cache_count vm.stats.vm.v_free_count", $memory);

       $totalMem = $memory[0];
       $availMem = $memory[1] + $memory[2] + $memory[3];
       $usedMem = $totalMem - $availMem;
       $memUsage = round(($usedMem * 100) / $totalMem, 0);
      
       return $memUsage;
      

      }

      // Get memory and SWAP usage value
      $memusage_pct = mem_usage();
      $swapusage_pct = swap_usage();

      // Display memory usage
      echo "Memory Usage: " . $memusage_pct . "%" . PHP_EOL;
      echo "SWAP Usage: " . $swapusage_pct . "%" . PHP_EOL;

      // If memory usage is above 90%, stop and restart squid services.
      if (($memusage_pct > 90) or ($swapusage_pct > 80)) {
      squid_stop_monitor();
      if (is_service_running('squid')) {
      stop_service("squid");
      }
      squid_restart_services();
      log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent, Swap Usage is $swap_usage percent, stopping and restarting services.")));
      }
      log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent and Swap Usage is $swapusage_pct")));
      ?>