Problem with Squid + HTTPS/SSL interception consuming all memory

[alesilvam]

Hi Guys,

The pfSense of my work is reaching 99% of its memory consumption, at the beginning of the day the memory consumption is 5...6%, the consumption starts to increase during the day, until the half of the day is already at 45...50% and after a few minutes shoot 55...60...70...85...99% in a matter of seconds, if you leave the pfSense collapses and crashes.

For this problem, the squid is the service that is consuming more memory. I have noticed with the tests that the villain is related with the option "HTTPS/SSL Interception: Enable SSL filtering" + "SSL/MITM Mode: Splice All", I leaved this option disabled for 2 days for testing and the memory consumption was stable between 5...10%.

One way to avoid the crash is to click "Clear Disk Cache NOW" from the menu Services / Squid Proxy Server / Local Cache / Squid Hard Disk Cache Settings, with that memory consumption returns to its normal.

I don't know what else to do, i tried several suggestions of cache settings and nothing solves it, also tried to do a new installation of pfSense and did not solve the problem.

SETTINGS:

pfSense 2.4.4-RELEASE (amd64)
squid 0.4.44_6
squidGuard 1.16.18_1
Lightsquid 3.0.6_4

CPU Type:
Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
Current: 3000 MHz, Max: 3001 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (inactive)

Memory: 16GB
Hard Disk: 1TB

SQUID PROXY SERVER
Transparent HTTP Proxy: Enable
HTTPS/SSL Interception: Enable SSL filtering
SSL/MITM Mode: Splice All

LOCAL CACHE
== Squid Cache General Settings ==
Cache Replacement Policy: Heap LFUDA
Low-Water Mark in %: 60
High-Water Mark in %: 65

== Squid Hard Disk Cache Settings ==
Hard Disk Cache Size: 100
Hard Disk Cache System: ufs
Level 1 Directories: 16
Minimum Object Size: 0
Maximum Object Size: 4

== Squid Memory Cache Settings ==
Memory Cache Size: 64
Maximum Object Size in RAM: 256
Memory Replacement Policy: Heap GDSF

Can anyone help me, please?

[GixD]

same issue, its solved?

[alesilvam]

Unfortunately no, we are thinking of replacing pfSense.
An interim solution was to install the Cron package and set up to restart the Squid service 5 times during the day:

/usr/local/etc/rc.d/squid.sh restart

[GixD]

i see, im try to disable access.log on webconfig and its helped for decreasing ram for this issue.

[alesilvam]

I disabled the log of Squid and Squidguard by Webconfig and in my case I noticed little difference in memory consumption.

[atom1983]

I have the same problem as this problem and I have no chance to help

[GixD]

@atom1983 lets pray togther

[beauw]

All..... The script came from user Remzej. I have it on a cron job to check every 5 minutes (we are a busy proxy environment)...

*/2 * * * * root /usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/monitor_memory_usage.php

#!/usr/local/bin/php-cgi -f
<?php
/*

• monitor_memory_usage.php

• part of pfSense (https://www.pfsense.org)

• Copyright (c) 2011-2015 Rubicon Communications, LLC (Netgate)

• All rights reserved.

• Licensed under the Apache License, Version 2.0 (the "License");

• you may not use this file except in compliance with the License.

• You may obtain a copy of the License at

• http://www.apache.org/licenses/LICENSE-2.0

• Unless required by applicable law or agreed to in writing, software

• distributed under the License is distributed on an "AS IS" BASIS,

• WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

• See the License for the specific language governing permissions and

• limitations under the License.
  */
  require_once('config.inc');
  require_once('util.inc');
  require_once('squid.inc');
  global $config;

  // Monitor memory usage by remzej
  // Get SWAP usage funtion
  function swap_usage() {
  exec("/usr/sbin/swapinfo", $swap_info);
  $swap_used = "";
  foreach ($swap_info as $line) {
  if (preg_match('/(\d+)%$/', $line, $matches)) {
  $swap_used = $matches[1];
  break;
  }
  }
  return $swap_used;
  }

  // Get memory usage function
  function mem_usage() {
  $memory = "";
  exec("/sbin/sysctl -n vm.stats.vm.v_page_count vm.stats.vm.v_inactive_count " .
  "vm.stats.vm.v_cache_count vm.stats.vm.v_free_count", $memory);

   $totalMem = $memory[0];
   $availMem = $memory[1] + $memory[2] + $memory[3];
   $usedMem = $totalMem - $availMem;
   $memUsage = round(($usedMem * 100) / $totalMem, 0);
  
   return $memUsage;
  

  }

  // Get memory and SWAP usage value
  $memusage_pct = mem_usage();
  $swapusage_pct = swap_usage();

  // Display memory usage
  echo "Memory Usage: " . $memusage_pct . "%" . PHP_EOL;
  echo "SWAP Usage: " . $swapusage_pct . "%" . PHP_EOL;

  // If memory usage is above 90%, stop and restart squid services.
  if (($memusage_pct > 90) or ($swapusage_pct > 80)) {
  squid_stop_monitor();
  if (is_service_running('squid')) {
  stop_service("squid");
  }
  squid_restart_services();
  log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent, Swap Usage is $swap_usage percent, stopping and restarting services.")));
  }
  log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent and Swap Usage is $swapusage_pct")));
  ?>