Squid use all memory ram

remzej

The swapstate_check.php won't execute because there is no cache partition mounted found in the filesystem. I checked the source code line by line and by creating a test.php from there I can tell that it will never execute because of the conditions doesn't meet. We can modify the swapstate_check.php to monitor the swap.state file size and clean the cache if this file exceeds the specified amount we set in the script.

emax4

Yes, for now I think the best solution is to use a cron to stop and knit the squid, as says miquim. I think we have to wait for an update to see if they solve the problem. If someone finds the solution, please advise

remzej

It's hard to monitor the swap.state filesize and the SWAP usage percentage because it grows dynamically. In my case I modified the swapstate_check.php code to execute if the cache folder size reach 250MB or the swap.state filesize reach 640KB.

TomS

I'm also seeing this memory exhaustion problem on my box. I do not use any disk cache and only want to be prepared to be able to block web sites just in case I'm told to do so by government authorities or law enforcement. Currently, I'm not blocking any websites. One more thing I'm currently doing with Squid is creating log files (i.e. which web sites have been opened by the users)
The system is an anonymous WiFi hotspot with approx. 300 users per day.
For some reason, the memory consumption is slowly increasing. At first only the memory itself, but later also the swap space until all the free memory is occupied and the pfSense is crashing.

I saw bbassotti's post and the configure differences in the squid versions… I'm using squid V 3.5.24 which has exactly the same configure options as V3.5.23 and it shows the same behavior... Could this be the root cause?
What else could be the reason for this strange behavior?
Is there anyone working on this issue? If there's a way I can contribute or help finding and fixing this issue, please just let me know.

Best,
TomS

remzej

@TomS:

I'm also seeing this memory exhaustion problem on my box. I do not use any disk cache and only want to be prepared to be able to block web sites just in case I'm told to do so by government authorities or law enforcement. Currently, I'm not blocking any websites. One more thing I'm currently doing with Squid is creating log files (i.e. which web sites have been opened by the users)
The system is an anonymous WiFi hotspot with approx. 300 users per day.
For some reason, the memory consumption is slowly increasing. At first only the memory itself, but later also the swap space until all the free memory is occupied and the pfSense is crashing.

I saw bbassotti's post and the configure differences in the squid versions… I'm using squid V 3.5.24 which has exactly the same configure options as V3.5.23 and it shows the same behavior... Could this be the root cause?
What else could be the reason for this strange behavior?
Is there anyone working on this issue? If there's a way I can contribute or help finding and fixing this issue, please just let me know.

Best,
TomS

This was posted as a bug for pfSense v2.3.x I read someone posted that this issue doesn't exist in pfSense v2.4 beta. We hope they will release an update soon to fix this problem.

emax4

@remzej:

@TomS:

I'm also seeing this memory exhaustion problem on my box. I do not use any disk cache and only want to be prepared to be able to block web sites just in case I'm told to do so by government authorities or law enforcement. Currently, I'm not blocking any websites. One more thing I'm currently doing with Squid is creating log files (i.e. which web sites have been opened by the users)
The system is an anonymous WiFi hotspot with approx. 300 users per day.
For some reason, the memory consumption is slowly increasing. At first only the memory itself, but later also the swap space until all the free memory is occupied and the pfSense is crashing.

I saw bbassotti's post and the configure differences in the squid versions… I'm using squid V 3.5.24 which has exactly the same configure options as V3.5.23 and it shows the same behavior... Could this be the root cause?
What else could be the reason for this strange behavior?
Is there anyone working on this issue? If there's a way I can contribute or help finding and fixing this issue, please just let me know.

Best,
TomS

This was posted as a bug for pfSense v2.3.x I read someone posted that this issue doesn't exist in pfSense v2.4 beta. We hope they will release an update soon to fix this problem.

Hopefully they will soon solve the problem, if anyone knows anything please let me know

remzej

I want to share my simple PHP code to monitor the memory and SWAP usage every 5 minutes using cron.

This PHP code will automatically stop and restart squid services when memory and SWAP usage goes beyond 90% and 75% respectively without deleting the existing hard disk cache and swap.state file. The swapstate_check.php script will handle it if swap.state file goes beyond 1GB size.

Attached here is the monitor_memory_usage.txt file you can download and save to your preferred location on your pfSense box. Don't forget to change the file extention from .txt to .php.

In my case I saved it in /usr/local/pkg/ directory. Don't forget to set the file permission to rwxr-xr-r.

Next thing to do, is to add a cron job that will look like this: (Be sure to install cron package to be able to do this)

*/5 * * * * root /usr/local/pkg/monitor_memory_usage.php

then save it. That's it! I hope this simple hard work and research will help some of you that has high memory usage problem.

Thanks!
remzej

monitor_memory_usage.txt

marcelloc

Thanks for the contribution. :)

gersonofstone

@marcelloc:

Thanks for the contribution. :)

this change was in last update for squid?

marcelloc

@:

this change was in last update for squid?

Not sure, it was updated recently. Did you updated the package?

gersonofstone

Update squid to 1.16.2

And make changes

SystemAdvancedFirewall & NAT
Firewall Optimization Options Agrresive
IP Do-Not-Fragment compatibility
Clear invalid DF bits instead of dropping the packets

*Disable Firewall Scrub
Disables the PF scrubbing option which can sometimes interfere with NFS traffic.

SystemAdvancedNetworking

Hardware Checksum Offloading *Disable
Hardware TCP Segmentation Offloading *Disable
Hardware Large Receive Offloading *Disable

this for NIC realteck

:)

remzej

Latest squid package v0.4.37 already fixed this bug.

Gravitator

Hello!

Version PFSense:

2.3.4-RELEASE (i386)
built on Wed May 03 15:22:11 CDT 2017
FreeBSD 10.3-RELEASE-p19

Latest squid package v0.4.37 with SquidGuard 1.16.2 in transparent mode with SSL interception (strip all).

Gradually increasing memory consumption, the overflow resets.

@:

Update squid to 1.16.2

And make changes

SystemAdvancedFirewall & NAT
Firewall Optimization Options Agrresive

IP Do-Not-Fragment compatibility
Clear invalid DF bits instead of dropping the packets

*Disable Firewall Scrub
Disables the PF scrubbing option which can sometimes interfere with NFS traffic.

SystemAdvancedNetworking

Hardware Checksum Offloading *Disable
Hardware TCP Segmentation Offloading *Disable
Hardware Large Receive Offloading *Disable

this for NIC realteck

:)

These settings did not help.

Will the transition to x64?

emax4

Hello gravitator

With the update of Squid is enough for the solution of the problem, greetings!

Gravitator

@emax4:

Hello gravitator

With the update of Squid is enough for the solution of the problem, greetings!

Hello…

Unfortunately, you have the latest version of package squid - 0.4.37. The problem is not resolved.
Pfsense is also updated to the version from 20.07.2017.

emax4

@Gravitator:

@emax4:

Hello gravitator

With the update of Squid is enough for the solution of the problem, greetings!

Hello…

Unfortunately, you have the latest version of package squid - 0.4.37. The problem is not resolved.
Pfsense is also updated to the version from 20.07.2017.

Hello

I have version 0.4.37 of squid and 2.3.4 of Pfsense, I no longer have the problem.

With the update resolved

Sko

@Impatient:

I have had the issue since pfSense version 2.3 and the only way to help alleviate the issue was to set the Maximum Object size in Ram back to 256 kb.

Thank you, this helped me a lot. I was fiddling with squid settings, but it always ate all the RAM eventually. Setting 'Maximum Object Size in RAM' to default stopped it.

pfSense 2.3.4_p1 (amd64)
squid 0.4.37

Gravitator

Hello!
Рroblem solved!

This problem is observed only on i386. On amd-64 squid is working fine. Memory consumption was normal.

PFSense 2.3.4-1 (x64).
Squid packages 0.4.37.

doktornotor

If you are seeing this with HTTPS, see http://bugs.squid-cache.org/show_bug.cgi?id=4005

No fix for 3.5.x. Complain upstream.

Note: Setting "SSL Certificate Deamon Children" to 1 (or just something lower that the default 5) should help due to the way how dynamic_cert_mem_cache_size is configured by the package, but obviously will cause issues with busy proxies.

beauw

Okay - remzej's script saved me so much grief.

I run PFsense with about 1200 clients and the squid memory continually increases over the day until it locks up with no reboot. Even on the newest versions.

I adjusted the script to reboot only on memory (65% for me) and it works like a charm.

Once I got the cron running, it checks every 15 mins. At the most, the reboot results in 15 seconds downtime for the network which is palatable.

Thanks again remzej for your contribution….