Squid possible memory leak
-
All,
Running an XG-1450 (and a backup) with CARP failover in a school environment now for 9 months with minimal issues and fantastic performance.
800+ clients, HTTPS transparent filtering (MITM) using squidguard, suricata in-line, 10 interfaces.
I've scoured message boards to learn about the underlying concepts of Squid to make sure I understand how memory is allocated (and possibly released) but I'm still running into memory consumption on a massive scale that isn't being released back to the available pool.
Essentially memory creeps up throughout the entire day until both the memory (32 GB) and swap memory (4GB) is completely consumed.
Memory and SWAP run at 99%. This causes chaos until reset.I've changed between no cache, large cache, log rotations, etc etc. Can't seem to find the issue.
The only remediation is rebooting the Squid service (not squidguard but squid). Results in a network halt for about 10-20 seconds. Would prefer to avoid.
I tried a CRON job also to restart squid, but it seems to have no effect.
- 0,6,12,18 * * * root /usr/local/etc/rc.d/squid restart
Version 2.4.2
Firewall optimization is normal
Power saving mode is Maximum (meaning highest performance)Details…..
20GB cache (aufs)
Mostly standard options in the cache settings.Integrations on squid:
http_port 10.10.10.1:3128;;url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf;url_rewrite_bypass off;url_rewrite_children 16 startup=8 idle=4 concurrency=0Below is a normal (ps uxawwd).
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 0 0.0 0.0 0 976 - DLs 1Jan18 0:07.38 [kernel]
root 11 1547.8 0.0 0 256 - RL 1Jan18 248857:52.83 - [idle]
root 12 0.7 0.0 0 1248 - WL 1Jan18 706:23.65 - [intr]
root 1 0.0 0.0 5024 132 - ILs 1Jan18 0:00.05 - /sbin/init –
nobody 83610 0.9 0.0 31868 4508 - S Thu08 13:41.87 |-- /usr/local/sbin/dnsmasq --all-servers -C /dev/null --dns-forward-max=5000 --cache-size=10000 --local-ttl=1
root 96600 0.8 2.2 1266252 723272 - Ss 4Jan18 558:32.04 |-- /usr/local/bin/suricata -i igb0 -D -c /usr/local/etc/suricata/suricata_61396_igb0/suricata.yaml --pidfile /var/run/suricata_igb061396.pid
root 93337 0.1 1.6 1020492 526956 - Ss 4Jan18 286:47.81 |-- /usr/local/bin/suricata -i igb1 -D -c /usr/local/etc/suricata/suricata_19793_igb1/suricata.yaml --pidfile /var/run/suricata_igb119793.pid
root 628 0.0 0.0 280396 5828 - Ss 1Jan18 0:26.38 |-- php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root 37781 1.9 0.1 290640 27940 - S 12:10 0:00.42 | |-- php-fpm: pool nginx (php-fpm)
root 26368 0.0 0.1 286544 23324 - S 12:10 0:00.03 |-- php-fpm: pool nginx (php-fpm) root 27417 0.0 0.0 21104 2644 - R 12:10 0:00.00 |-- ps uxawwd
root 642 0.0 0.0 19440 1112 - INs 1Jan18 0:00.20 |-- /usr/local/sbin/check_reload_status
root 644 0.0 0.0 19440 0 - IWN - 0:00.00 |-- check_reload_status: Monitoring daemon of check_reload_status root 658 0.0 0.0 9556 80 - Is 1Jan18 0:00.57 |-- /sbin/devd -q -f /etc/pfSense-devd.conf root 14390 0.0 0.0 10472 2000 - Ss 1Jan18 29:30.47 |-- /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf root 56843 0.0 0.0 10580 1940 - Is 1Jan18 0:00.02 |-- /usr/local/sbin/sshlockout_pf 15
root 22436 0.0 0.0 53492 4128 - Is 1Jan18 0:00.00 |-- /usr/sbin/sshd
root 26295 0.0 0.0 13084 844 - S Mon12 0:03.24 |-- /bin/sh /usr/local/pkg/sqpmon.sh
root 28561 0.0 0.0 6172 1928 - S 12:10 0:00.00 |-- sleep 55 root 27333 0.0 0.0 24448 4516 - Ss 1Jan18 0:39.85 |-- /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 28035 0.0 0.0 12696 2448 - Ss 1Jan18 20:13.91 |-- /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid root 38272 0.0 0.0 154244 3172 - Is 1Jan18 0:07.25 |-- /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 root 43348 0.0 0.0 8224 0 - IWs - 0:00.00 |-- /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 43813 0.0 0.0 8224 336 - I 1Jan18 0:00.36 |-- minicron: helper /usr/local/bin/ping_hosts.sh (minicron)
root 43969 0.0 0.0 8224 0 - IWs - 0:00.00 |-- /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root 44272 0.0 0.0 8224 336 - I 1Jan18 0:00.03 |-- minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) root 44581 0.0 0.0 8224 0 - IWs - 0:00.00 |-- /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data root 45019 0.0 0.0 8224 336 - I 1Jan18 0:00.00 |-- minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron)
root 48250 0.0 0.0 63052 6400 - Ss 2Jan18 7:13.61 |-- /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid
root 55874 0.0 0.1 66472 25592 - Is 11:46 0:00.00 |-- /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
squid 56059 75.9 16.2 5972892 5420300 - S 11:46 11:38.00 |-- (squid-1) -f /usr/local/etc/squid/squid.conf (squid) squid 56310 1.5 0.1 38216 17688 - S 11:46 0:20.96 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 56063 0.0 0.0 26984 6192 - I 11:46 0:00.01 | |-- (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd) squid 56116 0.0 0.0 26984 6192 - I 11:46 0:00.02 | |-- (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd) squid 56123 0.0 0.0 26984 6192 - I 11:46 0:00.01 | |-- (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd) squid 56216 0.0 0.0 26984 6192 - I 11:46 0:00.02 | |-- (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd) squid 56237 0.0 0.0 26984 6192 - I 11:46 0:00.01 | |-- (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd) squid 56460 0.0 0.1 38216 17692 - S 11:46 0:05.80 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 56648 0.0 0.1 38216 17688 - S 11:46 0:02.33 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 56777 0.0 0.1 38216 17688 - S 11:46 0:01.14 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 57101 0.0 0.1 38216 17684 - S 11:46 0:00.64 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 57142 0.0 0.1 38216 17676 - S 11:46 0:00.37 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 57371 0.0 0.1 38216 17684 - S 11:46 0:00.24 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 57589 0.0 0.1 38216 17672 - I 11:46 0:00.17 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 57696 0.0 0.1 38216 17628 - I 11:46 0:00.09 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58042 0.0 0.1 38216 17524 - I 11:46 0:00.07 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58120 0.0 0.0 38216 16632 - I 11:46 0:00.05 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58131 0.0 0.0 38216 16544 - I 11:46 0:00.04 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58390 0.0 0.0 36168 16104 - I 11:46 0:00.03 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58446 0.0 0.0 36168 15616 - I 11:46 0:00.03 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58569 0.0 0.0 34120 14608 - I 11:46 0:00.03 | |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) squid 58759 0.0 0.0 34120 14608 - I 11:46 0:00.03 |-- (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
root 56152 0.0 0.0 12496 676 - Is 1Jan18 0:06.36 |-- /usr/sbin/cron -s
root 57331 0.0 0.0 10288 0 - IWs - 0:00.00 |-- /usr/local/sbin/qstats -p /var/run/qstats.pid
root 68631 0.0 0.0 35660 0 - IWs - 0:00.00 |-- nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root 68677 0.0 0.0 37708 4784 - S 4Jan18 0:28.51 | |-- nginx: worker process (nginx)
root 68753 0.0 0.0 37708 4720 - S 4Jan18 0:35.38 | |-- nginx: worker process (nginx)
root 68852 0.0 0.0 37708 5116 - S 4Jan18 0:46.82 | |-- nginx: worker process (nginx)
root 69079 0.0 0.0 37708 5156 - S 4Jan18 0:58.66 |-- nginx: worker process (nginx) root 78850 0.0 0.0 10368 1752 - Ss 1Jan18 6:20.01 |-- /usr/sbin/powerd -b hadp -a max -n hadp dhcpd 87140 0.0 0.2 108808 82324 - Ss Thu08 0:30.95 |-- /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igb0 igb0.7 igb0.104 igb0.2 igb0.3 igb0.101 igb0.80 igb0.109 igb0.4 igb0.81 igb0.6 root 88675 0.0 0.0 24608 12428 - Ss Thu08 0:08.02 |-- /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 93181 0.0 0.0 39432 0 u0 IWs - 0:00.00 |-- login [pam] (login) root 93712 0.0 0.0 13084 0 u0 IW - 0:00.00 |– -sh (sh)
root 93957 0.0 0.0 13084 1684 u0 I+ 1Jan18 0:00.00 |-- /bin/sh /etc/rc.initial root 91355 0.0 0.0 39432 0 v0 IWs - 0:00.00 |-- login [pam] (login) root 93470 0.0 0.0 13084 0 v0 IW - 0:00.00 |– -sh (sh)
root 93851 0.0 0.0 13084 1684 v0 I+ 1Jan18 0:00.00 |-- /bin/sh /etc/rc.initial root 91523 0.0 0.0 10388 1684 v1 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv1 root 91821 0.0 0.0 10388 1684 v2 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv2 root 92148 0.0 0.0 10388 1684 v3 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv3 root 92425 0.0 0.0 10388 1684 v4 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv4 root 92536 0.0 0.0 10388 1684 v5 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv5 root 92606 0.0 0.0 10388 1684 v6 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv6 root 92866 0.0 0.0 10388 1684 v7 Is+ 1Jan18 0:00.00 |-- /usr/libexec/getty Pc ttyv7 root 69676 0.0 0.0 13084 964 0- SN 4Jan18 1:47.89-- /bin/sh /var/db/rrd/updaterrd.sh
root 10704 0.0 0.0 6172 1928 - SN 12:10 0:00.00 `-- sleep 60
root 2 0.0 0.0 0 16 - DL 1Jan18 0:00.00 - [crypto]
root 3 0.0 0.0 0 16 - DL 1Jan18 0:00.00 - [crypto returns]
root 4 0.0 0.0 0 64 - DL 1Jan18 12:01.41 - [cam]
root 5 0.0 0.0 0 16 - DL 1Jan18 0:00.16 - [soaiod1]
root 6 0.0 0.0 0 16 - DL 1Jan18 0:00.16 - [soaiod2]
root 7 0.0 0.0 0 16 - DL 1Jan18 0:00.16 - [soaiod3]
root 8 0.0 0.0 0 16 - DL 1Jan18 0:00.16 - [soaiod4]
root 9 0.0 0.0 0 16 - DL 1Jan18 0:00.00 - [sctp_iterator]
root 10 0.0 0.0 0 16 - DL 1Jan18 0:00.00 - [audit]
root 13 0.0 0.0 0 256 - DL 1Jan18 0:00.00 - [ng_queue]
root 14 0.0 0.0 0 48 - DL 1Jan18 0:00.01 - [geom]
root 15 0.0 0.0 0 240 - DL 1Jan18 1:40.68 - [usb]
root 16 0.0 0.0 0 16 - DL 1Jan18 5:35.42 - [pf purge]
root 17 0.0 0.0 0 16 - DL 1Jan18 2:37.99 - [rand_harvestq]
root 18 0.0 0.0 0 16 - DL 1Jan18 0:00.72 - [enc_daemon0]
root 19 0.0 0.0 0 48 - DL 1Jan18 308:15.42 - [pagedaemon]
root 20 0.0 0.0 0 16 - DL 1Jan18 1:41.45 - [vmdaemon]
root 21 0.0 0.0 0 16 - DL 1Jan18 0:00.01 - [pagezero]
root 22 0.0 0.0 0 16 - DL 1Jan18 0:07.25 - [bufspacedaemon]
root 23 0.0 0.0 0 32 - DL 1Jan18 0:28.88 - [bufdaemon]
root 24 0.0 0.0 0 16 - DL 1Jan18 0:06.23 - [vnlru]
root 25 0.0 0.0 0 16 - DL 1Jan18 8:06.42 - [syncer]
root 58 0.0 0.0 0 16 - DL 1Jan18 0:00.73 - [md0] -
Switched cache to diskd and disabled remote certificate checks.
All stabilized now at 40% memory on a 60GB cache.
Still don't think the dashboard is an accurate representation of memory as it includes laundry in the active memory….addressed in separate thread.
-
Nope. Didn't work.
Still extremely unstable. Sigh…..
Looking into memory pools now and certificate memory issues. Any ideas welcome still....