ClamAV spikes cpu usage after changing Squid setting.
-
Hey, I've been using Squid + ClamAV for a long time, for some time I'm having issues with both. Every time I change anything in the Squid conf, like a single insertion or deletion on ACL Whitelist and click save, the clamav proccess spikes to 100% and internet through Squid stops responding. It stays at 100% for 2-3 minutes, then it just gets backs to normal, with low cpu usage. It's not freshclam (which runs pretty fast), it's just that time after saving a new setting that makes the cpu spike.
I use SSL filtering, with splice whitelist, bump otherwise.
My pfSense setup is "more than enough" for the job.
CPU Type Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
4 CPUs: 4 package(s)
AES-NI CPU Crypto: Yes (active)
With 6gb of Ram dedicated.
After that temporary spike Squid+ClamAV runs smoothly "forever", until I change any Squid setting.
Has anyone ever experienced anything like that?
I've performed a fresh install (restoring the xml) and now I'm out of options.
BTW, I'm on 2.4.4-RELEASE-p3 (amd64) with squid package up to date.
Any suggestions?
Thanks in advance -
Unfortunately that's the normal behavior of ClamAV at the moment, it takes 2- 3 minutes to start or reload.
https://lists.clamav.net/pipermail/clamav-users/2019-October/thread.html
https://lists.clamav.net/pipermail/clamav-users/2019-September/thread.html[clamav-users] Continuous increase of startup time (is daily.cld broken?)
[clamav-users] How to boost clamav? Reloading database results in a talking timeout?I'm using a patched version of ClamAV, which offloads the reloading to a second thread, so no interruption of the Internet and so on.
You can get it from here: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c13
-
@Bismarck said in ClamAV spikes cpu usage after changing Squid setting.:
Unfortunately that's the normal behavior of ClamAV at the moment, it takes 2- 3 minutes to start or reload.
https://lists.clamav.net/pipermail/clamav-users/2019-October/thread.html
https://lists.clamav.net/pipermail/clamav-users/2019-September/thread.html[clamav-users] Continuous increase of startup time (is daily.cld broken?)
[clamav-users] How to boost clamav? Reloading database results in a talking timeout?I'm using a patched version of ClamAV, which offloads the reloading to a second thread, so no interruption of the Internet and so on.
You can get it from here: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c13
So, you replaced the mentioned server.h in /usr/local/include ?
Did you also apply any system patch? -
@do1984 said in ClamAV spikes cpu usage after changing Squid setting.:
So, you replaced the mentioned server.h in /usr/local/include ?
No, you need to setup eg. a VM with FreeBSD 11.2 and fetch the source via Ports, replace the files with the one from bugzilla and build a new pkg, which can be installed like:
pkg add -f -M /root/tmp/clamav-0.101.4,1.txz pkg add -f http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/json-c-0.13.1_1.txz(newer json-c version is required to run clamav 0.101.4)
@do1984 said in ClamAV spikes cpu usage after changing Squid setting.:
Did you also apply any system patch?
Yes and no, because the rc file for clamd which ships with pfSense is a bit harsh, it does not really restart rather kill the process.
I can give you the ready compiled clamav-0.101.4,1.txz package, but you should not trust a stranger from the internet, anyway there you go... :D
https://github.com/spec1re/stuff/raw/master/clamav-0.101.4%2C1.txz
With this patched ClamAV and a little edit to /usr/local/pkg/squid_antivirus.inc to use the original rc file for clamd, no more internet interruption when doing freshclam or save a setting in squid.
-
@Bismarck said in ClamAV spikes cpu usage after changing Squid setting.:
@do1984 said in ClamAV spikes cpu usage after changing Squid setting.:
So, you replaced the mentioned server.h in /usr/local/include ?
No, you need to setup eg. a VM with FreeBSD 11.2 and fetch the source via Ports, replace the files wit the one from bugzilla and build a new pkg, which can be installed like:
pkg add -f -M /root/tmp/clamav-0.101.4,1.txz pkg add -f http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/json-c-0.13.1_1.txz(newer json-c version is required to run clamav 0.101.4)
@do1984 said in ClamAV spikes cpu usage after changing Squid setting.:
Did you also apply any system patch?
Yes and no, because the rc file for clamd which ships with pfSense is a bit harsh, it does not really restart rather kill the process.
I can give you the ready compiled clamav-0.101.4,1.txz package, but you should not trust a stranger from the internet, but anyway there you go... :D
https://github.com/spec1re/stuff/raw/master/clamav-0.101.4%2C1.txz
With this patched ClamAV and a little edit to /usr/local/pkg/squid_antivirus.inc to use the original rc file for clamd, no more internet interruption when doing freshclam or save a setting in squid.
I had already tried installing the 0.101.4,1.txz from FreeBSD repository directly, but it warned me because of different kernels. Ignoring that, it said 101.2 was already installed. I missed the -f (force) part. Anyway, I trusted your compiled version for testing, and it worked flawlessly, I didn't even had to edit the squid_antivirus.inc. I just force installed, ran the clamd.sh e it worked. I've tried messing with the acls on Squid, the clamd process shows up for with a small cpu usage and it finishes the task right away, with no service disruption at all. God you made my day. This issue was driving me nuts, I've tried adding squid workers to try to make the service available during the clamav "refresh", but nothing worked. I've even tried to use the pfSense 2.5 devel on another VM, wondering if the problem was solved there, but I wasn't really successful, since with my actual setup, squid crashed immediately when trying to use ssl interception.
Basically,
pkg add -f https://github.com/spec1re/stuff/raw/master/clamav-0.101.4%2C1.txz
Fetching clamav-0.101.4%2C1.txz: 100% 1 MiB 1.2MB/s 00:01
Installing clamav-0.101.4,1...
package clamav is already installed, forced install
===> Creating groups.
Using existing group 'clamav'.
Using existing group 'mail'.
===> Creating users
Using existing user 'clamav'.
Extracting clamav-0.101.4,1: 100%
Thank you so much, I wonder how many users are facing this exact issue and have no idea of what's going on.
Hope the clamav team fix this as soon as possible, and that pfSense updates its repositories using those fixed versions. -
Glad that I could help you!
I was in the same boot, scheduled feshclam updates and acl changes for Squid in the night hours, so users wouldn't disturbed to much, but now no problem need a change just do it.
