Dansguardian package for 2.0

[wheelz]

Somehow with the that script I ended up with an install that wouldn't boot.  So I just wiped and started over.  So far I haven't seen the memory errors so that is good.  I'll look for residual packages with pkg_info next time.

[dig1234]

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

[wheelz]

@dig1234:

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

With google, facebook, and others going https, this is really needed.  I want to encourage anyone interested in this to help post bounty:  http://forum.pfsense.org/index.php/topic,58368.0.html.  I'll give mine but we need more than I can afford for my personal use to get a priority on this feature.

[ZGruk]

I'm trying to install dansguardian on the latest 2.1 build, and I'm not getting it to work. I installed the squid and dansguardian packages and set them up, but when I test to see if its blocking anything, it doesn't block.

When I reboot pfSense, it spits out a bunch of errors when it tries to start dansguardian. I tried to capture them all, but I may have missed some.

Warning: file_put_contents(/usr/pbi/dansguardian-i386/dansguardian/
contentscanners/<variouslists>): failed to open stream: Read-only 
file system in /usr/local/pkg/dansguardian.inc on line <various lines="" see="" list="" below=""></various></variouslists> 

I'm not sure I got all the lines, but the ones I got were:
647
662
669
676
683
702
–- a gap where I might have missed some ---
909
911
915
919

Then there's another series of errors

Warning: closedir() expects parameter 1 to be resource, null given in 
/usr/local/pkg/dansguardian.inc on line 69

Warning: file(/usr/local/share/certs/ca-root-mss.crt): failed to open stream:
no such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.inc on lind 76

So apparently there's something wrong with /usr/local/pkg/dansguardian.inc?
Suggestions are appreciated.

EDIT:
I changed the permissions on the dansguardian.inc file to make sure it wasn't read only, and all of the read only errors went away. The last three errors are still there however, and its not blocking sites.
Also, I don't know if its related, but squid is taking what experience tells me is an exceptionally long time to startup. It doesn't give any errors, just takes awhile to start.

Third, possibly not related, after installing dansguardian and squid and rebooting, I can no longer access my webConfigurator. It just times out. if I reset to factory settings, it will work fine, but after installing and rebooting again, it once again times out. I haven't tried to figure out whether its dansguardian, squid, 2.1 BETA, or some combination thereof thats causing this. The first time I had this problem I assumed it was the beta version of squid3 that I had installed, but now its happening again with the normal squid package.

[marcelloc]

@ZGruk:

Third, possibly not related, after installing dansguardian and squid and rebooting, I can no longer access my webConfigurator.

Startup erros on packages breaks webconfigurator and some rules load.
Can you check on console what errors are you getting.

I've tested dansguardian on 2.1 before pushing the code, I'll start a new test run.

[ZGruk]

I rebooted just now and discovered that all of the read-only errors in my previous post are back, plus some errors from squid and I think more that I hadn't seen before from dansguardian. Is there a log I can find that shows these errors? I've been trying to take pictures of the screen as they flash by, but obviously that doesn't work very well.

[marcelloc]

I think it's on system logs but you can use scroll lock key and up and down arrow to move on screen to see what errors you got.

[ZGruk]

Here are the errors that appear during booting.

Starting package squid...

Warning: chown() Read-only file system in /usr/local/pkg/squid.inc on line 77

Warning: chgrp() Read-only file system in /usr/local/pkg/squid.inc on line 78
9

Those errors are repeated many times (probably over 100)

Warning: file_put_contents(/usr/pbi/squid-i386/etc/squid/squid.conf): failed to
open stream: Read only file system in /usr/local/pkg/squid.inc on line 1159
done.
Starting package Dansguardian...
Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionfilesitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 409

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/ban
nedsitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 417

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/gre
ysitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 424

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/log
sitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 431

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/ban
nedurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 467

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionfileurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 474

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionregexpurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 481

And so on and so forth down to line 919. Then there are some different errors:

Warning: closedir()expects parameter 1 to be resource, null given in /usr/local
/pkg/dansguardian.inc on line 69

Warning: file(/usr/local/share/certs/ca-root-nss.crt): failed to open stream: No
 such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.
inc on line 76
done.

It almost all appears to be related to the supposedly read only files. I've checked to make sure they are writable.
The permissions on squid.inc are -rwxr-xr-x and the permissions on dansguardian.inc (I changed them the first time I got the errors) are -rwxrwxrwx.

[marcelloc]

Are you using nanobsd?

[ZGruk]

Yes, I am. You think that might be the issue?

[marcelloc]

Probably yes. I'm calling the mount_rw feature but I did not tested this package on nanobsd yet.

[awsiemieniec]

pfSense: 2.0.2-RELEASE (amd64)
Dansguardian: 2.12.0.3 pkg v.0.1.7_3

So the solution to the out of memory error during a save in DS is to uninstall, look for strays via pkg_info, delete any if found then reinstall?

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023308 bytes) in /usr/local/pkg/dansguardian.inc on line 1151

[rjcrowder]

I'm not sure what's going on with this, but I had the same error. I discovered that the clamav startup script in /usr/local/etc/rc.d had tons of duplicated lines in it! Each line was duplicated thousounds of times! Here's what I did that seemed to fix it (at least for the moment).
1.) Bumped up the memory limits listed previously in this thread
2.) Fixed the clamav startup script (not sure necessary - think it is rewritten each startup? not sure).

After reboot, everything started up fine.

@awsiemieniec:

pfSense: 2.0.2-RELEASE (amd64)
Dansguardian: 2.12.0.3 pkg v.0.1.7_3

So the solution to the out of memory error during a save in DS is to uninstall, look for strays via pkg_info, delete any if found then reinstall?

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023308 bytes) in /usr/local/pkg/dansguardian.inc on line 1151

[marcelloc]

I'll take a look on clamav startup script.

[ZGruk]

@marcelloc:

Probably yes. I'm calling the mount_rw feature but I did not tested this package on nanobsd yet.

Apparently that is the problem. I checked the "Keep media mounted read/write at all times" box under Diagnostics > NanoBSD, and all the read-only errors went away. There were still three errors showing up during boot however:

Warning: closedir() expects parameter 1 to be resource, null given in /usr/local/pkg/dansguardian on line 69

Warning: file(/usr/local/share/certs/ca-root-nss.crt): failed to open stream: No such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.inc on line 76

I ran "pkg_add -r ca-root-nss" and rebooted, and now there are no errors at all. So apparently all the problems have been solved.

Now a different question. How does the process of adding banned sites, urls, etc work exactly? I added a second list to the Site Lists under ACLs with a couple in the "Blocked" section to test if it was working. After saving, it's still not blocking them. I did make sure the "Enable" box for banned was checked. Am I doing something wrong, or is there still a problem that's just not showing up during boot?

[Legion]

First issue I'm having (a minor one): crontab is filled up with at least 100 entries of:

0 0 */7 * * root /usr/local/bin/php /usr/local/www/dansguardian.php fetch_blacklist

Maybe it writes another entry each time I save a config? I have manually deleted all these entries (except one) a couple of times now but it keeps filling up.

Second issue is getting clamav to work. Out of the box I kept getting a lot of:

Error connecting to ClamD socket
Unknown return code from content scanner: -1

To start with, after installing DG it seemed a bunch of files and directories are missing. So e.g. I had to manually create /var/run/clamav and chown to clamav. And then touch clamd.sock inside that directory and make sure it had 755 permissions and clamav owner. Also maybe some /var/log/clamav settings.

I tried a number of things after that, like manually running freshclam (OK), manually restarting DG (OK as long as I created clamd.sock as above), manually installing the latest version of clamav I could find (pkg_add -r http://files.pfsense.org/packages/8/All/clamav-0.97.6.tbz). Still nothing.

The final thing that got it working for me was to restart clamav-clamd myself. I'm not sure why this works since the system logs show it "starting" when I enable clamav via the GUI config:

Maybe it's a restart that is necessary? With stop/start? Because that's what I did.

Anyway, right now my system is working fine with pfSense -> DG w/ clamav -> Squid3 -> Internet using DHCP/wpad but I'll be interested to see if I have to manually set up the services in the right order again after rebooting.

[Legion]

Just rebooted and it works fine, so maybe my installation had gotten out of sync or something and I was running some older version of clamav. At any rate, the manually installed version fixed it for me. Now to get https AV working.

[LokisMischief]

Firstly I would like to thank Marcello for this great package, saved me so much time!

I have a suggestion, I guess you could call it a feature request.

On the ACL's when creating new site lists, phrase lists, etc. A button to create a new list based on the default would be handy, similar to that for firewall rules, it would just make life simpler!
Oh and separate html templates for the denied page… falls under the htmltemplate= variable in the dansguardianfx.conf

How dificult would it be to run two copies of dansguardian on the same server? (listening to different ports of course!) I'm wanting some traffic from one vlan transparently filtered and another explicitly.

[rjcrowder]

So… I was still having problems with the lines in /usr/local/etc/rc.d/clamav-clamd being duplicated. Unless I'm missing something, I think the fix is to change /usr/local/pkg/dansguardian.inc lines 1150-51 as follows:

$new_script_line=preg_replace("/NO/","YES",$script_line);
$new_clamav_startup.=preg_replace("@/usr/local@",DANSGUARDIAN_DIR,$new_script_line);

in other words… the old code did to "preg_replace" in a row appending the same script line... it was causing duplicate lines.

@marcelloc:

I'll take a look on clamav startup script.

[marcelloc]

@rjcrowder:

in other words… the old code did to "preg_replace" in a row appending the same script line... it was causing duplicate lines.

I've fixed the code today.

I'm including new sync options to push noew package version to github.