Dansguardian package for 2.0

[mschiek01]

New AMD64 8gig memory 2.03 build

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023308 bytes) in /usr/local/pkg/dansguardian.inc on line 1151
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023067 bytes) in /usr/local/pkg/mailscanner.inc on line 593

I have tried increasing the memory /etc/rc.php_ini_setup

In /etc/inc/config.inc

& both mailscanner.inc and dansguardian.inc.

The error still exists neither will start.

[marcelloc]

It will need more, try 384MB on both files

I'm debuggin it and trying to reduce memory load during config apply.

[wheelz]

Hmm.. yea, just to confirm from another system, I tried setting 512M and then 1024M (rebooted after each) and I still got this on save:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 42 bytes) in /usr/local/pkg/dansguardian.inc on line 1138

[elemay]

Hi,

i have resolved our issue for the php and blacklist problem.

I deleted /usr/local/pkg/blacklist.tgz and everything else regarding blacklist.

I then deinstalled dansguardian and reinstalled it.

That has it solved.

[elemay]

I have another problem  :o

Since a few days internal dns resolving doesn't work anymore.

I can resolve forum.pfsense.org with not problems but, for example, not http://wpad.

That causes my automatic proxy configuration to stop and our clients have no internet access anymore.

If i bypass the proxy, it works like before.

Any hints in the right direction?

Cheers.

[mschiek01]

Do you have at least one of your DNS server entries on the box pointing to your local machine. Also is your DHCP server assign you local machine as a DNS server?  If not that is most likely your problem.

[wheelz]

@elemay:

Hi,

i have resolved our issue for the php and blacklist problem.

I deleted /usr/local/pkg/blacklist.tgz and everything else regarding blacklist.

I then deinstalled dansguardian and reinstalled it.

That has it solved.

You did?… that sounds really familiar:

@wheelz:

@elemay:

and also it doesn't continue installing the package after downloading blacklists file.

Not sure what the issue here was but I was able to get it to install by first deleting the blacklists file and then doing an uninstall and reinstall of dansguardian and squid3 (I use squid3 in my case).

It's all good, just had to harrass you a little. ;)  Could have saved you some trial and error.

[AJungleDog]

Hi,

I'm installing several new VSATs in the near future and would like to use pfSense with DansGuardian.  I would also like to have the time groups features of SquidGuard.  Is it possible to run both of these packages on pfSense?  I tried installing them both but once I did DG didn't work anymore.  I'm guessing it was a configuration issue.  If I can run both at the same time, how should they be configured to pass traffic properly between them?

My ultimate goal is to have the filtering of DG and be able to block certain websites/file types/mime types during certain times of the day.  An example would be to block Facebook from 7am until 5pm and all video files from 5am until midnight.

Thanks for any help.

[aru]

Hi marcelloc,

I increased memory to 384MB as you suggested,  seems working, will report back if any issues.

I have another issue here.  MS Outlook isn't working behind pfsense.  I have setup wpad and tried manually setting proxy in ie also.

I have squid3 in non-transparent mode with DG.

Any suggestions..?

[marcelloc]

@aru:

I have another issue here.  MS Outlook isn't working behind pfsense.  I have setup wpad and tried manually setting proxy in ie also.

You mean owa or msoutlook communication to exchange server?

[aru]

I mean outlook pop3 mail fetching.  No exchange server.  Clients in lan configured pop3 are unable to download.  server authentication not happening.

[marcelloc]

Pop3 is not http. Dansguardian  does not affect it.

[wheelz]

@aru:

Hi marcelloc,

I increased memory to 384MB as you suggested,  seems working, will report back if any issues.

I have another issue here.  MS Outlook isn't working behind pfsense.  I have setup wpad and tried manually setting proxy in ie also.

I have squid3 in non-transparent mode with DG.

Any suggestions..?

I did this and I still get

Fatal error: Allowed memory size of 402653184 bytes exhausted (tried to allocate 71 bytes) in /usr/local/pkg/dansguardian.inc on line 1138

.  However when I bump it to 512 or higher it goes back to saying my allowed memory size is 134217728….   ???

[marcelloc]

The  hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

[wheelz]

It turns out to be exactly 488 MB is the most I can give it.  If I do 489 MB or higher the error seems to indicate that it didn't actually do it and it is still set to 128MB.  I still get the memory errors with 488 MB as well.

[wheelz]

Any ideas on this?  I guess I'm kind of stuck in my configuration until this is resolved.

[marcelloc]

@marcelloc:

The  hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

[samham]

I was able to setup the groups feature, and I'm glad to report it's working as expected, except for one problem: every time I make a change to dansguardian the content of the "ipgroups" file is wiped out. any suggestions

Dansguardian 2.12.0.3 pkg v.0.1.7_3
2.1-BETA1 (amd64)  built on Sat Feb 23 11:00:21 EST 2013

[Rossi]

Dear Community,

I am trying to setup dansguardian with groups / user authentification.

Dansguardian already works but I'm not really sure about how to set it up including authentification.
When I enable auth at the squid proxy and directly connect the browser to it I get the auth prompt and I am able to login and use the proxy.
The problem is now when I switch back the browser proxy setting to the dansguardian 8080 port I can browse the web with dansguardian rules active but without any auth prompt. Even when I change the "Auth Plugins" setting under general it stays the same.
Well… am I doing it wrong?
Thx for your help.

[wheelz]

@marcelloc:

@marcelloc:

The  hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

There has got to be a memory leak or something to that effect.  I found that if you set that value in /etc/rc.php_ini_setup to 2 GB or higher then it doesn't take it and you go back to 128 MB.  So I set it to a max of 1.99 GB.  I then kept bumping up the other values listed earlier and eventually got to 2000 MB (not quite 2GB) and I still get this almost all of the time:

Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 136184137 bytes) in /usr/local/pkg/dansguardian.inc on line 1150

I used top and watched the php processes.  I didn't even make any changes to any settings, just picked an ACL in the DG config and hit save.  I saw 2 php processes consume 100% of CPU1 and CPU2 and memory go up to close to 2GB and that's when I get the error.  It took a good 30 seconds to a minute afterwards for it to drop off.  The very first time I hit save it did not error, but even after a reboot for some reason it has done it ever since.  Any ideas?  This seems like more than just it using more memory as I can't see how saving some config files would eat up this much resources.

[marcelloc]

fetch this dansguardian.inc file from my repo, it has a lot o debug

http://e-sac.siteseguro.ws/packages/dansguardian/dansguardian.inc.txt

download it and save on /usr/local/pkg/dansguardian.inc

this inc file will stop process to show memory usage, if you want to test a full save config, remove the exit at line 1210.

on my testes, memory usage stays below 128Mb.

[mschiek01]

Here is what I get using this as it still errors out.

debug1 - start sync 3302080

debug2 - check xml values and sample files 3332920

debug3 - check ssl certificates 3341056

debug4 - memory load before phrase ACL 3343848

debug5 - check phrase ACL 3344448

debug6 - check site ACL 3352040

debug7 - check URL ACL  3344728

debug8 - check pics and search ACL 3358944

debug9 - check file ACL  3359088

debug10 - check header ACL  3359232

debug11 - check content ACL  3359376

debug12 - antivirus ACL and report log 3359520

debug13 - memory usage before filtergroups 3458528

debug14 3498576

debug15 3498576

debug14 3518144

debug15 3518144

debug14 3516760

debug15 3516760

debug14 3518816

debug15 3518816

debug16  - check filtergroups 3530456

debug17 - check blacklists ACL 3530456

debug18 - check clamav 3581648

debug19 - check cron 3585200

debug20 - check cron 3590328

debug21 - second write config  3695688

debug21 3645808

debug22 3589016

debug23 3587592
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023311 bytes) in /usr/local/pkg/dansguardian.inc on line 1156

[mschiek01]

@wheelz:

@marcelloc:

@marcelloc:

The  hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

There has got to be a memory leak or something to that effect.  I found that if you set that value in /etc/rc.php_ini_setup to 2 GB or higher then it doesn't take it and you go back to 128 MB.  So I set it to a max of 1.99 GB.  I then kept bumping up the other values listed earlier and eventually got to 2000 MB (not quite 2GB) and I still get this almost all of the time:

Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 136184137 bytes) in /usr/local/pkg/dansguardian.inc on line 1150

I used top and watched the php processes.  I didn't even make any changes to any settings, just picked an ACL in the DG config and hit save.  I saw 2 php processes consume 100% of CPU1 and CPU2 and memory go up to close to 2GB and that's when I get the error.  It took a good 30 seconds to a minute afterwards for it to drop off.  The very first time I hit save it did not error, but even after a reboot for some reason it has done it ever since.  Any ideas?  This seems like more than just it using more memory as I can't see how saving some config files would eat up this much resources.

I was having these same problems on both Mailscanner and Dansguardian.  I was finally able to get this fixed.  What I figured out is that there were some old versions of the packages still installed.

I unistalled the programs in question from the package manager. Mailscanner and Dansguardian as well as squid.
Go to the command line and enter pkg_info
Look for previous versions of these packages and do a pkg_delete -f "package name"
make sure to manually delete /usr/local/pkg/blacklist.tgz
Then go back to package manager and reinstall. In my case Mailscanner then squid the Dansguardian.

Go back into the gui and save the configs, no more memory errors.

[marcelloc]

@mschiek01:

Go back into the gui and save the configs, no more memory errors.

great troubleshooting, I'll test it here ASAP.

[wheelz]

Somehow with the that script I ended up with an install that wouldn't boot.  So I just wiped and started over.  So far I haven't seen the memory errors so that is good.  I'll look for residual packages with pkg_info next time.

[dig1234]

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

[wheelz]

@dig1234:

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

With google, facebook, and others going https, this is really needed.  I want to encourage anyone interested in this to help post bounty:  http://forum.pfsense.org/index.php/topic,58368.0.html.  I'll give mine but we need more than I can afford for my personal use to get a priority on this feature.

[ZGruk]

I'm trying to install dansguardian on the latest 2.1 build, and I'm not getting it to work. I installed the squid and dansguardian packages and set them up, but when I test to see if its blocking anything, it doesn't block.

When I reboot pfSense, it spits out a bunch of errors when it tries to start dansguardian. I tried to capture them all, but I may have missed some.

Warning: file_put_contents(/usr/pbi/dansguardian-i386/dansguardian/
contentscanners/<variouslists>): failed to open stream: Read-only 
file system in /usr/local/pkg/dansguardian.inc on line <various lines="" see="" list="" below=""></various></variouslists> 

I'm not sure I got all the lines, but the ones I got were:
647
662
669
676
683
702
–- a gap where I might have missed some ---
909
911
915
919

Then there's another series of errors

Warning: closedir() expects parameter 1 to be resource, null given in 
/usr/local/pkg/dansguardian.inc on line 69

Warning: file(/usr/local/share/certs/ca-root-mss.crt): failed to open stream:
no such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.inc on lind 76

So apparently there's something wrong with /usr/local/pkg/dansguardian.inc?
Suggestions are appreciated.

EDIT:
I changed the permissions on the dansguardian.inc file to make sure it wasn't read only, and all of the read only errors went away. The last three errors are still there however, and its not blocking sites.
Also, I don't know if its related, but squid is taking what experience tells me is an exceptionally long time to startup. It doesn't give any errors, just takes awhile to start.

Third, possibly not related, after installing dansguardian and squid and rebooting, I can no longer access my webConfigurator. It just times out. if I reset to factory settings, it will work fine, but after installing and rebooting again, it once again times out. I haven't tried to figure out whether its dansguardian, squid, 2.1 BETA, or some combination thereof thats causing this. The first time I had this problem I assumed it was the beta version of squid3 that I had installed, but now its happening again with the normal squid package.

[marcelloc]

@ZGruk:

Third, possibly not related, after installing dansguardian and squid and rebooting, I can no longer access my webConfigurator.

Startup erros on packages breaks webconfigurator and some rules load.
Can you check on console what errors are you getting.

I've tested dansguardian on 2.1 before pushing the code, I'll start a new test run.

[ZGruk]

I rebooted just now and discovered that all of the read-only errors in my previous post are back, plus some errors from squid and I think more that I hadn't seen before from dansguardian. Is there a log I can find that shows these errors? I've been trying to take pictures of the screen as they flash by, but obviously that doesn't work very well.

[marcelloc]

I think it's on system logs but you can use scroll lock key and up and down arrow to move on screen to see what errors you got.

[ZGruk]

Here are the errors that appear during booting.

Starting package squid...

Warning: chown() Read-only file system in /usr/local/pkg/squid.inc on line 77

Warning: chgrp() Read-only file system in /usr/local/pkg/squid.inc on line 78
9

Those errors are repeated many times (probably over 100)

Warning: file_put_contents(/usr/pbi/squid-i386/etc/squid/squid.conf): failed to
open stream: Read only file system in /usr/local/pkg/squid.inc on line 1159
done.
Starting package Dansguardian...
Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionfilesitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 409

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/ban
nedsitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 417

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/gre
ysitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 424

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/log
sitelist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 431

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/ban
nedurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 467

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionfileurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 474

Warning: file_put_contents(/usr/pbi/dansguardian-i386/etc/dansguardian/lists/exc
eptionregexpurllist.Default): failed to open stream: Read-only file system in /us
r/local/pkg/dansguardian.inc on line 481

And so on and so forth down to line 919. Then there are some different errors:

Warning: closedir()expects parameter 1 to be resource, null given in /usr/local
/pkg/dansguardian.inc on line 69

Warning: file(/usr/local/share/certs/ca-root-nss.crt): failed to open stream: No
 such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.
inc on line 76
done.

It almost all appears to be related to the supposedly read only files. I've checked to make sure they are writable.
The permissions on squid.inc are -rwxr-xr-x and the permissions on dansguardian.inc (I changed them the first time I got the errors) are -rwxrwxrwx.

[marcelloc]

Are you using nanobsd?

[ZGruk]

Yes, I am. You think that might be the issue?

[marcelloc]

Probably yes. I'm calling the mount_rw feature but I did not tested this package on nanobsd yet.

[awsiemieniec]

pfSense: 2.0.2-RELEASE (amd64)
Dansguardian: 2.12.0.3 pkg v.0.1.7_3

So the solution to the out of memory error during a save in DS is to uninstall, look for strays via pkg_info, delete any if found then reinstall?

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023308 bytes) in /usr/local/pkg/dansguardian.inc on line 1151

[rjcrowder]

I'm not sure what's going on with this, but I had the same error. I discovered that the clamav startup script in /usr/local/etc/rc.d had tons of duplicated lines in it! Each line was duplicated thousounds of times! Here's what I did that seemed to fix it (at least for the moment).
1.) Bumped up the memory limits listed previously in this thread
2.) Fixed the clamav startup script (not sure necessary - think it is rewritten each startup? not sure).

After reboot, everything started up fine.

@awsiemieniec:

pfSense: 2.0.2-RELEASE (amd64)
Dansguardian: 2.12.0.3 pkg v.0.1.7_3

So the solution to the out of memory error during a save in DS is to uninstall, look for strays via pkg_info, delete any if found then reinstall?

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023308 bytes) in /usr/local/pkg/dansguardian.inc on line 1151

[marcelloc]

I'll take a look on clamav startup script.

[ZGruk]

@marcelloc:

Probably yes. I'm calling the mount_rw feature but I did not tested this package on nanobsd yet.

Apparently that is the problem. I checked the "Keep media mounted read/write at all times" box under Diagnostics > NanoBSD, and all the read-only errors went away. There were still three errors showing up during boot however:

Warning: closedir() expects parameter 1 to be resource, null given in /usr/local/pkg/dansguardian on line 69

Warning: file(/usr/local/share/certs/ca-root-nss.crt): failed to open stream: No such file or directory in /usr/local/pkg/dansguardian.inc on line 74

Warning: Invalid argument supplied for foreach() in /usr/local/pkg/dansguardian.inc on line 76

I ran "pkg_add -r ca-root-nss" and rebooted, and now there are no errors at all. So apparently all the problems have been solved.

Now a different question. How does the process of adding banned sites, urls, etc work exactly? I added a second list to the Site Lists under ACLs with a couple in the "Blocked" section to test if it was working. After saving, it's still not blocking them. I did make sure the "Enable" box for banned was checked. Am I doing something wrong, or is there still a problem that's just not showing up during boot?

[Legion]

First issue I'm having (a minor one): crontab is filled up with at least 100 entries of:

0 0 */7 * * root /usr/local/bin/php /usr/local/www/dansguardian.php fetch_blacklist

Maybe it writes another entry each time I save a config? I have manually deleted all these entries (except one) a couple of times now but it keeps filling up.

Second issue is getting clamav to work. Out of the box I kept getting a lot of:

Error connecting to ClamD socket
Unknown return code from content scanner: -1

To start with, after installing DG it seemed a bunch of files and directories are missing. So e.g. I had to manually create /var/run/clamav and chown to clamav. And then touch clamd.sock inside that directory and make sure it had 755 permissions and clamav owner. Also maybe some /var/log/clamav settings.

I tried a number of things after that, like manually running freshclam (OK), manually restarting DG (OK as long as I created clamd.sock as above), manually installing the latest version of clamav I could find (pkg_add -r http://files.pfsense.org/packages/8/All/clamav-0.97.6.tbz). Still nothing.

The final thing that got it working for me was to restart clamav-clamd myself. I'm not sure why this works since the system logs show it "starting" when I enable clamav via the GUI config:

Maybe it's a restart that is necessary? With stop/start? Because that's what I did.

Anyway, right now my system is working fine with pfSense -> DG w/ clamav -> Squid3 -> Internet using DHCP/wpad but I'll be interested to see if I have to manually set up the services in the right order again after rebooting.