Dansguardian package for 2.0

AJungleDog · Feb 22, 2013, 5:50 AM

Hi,

I'm installing several new VSATs in the near future and would like to use pfSense with DansGuardian. I would also like to have the time groups features of SquidGuard. Is it possible to run both of these packages on pfSense? I tried installing them both but once I did DG didn't work anymore. I'm guessing it was a configuration issue. If I can run both at the same time, how should they be configured to pass traffic properly between them?

My ultimate goal is to have the filtering of DG and be able to block certain websites/file types/mime types during certain times of the day. An example would be to block Facebook from 7am until 5pm and all video files from 5am until midnight.

Thanks for any help.

aru · Feb 22, 2013, 12:24 PM

Hi marcelloc,

I increased memory to 384MB as you suggested, seems working, will report back if any issues.

I have another issue here. MS Outlook isn't working behind pfsense. I have setup wpad and tried manually setting proxy in ie also.

I have squid3 in non-transparent mode with DG.

Any suggestions..?

marcelloc · Feb 22, 2013, 3:57 PM

@aru:

I have another issue here. MS Outlook isn't working behind pfsense. I have setup wpad and tried manually setting proxy in ie also.

You mean owa or msoutlook communication to exchange server?

aru · Feb 23, 2013, 3:35 AM

I mean outlook pop3 mail fetching. No exchange server. Clients in lan configured pop3 are unable to download. server authentication not happening.

marcelloc · Feb 23, 2013, 4:09 AM

Pop3 is not http. Dansguardian does not affect it.

wheelz · Feb 23, 2013, 9:47 PM

@aru:

Hi marcelloc,

I increased memory to 384MB as you suggested, seems working, will report back if any issues.

I have another issue here. MS Outlook isn't working behind pfsense. I have setup wpad and tried manually setting proxy in ie also.

I have squid3 in non-transparent mode with DG.

Any suggestions..?

I did this and I still get

Fatal error: Allowed memory size of 402653184 bytes exhausted (tried to allocate 71 bytes) in /usr/local/pkg/dansguardian.inc on line 1138

. However when I bump it to 512 or higher it goes back to saying my allowed memory size is 134217728…. ???

marcelloc · Feb 23, 2013, 10:43 PM

The hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

wheelz · Feb 23, 2013, 11:20 PM

It turns out to be exactly 488 MB is the most I can give it. If I do 489 MB or higher the error seems to indicate that it didn't actually do it and it is still set to 128MB. I still get the memory errors with 488 MB as well.

wheelz · Feb 26, 2013, 4:33 PM

Any ideas on this? I guess I'm kind of stuck in my configuration until this is resolved.

marcelloc · Feb 26, 2013, 5:20 PM

@marcelloc:

The hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

samham · Feb 28, 2013, 5:09 AM

I was able to setup the groups feature, and I'm glad to report it's working as expected, except for one problem: every time I make a change to dansguardian the content of the "ipgroups" file is wiped out. any suggestions

Dansguardian 2.12.0.3 pkg v.0.1.7_3
2.1-BETA1 (amd64) built on Sat Feb 23 11:00:21 EST 2013

Rossi · Feb 27, 2013, 1:27 PM

Dear Community,

I am trying to setup dansguardian with groups / user authentification.

Dansguardian already works but I'm not really sure about how to set it up including authentification.
When I enable auth at the squid proxy and directly connect the browser to it I get the auth prompt and I am able to login and use the proxy.
The problem is now when I switch back the browser proxy setting to the dansguardian 8080 port I can browse the web with dansguardian rules active but without any auth prompt. Even when I change the "Auth Plugins" setting under general it stays the same.
Well… am I doing it wrong?
Thx for your help.

wheelz · Feb 27, 2013, 8:27 PM

@marcelloc:

@marcelloc:

The hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

There has got to be a memory leak or something to that effect. I found that if you set that value in /etc/rc.php_ini_setup to 2 GB or higher then it doesn't take it and you go back to 128 MB. So I set it to a max of 1.99 GB. I then kept bumping up the other values listed earlier and eventually got to 2000 MB (not quite 2GB) and I still get this almost all of the time:

Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 136184137 bytes) in /usr/local/pkg/dansguardian.inc on line 1150

I used top and watched the php processes. I didn't even make any changes to any settings, just picked an ACL in the DG config and hit save. I saw 2 php processes consume 100% of CPU1 and CPU2 and memory go up to close to 2GB and that's when I get the error. It took a good 30 seconds to a minute afterwards for it to drop off. The very first time I hit save it did not error, but even after a reboot for some reason it has done it ever since. Any ideas? This seems like more than just it using more memory as I can't see how saving some config files would eat up this much resources.

marcelloc · Feb 27, 2013, 9:24 PM

fetch this dansguardian.inc file from my repo, it has a lot o debug

http://e-sac.siteseguro.ws/packages/dansguardian/dansguardian.inc.txt

download it and save on /usr/local/pkg/dansguardian.inc

this inc file will stop process to show memory usage, if you want to test a full save config, remove the exit at line 1210.

on my testes, memory usage stays below 128Mb.

mschiek01 · Feb 28, 2013, 9:01 PM

Here is what I get using this as it still errors out.

debug1 - start sync 3302080

debug2 - check xml values and sample files 3332920

debug3 - check ssl certificates 3341056

debug4 - memory load before phrase ACL 3343848

debug5 - check phrase ACL 3344448

debug6 - check site ACL 3352040

debug7 - check URL ACL  3344728

debug8 - check pics and search ACL 3358944

debug9 - check file ACL  3359088

debug10 - check header ACL  3359232

debug11 - check content ACL  3359376

debug12 - antivirus ACL and report log 3359520

debug13 - memory usage before filtergroups 3458528

debug14 3498576

debug15 3498576

debug14 3518144

debug15 3518144

debug14 3516760

debug15 3516760

debug14 3518816

debug15 3518816

debug16  - check filtergroups 3530456

debug17 - check blacklists ACL 3530456

debug18 - check clamav 3581648

debug19 - check cron 3585200

debug20 - check cron 3590328

debug21 - second write config  3695688

debug21 3645808

debug22 3589016

debug23 3587592
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 17023311 bytes) in /usr/local/pkg/dansguardian.inc on line 1156

mschiek01 · Mar 1, 2013, 5:15 PM

@wheelz:

@marcelloc:

@marcelloc:

The hard limit is configured to 512. Try 500.

I'll find the option to increase it and post here.

The other value to increase is
suhosin.memory_limit = 512435456 on /etc/rc.php_ini_setup

Until I find a way to reduce dansguardian memory load during config save, you may need to increase these values.

There has got to be a memory leak or something to that effect. I found that if you set that value in /etc/rc.php_ini_setup to 2 GB or higher then it doesn't take it and you go back to 128 MB. So I set it to a max of 1.99 GB. I then kept bumping up the other values listed earlier and eventually got to 2000 MB (not quite 2GB) and I still get this almost all of the time:

Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 136184137 bytes) in /usr/local/pkg/dansguardian.inc on line 1150

I used top and watched the php processes. I didn't even make any changes to any settings, just picked an ACL in the DG config and hit save. I saw 2 php processes consume 100% of CPU1 and CPU2 and memory go up to close to 2GB and that's when I get the error. It took a good 30 seconds to a minute afterwards for it to drop off. The very first time I hit save it did not error, but even after a reboot for some reason it has done it ever since. Any ideas? This seems like more than just it using more memory as I can't see how saving some config files would eat up this much resources.

I was having these same problems on both Mailscanner and Dansguardian. I was finally able to get this fixed. What I figured out is that there were some old versions of the packages still installed.

I unistalled the programs in question from the package manager. Mailscanner and Dansguardian as well as squid.
Go to the command line and enter pkg_info
Look for previous versions of these packages and do a pkg_delete -f "package name"
make sure to manually delete /usr/local/pkg/blacklist.tgz
Then go back to package manager and reinstall. In my case Mailscanner then squid the Dansguardian.

Go back into the gui and save the configs, no more memory errors.

marcelloc · Mar 1, 2013, 5:42 PM

@mschiek01:

Go back into the gui and save the configs, no more memory errors.

great troubleshooting, I'll test it here ASAP.

wheelz · Mar 4, 2013, 9:41 PM

Somehow with the that script I ended up with an install that wouldn't boot. So I just wiped and started over. So far I haven't seen the memory errors so that is good. I'll look for residual packages with pkg_info next time.

dig1234 · Mar 7, 2013, 5:15 AM

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

wheelz · Mar 7, 2013, 7:17 PM

@dig1234:

Hi, what is the status of SSL MITM filtering?
In my tests with latest package browser just hangs with MITM enabled?

With google, facebook, and others going https, this is really needed. I want to encourage anyone interested in this to help post bounty: http://forum.pfsense.org/index.php/topic,58368.0.html. I'll give mine but we need more than I can afford for my personal use to get a priority on this feature.