Dansguardian 2.12.0.3 Signal 11



  • Hi guys,

    So I recently upgraded our pfSense install from 2.0.1 to 2.0.3 and upon reboot it reinstalled all the packages as well. When it did Dansguardian was upgraded from the 2.12.0.0 package to the 2.12.0.3 package. However, it appears as though the Dansguardian package is now broken or I have a library with an incorrect version that Dansguardian does not like. Here's what my log looks like and is causing the Dansguardian forked processes to crash causing a slow down in internet connection. We also did not change our configuration when updating and I'm wondering if there were some new options that is causing it to bug out on us. The service starts up fine and seems to run for about 10 minutes before generating the errors.

    Anyone else having this issue or can help me troubleshoot this?

    Apr 30 11:34:20 kernel: pid 11142 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:34:12 kernel: pid 11888 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:39 kernel: pid 47657 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:39 kernel: pid 2670 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:23 kernel: pid 47536 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:23 kernel: pid 20915 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 47179 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 46320 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 46639 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 16933 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 15729 (dansguardian), uid 106: exited on signal 11
    Apr 30 11:33:21 kernel: pid 26031 (dansguardian), uid 106: exited on signal 11



  • Did you updated dansguardian binaries after 2.12.0.3 to fix web upload bug?



  • Here's a bit more info to help with the troubleshooting.

    System Information:
    Version 2.0.3-RELEASE (amd64) built on Fri Apr 12 10:27:56 EDT 2013 FreeBSD 8.1-RELEASE-p13
    Platform pfSense
    CPU Type Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
    State table size 64707/3160000
    MBUF Usage 26438/655356

    Installed pfSense Packages:
    Dansguardian 2.12.0.3 pkg v.0.1.8
    darkstat 3.0.714
    Dashboard Widget: Snort Avaliable: 0.3.4, Installed: 0.3.2
    iperf 2.0.5
    nut 2.6.4 pkg 2.0
    pfBlocker 1.0.2
    Sarg 2.3.2 pkg v.0.6.1
    snort 2.9.4.1 pkg v. 2.5.7
    squid 2.7.9 pkg v.4.3.3
    widescreen 0.2

    PKG_INFO:
    arc-5.21p          Create & extract files from DOS .ARC files
    arj-3.10.22_4      Open-source ARJ
    barnyard2-1.12      Interpreter for Snort unified2 binary output files
    bsdinstaller-2.0.2011.1212 BSD Installer mega-package
    ca_root_nss-3.14.1  The root certificate bundle from the Mozilla Project
    clamav-0.97.6      Command line virus scanner written entirely in C
    cyrus-sasl-2.1.26_2 RFC 2222 SASL (Simple Authentication and Security Layer)
    dansguardian-2.12.0.3 A fast, feature-rich web content filter for Squid proxy ser
    daq-2.0.0          Data Acquisition abstraction library for snort 2.9+
    darkstat-3.0.714    Network statistics gatherer and reporter
    expat-2.0.1_2      XML 1.0 parser written in C
    freetype2-2.4.7    A free and portable TrueType font rendering engine
    gd-2.0.35_7,1      A graphics library for fast creation of images
    gettext-0.18.1.1    GNU gettext package
    iperf-2.0.5        A tool to measure maximum TCP and UDP bandwidth
    jpeg-8_3            IJG's jpeg compression utilities
    lha-1.14i_6        Archive files using LZSS and Huffman compression (.lzh file
    libdnet-1.11_3      A simple interface to low level networking routines
    libiconv-1.13.1_1  A character set conversion library
    libiconv-1.14      A character set conversion library
    libidn-1.22        Internationalized Domain Names command line tool
    libltdl-2.4.2      System independent dlopen wrapper
    libnet11-1.1.2.1_4,1 A C library for creating IP packets
    libnet11-1.1.6,1    A C library for creating IP packets
    libpcap-1.1.1_1    Ubiquitous network traffic capture library
    libpcap-1.3.0      Ubiquitous network traffic capture library
    libwww-5.4.0_4      The W3C Reference Library
    mysql-client-5.5.30 Multithreaded SQL database (client)
    nano-2.2.6          Nano's ANOther editor, an enhanced free Pico clone
    neon29-0.29.6_4    An HTTP and WebDAV client library for Unix systems
    net-snmp-5.7.1_7    An extendable SNMP implementation
    nut-2.6.4          Network UPS Tools
    openldap-client-2.4.33_1 Open source LDAP client implementation
    pcre-8.21          Perl Compatible Regular Expressions library
    pcre-8.30_2        Perl Compatible Regular Expressions library
    pcre-8.32          Perl Compatible Regular Expressions library
    perl-5.14.2_2      Practical Extraction and Report Language
    pkg-config-0.25_1  A utility to retrieve information about installed libraries
    pkgconf-0.8.7_2    pkg-config compatible utility which does not depend on glib
    png-1.4.8          Library for manipulating PNG images
    py26-openssl-0.10  Python interface to the OpenSSL library
    py26-setuptools-0.6c11 Download, build, install, upgrade, and uninstall Python pac
    py26-twistedCore-9.0.0 An asynchronous networking framework for Python - Core modu
    py26-twistedWeb-9.0.0 An HTTP protocol implementation together with clients and s
    py26-zopeInterface-3.5.3 Zope.interface package from Zope 3
    python26-2.6.5      An interpreted object-oriented programming language
    sarg-2.3.2_2        Squid log analyzer and HTML report generator
    snort-2.9.4.1      Lightweight network intrusion detection system
    squid-2.7.9_3      HTTP Caching Proxy
    squid_radius_auth-1.10 RADIUS authenticator for squid proxy 2.5 and later
    unzoo-4.4_2        A zoo archive extractor
    wget-1.13.4_1      Retrieve files from the Net via HTTP(S) and FTP



  • Marcelloc, yes I did try the package in this thread http://forum.pfsense.org/index.php?topic=58442.0 however it did not resolve the signal 11 issue.



  • I'm thinking about just rolling the Dansguardian package back to 2.12.0.0 however I do not see a tbz for this version under files.pfsense.com anyone know how I might roll this version back or how to install an older PBI?



  • Nevermind I just downgraded Dansguardian to 2.12.0.2 and it is working without an issue. Seems like there is a problem with the 2.12.0.3 package just an FYI.



  • @tj.krause:

    Nevermind I just downgraded Dansguardian to 2.12.0.2 and it is working without an issue. Seems like there is a problem with the 2.12.0.3 package just an FYI.

    I'll try to compile latest alpha version to do some tests.



  • I've pushed dansguardian non oficial version 2.12.0.5 to my repo.

    and64
    http://e-sac.siteseguro.ws/packages/amd64/8/All/dansguardian-2.12.0.5.tbz



  • Marcelloc, I'd like to install the 2.12.0.5 version of Dans Guardian as the sites that I support
    are both experiencing the Signal 11 issue with Dans Guardian.
    The sites are running 2.12.0.3 v.0.1.8 with your patched DansGuardian binary (to enable web uploads)

    I see that the new version that you have made available  is a .tbz file which looks like the complete package.

    How do I install this over an existing running copy of DG?
    Is it a simple case of pkg_add dansguardian-2.0.5.tbz?
    Do I need to uninstall DG first and if so what happens to the current configuration files (lists, groups etc)?

    Thanks
    Neil

    ???



  • @neil:

    How do I install this over an existing running copy of DG?
    Is it a simple case of pkg_add dansguardian-2.0.5.tbz?
    Do I need to uninstall DG first and if so what happens to the current configuration files (lists, groups etc)?

    pkg_delete current dansguardian package and then pkg_add -r with full url path.

    If you are using squid3, then check if dansguardian install process forced squid2 install. If so, reinstall squid3.

    And let me know if signal11 is gone on your install with this version.



  • have you got an i386 version of this by any chance??



  • @LokisMischief:

    have you got an i386 version of this by any chance??

    i386
    http://e-sac.siteseguro.ws/packages/8/All/dansguardian-2.12.0.5.tbz



  • Cheers,

    It seems to require squid 3.4 among other things.

    Anyway, I get the follwing error when starting it.

    Config problem; check allowed values for pcontimeout
    
    

    Yet to find an error in one of the conf files… I presume its complaining about the proxy connection timeout value... but that's set to 30 by default anyway.



  • oops, seems dansguardian needs three new values in the conf file come 2.12.0.4

    # Proxy timeout
    # Set tcp timeout between the Proxy and DansGuardian
    # Min 5 - Max 100
    proxytimeout = 20
    
    # Proxy header exchange
    # Set timeout between the Proxy and DansGuardian
    # Min 20 - Max 300
    proxyexchange = 20
    
    # Pconn timeout
    # how long a persistent connection will wait for other requests
    # squid apparently defaults to 1 minute (persistent_request_timeout),
    # so wait slightly less than this to avoid duff pconns.
    # Min 5 - Max 300
    pcontimeout = 55
    

    First signs are promising, no sig 11 yet.



  • Well, so far a child process hasn't dropped out, however I now have a load of ntlm failed auth's?

    May 7 13:32:49	dansguardian[7775]: Auth plugin returned error code: -3
    May 7 13:32:49	dansguardian[7775]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:48	dansguardian[10252]: Auth plugin returned error code: -3
    May 7 13:32:48	dansguardian[10252]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:48	dansguardian[30017]: Auth plugin returned error code: -3
    May 7 13:32:48	dansguardian[30017]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:48	dansguardian[29811]: Auth plugin returned error code: -3
    May 7 13:32:48	dansguardian[29811]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:42	dansguardian[9835]: Auth plugin returned error code: -3
    May 7 13:32:42	dansguardian[9835]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:42	dansguardian[12316]: Auth plugin returned error code: -3
    May 7 13:32:42	dansguardian[8234]: Auth plugin returned error code: -3
    May 7 13:32:42	dansguardian[8234]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:42	dansguardian[12316]: NTLM - Invalid message of length 42, message was: NTLMSSP
    May 7 13:32:42	dansguardian[9390]: Auth plugin returned error code: -3
    May 7 13:32:42	dansguardian[9390]: NTLM - Invalid message of length 42, message was: NTLMSSP
    May 7 13:32:41	dansguardian[11054]: Auth plugin returned error code: -3
    May 7 13:32:41	dansguardian[11054]: NTLM - Invalid message of length 0, message was:
    May 7 13:32:18	dansguardian[8848]: Auth plugin returned error code: -3
    May 7 13:32:18	dansguardian[8848]: NTLM - Invalid message of length 42, message was: NTLMSSP
    May 7 13:27:07	dansguardian[48709]: Auth plugin returned error code: -3
    May 7 13:27:07	dansguardian[48709]: NTLM - Invalid message of length 0, message was:
    May 7 13:27:07	dansguardian[49351]: Auth plugin returned error code: -3
    May 7 13:27:07	dansguardian[49351]: NTLM - Invalid message of length 0, message was:
    


  • @LokisMischief:

    Well, so far a child process hasn't dropped out, however I now have a load of ntlm failed auth's?

    Do you have ntlm auth set? I'ts working and logging some failures or it's not working?

    This version is compiled for high load, do you think it's running faster?





  • @marcelloc:

    @LokisMischief:

    Well, so far a child process hasn't dropped out, however I now have a load of ntlm failed auth's?

    Do you have ntlm auth set? I'ts working and logging some failures or it's not working?

    This version is compiled for high load, do you think it's running faster?

    I do have ntlm auth set, did have it in conjunction with basic, but it doesnt seem to matter if thats enabled or not.
    NTLM auth is working, I am getting usernames in the logs, nobody has complained they cant get on yet… I wonder if its a piece of software attempting to auth..

    Well, it seems marginally faster. Still getting the occasional redirect not being followed. I have all the tunables in the dansguardian.conf set for "suggested for large site" settings.

    I wonder if I should upgrade squid.



  • well, that worked yesterday (despite the ntlm auth errors), but today we are back to the same signal 11's.

    I have gone back to 2.12.0.2 for now.



  • I also have some of these errors - although it sounds like you're seeing it more often. I did a little googling and it seems that this issue with DG under freeBSD has existed for a long time. I didn't find any definitive answers, but most suggestions for fixing it centered around changing the DG settings - such as max children, max spare children, and max age of children. I bumped some of these settings up yesterday and will let you know the results…

    @LokisMischief:

    well, that worked yesterday (despite the ntlm auth errors), but today we are back to the same signal 11's.

    I have gone back to 2.12.0.2 for now.



  • With the latest version you can adjust maxchildren (maximun value) with your system
    For example on linux :

    ulimit -n 8192 -> new ./configure option = with-filedescriptors=8192 = dansguardian.conf maxchildren=8192

    Maybe this is a clue ? Perhaps this version was compiled with too much high value for the system ? Can you play with ulimit ?
    How many process are running when the crash appear ? ps -edf | grep dansguard | wc -l



  • @Fredb:

    With the latest version you can adjust maxchildren (maximun value) with your system
    For example on linux :

    ulimit -n 8192 -> new ./configure option = with-filedescriptors=8192 = dansguardian.conf maxchildren=8192

    Maybe this is a clue ? Perhaps this version was compiled with too much high value for the system ? Can you play with ulimit ?
    How many process are running when the crash appear ? ps -edf | grep dansguard | wc -l

    Well you can adjust the max/min children in the conf file, but it didn't seem to make much difference, same config file with the previous version (minus the bits added for that particular version) works. I'm afraid I cant count the processes, rolled back to 2.12.0.2 and don't currently have a dev box running only production.
    If I get a chance I will run up a vm for it "later"



  • @LokisMischief:

    Well you can adjust the max/min children in the conf file, but it didn't seem to make much difference, same config file with the previous version (minus the bits added for that particular version) works. I'm afraid I cant count the processes, rolled back to 2.12.0.2 and don't currently have a dev box running only production.
    If I get a chance I will run up a vm for it "later"

    Yea, I'm still having he problem. About ever other day I get a half dozen or so DG processes ending with signal 11. Are you saying one of the versions doesn't do this? If so, which one?



  • Signal 11 means that the program accessed a memory location that was not assigned to it, the strange thing that there is no problem in Linux (with dansguardian 2.12.0.5)

    Please, Can you post your maxchildren value ? More than 1024 ?
    And if someone know the value of FD_SETSIZE in types.h (or posix_types.h) and typesizes.h with FreeBSD ?
    Also can you post the compilation option (dansguardian -v)

    No problem at all with 2.12.0.2 ?

    Thanks



  • @Fredb:

    Signal 11 means that the program accessed a memory location that was not assigned to it, the strange thing that there is no problem in Linux (with dansguardian 2.12.0.5)

    Please, Can you post your maxchildren value ? More than 1024 ?
    And if someone know the value of FD_SETSIZE in types.h (or posix_types.h) and typesizes.h with FreeBSD ?
    Also can you post the compilation option (dansguardian -v)

    No problem at all with 2.12.0.2 ?

    Thanks

    Based on some notes here http://contentfilter.futuragts.com/wiki/doku.php?id=faq (see FAQ 26b)
    I have bumped the following sysctl values (in loader.conf.local):
     kern.ipc.shmseg=512
     kern.ipc.shmmni=512
     kern.ipc.semmni=512
     kern.ipc.msgssz=64
     kern.ipc.shm_use_phys=1

    at the moment, I have maxchildren set to 120 and maxsparechildren at 48



  • I am running 2.12.0.3 pkg v.0.1.7_3 and have not seen this issue at all.

    All of the setting I am using are the default.



  • @mschiek01:

    I am running 2.12.0.3 pkg v.0.1.7_3 and have not seen this issue at all.

    mschiek01 told me some time ago an issue with a specific perl version.

    Try to unistall package, remove all perl versions using pkg_delete on console and then try a dansguardian package reinstall.



  • I'm also suffering this issue.

    Lots of Signal 11 messages show up when the system is under load - about 40 office users with normal daily activities such as web browsing, email,…

    I am using 2.0.2-RELEASE (i386) with patched Dans for web uploads



  • @rjcrowder:

    Based on some notes here http://contentfilter.futuragts.com/wiki/doku.php?id=faq (see FAQ 26b)
    I have bumped the following sysctl values (in loader.conf.local):
     kern.ipc.shmseg=512
     kern.ipc.shmmni=512
     kern.ipc.semmni=512
     kern.ipc.msgssz=64
     kern.ipc.shm_use_phys=1

    at the moment, I have maxchildren set to 120 and maxsparechildren at 48

    Still getting them…

    I'm not really wanting to try a "reinstall" though... This is a fresh install of pfSense 2.0.3 64 bit and the only packages that I've added are:

    • Cron

    • File Manager

    • vHosts

    • Dansguardian

    • Squid 3

    I'm currently using the patched dansguardian 2.12.0.3 (just copied over the executable).



  • Please, can you try this latest version and let me know if it works (better) for you ? http://numsys.eu/search.php?search=Squid



  • @Fredb:

    Please, can you try this latest version and let me know if it works (better) for you ? http://numsys.eu/search.php?search=Squid

    I'll compile it and push to my repo.

    Fredb, nice to see you on pfsense forum  :)

    Most work I did on dansguardian 2.12 was for this package on pfsense.



  • Hi,
    Your work is included in "my" dansguardian version

    I hope, if I can …, rewrite the engine with kqueue for *BSD and epool for Linux and remove the old select() call, maybe this point is a part of problem signal 11



  • 2.12.0.6 compiled and pushed to my repo.

    amd64
    http://e-sac.siteseguro.ws/packages/amd64/8/All/dansguardian-2.12.0.6.tbz

    i386
    http://e-sac.siteseguro.ws/packages//8/All/dansguardian-2.12.0.6.tbz

    both complied with maxfiles=8192

    Also, I've removed squid ports compile depend. It will not force any squid version anymore.



  • Configuration files http://numsys.eu/dansguardian/

    Requires

    Proxy timeout

    Set tcp timeout between the Proxy and DansGuardian

    Min 5 - Max 100

    proxytimeout = 20

    Proxy header exchange

    Set timeout between the Proxy and DansGuardian

    Min 20 - Max 300

    proxyexchange = 20

    Pconn timeout

    how long a persistent connection will wait for other requests

    squid apparently defaults to 1 minute (persistent_request_timeout),

    so wait slightly less than this to avoid duff pconns.

    Min 5 - Max 300

    pcontimeout = 55

    Now you can can disabled some (if) unused values, like maxcontentramcachescansize, I think It should be interesting about signal 11 and a potential memory leak.



  • @Fredb:

    Configuration files http://numsys.eu/dansguardian/

    Requires

    Proxy timeout

    Set tcp timeout between the Proxy and DansGuardian

    Min 5 - Max 100

    proxytimeout = 20

    Proxy header exchange

    Set timeout between the Proxy and DansGuardian

    Min 20 - Max 300

    proxyexchange = 20

    Pconn timeout

    how long a persistent connection will wait for other requests

    squid apparently defaults to 1 minute (persistent_request_timeout),

    so wait slightly less than this to avoid duff pconns.

    Min 5 - Max 300

    pcontimeout = 55

    Now you can can disabled some (if) unused values, like maxcontentramcachescansize, I think It should be interesting about signal 11 and a potential memory leak.

    Marcello… are you going to add these config settings to the UI? If I manually add them to the config files, will they be dropped when I save via the UI?



  • @rjcrowder:

    Marcello… are you going to add these config settings to the UI?

    yes. you can force it on dansguardian.inc near

    $proxytimeout=($dansguardian['proxytimeout']?$dansguardian['proxytimeout']:"30");
    

    @rjcrowder:

    If I manually add them to the config files, will they be dropped when I save via the UI?

    Yes.



  • @marcelloc:

    yes. you can force it on dansguardian.inc near

    $proxytimeout=($dansguardian['proxytimeout']?$dansguardian['proxytimeout']:"30");
    

    Thanks - No problem… I'll add them on my setup.



  • @marcelloc:

    yes. you can force it on dansguardian.inc near

    The easiest thing to do was to just add them into dansguardian.conf.template - so that's what I did for now.

    Up and running with 2.12.0.6 - I'll keep you posted…!!!

    Thanks!



  • If You See Something, please try to reduce your unused values to 0



  • @Fredb:

    If You See Something, please try to reduce your unused values to 0

    Besides "maxcontentramcachescansize", which ones are no longer used?


Log in to reply