Squid + Dansguardian issue



  • Repost from general help.

    Hi

    I have a problem with my Dansguardian + Squid Setup. At the moment I have the most basic of Adblocking + Url filtering setup but I have a problem. I have a NAT rule to send anything  is not a Lan Subnet on Port 80 to 192.168.1.1:3128 (Dansguardian).

    Most things seem to be working but if I head over to google.co.uk, I'm redirected to https://www.google.co.uk (fine), I search for 'pfsense' the results are displayed as usual but every link I click times out. The top result is pfsense.org (acutal link www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDEQFjAA&url=http%3A%2F%2Fwww.pfsense.org%2F&ei=yhmfUr60A9Ly7AaYrIDQBA&usg=AFQjCNFmdoam9UPDzW72Y1FjKVDI2Vd47Q&bvm=bv.57155469,d.d2k&cad=rja). If I go direct to pfsense.org the page loads as normal.

    Pfsense-2.1 Release (amd64) is running on an i3 with 4gb of RAM installed onto a 250GB HDD.

    Any help would be much appreciated.



  • Anyone?

    Do I need to provide more info? If so, what?



  • 12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CC4QFjAA&url=http%3A%2F%2Fwww.pfsense.org%2F&ei=ddymUqfrCYWp7AbG-IGgDQ&usg=AFQjCNFmdoam9UPDzW72Y1FjKVDI2Vd47Q&bvm=bv.57799294,d.d2k CONTENTMOD  GET 385 0  1 200 text/html  Default  - -
    12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://clients1.google.com/ocsp CONTENTMOD  POST 3605 0  1 503 text/html  Default  - - application/ocsp-request,,107,0,,0;
    12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://clients1.google.com/ocsp CONTENTMOD  POST 3605 0  1 503 text/html  Default  - - application/ocsp-request,,107,0,,0;

    Copy of 3 entries from syslog server when going to gloogle.co.uk, searching for pfsesne and then clicking the link.



  • 12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CC4QFjAA&url=http%3A%2F%2Fwww.pfsense.org%2F&ei=ddymUqfrCYWp7AbG-IGgDQ&usg=AFQjCNFmdoam9UPDzW72Y1FjKVDI2Vd47Q&bvm=bv.57799294,d.d2k CONTENTMOD  GET 385 0  1 200 text/html  Default  - -
    12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://clients1.google.com/ocsp CONTENTMOD  POST 3605 0  1 503 text/html  Default  - - application/ocsp-request,,107,0,,0;
    12-10-2013 09:19:03 User.Info 192.168.1.1 Dec 10 09:18:53 dansguardian[73148]: 2013.12.10 9:18:53 - 192.168.1.15 http://clients1.google.com/ocsp CONTENTMOD  POST 3605 0  1 503 text/html  Default  - - application/ocsp-request,,107,0,,0;

    Copy of 3 entries from syslog server when going to gloogle.co.uk, searching for pfsesne and then clicking the link.

    1384976437.027  13244 127.0.0.1 TCP_MISS/503 0 CONNECT www.google.com:443 - DIRECT/2a00:1450:4009:807::1011 -
    1384976437.027  12965 127.0.0.1 TCP_MISS/503 0 CONNECT www.google.com:443 - DIRECT/2a00:1450:4009:807::1011 -
    1384976484.611  59615 127.0.0.1 TCP_MISS/503 0 CONNECT www.google.com:443 - DIRECT/2a00:1450:4009:807::1011 -

    Copy from Squid Log.



  • I tried this on my squid/dg setup and didn't have any issue…

    From this site -http://contentfilter.futuragts.com/wiki/doku.php?id=the_access.log_files  it appears that perhaps you have something in your content regular expression list that is modifying the returned content?



  • I've added the following http://forum.pfsense.org/index.php?topic=68975.0 to ACl' >URL Lists > Default Url Access List > Modify Section, Enable is ticked.



  • I will look at my rewrite rules when I get home… The difference in how I'm setup vs. what you are doing is that I force non-SSL google search using DNS overrides.

    Regardless... it seems that the rewrite stuff is what is messing you up. Can you disable it and test that things work?



  • I've just gone in and unchecked the Enable tick box, restarted Dansguardian Server and tried again.

    12-11-2013 08:19:04 User.Info 192.168.1.1 Dec 11 08:18:51 dansguardian[79155]: 2013.12.11 8:18:51 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.joules.com%2F&ei=6R-oUpH4GfLQ7Aa6tIDIAQ&usg=AFQjCNGogxNmwosX9d770DUhTMpRsazJXQ&bvm=bv.57799294,d.ZGU CONTENTMOD  GET 383 0  1 200 text/html  Default  - -
    12-11-2013 08:18:51 Local0.Info 192.168.1.1 Dec 11 08:18:38 pf:    192.168.1.15 > 224.0.0.252: igmp v2 report 224.0.0.252
    12-11-2013 08:18:51 Local0.Info 192.168.1.1 Dec 11 08:18:38 pf: 00:00:04.898251 rule 80/8(ip-option): pass in on re0: (tos 0x0, ttl 1, id 13621, offset 0, flags [none], proto IGMP (2), length 32, options (RA))
    12-11-2013 08:18:48 User.Info 192.168.1.1 Dec 11 08:18:36 dansguardian[79155]: 2013.12.11 8:18:36 - 192.168.1.15 http://clients1.google.com/ocsp CONTENTMOD  POST 3613 0  1 504 text/html  Default  - - application/ocsp-request,,107,0,,0;
    12-11-2013 08:18:47 Local0.Info 192.168.1.1 Dec 11 08:18:34 pf:    0.0.0.0 > 224.0.0.1: igmp query v2
    12-11-2013 08:18:47 Local0.Info 192.168.1.1 Dec 11 08:18:34 pf: 00:02:02.601002 rule 3/0(match): block in on re0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA))
    12-11-2013 08:18:37 User.Info 192.168.1.1 Dec 11 08:18:24 dansguardian[79155]: 2013.12.11 8:18:24 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.joules.com%2F&ei=oB-oUpCiEYr17Ab-x4GQBw&usg=AFQjCNGogxNmwosX9d770DUhTMpRsazJXQ&bvm=bv.57799294,d.ZGU CONTENTMOD  GET 383 0  1 200 text/html  Default  - -

    From the sys log it looks like the content mod is still being picked up but I cant find where….



  • Can anyone offer any more advice?



  • Apologize that I cannot be of more help on this but I can't replicate the problem. As a matter of fact, I can't seem to get anything to show up as "CONTENTMOD" in my logs… It makes me question whether my rules are even working!

    I posted in the thread you referenced previously and asked if others were seeing these log entries and got no response...



  • Does your relevant dansguardianfX.conf file look correct? If not, please try this fix, re-save your ACLs, and try again. Make absolutely sure you re-save your rules. Really, it should not need this fix since it was merged before release - but you never know.



  • I'll check the conf file when I get home… pretty sure it is right though because I remember viewing the thread you reference and also remember checking that it was fixed...

    However... can you confirm - should I be seeing "CONTENTMOD" (or something similar) in my DG access log? Are you seeing them for situations where the query string is being modified? Thanks!



  • @timthetortoise:

    Does your relevant dansguardianfX.conf file look correct? If not, please try this fix, re-save your ACLs, and try again. Make absolutely sure you re-save your rules. Really, it should not need this fix since it was merged before release - but you never know.

    Hi

    I had a quick look into that fix and my  /usr/local/pkg/dansguardian.inc file already looks like the one that's been 'fixed'.

    Any more ideas?



  • Please post screens of your ACLs that you're using, or the configuration files for them. It sounds like something is still enabled that shouldn't be.



  • I'll upload the logs, where about's can they be found?



  • Er, the logs aren't as important as your actual configs. Screenshots of your ACLs in your GUI config would suffice.



  • Hi

    I've attached a zip file of all the ACL lists (there are about 30), I hope these are not overlay complicated to follow. I can upload specific screen shots if needed.

    https://www.dropbox.com/s/8lb8w4g7do853bm/ScreenShots.rar

    Thanks



  • Could you post your $dansguardian_dir/etc/dansguardian/lists/contentregexplist.* files, and your $dansguardian_dir/etc/dansguardian/dansguardianf*.conf files? This is definitely a case of something getting mangled by something in the "Content Lists" ACL.



  • I've only just seen this reply!

    I have done some work, I removed DG and reinstalled it, removed all the config files, ACL's and blocked lists and started again. I have been through all the files that have regex in the file name and turned everything off but the problem still persists. I have just tried google.com and got the below in my syslog.

    2014-01-11 08:51:31 User.Info 192.168.1.1 Jan 11 08:51:29 dansguardian[16727]: 2014.1.11 8:51:29 - 192.168.1.37 http://google.com CONTENTMOD  GET 219 0  1 301 -  Default  - -

    I'll try and post the files later.





  • So looking at your configs, you have a lot of uncommented lines. My first recommendation would be to completely empty your contentregexplist.Default list (cat /dev/null > /usr/pbi/dansguardian-xxx/etc/dansguardian/lists/contentregexplist.Default) and your contentregexplist.g_Default list. Reload DG, test. I don't necessarily see anything in the files that would be doing this, but for testing purposes it's best to just have blank files in general. Let me know what effect, if any, that has, and we can go from there.



  • Done that and tested.

    Now nothing shows up in the logs regarding Dansgaurdian/content mod but I still get the same results, Google page loads with the results, click a link, a long Google link is generated and then the page says 'The connection was reset' .



  • Tested it again and found some logs.

    Buda.txt



  • It's interesting that your log looks so different from what I'd expect. Are you logging to syslog instead of access.log?

    When I go to google.co.uk, I am not redirected at all. Here is my log output for searching for "test" and clicking the subsequent Wikipedia link.

    1389894645.231    323 1.2.3.4 TCP_MISS/301 221 GET http://google.co.uk abcd DEFAULT_PARENT/127.0.0.1 -
    1389894645.404     81 1.2.3.4 TCP_MISS/302 222 GET http://www.google.co.uk abcd DEFAULT_PARENT/127.0.0.1 -
    1389894653.163    313 1.2.3.4 TCP_MISS/200 421 GET http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest_cricket&ei=-RvYUsz8CdWgsQTq84CoDw&usg=AFQjCNGPXOKClui7vHgzV25lOsr4nAq50g&bvm=bv.59568121,d.cWc abcd DEFAULT_PARENT/127.0.0.1 text/html
    1389894653.680    379 1.2.3.4 TCP_MISS/200 122872 GET http://en.wikipedia.org/wiki/Test_cricket abcd DEFAULT_PARENT/127.0.0.1 text/html
    1389894653.924    195 1.2.3.4 TCP_MISS/200 117 GET http://bits.wikimedia.org/geoiplookup abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894653.983    254 1.2.3.4 TCP_MISS/200 66560 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector&* abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894653.984    255 1.2.3.4 TCP_MISS/200 21414 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=site&only=styles&skin=vector&* abcd DEFAULT_PARENT/127.0.0.1 text/css
    1389894653.998    269 1.2.3.4 TCP_MISS/200 181864 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.gadget.DRN-wizard%2CReferenceTooltips%2Ccharinsert%2Cteahouse%7Cext.rtlcite%2Cwikihiero%7Cext.uls.nojs%7Cext.visualEditor.viewPageTarget.noscript%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmw.PopUpMediaTransform%7Cskins.common.interface%7Cskins.vector.styles&only=styles&skin=vector&* abcd DEFAULT_PARENT/127.0.0.1 text/css
    1389894654.056     55 1.2.3.4 TCP_MISS/200 9297 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=site&only=scripts&skin=vector&* abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894654.152    142 1.2.3.4 TCP_MISS/200 180709 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki%2CSpinner%7Cjquery.triggerQueueCallback%2CloadingSpinner%2CmwEmbedUtil%7Cmw.MwEmbedSupport&only=scripts&skin=vector&version=20140116T030122Z abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894654.291    103 1.2.3.4 TCP_MISS/200 188560 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.centralNotice.bannerController%7Cext.centralauth.centralautologin%7Cext.uls.i18n%2Cinit%2Cinterface%2Cmessages%2Cpreferences%2Cwebfonts%7Cext.uls.webfonts.repository%7Cext.visualEditor.viewPageTarget.init%7Cext.wikimediaShopLink.core%7Cjquery.byteLength%2Cclient%2Ccookie%2CdelayedBind%2Ci18n%2CjStorage%2Cjson%2CmwExtension%2Ctipsy%2Cwebfonts%7Cmediawiki.Title%2CUri%2Capi%2Ccldr%2CjqueryMsg%2Clanguage%2Cnotify%2Cuser%2Cutil%7Cmediawiki.language.data%2Cinit%7Cmediawiki.legacy.ajax%2Cwikibits%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.startup%7Cskins.vector.js%7Cwikibase.client.init&skin=vector&version=20140116T030123Z&* abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894654.898    148 1.2.3.4 TCP_MISS/200 71 GET http://meta.wikimedia.org/wiki/Special:BannerRandom?uselang=en&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=0&country=US&device=desktop&slot=10 abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894655.230    470 1.2.3.4 TCP_MISS/200 98901 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.articleFeedbackv5.startup%7Cext.cite%2CeventLogging%2CnavigationTiming%7Cext.gadget.DRN-wizard%2CReferenceTooltips%2Ccharinsert%2Cteahouse%7Cext.uls.eventlogger%7Cext.wikimediaEvents.moduleStorage%7Cjquery.articleFeedbackv5.utils%7Cjquery.autoEllipsis%2CcheckboxShiftClick%2Chidpi%2ChighlightText%2CmakeCollapsible%2Cmw-jump%2Cplaceholder%2Csuggestions%2CtabIndex%7Cmediawiki.action.view.postEdit%7Cmediawiki.hidpi%2Cinspect%2CsearchSuggest%7Cmediawiki.page.ready%7Cmobile.desktop%7Cmw.MwEmbedSupport.style%7Cmw.PopUpMediaTransform%7Cschema.ModuleStorage%2CNavigationTiming%2CUniversalLanguageSelector%7Cskins.vector.collapsibleNav&skin=vector&version=20140116T030123Z&* abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894655.611     57 1.2.3.4 TCP_MISS/200 23313 GET http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=jquery.tablesorter%7Cmediawiki.language.months&skin=vector&version=20140116T030123Z&* abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    1389894657.825   3130 1.2.3.4 TCP_MISS/200 261 GET http://login.wikimedia.org/wiki/Special:CentralAutoLogin/checkLoggedIn?type=script&wikiid=enwiki&proto=http abcd DEFAULT_PARENT/127.0.0.1 text/javascript
    
    

    I am beyond baffled at your results at this point, it doesn't seem to make any sense. If I were you, I'd try completely removing everything from your ACL configs, confirming that everything is removed with cat /usr/pbi/dansguardian-xxx/etc/dansguardian/lists/*, reloading (dansguardian -r) and retesting. If you still see the same behavior, there is a much deeper issue here. If you do NOT see the same behavior, test after every ACL change that you make to find the culprit. My guess would be some residual rule not getting taken out correctly, but it's very hard to say.



  • @LordCadbury:

    I've only just seen this reply!

    I have done some work, I removed DG and reinstalled it, removed all the config files, ACL's and blocked lists and started again. I have been through all the files that have regex in the file name and turned everything off but the problem still persists. I have just tried google.com and got the below in my syslog.

    2014-01-11 08:51:31 User.Info 192.168.1.1 Jan 11 08:51:29 dansguardian[16727]: 2014.1.11 8:51:29 - 192.168.1.37 http://google.com CONTENTMOD  GET 219 0  1 301 -  Default  - -

    I'll try and post the files later.

    problem still persists on a clean install? Are you sure there isn't something not being removed on the uninstall?

    find / -name *dansguardian*
    find / -name *dans*



  • I have done a complete re-install, I edited the config.xml (i think?) and deleted all the sections related to DG. I have configured the new ACl's to only block ADs, Tracking and Spyware URL's/Sites. There is no filtering for Porn or Adult material, no regex stuff no content other stuf. Previously I was logging to syslog server but now I'm not, below is a snippet from googling 'test' and choosing the top result (Wikipedia) which fails to load ( the connection was reset).

    
    2014.1.17 13:26:02 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC8QFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest_cricket&ei=VC_ZUoL0Eumq7Qb_rYDoAQ&usg=AFQjCNGPXOKClui7vHgzV25lOsr4nAq50g&bvm=bv.59568121,d.ZGU  GET 421 0  1 200 text/html  Default   - - 
    2014.1.17 14:22:30 - 192.168.1.15 http://clients1.google.com/ocsp  POST 3613 0  1 504 text/html  Default   - - application/ocsp-request,,107,0,,0;
    2014.1.17 14:22:33 - 192.168.1.15 http://clients1.google.com/ocsp  POST 3613 0  1 504 text/html  Default   - - application/ocsp-request,,107,0,,0;
    2014.1.17 14:23:11 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC8QFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest_cricket&ei=aTzZUoyiNI6ihge74oGQCw&usg=AFQjCNGPXOKClui7vHgzV25lOsr4nAq50g&bvm=bv.59568121,d.ZG4  GET 421 0  1 200 text/html  Default   - - 
    2014.1.17 14:29:32 - 192.168.1.15 http://clients1.google.com/ocsp  POST 3613 0  1 504 text/html  Default   - - application/ocsp-request,,107,0,,0;
    2014.1.17 14:30:14 - 192.168.1.15 http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC8QFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest_cricket&ei=ED7ZUoX2HI-shQfD_IGwAw&usg=AFQjCNGPXOKClui7vHgzV25lOsr4nAq50g&bvm=bv.59568121,d.ZG4  GET 421 0  1 200 text/html  Default   - - 
    
    


  • Does this happen if you don't proxy SSL traffic? I was having a problem with timeouts on SSL traffic in DG at one point and never found an exact cause (and now just let SSL out unfiltered). Adding "dns_v4_first on" to my custom squid directives helped a bit, but didn't 100% fix it. I would try pushing SSL traffic out unproxied and seeing if it exhibits the same behavior.



  • My test Firefox box is configured like so

    https://db.tt/GFMeUMC5



  • Working to reproduce your issue, give me a little bit.

    Do you have a secondary machine you can set up as a proxy and test on? I've put in your initial rules and still can't reproduce this behavior. I'm thinking there's something going on elsewhere here.

    EDIT: AHA! REPRODUCED IT! Steps to fix incoming.

    EDIT2: Edit your /usr/pbi/dansguardian-amd64/etc/dansguardian/lists/contentregexplist.g_Default file. Is there text in it? If so, find a line that has this in it:

    "
    


  • The file is complete blank and under Content Lists > Default content Setup the 'Enable' option is not checked…..

    :'(

    Thanks for your help so far!



  • Don't know what to tell you at this point. I've tried to reproduce it under different conditions, but the only way was with the aforementioned contentregexplist contents. If you find a fix, please do update this thread. I'm very curious on why this is happening.



  • @timthetortoise:

    Don't know what to tell you at this point. I've tried to reproduce it under different conditions, but the only way was with the aforementioned contentregexplist contents. If you find a fix, please do update this thread. I'm very curious on why this is happening.

    I've drawn the conclusion that I'm going to wipe/rebuild my pfsense box and see how far I get.

    Thanks again for the help!



  • Just thought I'd report back. I  did a wipe/rebuild and everything is working as expected.