https filter with https://http:/*



  • Re: https filter with https://http:/*

    I'm having the same issues when navigating to an entry in the Common ACL that's set to "deny"

    • pf v2.4.4_2
    • Squid package v0.4.44_7
    • SquidGuard package v1.16.18_1

    Here's the gist:

    • when using the proxy in transparent mode, everything works as expected, SSL interception and all.
    • when using it with defined proxy servers in the browser, http works as expected, as does everything SSL as long as the site isn't blocked in SquidGuard

    When using defined proxy servers, and the entry is blocked in SquidGuard, I get the following Squid error message. Stolen from another post, but it's exactly the same thing.

    alt text

    So it seems that Squid is getting bad data back from the SquidGuard callout. I tried to test the URL directly and it seems that squidGuard (the binary) errors out:

    [2.4.4-RELEASE][admin@pf]: echo "https://www.testblockedurl.com/" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
    
    BH message="squidGuard error parsing squid line"
    

    Seems like this is a bug. Yes, the site gets blocked as expected, so it's not a total loss, but the logging doesn’t trigger and the enduser has no idea what happened.

    Edit 1: I tried following the other advice regarding disabling "prevent access by IP address" and it did not alleviate the problem.

    Thoughts?



  • Try:

    echo "https://www.testblockedurl.com/ 10.10.2.1 – – GET" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
    

    Where 10.10.2.1 is the IP address of the client making the request. I tried it with playboy.com and received:

    OK rewrite-url="http://10.10.4.1:80/sgerror.php?url=403%20&a=10.10.2.1&n=&i=&s=default&t=blk_BL_porn&u=https://www.playboy.com/"
    


  • @kom said in https filter with https://http:/*:

    echo "https://www.testblockedurl.com/ 10.10.2.1 – – GET" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf

    Thanks. Tried that but get ERR on a newline with no details. Nothing in /var/squidGuard/log/squidGuard.log either.



  • @kom Tried a bunch of other options, and I do get a valid redirect link, but only when there should be a redirect. If the URL I pass is one that the proxy should allow, I get "ERR" which I can't imagine is correct?

    EDIT: This is correct behavior according to https://wiki.squid-cache.org/Features/Redirectors

    SO, my question now becomes, what's the best way to log what is being sent to the rewrite provider? I'm guessing that Squid isn't sending what it should be. Thoughts?



  • I also get ERR for valid URLs when running that test. No idea why, but my config works just fine.



  • Right, the ERR is normal. Means "don't rewrite the URL". So now I need to see what Squid is passing the squidGuard binary. Any ideas? I've narrowed it down to a small use-case now and I think this is a legit bug in Squid's treatment of external rewrite providers



  • Isn't that enclosed within the quotes? "https://www.testblockedurl.com/ 10.10.2.1 – – GET" I don't know for sure but you would think they use the same method they use for the test.



  • The method itself seems to be OK, but I have the feeling that Squid is sending over something that looks more like this:

    "https://https//www.testblockedurl.com/ 10.10.2.1 – – GET"

    and that's what's causing the error message to be

    The following error was encountered while trying to retrieve the URL: https://https/*
    
        Unable to determine IP address from host name "https"
    
    The DNS server returned:
    
        Server Failure: The name server was unable to process this query.
    
    This means that the cache was not able to resolve the hostname presented in the URL.
    

    Something is causing Squid to spit out a wrongly-concatenated URL when SSL inspection is turned on and everything's bumped by default.



  • I run explicit with WPAD and no SSL intercept. I've never seen this problem. Try increasing the log level for squid and see if it give you any more data.



  • Looking into that now actually. FWIW this is in a tightly controlled RDS environment so I'm able to specify proxy settings via Group Policy without having to worry about WPAD "fun" yet. I'll report back if I find anything. Thanks for the insight



  • @bfu said in https filter with https://http:/*:

    Looking into that now actually. FWIW this is in a tightly controlled RDS environment so I'm able to specify proxy settings via Group Policy without having to worry about WPAD "fun" yet. I'll report back if I find anything. Thanks for the insight

    Have you tried using e2guardian?
    much more stable as intercept
    other option works fine when ssl mitm is set



  • @bfu

    Hey! Same problem here. Anyone found the solution?



  • Also curious if there was a solution without wpad. I really don't want to host a wpad file using "http" admin on the pfsense box instead of "https". I hope transparent mode HTTPS & HTTP will work without WPAD files.



  • I found this and haven't been able to test yet.

    SquidGuard is broken for https out of the box. You need configure Common ACL Target Rules List Default access [all] to Allow, save. Then click Apply in General settings tab.

    My best bet is that Default access has no block page configured for some reason. If anyone knows how to get Default access to deny working please let me know.

    Here is my working SquidGuard configuration step by step tested on pfSense 2.3.4-RELEASE-p1 (amd64):

    Download any blacklist - shallalist. for example.
    General Settings -> Blacklist options -> check to enable blacklist
    Put in Blacklist URL: shalla list
    Go to Blacklist tab.
    Hit download (Black list url is already there)
    Wait for it to finish downloading.
    2. You need to configure your blacklist default to Allow state (The default state which is Deny all is what causes https://http/* error)

    Go to Common ACL Tab
    Hit plus button on Target Rules List
    Scroll down to Default access [all], set access to allow
    Set other categories that you want to be blocked to deny.
    Hit save at the bottom of the page.
    Go to General settings Tab.
    Click Apply at to Top of the page so your settings will be applied from Common ACL Tab.
    Check if https sites load properly now.

    Remember to clear cache from before playing with pfsense from your browser or it will show you old state of web filtering.



  • @jonathanp123 I had that same thought. I discovered this "Unofficial WPAD package" that allows you to use WPAD without changing the protocol to HTTP. I haven't tested it yet because I'm spending my time to find a solution to the error in question, about "The following error was encountered while trying to retrieve the URL: https: // http / * [...]".



  • @jonathanp123 Well, I had already tested this solution but the error persists :/



  • @mguarienti I tested as well. I gave up on "transparent" mode. I ended up using a GPO in active directory to force the proxy settings on the client. I guess transparent mode doesn't support auth either. I want to try pf2ad so users will show up in the reporting.



  • @jonathanp123 I gave up on transparent mode too. i'm still running without the wpad for a moment. But when blocking a website with HTTPS pfSense tries to resolve a host 'https://http*', like the error. With HTTP it displays the correct page.


Log in to reply