https filter with https://http:/*

bfu · Mar 12, 2019, 6:59 PM

I'm having the same issues when navigating to an entry in the Common ACL that's set to "deny"

pf v2.4.4_2
Squid package v0.4.44_7
SquidGuard package v1.16.18_1

Here's the gist:

when using the proxy in transparent mode, everything works as expected, SSL interception and all.
when using it with defined proxy servers in the browser, http works as expected, as does everything SSL as long as the site isn't blocked in SquidGuard

When using defined proxy servers, and the entry is blocked in SquidGuard, I get the following Squid error message. Stolen from another post, but it's exactly the same thing.

So it seems that Squid is getting bad data back from the SquidGuard callout. I tried to test the URL directly and it seems that squidGuard (the binary) errors out:

[2.4.4-RELEASE][admin@pf]: echo "https://www.testblockedurl.com/" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf

BH message="squidGuard error parsing squid line"

Seems like this is a bug. Yes, the site gets blocked as expected, so it's not a total loss, but the logging doesn’t trigger and the enduser has no idea what happened.

Edit 1: I tried following the other advice regarding disabling "prevent access by IP address" and it did not alleviate the problem.

Thoughts?

KOM · Mar 12, 2019, 8:02 PM

Try:

echo "https://www.testblockedurl.com/ 10.10.2.1 – – GET" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf

Where 10.10.2.1 is the IP address of the client making the request. I tried it with playboy.com and received:

OK rewrite-url="http://10.10.4.1:80/sgerror.php?url=403%20&a=10.10.2.1&n=&i=&s=default&t=blk_BL_porn&u=https://www.playboy.com/"

bfu · Mar 14, 2019, 3:56 PM

@kom said in https filter with https://http:/*:

echo "https://www.testblockedurl.com/ 10.10.2.1 – – GET" | /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf

Thanks. Tried that but get ERR on a newline with no details. Nothing in /var/squidGuard/log/squidGuard.log either.

bfu · Mar 14, 2019, 4:07 PM

@kom Tried a bunch of other options, and I do get a valid redirect link, but only when there should be a redirect. If the URL I pass is one that the proxy should allow, I get "ERR" which I can't imagine is correct?

EDIT: This is correct behavior according to https://wiki.squid-cache.org/Features/Redirectors

SO, my question now becomes, what's the best way to log what is being sent to the rewrite provider? I'm guessing that Squid isn't sending what it should be. Thoughts?

KOM · Mar 14, 2019, 5:30 PM

I also get ERR for valid URLs when running that test. No idea why, but my config works just fine.

bfu · Mar 14, 2019, 9:08 PM

Right, the ERR is normal. Means "don't rewrite the URL". So now I need to see what Squid is passing the squidGuard binary. Any ideas? I've narrowed it down to a small use-case now and I think this is a legit bug in Squid's treatment of external rewrite providers

KOM · Mar 15, 2019, 1:38 PM

Isn't that enclosed within the quotes? "https://www.testblockedurl.com/ 10.10.2.1 – – GET" I don't know for sure but you would think they use the same method they use for the test.

bfu · Mar 15, 2019, 6:12 PM

The method itself seems to be OK, but I have the feeling that Squid is sending over something that looks more like this:

"https://https//www.testblockedurl.com/ 10.10.2.1 – – GET"

and that's what's causing the error message to be

The following error was encountered while trying to retrieve the URL: https://https/*

    Unable to determine IP address from host name "https"

The DNS server returned:

    Server Failure: The name server was unable to process this query.

This means that the cache was not able to resolve the hostname presented in the URL.

Something is causing Squid to spit out a wrongly-concatenated URL when SSL inspection is turned on and everything's bumped by default.

KOM · Mar 15, 2019, 6:19 PM

I run explicit with WPAD and no SSL intercept. I've never seen this problem. Try increasing the log level for squid and see if it give you any more data.

bfu · Mar 15, 2019, 6:27 PM

Looking into that now actually. FWIW this is in a tightly controlled RDS environment so I'm able to specify proxy settings via Group Policy without having to worry about WPAD "fun" yet. I'll report back if I find anything. Thanks for the insight

susamlicubuk · Mar 15, 2019, 11:02 PM

@bfu said in https filter with https://http:/*:

Looking into that now actually. FWIW this is in a tightly controlled RDS environment so I'm able to specify proxy settings via Group Policy without having to worry about WPAD "fun" yet. I'll report back if I find anything. Thanks for the insight

Have you tried using e2guardian?
much more stable as intercept
other option works fine when ssl mitm is set

mguarienti · Jan 20, 2020, 6:45 PM

@bfu

Hey! Same problem here. Anyone found the solution?

jonathanp123 · Jan 24, 2020, 9:27 PM

Also curious if there was a solution without wpad. I really don't want to host a wpad file using "http" admin on the pfsense box instead of "https". I hope transparent mode HTTPS & HTTP will work without WPAD files.

jonathanp123 · Jan 24, 2020, 9:40 PM

I found this and haven't been able to test yet.

SquidGuard is broken for https out of the box. You need configure Common ACL Target Rules List Default access [all] to Allow, save. Then click Apply in General settings tab.

My best bet is that Default access has no block page configured for some reason. If anyone knows how to get Default access to deny working please let me know.

Here is my working SquidGuard configuration step by step tested on pfSense 2.3.4-RELEASE-p1 (amd64):

Download any blacklist - shallalist. for example.
General Settings -> Blacklist options -> check to enable blacklist
Put in Blacklist URL: shalla list
Go to Blacklist tab.
Hit download (Black list url is already there)
Wait for it to finish downloading.
2. You need to configure your blacklist default to Allow state (The default state which is Deny all is what causes https://http/* error)

Go to Common ACL Tab
Hit plus button on Target Rules List
Scroll down to Default access [all], set access to allow
Set other categories that you want to be blocked to deny.
Hit save at the bottom of the page.
Go to General settings Tab.
Click Apply at to Top of the page so your settings will be applied from Common ACL Tab.
Check if https sites load properly now.

Remember to clear cache from before playing with pfsense from your browser or it will show you old state of web filtering.

mguarienti · Jan 27, 2020, 5:34 PM

@jonathanp123 I had that same thought. I discovered this "Unofficial WPAD package" that allows you to use WPAD without changing the protocol to HTTP. I haven't tested it yet because I'm spending my time to find a solution to the error in question, about "The following error was encountered while trying to retrieve the URL: https: // http / * [...]".

mguarienti · Jan 27, 2020, 5:37 PM

@jonathanp123 Well, I had already tested this solution but the error persists :/

jonathanp123 · Jan 27, 2020, 5:38 PM

@mguarienti I tested as well. I gave up on "transparent" mode. I ended up using a GPO in active directory to force the proxy settings on the client. I guess transparent mode doesn't support auth either. I want to try pf2ad so users will show up in the reporting.

mguarienti · Jan 27, 2020, 5:46 PM

@jonathanp123 I gave up on transparent mode too. i'm still running without the wpad for a moment. But when blocking a website with HTTPS pfSense tries to resolve a host 'https://http*', like the error. With HTTP it displays the correct page.