[SOLVED] Several questions regarding Squid + SquidGuard + SSL certificates

TemplarLord

Hi folks,

Right off the bat I'd like to thank the pfSense guys, the Squid guys, the SquidGuard guys, the Shallalist guys and the people willing to help on this forum. Because of people like you we are able to filter pretty much anything we want, whether it be for a reasonable cause or just to make everyone at home mad because Facebook no longer works. God indeed exists.

Rather than just telling you how awesome you are, I do have some questions regarding this software and how exactly it should work. I was tasked with filtering adult/malware sites and would greatly appreciate your help. I also promise not to abuse this power for before-mentioned purposes, let God be my witness.

I have only a few desktop computers for now which need an active filter(thinking about all WiFi devices but that's future-proofing and not relevant so much right now). I envisioned a simple proxy server running somewhere in my network. I would not be using this as a gateway between my ISP router and my network. If you think this is a grave mistake on my part, please let me know.

So, my setup right now is this:

VmWare VM, running on Workstation 12

Hardware on the host-side

CPU:		AMD Athlon FX-4100 3.6GHz
RAM:		16GB
HDD:		WD Black 1TB

Software on the host-side

OS:		Windows 10 Professional LTSB N

Hardware on the guest VM

CPU:		AMD Athlon FX-4100 3.6GHz (1 core)
RAM:		1GB
HDD:		20GB

Software on the VM (I'll just copy and paste the System Information info

Name 		pfSense. <classified>Version 	2.2.6-RELEASE (amd64)
		built on Mon Dec 21 14:50:08 CST 2015
		FreeBSD 10.1-RELEASE-p25
Platform 	pfSense
CPU Type 	AMD FX(tm)-4100 Quad-Core Processor
Current: 	452 MHz, Max: 3616 MHz
DNS server(s) 	127.0.0.1
		 <classified><classified>8.8.8.8

Packages which are installed and running

Lightsquid 	Network Management 	2.43
Open-VM-Tools 	System 			1280544.13
squid3 		Services 		0.4.7
squidGuard 	Network Management 	1.9.18</classified></classified></classified>

Current setup works like this:

2 network cards, one is used as WAN, the other is LAN, but both are really part of the same network
Transparent HTTP proxy is enabled
SSL middleman tactics are enabled
Self-signed SSL certificate created, imported on some test machines
SquidGuard setup with Shallalist blacklist, some common ACL categories setup(porn, warez), also setup a dummy target category so to make SquidGuard boot up
SquidGuard is setup to just show a blank page. I did this because when an average user accesses a website which is blocked, they don't really know what's going on or what to do, and generally think the Internet is at fault. This approach suits me just fine

If any more info is required please let me know.

So, time for questions!

1. Why on God's green Earth do I get the following error message when I access only some HTTPS sites? For instance if I try to reach " https://palemoon.start.me/start " I get the following error message:

The following error was encountered while trying to retrieve the URL: ://54.230.94.85:443

    Failed to establish a secure connection to 54.230.94.85

The system returned:

    (92) Protocol error (TLS code: SQUID_ERR_SSL_HANDSHAKE)

    Handshake with SSL server failed: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

While Facebook, Google, Paypal work just fine and dandy. Ordinarily, when SquidGuard blocks a website, it's SquidGuard who does it. This error message defaults back to Squid. Why does this happen? Show me my error and make me smarter.

2. For some reason, Squid likes to make fun of me and my filters. When accessing a blocked website which resides on HTTPS I don't get the blank webpage which is supposed to be shown, but rather a certificate error. And if I accept said certificate, I get another error like the one above:

The following error was encountered while trying to retrieve the URL: ://141.101.118.194:443

Failed to establish a secure connection to 141.101.118.194

The system returned:

(92) Protocol error (TLS code: SQUID_ERR_SSL_HANDSHAKE)
Handshake with SSL server failed: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error

This proxy and the remote host failed to negotiate a mutually acceptable security settings for handling your request. It is possible that the remote host does not support secure connections, or the proxy is not satisfied with the host security credentials.

Your cache administrator is admin@localhost.

The thing works just fine if I access a HTTP site. Blank webpage and everything. You probably already know what I'm doing wrong, please enlighten me.

3. Transparent vs. non-transparent – which is better and why? Discuss!!!

My understanding is this, transparent is great because it works on everything and requires almost no additional configuration on the network(except changing the gateway from the existing one(eg ISP router) to the pfSense LAN IP). It's bad because it doesn't work with self-signed SSL certificates, so whenever you access a HTTPS site, you get an error saying the certificate is not valid or some-such nonsense. Would this be remedied by a valid SSL certificate bought from a site such as GoDaddy or somesuch?

If I enable non-transparent, I didn't really get any traffic on my setup, probably did something wrong. Are the SSL problems existent on non-transparent as well? I am open to non-transparent, thanks to this helpful answer: https://forum.pfsense.org/index.php?topic=107909.msg601236#msg601236

Enabling this WPAD thingy sounds like a lot of fun. But I read somewhere it doesn't work with Android phones. Is this really true? Is really nothing perfect? Are there any other drawbacks of this approach?

4. In my current setup, Skype and Outlook do not work. I am unable to sign into Skype, and Outlook just asks for a bunch of certificates and refuses to connect to the server even after adding the certificates to trusted root. Why?

I think that's pretty much it at this point. I guess what I'm really looking for are real-world examples and what would be the best way to go in regards to what I need – blocking a few categories of websites and a rudimentary report system(which LightSquid provides). If you made it this far, thanks! You're a bro.

KOM

1,2: Looks like that site doesn't accept SSLv3, so squid can't complete the SSL handshake.

3: Non-transparent, aka explicit, is best. Transparent mode will not work with HTTPS sites unless you install a pfSense certificate on every single client that will access the proxy. Better to not break HTTPS by using explicit mode in conjunction with WPAD to allow clients to discover the proxy on their own. Clients that fail to auto-detect the proxy (like Android, for instance) will have to be configured manually which is not really that big a deal.

4: Use explicit mode and these problems may just disappear on their own.

TemplarLord

Hi KOM,

thanks for your prompt and concise reply, I apologize for my late reply.

I have deferred to your expert judgement and tried out explicit mode. But, I'm having some problems with it so far. First, I'll explain what I changed to have non-transparent mode:

1. Unticked Transparent HTTP Proxy in <pfsense -="">Services -> Squid Proxy

That's it. :) Unfortunately doing this resulted in a ridiculous slowdown of web performance, so I thought the problem must be in the VM. I then spawned an older machine that's been lying around:

CPU:		Intel Core 2 Duo E7600
RAM:		2GB
HDD:		WD Red 1TB
LAN:		Intel dedicated add-in card

Thinking this would solve the problem, but unfortunately it did not. It's still a very noticeable crawl to browse HTTP sites, while I haven't even touched HTTPS yet.

I only installed the Squid3 package onto it and configured "Local Cache", and pointed my web browser onto it.

Can you tell me what I did wrong with setup and why is it so much slower than transparent proxy filtering? Or just point me in the direction of a good guide, most of what I found is for transparent proxy filtering.</pfsense>

KOM

Transparent vs explicit should have no bearing on the speed of the proxy. To get a look into what's going on, shell in and run:

squidclient -h 192.168.111.180 -p 3128 mgr:info

Look at the Median Service Times. These are the timings measured in seconds for squid operations. Anything look too big as compared to the rest? Here is my output as an example:

Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.20843 0.09736
Cache Misses: 0.22004 0.14252
Cache Hits: 0.00102 0.00091
Near Hits: 0.00307 0.00379
Not-Modified Replies: 0.00091 0.00091
DNS Lookups: 0.03374 0.03374
ICP Queries: 0.00000 0.00000

TemplarLord

Hey KOM,

this is what I got from running the command in shell:

[2.2.6-RELEASE][admin@pfSense2.<confidential>]/root: squidclient -h 192.168.111.180 -p 3128 mgr:info
Sending HTTP request ... done.
HTTP/1.1 403 Forbidden
Server: squid/3.4.10
Mime-Version: 1.0
Date: Mon, 14 Mar 2016 07:17:24 GMT
Content-Type: text/html
Content-Length: 3109
X-Squid-Error: ERR_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from localhost
X-Cache-Lookup: NONE from localhost:3128
Via: 1.1 localhost (squid/3.4.10)
Connection: close

<title>ERROR: The requested URL could not be retrieved</title>

# ERROR

## The requested URL could not be retrieved

* * *

The following error was encountered while trying to retrieve the URL: [cache_object://192.168.111.180/info](cache_object://192.168.111.180/info)

> **Access Denied.**

Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.

Your cache administrator is [admin@localhost](mailto:admin@localhost?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20localhost%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Mon,%2014%20Mar%202016%2007%3A17%3A24%20GMT%0D%0A%0D%0AClientIP%3A%20192.168.111.30%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AHost%3A%20192.168.111.180%0D%0AUser-Agent%3A%20squidclient%2F3.4.10%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0A%0D%0A%0D%0A).

* * *

Generated Mon, 14 Mar 2016 07:17:24 GMT by localhost (squid/3.4.10)</confidential>

Unfortunately, I did not get the Median Service Times thingy from the output. Have I done something wrong?

KOM

Go to Services - Squid Proxy Server. Under the Local Cache tab, go to the External Cache Managers field and make sure it has 127.0.0.1 and your squid LAN IP like this:

127.0.0.1;10.10.4.1

Save it and then try again. Note that 10.10.4.1 is my LAN IP so replace it with your own.

TemplarLord

Hey KOM,

I got it working somehow. Not the Median Service Times thingy, but the proxy itself is lighting fast, if I dare say so my myself. Stuff I changed:

I ticked: Resolve DNS IPv4 First (under Services -> Squid Proxy -> General -> Squid General Settings)

And voila, stuff just works! Installed SquidGuard after I did quite some testing on it with just Squid installed. Managed to get the blacklist to load, setup some categories to block for testing and for now it works pretty nice. Both HTTP and HTTPS is filtered. The only quirk I have so far is when blocking HTTPS sites I don't get the SquidGuard error page but rather an error page from my browser.

Now, if you are so inclined to help me troubleshoot the Media Service Times thingy issue further, I tried adding the IP addresses to External Cache Managers. Here's the output from shell:

[2.2.6-RELEASE][admin@pfSense2.bcs]/root: squidclient -h 192.168.111.180 -p 3128 mgr:info
Sending HTTP request ... done.
HTTP/1.1 403 Forbidden
Server: squid/3.4.10
Mime-Version: 1.0
Date: Tue, 15 Mar 2016 10:10:38 GMT
Content-Type: text/html
Content-Length: 3109
X-Squid-Error: ERR_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from localhost
X-Cache-Lookup: NONE from localhost:3128
Via: 1.1 localhost (squid/3.4.10)
Connection: close

<title>ERROR: The requested URL could not be retrieved</title>

# ERROR

## The requested URL could not be retrieved

* * *

The following error was encountered while trying to retrieve the URL: [cache_object://192.168.111.180/info](cache_object://192.168.111.180/info)

> **Access Denied.**

Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.

Your cache administrator is [admin@localhost](mailto:admin@localhost?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20localhost%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Tue,%2015%20Mar%202016%2010%3A10%3A38%20GMT%0D%0A%0D%0AClientIP%3A%20192.168.111.30%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AHost%3A%20192.168.111.180%0D%0AUser-Agent%3A%20squidclient%2F3.4.10%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0A%0D%0A%0D%0A).

* * *

Generated Tue, 15 Mar 2016 10:10:38 GMT by localhost (squid/3.4.10)

Thanks for all the help so far KOM, you're a bro.

KOM

If it's working quickly now then I wouldn't spend a lot of time trying to get the squidclient output working. Start a new thread about your new problem.

TemplarLord

Not just quickly, it's working like there's nothing in between. Just as it should be.

To summarize for everyone who might have this or a similar issue, I got this problem fixed by going the route of non-transparent proxy(or explicit if you will). Some bumps along the road, but comrade KOM helped me see the error of my ways and set me on the right path.

Phase 1 complete, Phase 2 of my "Ban-Facebook-and-Youtube-for-EVERYONE" is just starting…