SquidGuard not working with some HTTPS sites

benc

Hello,

We've got a new PF Sense box (a random Dell PC, i5, 8GB, 1TB, 3x 1gbit NICs). I just put it into service today to replace the older PF Sense box we had. Internet and routing and local network stuff is perfect, but this is a public school and we have to be able to filter out porn. I tried implementing SquidGuard on the old box and kept running into problems. Part of the reason for the upgrade to the PF Sense hardware is due to SquidGuard not working and we don't know what else to do.

I am very frustrated after spending weeks researching and experimenting and troubleshooting. SquidGuard documentation sucks. It's a piece of cake to get HTTP filtering working. HTTPS/SSL is a different story. Explicit mode is not an option because this is a school and we must support Windows, Apple, Andriod, etc etc etc. WPAD is not an option because of this. There are many options in Squid/SquidGuard that have to do with HTTPS/SSL but are poorly explained or documented.

Here is where we stand now: HTTP filtering is perfect. SSL filtering works on 99% of websites, i.e. it will allow 99% of the good sites through while blocking the bad sites. That 1% of websites that don't work just happens to be big sites, the main two being Amazon and Netflix. When I say they don't work I mean I get ERR_SSL_PROTOCOL_ERROR from Chrome when I try to visit these sites. This is the same error I get when a site is blocked. It seems to me that SquidGuard is blocking Amazon and Netflix randomly and I cannot figure out why. I've read that part of the issue with these sites is that they use anycast or a round robin DNS scheme for load balancing, and as a result the public IP of the site does not stay the same. I'm not sure why that is relevant because all other big sites seem to work fine. Microsoft, Facebook, ebay, etc are all fine.

This is a fresh install of PF Sense using the latest ISO (2.4.4p3). No firewall rules defined except the any/any to allow the LAN to access the internet. We have a 1 gig internet connection and bandwidth is great. Performance is amazing with the new PF Sense box.

I'm not sure what info is relevant but here is my process. Install PF Sense with USB, change password, set up interface IPs, set up LAN firewall rule any/any, install Squid, install SquidGuard.

Squid config: (waited until the end to check the box to enable Squid)
Enable DNS IPv4 lookup first
Enable transparent mode
Do not forward traffic to Private Address Space
Enable SSL filtering
Splice all
Created Certificate Authority in PF Sense
Local Cache tab: Set hard disk cache system to null (don't care about caching)

SquidGuard config: (waited until the end to check the box to enable SquidGuard and hit Apply)
Enable Blacklist
Blacklist URL: Shalla's black list.
Target categories tab: Created a dummy category named DummyCategory
Redirect mode: ext url err page
Redirect: [a web page coming from a local windows server]
Enable logging for this ACL
Blacklist tab: downloaded the blacklist
Common ACL tab: Expand target rules. Set DummyCategory to whitelist. Set porn to deny. Set default access [all] to allow. Set up redirect options the same way I set them up in the Target Categories tab. Not sure which redirect settings it's actually using but it doesn't matter as they are identical.
Enable logging for this ACL
Go back to Squid and enable it, hit apply.

I wish I had the expertise to make sense out of all this. Half of the time Amazon and Netflix work fine. The next time you load the page it may give the ERR_SSL_PROTOCOL_ERROR. I've tried with IE and it does essentially the same thing. We have about 50 computers, and the behavior is the same on all of them. I've tried whitelisting the public IP of a website for testing but whitelisting IPs does not work in SquidGuard. At least not the way I understand it.

I've put too much time and effort into building a simple URL filtering solution. I'm ready to pull my hair out because I can't find documentation that makes sense. Much of the info out there is rather old and is focused on HTTP only.

I hope someone out there can point me in the right direction.

Ben

edit: had to remove "http://" from the URLs in this message because it said my post was marked as spam?

benc

To be clear, we need access to Netflix and Amazon. When SquidGuard is enabled, sometimes they load and sometimes they don't. When SquidGuard is disabled, every site works fine first try.

aGeekhere

The best way is
Setup WPAD First
Then transparent proxy
Then transparent proxy MITM with splice all
Then manually set any device that cannot work with the WPAD

Why do this
The WPAD automatically configure a device to use the proxy (things like windows updates or some websites do not work well using a transparent proxy).
Chrome, firefox IE etc should be using the WPAD or manually configure, NOT the transparent proxy!
Now if a device cannot use a WPAD you have to manually configure it.

But not all programs have a proxy setting so a transparent proxy plus MITM (for https) with splice all is there as a fail safe to catch any traffic that the WPAD/manual proxy setup misses.

To conclude to get the best result you must setup
WPAD
Manual proxy
Transparent proxy MITM with splice all

https://forum.netgate.com/topic/100342/guide-to-filtering-web-content-http-and-https-with-pfsense-2-3/176

benc

@aGeekhere I don't have time to finish reading the post right now but it looks like a lot of good info. I don't remember seeing that page.

The reason WPAD won't do us much good is because we have to be able to filter all internet access, including public WiFi.