How to configure squidGuard for HTTPS?

[IamArobot]

Hey Everyone. I am trying to enable HTTPS filtering for squidGuard to block categories in shallalist. I have it partially working, but there are some HTTPS sites being blocked that either do not use port 443 or are not in the blacklist. For example, some RDP connections for Internet servers are being blocked (using port 3389), and websites such as FedEx.com are blocked. I have to keep revising the whitelist to add these URL's to them, and even then, other things such as update services for Adobe are not working properly on client PC's. If I disable SSL/MITM it all works again but nothing for HTTPS sites is filtered. Am I missing a simple setting somewhere?

pfsense version: 2.4.5-p1
squid version: 0.4.44_26
squidGuard version: 1.16.18_6

I have DNS forwarder enabled on my firewall pointing to an internal DNS server that is also Active Directory.

squid proxy is enabled for IPv4 on LAN interface. Transparent HTTP Proxy is enabled for LAN interface. I had HTTPS/SSL Interception enabled for "Splice All" mode on LAN interface. It says on the firewall when using splice I do not need to install the certificate on clients.

For squidGuard I am using shallalist blacklist and I have a commonACL to block specific categories. I have a whitelist in Target categories that I was having to always update with more URL's and websites that shouldn't have been blocked. Most HTTPS sites if they aren't in a category were working properly but here are a few that were not working until I whitelist the URL's:
Fedex.com
All Microsoft O365 services (including Outlook on Desktop)
RDP to servers hosted on Internet (port 3389)
Remote PC sessions like GoToAssist, GoToWebinar

Am I required to set up DNS resolver on the firewall along with WPAD/PAC options on my DHCP server for the HTTPS filter to work properly?

I am also not receiving any error pages on HTTPS blocked sites which makes it more difficult to see why it's being blocked. If any screenshots of configuration or logs are required for further assistance please let me know what is desired. Thanks.

[IamArobot]

Just an update to my previous post:

The firewall is in an HA cluster using CARP. I am not using CARP status detection in squid so it is running on both firewalls. Normally my clients point to the LAN VIP as the gateway. When I switch my gateway explicitly to Firewall#2 on a test client and enable an outbound NAT rule to use the interface IP (to get Internet access) and I test squid, it doesn't exhibit the strange behavior where some sites won't load.

Going back to Firewall#1, if I clear disk cache (currently set to 0) in squid, or if I reapply the squid settings it seems to temporarily alleviate the problem.

So for example I will have my gateway on a client set to the HA VIP and have SSL Splice enabled on squid. Everything seems to work fine for a short while, then I can't load the ghx.com website. If I clear disk cache or reload the squid settings (make a trivial change and save) then it seems to load again for a short period before it stops loading. If I set the gateway on the same client explicitly to Firewall#2 (in CARP backup mode) and then enable the outbound NAT rule to use the Interface IP, I do not see these issues.

Does this possibly have something to do with CARP or squid caching?

[coffeelover]

I hope I can help you on some targets:

For complete squidguard support even for ssl sites you have to use the mode 'Splice whitelist, Bump otherwise'. The sites in the whitelist will be spliced which means no MITM is happening. The filter (squidguard) only gets the domain and not the complete url. Other sites are bumped (MITM is happening), so you need the CA-certificate of the proxy ca installed on the clients.

To set a real whitelist (don't try to filter the domains), you have to set

url_rewrite_access deny whitelist

in custom options. This means, that the a domain in the whitelist will not be checked by squidguard.

Next topic:
To get the redirect working even for SSL sites, you have to add the following settings to the custom options:

url_rewrite_access deny CONNECT
url_rewrite_access allow all

This means, that the initial connect session will not go through the filter but the first page served will be filtered.
(The browser only accepts a socket address like host:port in response to the connect. So a redirect url like http://pfsense/sgerror.php will just be parsed as the socket address http:80 (default port) and will throw an error.)

So the complete block should be:
url_rewrite_access deny whitelist
url_rewrite_access deny CONNECT
url_rewrite_access allow all

I set these in the field 'Custom options (before auth)'.

After applying these settings, the redirect should work. In our case it did not, because we only have 1 interface. We configured the redirect Mode to 'ext url move' using the sgerror.php: 'https://pfsense.domain/sgerror.php?url=403&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u'

HTH

[viberua]

Hi, all. Can anybody help with SG and HTTPS.
Before we used MITM as splice whitelist/bump otherwise. But few weeks ago squid with this settings starts crashing and first we disabled MITM at all.
We need filter https by SG so we enabled splice all.
But unfortunatelly when i type url with https:// site any blacklisted url is loading.
For example in shallalist rapid7.com is blocked (Target group: blk_BL_hacking) but if write it wth https:// is loading without any blocks. :(

This work the same as with disabled MITM filtering http:// url but allow any https://
Say again when MITM was working in splice whitelist/bump otherwise - all was working fine.

[coffeelover]

SG does not work in Mode "Splice all" with HTTPS.
You have to set Mode "Splice whitelist, bump otherwise".

You write "Squid is crashing", is there any error noticed in the cache log?

[aGeekhere]

https://forum.netgate.com/topic/100342/guide-to-filtering-web-content-http-and-https-with-pfsense-2-3/190

For crashing
https://forum.netgate.com/topic/153933/solved-squid-0-4-44_25-assertion-failed-http-cc-1533-comm-monitorsread-serverconnection-fd

[viberua]

@aGeekhere yeah i read those threads, hope soon squid will be patched
About error in cache log, i don't have time to check logs (all work is stopped), just searching solution, and found it when i disabled MITM mode.
May be i don't understand well what is splice all mode doing. But yellow marked text make me think that it should work. Because with disabled MITM is work the same as with enabled MITM Splicle All