Squid + Squidguard with WPAD. Filter doesn't work.

[WhiteTiger-IT]

I continue to have problems and every change I make new ones arise even where there weren't any.

If I enable "SSL filtering" in "HTTPS/SSL Interception" I no longer connect to any website, even if it is not in the blacklist.
If I disable "Transparent" everything goes through, including the websites in the blacklist.

Instead, with Transparent and SSL Filtering, if I go to a porn site, the URL is changed to
https://192.168.1.1/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD
So, the redirect works, but I see that the user name is missing (how can I pass it without forcing people to log in?) and the group is "default".
However, no message appears because there is a Timed Out error.

Instead, DNS always works.
At least that doesn't seem to be a cause.

In my opinion there is some misconfiguration in Squid+Squidguard, but which one?
Is it possible that there is no definitive guide? Nowadays almost all websites are HTTPS.

Even the certificates, after creation it is unthinkable to go around uploading them to all the PCs.
I haven't done it even in this testing phase because I won't be able to do it when fully operational.

==== Update
If I change the proxy settings in Firefox, inserting the days instead of taking them from the system, things change a little.

First the usual message appears

Some software is preventing Firefox from securely connecting to this site
192.168.1.1 is probably a trustworthy site, but a secure connection could not be established. This problem is caused by internal-ca, a software installed on your computer or network.

Then, accepting the risk and moving forward, this message appears:

ERROR

The requested URL could not be retrieved
The following error was encountered while trying to retrieve the URL: https://192.168.1.1/*
    Connection to 192.168.1.1 failed.
The system returned: (61) Connection refused
The remote host or network may be down. Please try the request again.
Your cache administrator is admin@localhost.

[JonathanLee]

@WhiteTiger-IT have you installed certificates on the devices? You would require them for SSL intercept/MITM to function.

Transparent is what I think you want. No certificates and URL blocking.

You said your redirect is now...

https://192.168.1.1/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD

However what is the port number your GUI uses?

https://192.168.1.1:port number here/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD

Did you use redirect replace URL under every category in Squidguard? It has to be under default and all Squidguard groups. It looks like a double redirect. Have you set the proxy to bypass Squidguard redirect also?

With transparent you don't need to worry about proxy auto detect.

I suspect you have no certificates installed on devices correct?

Test that link by itself past it in a browser window.

What port do you use when you log in to PfSense?? That port is also needed to access the WWW folder and that's what contains the sgerror file.

Example 192.168.1.1:8080 is what I have configured as a custom port.

I assumed you have done that as you configured WPAD already.

[WhiteTiger-IT]

@JonathanLee
I didn't realize that the port used for login was needed, I can't find it described anywhere.

As I was saying, I don't find it correct that the official guide is not aligned with the current situation.
I'm reading other guides on the net, more than 50. Most are generic, the most complete are years old, some relating to old versions of pfSense.
There are a lot of unclear things.
Some say to activate also Transparent Mode, some don't.
Some create a rule to block HTTP, some also HTTPS.
Port 3129 also popped up in una guide, should this be configured too? Nobody makes it clear.

Now the error message appears and I thank you for this.
I now have to configure the rest as well.
For example, changes to sgerror.php so that attempts to access a site blocked by squidguard are reported in the log.
The documentation describes which lines to insert, but not where they should be inserted.

I also want to avoid having to install certificates on devices because I can't do it with guests and I don't have an AD domain in Windows to transfer them with a GPO rule.

For now I thank you.

[JonathanLee]

@WhiteTiger-IT you said "The documentation describes which lines to insert, but not where they should be inserted."

You could always use the url that I use and change it to what ever error message you want. Or just redirect to an official company webpage maybe with corporate policy on LAN use.

I agree there is not much information on this topic. I found Netgate forum is the best unofficial guide for any questions you need.

Glad you got it working.

[WhiteTiger-IT]

@JonathanLee said in Squid + Squidguard with WPAD. Filter doesn't work.:

@WhiteTiger-IT you said "The documentation describes which lines to insert, but not where they should be inserted."

You could always use the url that I use and change it to what ever error message you want. Or just redirect to an official company webpage maybe with corporate policy on LAN use.

We are talking about two different things.
You are referring to the error message and your suggestion was helpful to me.

I'm now talking about the modification to be made to sgerror.php so that anything blocked by squidguard is inserted into the log.
In the documentation i find the portion of code to insert, but I didn't understand where this code should be inserted in sgerror.php.

Then, I think I understood that if I want to filter HTTPS I am obliged to pass the certificate created in pfSense to everyone who browses.
This is impossible for me because not everyone wants to do it or would not be capable of doing it.

Also, if the domain has its own certificate provided by the ISP or Let's Encrypt, what should I do?
Still create a certificate in pfSense for Squid?

[JonathanLee]

@WhiteTiger-IT You said, "I'm now talking about the modification to be made to sgerror.php so that anything blocked by squidguard is inserted into the log. In the documentation i find the portion of code to insert, but I didn't understand where this code should be inserted in sgerror.php."

Squidguard does have the ability to log blocks in the rules area click the log radio button. This will log the blocks you have created to Squid Proxy.

/var/squidGuard/log/squidGuard.log

You also said, "Then, I think I understood that if I want to filter HTTPS I am obliged to pass the certificate created in pfSense to everyone who browses. This is impossible for me because not everyone wants to do it or would not be capable of doing it."

You are still able to splice https get requests and block in transparent mode without certificates as the HTTPS is not broken only the get requests are inspected.

You also said "Also, if the domain has its own certificate provided by the ISP or Let's Encrypt, what should I do?
Still create a certificate in pfSense for Squid?"

I still created a certificate in Squid and passed it to PfSense so they trust each other. Follow this guide.

Ref:
https://forum.it-monkey.net/index.php?topic=23.0

This covers a lot I think this is a great guide. It is also still relevant with 25.05

[WhiteTiger-IT]

@JonathanLee
I followed the guide step by step.
I even reset pfsense to factory settings.
I also removed the IPv6 settings from Windows 11 although I don't use them and there are no settings for IP6 on pfSense.
Nonetheless, I can't navigate (Timeout error) and what's worse, with these settings I lose access to the Internet even from my PC (with alias IT_Admin).
I can ping or tracert any domain, but it's not reachable.

This is done using the network settings in Windows and Firefox. Instead, if I set the proxy directly in Firefox, there are no active filters and everything passes.

Also, in monitor I still see references to the Squidguard blacklist from the previous installation. How do I clear the Squid and Squidguard log?

Finally, in this guide I don't understand the need to create the CA from SSH instead of GUI. However, I created it from SSH.

To conclude, I don't know what to do anymore, except give up HTTPS control and leave the transparent proxy.
Tomorrow I'll do a new installation by formatting the disk, but I confess that I'm very disappointed.

These the rules.

[JonathanLee]

You will never get SSL intercept to fully work as some applications require different things. If you are not using IPv6 have you disabled it inside the DNS forwarder? You have WPAD set up in your DNS?

Here is copy my my custom IPv6 disable, my ISP does not allow IPv6, and my WPAD set up in DNS as a host override

Here is a copy of my ACLs if you want to check it out

NAT RULES for DNS and NTP

Squid ACL
.* for my allow all domians I let Squidguard do my blocks

Squid Proxy Custom options.

Some clients I splice everything for so they are transparent anyway.

Take note of the Regular Expression file it is pointing to. . .
This is my always splice file no matter what like banks phone messages etc, that way they are never intercepted ever.

I do not use certificate checks inside the proxy

Squid's CERT is also a root authority see CA-Cert

Same here

Xforwarder mode is set to transparent for mine as I am not using anything else just the firewall and the Dmark

I have a large do not cache list

I also have a Squidguard always allow list

If you use a Blacklist proxy you also have to set it to allow as a default outside of the blocks so it does not block everything.

You also have to point clients to it that are not set as transparent

I just noticed your Rule 2 you have a any any rule this will bypass everything as clients do not need to use the proxy at all with that rule.

[JonathanLee]

If you are just starting out follow a guide for transparent mode and just use it as a URL blocker. I personally inspect a lot of traffic and make reports all the time on my freetime.

[JonathanLee]

@WhiteTiger-IT You also would need SMTP and POP for email, I would look into your ACLS more you have a ANY ANY LAN rule that is a contradiction to the proxy.

[JonathanLee]

You could also try pfblocking. I have always used Squid and I like the puzzles, but a lot of users like pfblock over Squidguard because of the ease of use. I have always liked Squidguard and Squid so I just use that.

[WhiteTiger-IT]

@JonathanLee
I have already installed some pfSense and they work without problems without HTTPS configurations.
As I said previously, I need to manage traffic without being too intrusive with users because I don't always have permission to intervene on their laptops and smartphones.
So I need ClamAV and Squidguard filters to filter their traffic, preventing or limiting access to the network and the content they request on the network.
My case is for a small company, but the best example would be a school where you have to allow students to access the network without letting them do what they want and monitor "who does what".
I'm spending days looking for a correct configuration and frankly I don't understand all these difficulties for activities that in 2023 should already be taken for granted at first installation level.
Furthermore, I also find it difficult to understand the reasons.
Squidguard handles URLs (unlike Dansguardian which handles content), so you shouldn't even need to open the packet.
It makes sense to do this for ClamAV, but that still wouldn't be a fixable security service. So, I might as well not install it.

The guide you suggested to me the other day was over 5 years ago; The world has changed and perhaps the guide should also change.
However, it didn't help me even with a new installation.

Now I'll do the tests with your latest suggestions and then if it doesn't work, I'll leave it alone because you can't spend all this time in vain.

In any case, thank you for your help. I really appreciated it, also for your patience.

[JonathanLee]

@WhiteTiger-IT

I can see your access control lists need to be studied more. The proxy takes over 443 and 80 when it's being used however if you have a any any rule for approval for all ports and for all lan traffic that defeats the purpose of the proxy and use of port 3128. Again you also want to use port address translation if your using unbound DNS.

Don't give up I am sure you will get it. I highly suggest you only use transparent mode for URL filtering. SSL intercept is more of an advanced configuration. It took me a long time to get it right.

I wish you good luck. Sorry it's not working yet.

You asked about the log clearing it. Just go to command line and delete it if you need to clear it.

[WhiteTiger-IT]

@JonathanLee
I was trying to work on it now.

I can't do without https.
It's a small business firewall where I work, I deal with guests and people who use their laptop or smartphone.
I have to create filters to prevent them from doing what they want.
I should have also configured squidguard for subnets, but there's no point in thinking about it if I can't even configure the main network.

In your screenshots I don't see many differences regarding the main settings, but I see other differences compared to the documentation and guide from the other day. For example there are NAT rules.

I didn't understand the use of the ".*" whitelist. Quale è esattamente lo scopo?
Not even NoSSLintercept acls. I also have to avoid managing certain traffic such as that of banks.

As for DNS, in General Setup I have configured "DNS Server Override = On" and in DNS Resolver "DNS Query Forwarding = On".

Did you create the Certificate Authority from the GUI or from SSH as indicated in the guide?

[JonathanLee]

@WhiteTiger-IT command line after imported it into pfsense gui
whitelist is a wildcard to allow internet use

Do you use any dhcp options??

Ref:
https://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml
https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-networking-admin/dhcp/dhcp-options/predefined-dhcp-options

[WhiteTiger-IT]

@JonathanLee
This news makes our discussion purely philosophical.

[JonathanLee]

@WhiteTiger-IT

Looks updated to me. . .

I am going to continue to run it. I had issues with Intel Speed shift patch on my SG-2100 so I will stay with the last stable version. It was so slow when I went to 23.09

[JonathanLee]

@WhiteTiger-IT Netgate Does recommend you uninstall it and use something else.

Sorry It is pretty advanced to run and configure it.

[WhiteTiger-IT]

@JonathanLee
I had already seen that Squid only produced updates last week, so I was surprised by pfSense's decision.
Furthermore, in other posts we read that version 28 is close to release, so the problem is "just around the corner".
In the OPNSense forums we read more or less the same thing; they will remove Squid from the integrated system and place it among the optional plugins, therefore without support.

Squid is not a secondary element, but the basis of web traffic management.
I'm very perplexed.
If I focus on a firewall distribution it is certainly not to then go and implement its components on my own.

At this point it is difficult to decide what to do.
For the antivirus I can always decide to rely only on the one installed on the PCs.
For the cache it no longer becomes so important where there is good network connectivity (however in this company there is still an old ADSL because optical fiber is not available).
But it becomes difficult, if not impossible, to manage content filtering if you cannot rely on engines like Squidguard or Dansguardian since they rely on Squid. I don't know others.

[JonathanLee]

@WhiteTiger-IT there is a way to run Squid on a raspberry pi 5 or 4b or some other box, simply NAT traffic to it. Or use mitmproxy it's also open source.

But heed Netgate's warnings. If you use it do so at your own risk.