Squid + Squidguard with WPAD. Filter doesn't work.

[WhiteTiger-IT]

I followed the guides in the pfsense documentation and web pages online, but I can't get Squid and SquidGuard to work with a proxy setting that doesn't force the user to customize his device and everything happens automatically.
I created wpad.dat, wpad.da and proxy.pac as in this guide.
Again as suggested in the guide, I added a Host Override in DNS Resolver (I didn't use DNS Forwarder because I use the Resolver).
I also created the rule that blocks HTTP.
For the test I used https://www.lagado.com/tools/cache-test because the URL in the guide doesn't seem to be correct today. The test doesn't seem to work.
I set a Windows 10 PC's proxy to automatically detect network settings (even though I didn't want to do that).
I configured Squid by enabling Transparent HTTP Proxy and HTTPS/SSL Interception.
I created a new CA which I then pointed to SSL Man In the Middle Filtering / CA.
For the test in Squidguard I activated the block of Adult, Dating and Social Networks.
I would like to activate "Log Pages Denied by SquidGuard", but it is suggested to insert a code without giving any indication of where to insert this code in sgerror.php. So, I didn't enable this option.
On a private Firefox page I tried to call up Facebook, a porn site, a dating site and pfsense.com (without prefixing http:// or https://). All sites open without problems, both with in the proxy setting of Firefox automatic network setup or using system settings).
Instead, if I enter the pfsense address in the Proxy Firefox settings, the only page that can open is pfsense.com while the other 3 give me an address not found error.
In the Squidguard logs there is "Request(default/blk_blacklists_social_networks/-) - CONNECT REDIRECT", but in Squidguard I configured Redirect mode = "int error page (enter error message)" with in Redirect info the message "Site blocked. Ask the administrator what you need to do."

I can't understand what I did wrong.

As I said initially, I don't want to have to force people to customize PCs, smartphones and browsers because I have both users and guests on the network.
Furthermore, everyone has little familiarity and would not be able to do it anyway.

[JonathanLee]

Just to confirm you would like this to occur for blocks?

If so you can create a custom redirect entry as you likely using pfsense to serve WPAD, this can cause complication where the built in redirect is not fully functional.

In place used a customized external move

[JonathanLee]

use this file as a guide

Path is

/usr/local/www/sgerror.php.orig

<?php
include "globals.inc";
include "config.inc";
$page_info = <<<EOD
/*
 * sgerror.php
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2017-2023 Rubicon Communications, LLC (Netgate)
 * Copyright (c) 2006-2011 Serg Dvoriancev
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

----------------------------------------------------------------------------------------------------------------------
SquidGuard error page generator
----------------------------------------------------------------------------------------------------------------------
This program processes redirection requests to specified URL or generated error page for a standard HTTP error code.
Redirection supports HTTP and HTTPS protocols.
----------------------------------------------------------------------------------------------------------------------
Format:
	sgerror.php?url=[http://myurl]or[https://myurl]or[error_code[space_code]output-message][incoming SquidGuard variables]
Incoming SquidGuard variables:
	a=client_address
	n=client_name
	i=client_user
	s=client_group
	t=target_group
	u=client_url
Example:
	sgerror.php?url=http://myurl.com&a=..&n=..&i=..&s=..&t=..&u=..
	sgerror.php?url=https://myurl.com&a=..&n=..&i=..&s=..&t=..&u=..
	sgerror.php?url=404%20output-message&a=..&n=..&i=..&s=..&t=..&u=..
----------------------------------------------------------------------------------------------------------------------
Tags:
	myurl and output messages can include Tags
		[a] - client address
		[n] - client name
		[i] - client user
		[s] - client group
		[t] - target group
		[u] - client url
Example:
	sgerror.php?url=401 Unauthorized access to URL [u] for client [n]
	sgerror.php?url=http://my_error_page.php?cladr=%5Ba%5D&clname=%5Bn%5D // %5b=[ %d=]
----------------------------------------------------------------------------------------------------------------------
Special Tags:
	blank     - get blank page
	blank_img - get one-pixel transparent image (to replace images such as banners, ads, etc.)
Example:
	sgerror.php?url=blank
	sgerror.php?url=blank_img
----------------------------------------------------------------------------------------------------------------------
EOD;

define('ACTION_URL', 'url');
define('ACTION_RES', 'res');
define('ACTION_MSG', 'msg');

define('TAG_BLANK', 'blank');
define('TAG_BLANK_IMG', 'blank_img');

/* ----------------------------------------------------------------------------------------------------------------------
 * ?url=EMPTY_IMG
 *      Use this option to replace banners/ads with a transparent picture. This is better for web page rendering.
 * ----------------------------------------------------------------------------------------------------------------------
 * NULL GIF file
 * HEX: 47 49 46 38 39 61 - - -
 * SYM: G  I  F  8  9  a  01 00 | 01 00 80 00 00 FF FF FF | 00 00 00 2C 00 00 00 00 | 01 00 01 00 00 02 02 44 | 01 00 3B
 * ----------------------------------------------------------------------------------------------------------------------
 */
define('GIF_BODY', "GIF89a\x01\x00\x01\x00\x80\x00\x00\xFF\xFF\xFF\x00\x00\x00\x2C\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02\x44\x01\x00\x3B");

$url  = '';
$msg  = '';
$cl   = Array(); // squidGuard variables: %a %n %i %s %t %u
$err_code = array();

$err_code[301] = "301 Moved Permanently";
$err_code[302] = "302 Found";
$err_code[303] = "303 See Other";
$err_code[305] = "305 Use Proxy";

$err_code[400] = "400 Bad Request";
$err_code[401] = "401 Unauthorized";
$err_code[402] = "402 Payment Required";
$err_code[403] = "403 Forbidden";
$err_code[404] = "404 Not Found";
$err_code[405] = "405 Method Not Allowed";
$err_code[406] = "406 Not Acceptable";
$err_code[407] = "407 Proxy Authentication Required";
$err_code[408] = "408 Request Time-out";
$err_code[409] = "409 Conflict";
$err_code[410] = "410 Gone";
$err_code[411] = "411 Length Required";
$err_code[412] = "412 Precondition Failed";
$err_code[413] = "413 Request Entity Too Large";
$err_code[414] = "414 Request-URI Too Large";
$err_code[415] = "415 Unsupported Media Type";
$err_code[416] = "416 Requested range not satisfiable";
$err_code[417] = "417 Expectation Failed";

$err_code[500] = "500 Internal Server Error";
$err_code[501] = "501 Not Implemented";
$err_code[502] = "502 Bad Gateway";
$err_code[503] = "503 Service Unavailable";
$err_code[504] = "504 Gateway Time-out";
$err_code[505] = "505 HTTP Version not supported";

/* ----------------------------------------------------------------------------------------------------------------------
 * Functions
 * ----------------------------------------------------------------------------------------------------------------------
 */
function get_page($body) { ?>
<html>
	<body>
<?=$body?>
	</body>
</html>
<?php
}

/*
 * Generate an error page for the user
 */
function get_error_page($er_code_id, $err_msg='') {
	global $err_code, $cl;
	header("HTTP/1.1 " . $err_code[$er_code_id]);

?>
<html>
	<head>
		<title>squidGuard Error page</title>
	</head>
	<body>
	<?php if (config_get_path('installedpackages/squidguarddefault/config/0/deniedmessage')): ?>
		<h3><?= config_get_path('installedpackages/squidguarddefault/config/0/deniedmessage') ?>: <?= htmlspecialchars($err_code[$er_code_id]) ?></h3>;
	<?php else: ?>
		<h3>Request denied by <?= g_get('product_name') ?> proxy: <?= htmlspecialchars($err_code[$er_code_id]) ?></h3>
	<?php endif; ?>

	<?php if ($err_msg): ?>
		<b>Reason:</b> <?= htmlspecialchars($err_msg) ?>
	<?php endif; ?>

		<hr size="1" noshade>
	<?php if ($cl['a']): ?>
		<b> Client address: </b> <?= htmlspecialchars($cl['a']) ?><br/>
	<?php endif; ?>

	<?php if ($cl['n']): ?>
		<b> Client name:    </b> <?= htmlspecialchars($cl['n']) ?><br/>
	<?php endif; ?>

	<?php if ($cl['i']): ?>
		<b> Client user:    </b> <?= htmlspecialchars($cl['i']) ?><br/>
	<?php endif; ?>

	<?php if ($cl['s']): ?>
		<b> Client group:   </b> <?= htmlspecialchars($cl['s']) ?><br/>
	<?php endif; ?>

	<?php if ($cl['t']): ?>
		<b> Target group:   </b> <?= htmlspecialchars($cl['t']) ?><br/>
	<?php endif; ?>

	<?php if ($cl['u']): ?>
		<b> URL:            </b> <?= htmlspecialchars($cl['u']) ?><br/>
	<?php endif; ?>

		<hr size="1" noshade>
	</body>
</html>
<?php
}

function get_about() {
	global $err_code, $page_info; ?>
<?= str_replace("\n", "<br/>", $page_info); ?>
<br/>
<table>
	<tr><th><b>HTTP error codes (ERROR_CODE):</b></th></tr>
	<?php foreach ($err_code as $val): ?>
	<tr><td><?= htmlspecialchars($val) ?></td></tr>
	<?php endforeach; ?>
</table>
<?php
}


/* ----------------------------------------------------------------------------------------------------------------------
 * Check arguments
 * ----------------------------------------------------------------------------------------------------------------------
 */
if (count($_REQUEST)) {
	$url  = trim($_REQUEST['url']);
	$msg  = $_REQUEST['msg'];
	$cl['a'] = $_REQUEST['a'];
	$cl['n'] = $_REQUEST['n'];
	$cl['i'] = $_REQUEST['i'];
	$cl['s'] = $_REQUEST['s'];
	$cl['t'] = $_REQUEST['t'];
	$cl['u'] = $_REQUEST['u'];
} else {
	// Show 'About page'
	echo get_page(get_about());
	exit();
}

/* ----------------------------------------------------------------------------------------------------------------------
 * Process URLs
 * ----------------------------------------------------------------------------------------------------------------------
 */
if ($url) {
	$err_id = 0;

	// Check error code
	foreach ($err_code as $key => $val) {
		if (strpos(strtolower($url), strval($key)) === 0) {
			$err_id = $key;
			break;
		}
	}

	if ($url === TAG_BLANK) {
		// Output a blank page
		echo get_page('');
	} elseif ($url === TAG_BLANK_IMG) {
		// Return a blank image
		header("Content-Type: image/gif;"); // charset=windows-1251");
		echo GIF_BODY;
	} elseif ($err_id !== 0) {
		// Output an error code
		$er_msg = strstr($_GET['url'], ' ');
		echo get_error_page($err_id, $er_msg);
	} elseif ((strpos(strtolower($url), "http://") === 0) or (strpos(strtolower($url), "https://") === 0)) {
		// Redirect to the specified url
		header("HTTP/1.0");
		header("Location: $url", '', 302);
	} else {
		// Output an error
		echo get_page("sgerror: error arguments " . htmlspecialchars($url));
	}
} else {
	echo get_page($_SERVER['QUERY_STRING']); //$url . implode(" ", $_GET));
	// echo get_error_page(500);
}
?>

[JonathanLee]

You may also just want to use transparent mode for URL blocking.

[KOM]

Why are you configuring WPAD and then making Squid transparent?? Typically you either use transparent mode, or configure proxy autodetect. IME transparent mode gives me weird problems that I don't see when using squid in explicit mode.

[JonathanLee]

@KOM It is for use with both Transparent mode and SSL mode for devices like XBOX and Nintendo Switch Iphones etc. You must have them for Transparent on the Applications, plus you can have the certificates for the web browser. That way it works both ways. Spice and non-splice.

Xbox will not work unless transparent mode is enabled as an example plus you need WPAD and DHCP option 252 to be enabled for it to function correctly with the firewall. However, my other devices on the LAN use the SSL intercept. With or without the proxy used it functions this way. Some with SSL intercept get better cache rates and HTTPS Clam AV use.

You can get both working plus the Clam AV HTTPS use for your desktops that can have certificates installed

Best of both worlds, transparent use for URL blockers on the XBOX/Nintendo web browsers + the high security and caching for other devices that do not require transparent mode.

[WhiteTiger-IT]

@JonathanLee
@KOM

I didn't understand the suggestions.
I compared the sgerror.php file in the answer, but it is exactly the same as the one in pfsense.
Furthermore, the string in your redirect Info is unclear. What should I redirect it towards?

The error message is not the one you showed. Mine is a generic "URL not found" message.
Reading online it would seem that this is the norm when you want to filter HTTPS URLs, but today all sites are HTTPS.

Finally, Transparent Mode is only for HTTP.
For HTTPS I need to enable HTTPS/SSL Interception.
What should I do?

[KOM]

@JonathanLee I would just exempt my consoles from the proxy and let them go direct.

[JonathanLee]

@WhiteTiger-IT

I used that file to generate a custom useable redirect.

my example shows a line that says redirect info . . .

This line is where I have it set as the ext URL move (enter URL)

It is the same thing it is on PfSense however I made the custom file.

My URL for example is
https://192.168.1.1:8080/sgerror.php?url=403%20Blocked%20by%20Mom%20and%20Dad&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u

Reading that file this is basically access the sgerror.php script and use url=403 : forbidden
My message Blocked by Mom and Dad
a=client address
n=client name
i=client user
s=client group

It is just using that php file with the firewall port I have set. Mine is 8080

[JonathanLee]

@KOM I would however I need to block some URLs from being accessed still with Squidguard

[JonathanLee]

[WhiteTiger-IT]

I continue to have problems and every change I make new ones arise even where there weren't any.

If I enable "SSL filtering" in "HTTPS/SSL Interception" I no longer connect to any website, even if it is not in the blacklist.
If I disable "Transparent" everything goes through, including the websites in the blacklist.

Instead, with Transparent and SSL Filtering, if I go to a porn site, the URL is changed to
https://192.168.1.1/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD
So, the redirect works, but I see that the user name is missing (how can I pass it without forcing people to log in?) and the group is "default".
However, no message appears because there is a Timed Out error.

Instead, DNS always works.
At least that doesn't seem to be a cause.

In my opinion there is some misconfiguration in Squid+Squidguard, but which one?
Is it possible that there is no definitive guide? Nowadays almost all websites are HTTPS.

Even the certificates, after creation it is unthinkable to go around uploading them to all the PCs.
I haven't done it even in this testing phase because I won't be able to do it when fully operational.

==== Update
If I change the proxy settings in Firefox, inserting the days instead of taking them from the system, things change a little.

First the usual message appears

Some software is preventing Firefox from securely connecting to this site
192.168.1.1 is probably a trustworthy site, but a secure connection could not be established. This problem is caused by internal-ca, a software installed on your computer or network.

Then, accepting the risk and moving forward, this message appears:

ERROR

The requested URL could not be retrieved
The following error was encountered while trying to retrieve the URL: https://192.168.1.1/*
    Connection to 192.168.1.1 failed.
The system returned: (61) Connection refused
The remote host or network may be down. Please try the request again.
Your cache administrator is admin@localhost.

[JonathanLee]

@WhiteTiger-IT have you installed certificates on the devices? You would require them for SSL intercept/MITM to function.

Transparent is what I think you want. No certificates and URL blocking.

You said your redirect is now...

https://192.168.1.1/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD

However what is the port number your GUI uses?

https://192.168.1.1:port number here/sgerror.php?url=403%20My%20Message&a=192.168.1.100&n=touch-pc.DOMAIN.TLD&i=&s=default&t=blk_blacklist_adult&u=http://PORNSITE.TLD

Did you use redirect replace URL under every category in Squidguard? It has to be under default and all Squidguard groups. It looks like a double redirect. Have you set the proxy to bypass Squidguard redirect also?

With transparent you don't need to worry about proxy auto detect.

I suspect you have no certificates installed on devices correct?

Test that link by itself past it in a browser window.

What port do you use when you log in to PfSense?? That port is also needed to access the WWW folder and that's what contains the sgerror file.

Example 192.168.1.1:8080 is what I have configured as a custom port.

I assumed you have done that as you configured WPAD already.

[WhiteTiger-IT]

@JonathanLee
I didn't realize that the port used for login was needed, I can't find it described anywhere.

As I was saying, I don't find it correct that the official guide is not aligned with the current situation.
I'm reading other guides on the net, more than 50. Most are generic, the most complete are years old, some relating to old versions of pfSense.
There are a lot of unclear things.
Some say to activate also Transparent Mode, some don't.
Some create a rule to block HTTP, some also HTTPS.
Port 3129 also popped up in una guide, should this be configured too? Nobody makes it clear.

Now the error message appears and I thank you for this.
I now have to configure the rest as well.
For example, changes to sgerror.php so that attempts to access a site blocked by squidguard are reported in the log.
The documentation describes which lines to insert, but not where they should be inserted.

I also want to avoid having to install certificates on devices because I can't do it with guests and I don't have an AD domain in Windows to transfer them with a GPO rule.

For now I thank you.

[JonathanLee]

@WhiteTiger-IT you said "The documentation describes which lines to insert, but not where they should be inserted."

You could always use the url that I use and change it to what ever error message you want. Or just redirect to an official company webpage maybe with corporate policy on LAN use.

I agree there is not much information on this topic. I found Netgate forum is the best unofficial guide for any questions you need.

Glad you got it working.

[WhiteTiger-IT]

@JonathanLee said in Squid + Squidguard with WPAD. Filter doesn't work.:

@WhiteTiger-IT you said "The documentation describes which lines to insert, but not where they should be inserted."

You could always use the url that I use and change it to what ever error message you want. Or just redirect to an official company webpage maybe with corporate policy on LAN use.

We are talking about two different things.
You are referring to the error message and your suggestion was helpful to me.

I'm now talking about the modification to be made to sgerror.php so that anything blocked by squidguard is inserted into the log.
In the documentation i find the portion of code to insert, but I didn't understand where this code should be inserted in sgerror.php.

Then, I think I understood that if I want to filter HTTPS I am obliged to pass the certificate created in pfSense to everyone who browses.
This is impossible for me because not everyone wants to do it or would not be capable of doing it.

Also, if the domain has its own certificate provided by the ISP or Let's Encrypt, what should I do?
Still create a certificate in pfSense for Squid?

[JonathanLee]

@WhiteTiger-IT You said, "I'm now talking about the modification to be made to sgerror.php so that anything blocked by squidguard is inserted into the log. In the documentation i find the portion of code to insert, but I didn't understand where this code should be inserted in sgerror.php."

Squidguard does have the ability to log blocks in the rules area click the log radio button. This will log the blocks you have created to Squid Proxy.

/var/squidGuard/log/squidGuard.log

You also said, "Then, I think I understood that if I want to filter HTTPS I am obliged to pass the certificate created in pfSense to everyone who browses. This is impossible for me because not everyone wants to do it or would not be capable of doing it."

You are still able to splice https get requests and block in transparent mode without certificates as the HTTPS is not broken only the get requests are inspected.

You also said "Also, if the domain has its own certificate provided by the ISP or Let's Encrypt, what should I do?
Still create a certificate in pfSense for Squid?"

I still created a certificate in Squid and passed it to PfSense so they trust each other. Follow this guide.

Ref:
https://forum.it-monkey.net/index.php?topic=23.0

This covers a lot I think this is a great guide. It is also still relevant with 25.05

[WhiteTiger-IT]

@JonathanLee
I followed the guide step by step.
I even reset pfsense to factory settings.
I also removed the IPv6 settings from Windows 11 although I don't use them and there are no settings for IP6 on pfSense.
Nonetheless, I can't navigate (Timeout error) and what's worse, with these settings I lose access to the Internet even from my PC (with alias IT_Admin).
I can ping or tracert any domain, but it's not reachable.

This is done using the network settings in Windows and Firefox. Instead, if I set the proxy directly in Firefox, there are no active filters and everything passes.

Also, in monitor I still see references to the Squidguard blacklist from the previous installation. How do I clear the Squid and Squidguard log?

Finally, in this guide I don't understand the need to create the CA from SSH instead of GUI. However, I created it from SSH.

To conclude, I don't know what to do anymore, except give up HTTPS control and leave the transparent proxy.
Tomorrow I'll do a new installation by formatting the disk, but I confess that I'm very disappointed.

These the rules.

[JonathanLee]

You will never get SSL intercept to fully work as some applications require different things. If you are not using IPv6 have you disabled it inside the DNS forwarder? You have WPAD set up in your DNS?

Here is copy my my custom IPv6 disable, my ISP does not allow IPv6, and my WPAD set up in DNS as a host override

Here is a copy of my ACLs if you want to check it out

NAT RULES for DNS and NTP

Squid ACL
.* for my allow all domians I let Squidguard do my blocks

Squid Proxy Custom options.

Some clients I splice everything for so they are transparent anyway.

Take note of the Regular Expression file it is pointing to. . .
This is my always splice file no matter what like banks phone messages etc, that way they are never intercepted ever.

I do not use certificate checks inside the proxy

Squid's CERT is also a root authority see CA-Cert

Same here

Xforwarder mode is set to transparent for mine as I am not using anything else just the firewall and the Dmark

I have a large do not cache list

I also have a Squidguard always allow list

If you use a Blacklist proxy you also have to set it to allow as a default outside of the blocks so it does not block everything.

You also have to point clients to it that are not set as transparent

I just noticed your Rule 2 you have a any any rule this will bypass everything as clients do not need to use the proxy at all with that rule.

[JonathanLee]

If you are just starting out follow a guide for transparent mode and just use it as a URL blocker. I personally inspect a lot of traffic and make reports all the time on my freetime.