Squid/SquidGuard NONE/409 and DNS issue

aGeekhere · May 2, 2021, 2:05 AM

@shawn8888
To set auto config in windows go to
Control Panel - Internet Properties - connections - LAN settings and select Automatically detect settings.

Make sure programs are set to Use system proxy settings.
Test with chrome for now as firefox in the past had an outstanding bug with auto configuring a proxy which i am not sure if it was fixed.

"If the transparent mode works, why do you need wpad? Is that because transparent mode is bugy?"

From memory transparent proxy can break certificates resulting in a failed connection, though someone with more knowledge would have to confirm the technical details.

Also in your WPAD add to the top

if (shExpMatch(host, "ENTER YOUR PFSENSE DOMAIN HERE")) return "DIRECT";

use https://app.thorsen.pm/proxyforurl for testing the WPAD for errors

shawn8888 · May 2, 2021, 2:41 AM

@ageekhere

Thank you guys for the endless help. I haven't given up because of you!

My Internet properties look good.
Chrome also points to Auto detect.
My wpad.dat now looks like below and the proxy tester didn't complain
I also tested on my iPhone, iPad, same as my PC. As soon as I disable transparent mode, Internet drops.
I checked my iOS -> wifi -> HTTP PROXY -> Configure Proxy -> Automatic
All my devices can download the wpad.dat file from the browser:
http://192.168.100.1/wpad.dat

function FindProxyForURL(url,host)
{
if (shExpMatch(host, "pfsense.mydomain.com")) 
  return "DIRECT";

if (isInNet(myIpAddress(), "192.168.100.155", "255.255.255.0")) 
  return "DIRECT";

  return "PROXY 192.168.100.1:3128";
}

login-to-view

aGeekhere · May 2, 2021, 4:01 AM

@shawn8888
I think i see the issue

Under Additional BOOTP/DHCP Options change to

252 String "http://192.168.1.1/wpad.dat"
252 String "http://192.168.1.1/wpad.da"
252 String "http://192.168.1.1/proxy.pac"

Under DNS ResolverGeneral Settings Host Overrides add

wpad	YourPfsenseDomain	192.168.1.1	wpad

https://docs.netgate.com/pfsense/en/latest/recipes/http-client-proxy-wpad.html?highlight=wpad

"A WPAD host may be supplied via DHCP numbered option 252 (string value containing the entire URL to the WPAD file) or DNS, which is easy to do with the built-in DNS forwarder."

Make sure you get the correct YourPfsenseDomaid, something like pfsensedomain.local
You can check in windows by using cmd and look for Connection - specific DNS suffix

shawn8888 · May 2, 2021, 4:14 AM

@ageekhere
no luck.
From what I read, you only need DHCP or dns resolver. I tried both, and still the same.
Even though I am complete newbie about wireshark, I gave it a shot. And I cannot even find string "wpad" in the logs.

login-to-view

aGeekhere · May 2, 2021, 4:46 AM

@shawn8888 Try Resetting the States, Diagnostics States Reset States.
Also try restarting your pc and router.

KOM · May 2, 2021, 4:30 PM

@shawn8888 Transparent mode isn't buggy per se, it's just that you have to consider other things when using it. You're trying to intercept a data stream that is trying to prevent you from doing so. You either need to install a trusted cert on every device that will use your transparent proxy to void MitM browser errors when using Splice/Bump, or no certs and you use Splice All. If transparent mode works for you then use that but I find explicit mode has less hassles.

Are you sure that your proxy works? What happens if you manually set a browser to use it?

shawn8888 · May 3, 2021, 2:11 AM

@ageekhere
I reboot pfSense and client pc. wpad is still not working.

@KOM
"Splice All" is recommended in order to use squidguard, which is the main reason I use squid for.

My proxy ONLY works at Transparent mode.
I have tried turning off Transparent mode and setting proxy manually (set IP and port 3128) on my PC. But it seems that a lot of things are not working. For example, I can open speedteset.net, but when I hit "go" button, it never starts.

KOM · May 3, 2021, 2:11 AM

@shawn8888 When running in explicit mode, how does it not work exactly? What browser error are you getting? Squid normally just works right out of the box.

shawn8888 · May 3, 2021, 2:31 AM

@kom
What do you mean by "explicit mode"?
Does that mean:
Disable both "Transparent mode" and "Enable SSL filtering.", and manually set clients to use ip:3128 as proxy?

KOM · May 3, 2021, 3:21 AM

@shawn8888 Yes. Explicit means not transparent, no SSL interception. Clients can be manually set or use WPAD once you determine squid is working properly.

shawn8888 · May 4, 2021, 5:40 AM

@kom

My proxy now works on explicit mode.
I turned off "Transparent HTTP Proxy" and "HTTPS/SSL Interception".
I also turned off the lan rules, which you suggest for wpad bypass some devices earlier.

login-to-view

This allows all LAN devices to go Internet directly by default, right? All my LAN devices can access Internet at this moment.

I tested on my iPad to set wifi manually use proxy at IP:3128. It worked. the only thing not working so far is speedtest.net go button. It stays at connecting forever.

login-to-view

I tested my PC.
firefox, set proxy in option:

login-to-view

I find that only http websites get blocked by squidguard. And https blocked websites can passthrough. Is it because https websites are bypassed due to "HTTPS/SSL Interception" being off?

I also tried Chrome and set the proxy like this:
login-to-view

It's funny that Chrome can not only see squidguard blocks http websites, but also https ones!
speedtest.net go button still not working.
Another thing doesn't work is the local websites. It gives me Access Denied error:

login-to-view

So I guess the explicit mode works, but I still need wpad or transparent working so I don't have to deal with the clients...

KOM · May 4, 2021, 1:24 PM

@shawn8888 Lots to unpack here.

This allows all LAN devices to go Internet directly by default, right?

You didn't show your complete LAN rule set so I don't know but I can assume the last rule is the default Allow All rule. Btw you can do the block with just one rule by making a port alias called WebPorts that contains 80,443.

I tested on my iPad to set wifi manually use proxy at IP:3128. It worked. the only thing not working so far is speedtest.net go button.

No idea about that but it seems to be a common thing with speedtests and proxies.

I tested my PC. firefox, set proxy in option

I notice you didn't set the https proxy or check the checkbox to use the same proxy for ftp and https. You need to do one or the other.

I find that only http websites get blocked by squidguard.

Get squid working first before you worry about that. Disable squidguard until you have squid working.

Is it because https websites are bypassed due to "HTTPS/SSL Interception" being off?

That might be because you didn't set the https proxy.

Another thing doesn't work is the local websites.

Either use the site's LAN IP address, or use split DNS to resolve your domain to its LAN IP instead of its public IP.

shawn8888 · May 4, 2021, 3:49 PM

@kom

You are right! After I "check the checkbox to use the same proxy for ftp and https" Firefox works the same as Chrome for https websites.

If squid doesn't work for certain websites like speedtest.net, how to set them as exceptions?

The local website is an oddball:
The ping shows the dns is solved as a local IP, 192.168.100.155
But no matter dns or ip, it all shows as Access Denied by squid. And it only not working for this tt-rss site, other local websites with the same domain but different ports are OK:
Working ones:
http://syn1.mydomain.com:5000/
http://syn1.mydomain.com:8080/
http://syn1.mydomain.com:5076/

NOT Working:
http://syn1.mydomain.com:181/tt-rss/
http://192.168.100.155:181/tt-rss/

Logs:
login-to-view

KOM · May 4, 2021, 4:23 PM

@shawn8888 Squid only works with ports that are part of its safe_ports list. You can't just go to any port. In pfSense, you can find this under Services - Squid proxy - ACLs - Squid Allowed Ports.

shawn8888 · May 4, 2021, 4:33 PM

@kom
I added port 181 below and tt-rss finally works! Thanks again!
Squid Allowed Ports -> ACL SafePorts

KOM · May 4, 2021, 4:47 PM

@shawn8888 Now you can start trying to get wpad working, then squidguard.

shawn8888 · May 4, 2021, 5:09 PM

@kom
I will give it a couple of days to see if most of the problems can be solved in explicit mode before testing wpad again.
Thanks!

KOM · May 4, 2021, 5:12 PM

@shawn8888 WPAD solves the problem of having to manually configure proxy setting on every device so it's kind of important to get it working. You can also push a proxy via DHCP option 252. I would do both, and don't forget to add your block rule on LAN to prevent people from going around the proxy.

shawn8888 · May 10, 2021, 11:32 PM

After a week testing, I can confirm that NONE/409 issue is gone with the explicit mode. It seems that the issue only happens with the transparent mode.

WPAD is still a hit and miss. I will test it more later. Right now I set all the devices need to go through squid manually.

aGeekhere · May 11, 2021, 2:04 AM

@shawn8888 @shawn8888 You can always try the Wpad Unofficial package https://github.com/marcelloc/Unofficial-pfSense-packages/tree/master/pkg-wpad