Squid/SquidGuard NONE/409 and DNS issue

shawn8888

@shawn8888
I don't see any solution to this bug, which makes Squid totally useless. A lot of websites broke randomly because of this.
I uninstalled Squid...

shawn8888

I just upgraded pfSense to 2.5 and all the packages, and this issue still persist.
Sigh...

shawn8888

I found this:
host_verify_strict
http://www.squid-cache.org/Doc/config/host_verify_strict/

How can I set this option on/off in pfSense?

dbx

@shawn8888

Did you find a solution to this?

I have my clients using the pfsense firewall as their DNS server, but I still see 409 responses from squid

As you described, simply waiting a while and trying again does solve the problem but it seems that this shouldn’t be necessary

shawn8888

@dbx
There is no solution for this from what I've read.
I gave up on squid. And sadly, I haven't found an alternative.

dbx

@shawn8888

For me the idea of a transparent proxy seemed ideal, especially being able to filter content that my kids devices are able to access

Niggles like this make the experience feel like I’m a full time sysadmin at home, when really after some initial config it ought to just work

Simple things like clicking links from a google search on a mobile device not working turns a quiet 5 mins browsing into a headache

It certainly isn’t transparent anyway. Short of ditching the firewall completely and going direct to the World Wide Web are there any config changes we may not have thought of? I guess you the one you linked to above can’t be applied manually in squid.conf of similar?

shawn8888

@dbx
I did try the option "host_verify_strict" on and off. I put it in Advanced Feature -> Customer Options
It does NOT fixe the 409 problem.

I guess because this 409 doesn't happen 100% for all websites, so the developers don't care, even though they know it's a bug and they put it in the docs and call it a DNS/security issue and don't want to fix it.

rafaelrenan

Is there anyway to use SSL filtering in pfsense without having this issue?
I've tried to point the DNS to the same server as @shawn8888 did, but with no success too.
Could it be solved with non transparent proxy?

aGeekhere

I do not have this problem.
To fix you should not rely on just a transparent proxy but use a combination of both transparent proxy and non transparent. A WPAD (try the Unofficial WPAD package) can also be used to for auto configuration. But to test configure a PC/or chrome to use the proxy and see if you still get those errors.

KOM

Some browsers can reach out and use DNS that may be different from the client OS. To prevent this, you should be blocking DNS traffic to anything except pfSense:

Redirecting Client DNS Requests

You will also need to be aware of DoH and DoT and how to block them but one step at a time.

Caching the dynamic web with squid isn't very effective anymore. My hit rates were usually in the 4-7% range which is a waste of time. I ended up just disabling the cache and only using squid as base for squidguard.

I found transparent proxy to be a pain in the ass. Now I use WPAD to allow clients to autodetect squid themselves, and any other devices will have to be configured manually or else they don't get access. Modern wireless devices will allow you to configure a proxy per AP so they don't have to apply it globally.

shawn8888

@kom Thanks for the suggestion
I tried the "Redirecting Client DNS Requests" trick, but still many NONE/409 errors.

I don't know what WPAD is, guess I will give it a try when I have time.

KOM

@shawn8888

Setting up WPAD Autoconfigure for the Squid Package

It allows an OS to autodetect a proxy. When you use this method, you run squid in explicit mode (non-transparent). This way you don't have to screw around with certs or MitM splicing because everyone involved knows a proxy server is being used. Most OSes and devices support it. Older ones may have to be manually configured to use the proxy but it varies.

aGeekhere

@kom
However some software may not have support for a proxy so if you block traffic on port 80 and 443 then that program may have connection issues. A simple fix is to just run both, non-transparent for the majority of traffic and a transparent proxy to catch any traffic that is not supported by the WPAD. You may find that the number of programs not supporting a proxy is slowly declining.

The Wpad Unofficial package works very well https://github.com/marcelloc/Unofficial-pfSense-packages/tree/master/pkg-wpad

Maybe one day it will be pushed to an official package.

shawn8888

@kom @aGeekhere

I tried WPAD, and it worked!
I haven't seen any NONE/409 errors since then and it looks promising!

If this error is for non-WPAD, and transparent mode only, it's a bug, don't you think?
Do you know how to manually set some of my devices in LAN bypass the proxy? I have a Synology NAS and some other devices to access Internet directly.

Thanks!

aGeekhere

@shawn8888
1 Not sure if it is a bug or just a limitation

2 In you wpad you can bypass devices like this

if (isInNet(myIpAddress(), "192.168.1.99", "255.255.255.0"))
return "DIRECT";

KOM

@shawn8888 I put a block rule for tcp 80/443 on LAN above my Allow All rule, then above that I have an allow rule with an alias that holds IPs that I allow to tcp 80/443.

shawn8888

@kom

You mentioned that Transparent HTTP Proxy should be disabled. But in my case, I have to enable it to make proxy working.
In order to bypass some of my LAN IPs, I did it as you suggested. But it seems not working? Is there anything wrong in the screenshot below?

shawn8888

@ageekhere

I changed my wpad.dat to this:

function FindProxyForURL(url, host) {
    if (isInNet(myIpAddress(), "192.168.100.159", "255.255.255.0")) 
    return "DIRECT";

    return "PROXY 192.168.100.1:3128";
}

But somehow it doesn't bypass the the device I put in there. :(

aGeekhere

@shawn8888
in squid under
Bypass Proxy for These Source IPs
add your device there

shawn8888

@ageekhere said in Squid/SquidGuard NONE/409 and DNS issue:

@shawn8888
in squid under
Bypass Proxy for These Source IPs
add your device there

I tried that, not working either. Which makes wonder if I am doing something wrong.

How should I test if a device goes to Internet directly or though a proxy?
Right now, because I set a block web site in SquidGuard, such as youtube.com. So, if I can access google.com but not youtube.com, I assume the proxy is working, because SquidGuard needs Squid to work. If I can access both, then it accesses directly. Is there a better way?