Squid/SquidGuard NONE/409 and DNS issue

KOM

Setting up WPAD Autoconfigure for the Squid Package

It allows an OS to autodetect a proxy. When you use this method, you run squid in explicit mode (non-transparent). This way you don't have to screw around with certs or MitM splicing because everyone involved knows a proxy server is being used. Most OSes and devices support it. Older ones may have to be manually configured to use the proxy but it varies.

aGeekhere

@kom
However some software may not have support for a proxy so if you block traffic on port 80 and 443 then that program may have connection issues. A simple fix is to just run both, non-transparent for the majority of traffic and a transparent proxy to catch any traffic that is not supported by the WPAD. You may find that the number of programs not supporting a proxy is slowly declining.

The Wpad Unofficial package works very well https://github.com/marcelloc/Unofficial-pfSense-packages/tree/master/pkg-wpad

Maybe one day it will be pushed to an official package.

shawn8888

@kom @aGeekhere

I tried WPAD, and it worked!
I haven't seen any NONE/409 errors since then and it looks promising!

If this error is for non-WPAD, and transparent mode only, it's a bug, don't you think?
Do you know how to manually set some of my devices in LAN bypass the proxy? I have a Synology NAS and some other devices to access Internet directly.

Thanks!

aGeekhere

@shawn8888
1 Not sure if it is a bug or just a limitation

2 In you wpad you can bypass devices like this

if (isInNet(myIpAddress(), "192.168.1.99", "255.255.255.0"))
return "DIRECT";

KOM

@shawn8888 I put a block rule for tcp 80/443 on LAN above my Allow All rule, then above that I have an allow rule with an alias that holds IPs that I allow to tcp 80/443.

shawn8888

@kom

You mentioned that Transparent HTTP Proxy should be disabled. But in my case, I have to enable it to make proxy working.
In order to bypass some of my LAN IPs, I did it as you suggested. But it seems not working? Is there anything wrong in the screenshot below?

shawn8888

@ageekhere

I changed my wpad.dat to this:

function FindProxyForURL(url, host) {
    if (isInNet(myIpAddress(), "192.168.100.159", "255.255.255.0")) 
    return "DIRECT";

    return "PROXY 192.168.100.1:3128";
}

But somehow it doesn't bypass the the device I put in there. :(

aGeekhere

@shawn8888
in squid under
Bypass Proxy for These Source IPs
add your device there

shawn8888

@ageekhere said in Squid/SquidGuard NONE/409 and DNS issue:

@shawn8888
in squid under
Bypass Proxy for These Source IPs
add your device there

I tried that, not working either. Which makes wonder if I am doing something wrong.

How should I test if a device goes to Internet directly or though a proxy?
Right now, because I set a block web site in SquidGuard, such as youtube.com. So, if I can access google.com but not youtube.com, I assume the proxy is working, because SquidGuard needs Squid to work. If I can access both, then it accesses directly. Is there a better way?

aGeekhere

@shawn8888
You can look at squid real time to see if that device comes up

shawn8888

@ageekhere

Thanks!

So I did all the below, and the bypass finally works:

setup the rules in LAN like this:

change the wpad.dat like this:

function FindProxyForURL(url, host) {
    if (isInNet(myIpAddress(), "192.168.100.159", "255.255.255.0")) ||
      (isInNet(myIpAddress(), "192.168.100.155", "255.255.255.0")) 
    return "DIRECT";
    return "PROXY 192.168.100.1:3128";
}

add Bypass Proxy for These Source IPs

KOM

@shawn8888 Something is wrong with your config. You don't need WPAD if you're still running transparent mode. It's one or the other.

aGeekhere

@kom
I think if you go direct with the WPAD the transparent proxy will see traffic going directly through port 80/443 and will redirect the traffic through the proxy again. So by adding the bypass in squid it prevents it from being routed through the transparent proxy.

If you turn off the transparent proxy and just rely on the WPAD some software may have connection issues.

shawn8888

@kom
If I disable transparent mode, then all my devices lost Internet. So honestly, at this point, I don't know if WPAD or transparent mode is working.
However, twitter image/video now works both on my PC and my iPad, and it didn't work before.

aGeekhere

@shawn8888
A way to test if the WPAD is working is temporary turn off the transparent proxy and make sure autoconfig in turn on. If you internet browser uses the proxy then it is working.

shawn8888

@ageekhere

What do you mean "autoconfig"?
I followed the help here:

If I turn off transparent mode and change firefox network settings:

Still no Internet. So I guess WPAD never worked...

KOM

@shawn8888 What happens when you try to resolve 'wpad' or 'wpad.yourdomain.lol' on your network? Does it come back with the IP address of the server holding the wpad.dat file? IIRC your wpad.dat file to live on an http server or a trusted https server. Any cert errors will stop the wpad file from being read.

shawn8888

@kom
I think I have followed all the steps on doc page.
Ping/dns is fine, and I can download the wpad.dat file from the browser.

One question though:
If the transparent mode works, why do you need wpad? Is that because transparent mode is bugy?

aGeekhere

@shawn8888
To set auto config in windows go to
Control Panel - Internet Properties - connections - LAN settings and select Automatically detect settings.

Make sure programs are set to Use system proxy settings.
Test with chrome for now as firefox in the past had an outstanding bug with auto configuring a proxy which i am not sure if it was fixed.

"If the transparent mode works, why do you need wpad? Is that because transparent mode is bugy?"

From memory transparent proxy can break certificates resulting in a failed connection, though someone with more knowledge would have to confirm the technical details.

Also in your WPAD add to the top

if (shExpMatch(host, "ENTER YOUR PFSENSE DOMAIN HERE")) return "DIRECT";

use https://app.thorsen.pm/proxyforurl for testing the WPAD for errors

shawn8888

@ageekhere

Thank you guys for the endless help. I haven't given up because of you!

My Internet properties look good.
Chrome also points to Auto detect.
My wpad.dat now looks like below and the proxy tester didn't complain
I also tested on my iPhone, iPad, same as my PC. As soon as I disable transparent mode, Internet drops.
I checked my iOS -> wifi -> HTTP PROXY -> Configure Proxy -> Automatic
All my devices can download the wpad.dat file from the browser:
http://192.168.100.1/wpad.dat

function FindProxyForURL(url,host)
{
if (shExpMatch(host, "pfsense.mydomain.com")) 
  return "DIRECT";

if (isInNet(myIpAddress(), "192.168.100.155", "255.255.255.0")) 
  return "DIRECT";

  return "PROXY 192.168.100.1:3128";
}