WPAD Setup help [Solved]



  • Last updated May 23 2016

    please now refer to https://forum.pfsense.org/index.php?topic=112335.0

    How to setup a WPAD for non transparent web filtering

    Set all nodes to the same workgroup (may be not needed)

    1. Create wpad files, either use Diagnostics: Edit file or use ssh to add the files, TIP: make a symbolic link between the files

    
    vi /usr/local/www/wpad.dat
    vi /usr/local/www/wpad.da
    vi /usr/local/www/proxy.pac
    
    

    The following wpad code allows all local ip addresses to go direct but makes all external connections go through the proxy (you must adjust the IP information to suit your setup)
    The code for the 3 files are

    
    function FindProxyForURL(url, host) 
    { 
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
        return "PROXY 192.168.1.1:3128";
    }
    
    

    If you connect to a VPN you need to go direct for the VPN instead of the proxy, Remember you need to add the correct network class for the VPN  either A, B or C

    
    function FindProxyForURL(url, host) 
    { 
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
            if (isInNet(dnsResolve(host), "1.0.0.0",  "255.0.0.0" ))
            { return "DIRECT"; }
    
        return "PROXY 192.168.1.1:3128";
    }
    
    

    2. Enable DNS Resolver (not using DNS forwarder)

    3.Configure DNS Resolver
    add new Host Overrides

    
    Host: wpad
    Domain: mylocaldomain.local
    IP Address: 192.168.1.1
    Description: WPAD Autoconfigure Host
    
    

    save

    4. Configure DHCP server

    Services: DHCP server
    Additional BOOTP/DHCP Options

    number: 252 type: string value: "http://192.168.1.1/wpad.dat"
    number: 252 type: string value: "http://192.168.1.1/wpad.da"
    number: 252 type: string value: "http://192.168.1.1/proxy.pac"
    

    5. run some test
    a. can the file be download at the following link?

    http://wpad.mylocaldomain.local/wpad.dat
    

    If not try this address

    http://pfsense.mylocaldomain.local/wpad.dat
    

    b. run a nslookup in cmd (windows)

    nslookup wpad.mylocaldomain.local
    

    or try

    nslookup pfsense.mylocaldomain.local
    

    result should be something like this

    
    Server:  pfSense.mylocaldomain.local
    Address:  192.168.1.1
    Name:    wpad.mylocaldomain.local
    Address:  192.168.1.1
    
    

    6. Configure DHCP server
    Services: DHCP server
    Additional BOOTP/DHCP Options

    
    number: 252 type: string value: "http://192.168.1.1/wpad.dat"
    number: 252 type: string value: "http://192.168.1.1/wpad.da"
    number: 252 type: string value: "http://192.168.1.1/proxy.pac"
    
    

    set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
    System: Advanced: Admin Access Protocol http

    7. Block lan Destination port 80 and 443 (http, https)
    Firewall: Rules lan

    IPv4 TCP * * * 80 - 443 * none
    

    or add two rules

    setup web browsers
    For Chrome and IE make sure that autodiscovery is turned on in internet options.
    Try the same for firefox, if firefox does not find the wpad add automatic proxy configuration url to the url of the wpad (like http://pfsense.mylocaldomain.local/wpad.dat)

    Hope this helps some people



  • @aGeekHere:

    I do not think I have set up the LAN rules correctly, as it is not working.

    Could you please be more explicit, explaining what doesn't work?
    e.g. can you access http://wpad.yourdomain/proxy.pac



  • By not working I mean all http traffic is blocked and I can only access https sites.

    If I disable the firewall rule then nothing is being filtered but then I can access
    http://192.168.1.1/proxy.pac
    and the file downloads.

    I must have missed something.

    In Services: DNS forwarder do I need to Enable DNS forwarder? The guide only talks about adding a new Host Override and not what the other settings should be.

    If I try to Enable DNS forwarder i get an  error
    The following input errors were detected:
    The DNS Resolver is enabled using this port. Choose a non-conflicting port, or disable DNS Resolver.

    I must be missing something.

    [Update]
    ok it seems that when I try to enable Enable DNS forwarder and set the Listen Port to 80 the DNS Forwarder service will not start



  • The way it works (supposed to work) is that your browser searches for http://wpad.your_domain/proxy.pac which will provide you with required inputs so that your browser configures itself with the right settings (meaning proxy address, port and behaviour (e.g. exceptions).
    That is easy as this  8)

    Then everything else depends on your own implementation:

    • if your web site serving http://wpad.your_domain can't be accessed on port 80, it will not work (this refers potentially to FW rules)
    • if this FQDN can't be resolved, it will not work (this refers to DNS settings but also to your workstation settings: it has to be configured with domain otherwise this domain can't be guessed)
    • if it contains wrong inputs, it will not work (refers to proxy.pac content)

    troubleshooting is however quite simple:

    • you could e.g. test your proxy.pac by configuring your browser, manually, with this URL
    • you could try to resolve wpad.your_domain (hint: try also "wpad", this will ensure you also have the right "search domain"
    • you could try to access http://wpad.your_domain/proxy.pac (which should show you its content)

    I hope this helps  ;)

    EDIT: there is one last (perhaps critical) point I forgot => did you try to access internet relying on your proxy, i.e.  configuring your browser (hardcoded) to use your proxy?



  • Ok, I think I am getting closer, The main issue I get now is that I get locked out of pfsense web gui and all Local addresses.

    So what I have done is I have committed out the 3 files and only uncommitted the proxy.pac (which seems to be for IE, I have done this so I do not get locked out on chrome and firefox). When I test it on IE the blocking works :) even https sites :) but only for my PC and not the others on the LAN??? they have the same setups???. Fixed that

    However then IE gets locked out of pfsense and for that matter ALL local domains.

    The error message i get is:

    Request denied by pfSense proxy: 403 Forbidden
     Reason:  
     Client address:  192.168.1.55 
     Client group:  default 
     Target group:  in-addr 
     URL:  http://192.168.1.1/192.168.1.22/--GET 
    

    Any Ideas?



  • Did some updates to the first post

    [Update]
    Ok it seems that when I setup a manual proxy I get the same lockout issue (getting closer).
    So the issue does not seem to be with the WPAD setup

    Does anyone else have an issue with squid's non transparent proxy (manual proxy setup) access local addresses (like the pfsense GUI)??



  • Not the ultimate way of solving the issue, what you could (should ?) do is to customize your proxy.pac file so that you do not rely on proxy when accessing local files, including pfSense GUI.

    This is done adding "DIRECT" directive. Something like (copy from this link which contains interesting stuff)

    
    function FindProxyForURL(url, host) {
    
    // If the requested website is hosted within the internal network, send direct.
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
    // DEFAULT RULE: All other traffic, use below proxies, in fail-over order.
        return "PROXY 192.168.1.1:3128";
    
    }
    
    

    Notice that proxy.pac can contains other directives. I suggest you have a look at this once it works  ;)



  • Thanks, I will have a look at that to see if it works.



  • updated first post, do not know what to try next.



  • @aGeekHere:

    updated first post, do not know what to try next.

    Ouch  :o  meaning we have to read it again and guess how it differs from initial post, then for those discovering this thread, they may find answers meaningless if initial post changed significantly.
    IMHO, this is more confusing than helpful except if you managed to heavily comment what was written before and how it changed.



  • it was more of a clean up



  • Reading your first post again:

    1 - you could easily maintain one single file and create symlink
    2 - I'm not sure you can push multiple "DHCP option 252" values  :(
    3 - your proxy.pac file states:

    • if there is no domain extension (no FQDN but host only)
    • or if FQDN belongs to "local" domain (is it really your domain name ?)
    • OR if IP I tray to reach belongs to my LAN
      then go DIRECT (without proxy)

    Is it what you target?

    4 - I don't understand

    Can connect to the internet (local ips are inaccessible, filtering works)

    ???

    5 - would you mind replacing "cannot connect to internet" with something clearer like the error message you receive?



  • @aGeekHere:

    it was more of a clean up

    Sure, I did notice but if your read the whole thread again as if you were discovering it, posts after your initial post are totally meaningless isn't it?  ::)



  • Let's try something basic:

    Using Firefox (let's focus on on browser for the time being)

    • using "automatic proxy detect for this network" option, it doesn't work (although error message would be more than welcome here)
    • using manual proxy configuration (192.168.1.1 port 3128), it works and as there is no option for local IPs, these local servers can't be reached (still I don't understand why => this would mean that proxy can access or can't resolve such URL)
    • using "Automatic proxy configuration URL" option: this is not clear to me whether it works or not.

    I also have the feeling that there is something unclear with your DNS: I'm very confused with error/warning message about DNS forwarder and port 80  ??? ???



  • Hi all, I have time again to try and get the wpad working.

    I think I am starting to get my head around this.

    This is where i am at
    pfsense is using the wpad files but with issues.

    with this wpad example the issue is local address are blocked

    function FindProxyForURL(url,host)
     {
     return "PROXY 192.168.1.1:3128";
     }
    

    to solve this we use this to allow local addressed

    function FindProxyForURL(url, host) {
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
        return "PROXY 192.168.1.1:3128";
    }
    

    With this

    • sites in the squardguard blacklist are blocked with the squidguard error message (this is good)
    • navigating to normal sites have major issue, some only half load, some do not load at all.

    Getting very close to solving this, i think it could be the wpad code.

    Anyone have any ideas?



  • Like with my previous reply, I think it deserves clearer message: what does "with this wpad example the issue is local address are blocked" mean?
    Is it "blocked" because not found, time-out, not authorized by proxy?

    What you are describing shows that your WPAD set-up is working as expected:

    • without any specific rule, everything goes thorough proxy, including internal targets but internal servers can't be reached via proxy (this I don't know yet why)
    • if you set-up proxy.pac to do direct for internal servers, you can reach it.

    Thus issue is definitely not with WPAD, IMHO  8)



  • Hi chris4916 I will try to be more clear with the current issue I have.

    The wpad (goal is to allow local sites to bypass the proxy but make all external sites to go through the proxy)

    function FindProxyForURL(url, host) {
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
        return "PROXY 192.168.1.1:3128";
    }
    

    Current status
    pfsense is now passing the wpad onto the browsers (chrome and ie for now through the DNS forwarder).

    http://www.lagado.com/proxy-test shows that the browsers are connected through a proxy (good)

    sites in the squidguard blocklist are being blocked for both http and https connections (good)

    Current issue
    Normal websites (sites which are not blocked form squidguard) are not loading correctly (Images are missing, layout is not fully loaded and some sites do not load at all).

    "with this wpad example the issue is local address are blocked" mean?
    I was just outlining that if you use that example wpad from https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid that local addresses are blocked and by blocked i mean I get a proxy error (can't remember what the error was).

    What you are describing shows that your WPAD set-up is working as expected
    That is correct the WPAD is now working (however now have another issue with webpages not loading correctly)

    if you set-up proxy.pac to do direct for internal servers, you can reach it.
    I can reach them now by using the following wpad

    function FindProxyForURL(url, host) {
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
        return "PROXY 192.168.1.1:3128";
    }
    

    Let me know if I need to expand on anything



  • That's clearer (I hope  :))

    So your current issue has nothing to do with WPAD but with proxy (Squid) that is not loading pages as expected (or Squidguard blocking some parts of these pages)
    Did you look at squid log content already?

    Keep in mind that quite often web pages as you see it through your browser are built from different sources.
    I would suggest that you perform some tests without any Squidguard rules or even Squid ACL, just to ensure that from HTTP standpoint, everything is OK btween your browser and external web servers, then introduce Squid ACL if needed and Squidguard rules so that you can, perhaps, determine, what's wrong here.



  • I have updated the first post, I now use the strikethrough to show old settings. Very strange issue this is now, squidguard is not showing any blocked items for non blocked sites and even if I turn it off or allow all to squidguard I get the same issue (sites not loading correctly, missing content, not loading at all AND very very slow webpage loading times).

    Stuck atm (I wounder if it is squid3 that is the issue);

    Any ideas on what I should try next?



  • @aGeekHere:

    I have updated the first post, I now use the strikethrough to show old settings.

    From personal viewpoint, this is not something I use to do because it makes next posts most of the time out of context.

    Very strange issue this is now, squidguard is not showing any blocked items for non blocked sites and even if I turn it off or allow all to squidguard I get the same issue (sites not loading correctly, missing content, not loading at all AND very very slow webpage loading times).

    Stuck atm (I wounder if it is squid3 that is the issue);

    I'm not using Squid on pfSense but reading various threads here and there, it looks like there is a significant amount of problems with Squid on pfSense 2.x
    At least, you have clarified that Squidguard is not the issue here, neither WPAD  ;)
    Now you could perhaps focus on Squid logs…



  • reinstalled squid gui components, cleared the cache (one the user computers and squids cache), and now it works :)

    Will test it for a few days to see how i go.

    ;D

    [Update]
    If i have no issues with it i will write up a how to



  • I just want to update something here.
    if you are having issues with webpages only half loading (which is what I had), then you must first delete your squid cache then it should fix that issue.



  • if you are having issues with webpages only half loading

    This can also by symptomatic of an IP6/IP4 preference issue with dual-stack sites.



  • Hi all,

    I found an issue where some programs have an issue with the global "automatically detect settings" option (some programs with this issue do not even have a proxy setting setup page) and still want to go through port 80 (I think this could also be what is causing downthemall to only connect to 1 segment because when I open port 80 I can connect to more segments).

    So sometimes I have to reopen port 80 to let the traffic through temporary.

    I read somewhere that a NAT rule could be created to catch any leftover traffic going to port 80 and redirect it to port 3128.

    Tired playing around with a NAT rule
    LAN - TCP - LAN address - 80 - WAN address - 80 - 192.168.1.1 - 3128
    And in rules i put it at the top.

    However the traffic is not redirected.
    Tested with a troubled program using

    • no proxy
    • automatically detect settings
    • And even setting the proxy setting with the port set to 80

    Any thoughts on this?



  • @aGeekHere:

    2. Disable DNS Resolver Updated needs to be on

    3.Configure DNS Service
    add new Host Overrides

    
    Host: wpad
    Domain: mylocaldomain.local
    IP Address: 192.168.1.1
    Description: WPAD Autoconfigure Host
    
    

    save
    Updated change Listen Port to 3128
    Enable DNS forwarder
    save

    Quick question, for clarification on the above quote, please:
    I currently have DNS Resolver off and DNS forwarder on with the above Host Override configured on the DNS Forwarder.

    Which one did you apply the Host Override configuration?

    Which one did you change to listen on port 3128 (same as Squid3, unless you changed that.  I don't think you did, based on your proxy.pac code).

    I add the Firewall rules for blocking HTTP and HTTPS and I can't load internet pages.  I can still get to local servers.  Nothing's being filtered by Squidguard.  Or, at least, it's not showing up in the logs.  Wondering if that's my problem now.



  • It is the DNS forwarder that needs the Host Override configuration.



  • @User43617:

    @aGeekHere:

    2. Disable DNS Resolver Updated needs to be on

    3.Configure DNS Service
    add new Host Overrides

    
    Host: wpad
    Domain: mylocaldomain.local
    IP Address: 192.168.1.1
    Description: WPAD Autoconfigure Host
    
    

    save
    Updated change Listen Port to 3128
    Enable DNS forwarder
    save

    Quick question, for clarification on the above quote, please:
    I currently have DNS Resolver off and DNS forwarder on with the above Host Override configured on the DNS Forwarder.

    Which one did you apply the Host Override configuration?

    Which one did you change to listen on port 3128 (same as Squid3, unless you changed that.  I don't think you did, based on your proxy.pac code).

    I add the Firewall rules for blocking HTTP and HTTPS and I can't load internet pages.  I can still get to local servers.  Nothing's being filtered by Squidguard.  Or, at least, it's not showing up in the logs.  Wondering if that's my problem now.

    I'm also wondering do I put listen to 3128 in DNS Forwarder or in DNS Resolver?



  • I'm also wondering do I put listen to 3128 in DNS Forwarder or in DNS Resolver?

    see above post :)

    And while I am here, there is an issue that remains with this setup is that programs that have no proxy settings and want to connect directly will get blocked by the port 80 rule.

    If someone has any advice on how to solve this i'll update the first post.



  • If someone has any advice on how to solve this i'll update the first post.

    Add a firewall rule(s) to allow specified LAN IPs to connect to specified destination IPs via 80.  Put it above the HTTP/S block.  Use an alias to hold the LAN IPs of your clients that have apps that don't support proxy, and an alias to hold all the IPs of the servers they need to talk to.  That's it.



  • and an alias to hold all the IPs of the servers they need to talk to

    The problem is first finding the server that they need to talk to, then updating that rule when they change it. If the destination server is dynamic then you will never know the destination server.

    But I guess that is all we can do.



  • But I guess that is all we can do.

    Them's the breaks.



  • @aGeekHere:

    2. Disable DNS Resolver Updated needs to be on

    3.Configure DNS Service
    add new Host Overrides

    
    Host: wpad
    Domain: mylocaldomain.local
    IP Address: 192.168.1.1
    Description: WPAD Autoconfigure Host
    
    

    save
    Updated change Listen Port to 3128
    Enable DNS forwarder
    save

    This is working on my machine but I will like to know if you can help me understand why do you need to enable DNS forwarder and set listening port to 3128. I have pfsense 2.2.4 DNS Resolver is enable by default and DNS forwarder is disable. I have created the host override on DNS resolver for it to work and enable DNS Forwarder service on port 3128.

    Also why do you need the host override if dhcp is configured with the ip address and not a domain name? I will appreciate if you help me understand this.



  • Hi enrique.perezrul

    Hmm some hard questions there, will do my best, lets start with reading from the wiki

    https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid

    WPAD will take the domain name given to the machine, likely assigned by DHCP, and prepend wpad.. If the domain is example.com, it will look for wpad.example.com. This task may be accomplished with the DNS Forwarder/DNS Resolver in pfSense or with another internal DNS server used by client PCs.

    why do you need to enable DNS forwarder

    Because you need to create a Host Override for the wpad.

    …set listening port to 3128

    Because I wanted to use port 53 for the DNS Resolver (Will need to look more into why I used port 3128)

    Also why do you need the host override if dhcp is configured with the ip address and not a domain name

    Some web browsers use DNS to configure them others use dhcp so both are needed.



  • Looking for a well documented guide or video to help configure wpad. Do any exist?



  • @maverik1:

    Looking for a well documented guide or video to help configure wpad. Do any exist?

    With this Post i configurate WPAD, but if you have problems, create your own post. Problems never are the same.



  • Looking for a well documented guide or video to help configure wpad. Do any exist?

    Look at the post above yours, specifically the link to WPAD Autoconfigure for Squid.



  • I've checked that out but it's quite vague. For example, I run vlans on pfsense. My guest network is 10.0.0.0/24 and my home lan is 192.168.2.0/24.

    In the wpad file what do I need to specify as the proxy address?  I have configured pfsense such that guest vlan cannot access home vlan and vice versa. As I do not have a separate box I will have to host the wpad on the pfsense box.

    function FindProxyForURL(url,host)
    {

    return "PROXY 192.168.2.1:3128";  What does this need to be to support both vlans?
    }



  • try this

    function FindProxyForURL(url, host) 
    { 
        if (isPlainHostName(host) ||
            shExpMatch(host, "*.local") ||
            isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
            return "DIRECT";
    
        return "PROXY 192.168.1.1:3128";
    }
    


  • In the wpad file what do I need to specify as the proxy address?

    Whatever the IP address is of the interface that Squid is listening on.  Usually your LAN NIC.

    I have configured pfsense such that guest vlan cannot access home vlan and vice versa.

    Add a firewall rule above your vlan blocks that allows the guest vlan to talk to squid.

    As I do not have a separate box I will have to host the wpad on the pfsense box.

    So do that then.  I host wpad.dat and proxy.pac on my pfSense box.  Note that it won't work if you have WebGUI running in HTTPS mode.



  • Is there a command or configuration page to see what interface squid is listening on? As mentioned previously I have vlans running. The default LAN is disabled.

    Vlan10 is admin
    Vlan20 is guest
    Vlan30 is home

    I configured squid to bind to vlan20 and vlan30.

    You mention that if WebGUI is running over https I cannot host the proxy.pac. Can this be overcome by changing the port from 443 to 444?


Log in to reply