• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Guide to filtering web content (http and https) with pfsense 2.3

Scheduled Pinned Locked Moved Documentation
190 Posts 54 Posters 228.1k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    J7529
    last edited by Feb 19, 2018, 10:42 AM

    Hello,

    Just following the Guide, I am getting a issue where when searching on https i get "This site can't be reached" and when browsing on http I get "The requested URL could not be retrieved" I am only trying to get to Google which is let through!

    How I am currently setting up is behind a Router/Firewall and we already have a DNS server so I haven't enabled DNS Resolver, would this cause issues?

    Thanks!

    1 Reply Last reply Reply Quote 0
    • M
      MartinN
      last edited by Mar 10, 2018, 6:45 AM

      @aGeekHere:

      Go to Services/DNS/Resolver/General Settings
      in custom option enter

      server:
      include: /var/unbound/forecegoogle.conf
      

      You have mis-spelled the filename here, which prevents it from being saved by pfSense. Please fix it  8)

      Thanks for the guide, excellent stuff!

      1 Reply Last reply Reply Quote 0
      • A
        aGeekhere
        last edited by Mar 10, 2018, 10:36 AM

        Fixed spelling

        Never Fear, A Geek is Here!

        1 Reply Last reply Reply Quote 0
        • N
          nib01
          last edited by Mar 24, 2018, 11:07 PM

          @nib01:

          I got confused when you say add (blue). Did you mean add (blue) below yellow within the same wpad.da file? or..

          "Part 3
          Now we are going to set up a wpad read more here about wpad https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid
          ssh in to pfsense
          8
          cd /
          create the wpad.da file
          vi /usr/local/www/wpad.da
          wq

          Create two new symbolic link files
          Code: [Select]
          ln -s /usr/local/www/wpad.da /usr/local/www/wpad.dat
          ln -s /usr/local/www/wpad.da /usr/local/www/proxy.pac

          Then go Diagnostics /Edit File
          click browse
          user
          local
          www
          click wpad.da
          add

          Code: [Select]
          function FindProxyForURL(url, host)
          {
              if (isPlainHostName(host) ||
                  shExpMatch(host, "*.local") ||
                  isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
                  return "DIRECT";

          return "PROXY 192.168.1.1:3128";
          }

          save"

          1 Reply Last reply Reply Quote 0
          • A
            aGeekhere
            last edited by Mar 27, 2018, 12:31 AM

            You have to create 3 files in /usr/local/www/
            wpad.da
            wpad.dat
            proxy.pac

            Each having this code

            function FindProxyForURL(url, host) 
            { 
                if (isPlainHostName(host) ||
                    shExpMatch(host, "*.local") ||
                    isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
                    return "DIRECT";
            
                return "PROXY 192.168.1.1:3128";
            } 
            

            Instead of maintaining changes for all 3 files you can create a main file e.g. wpad.da and create a symbolic link for the other 2 files, that way all changes from wpad.da are copied over to the other files.

            So now you only need to make changes to wpad.da.

            Never Fear, A Geek is Here!

            1 Reply Last reply Reply Quote 0
            • K
              kpoman
              last edited by Apr 5, 2018, 11:41 PM

              Is there any way to force safesearch (google youtube etc…) if your DNS is on a Windows Server AD but your gateway pfSense is intercepting all http on a squid instance ?

              1 Reply Last reply Reply Quote 0
              • A
                aGeekhere
                last edited by Apr 6, 2018, 6:38 AM

                @kpoman:

                Is there any way to force safesearch (google youtube etc…) if your DNS is on a Windows Server AD but your gateway pfSense is intercepting all http on a squid instance ?

                Read the DNS resolver part in the guide and see if you can use that method.

                Never Fear, A Geek is Here!

                1 Reply Last reply Reply Quote 0
                • J
                  jopeme
                  last edited by Apr 6, 2018, 11:10 AM

                  I see a problem with this option and it is that many of my clients have problems to watch YouTube videos, when they search from Google, since it shows that they are restricted videos and that you have to register.

                  Click add under Host overrides
                  Host = www
                  Domain = youtube.com
                  IP =  216.239.38.120
                  Description = youtube
                  Save
                  NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

                  1 Reply Last reply Reply Quote 0
                  • V
                    vielfede
                    last edited by Apr 12, 2018, 10:53 AM

                    Hello,
                    Maybe I missed something, I read the entire topic, but I did not find any post about access url on different port then standard 80/443….
                    For example if you try to access the following site:

                    http://fatturazione.comieco.org:82/anagrafica.php

                    The proxy respond with an error message.
                    Loooking to the squid log I noticed that when I tried to connect to http://fatturazione.comieco.org:82/ squid tried to connect still to port 80…
                    To get it work, I inserted an exception  on wpad configuration, something like

                            if (shExpMatch(url, "http://fatturazione.comieco.org:82/*"))
                            { return "DIRECT"; }
                    
                    

                    That allows the client to go out straight through the FW on port 82…. and hence now it works...

                    Indeed I'd like a solution to allow "different ports" work without WPAD exceptions. Otherwise you have to insert an exception for every site not using std port.
                    I did not notice anyone pointing out this kind of issue... Can I be the only one with this kind of issue?
                    Thanks in advance.

                    1 Reply Last reply Reply Quote 1
                    • A
                      aGeekhere
                      last edited by Apr 13, 2018, 12:55 AM

                      I am not sure on this, if you use the transparent proxy then that only listens on port 80 and 443 for MITM.

                      Never Fear, A Geek is Here!

                      1 Reply Last reply Reply Quote 0
                      • V
                        vielfede
                        last edited by Apr 18, 2018, 7:36 AM Apr 13, 2018, 10:11 AM

                        Finally I found out the solution… (http and https)
                        https://forum.pfsense.org/index.php?topic=96782.0

                        1 Reply Last reply Reply Quote 1
                        • K
                          KARMA
                          last edited by Apr 27, 2018, 9:44 AM

                          I am trying to block facebook and youtube and some unwanted sites using Squid and SquidGuard filtering but those sites are't blocked. I can still access the blocked sites. Please can any one help me where i went wrong?

                          1 Reply Last reply Reply Quote 0
                          • pfrickrollP
                            pfrickroll
                            last edited by Jun 5, 2018, 9:19 PM

                            This post is deleted!
                            1 Reply Last reply Reply Quote 0
                            • S
                              stuck
                              last edited by stuck Jun 22, 2018, 5:24 PM Jun 21, 2018, 8:34 PM

                              @aGeekhere
                              First thank you for the detail guide. I'm running netgate 2.4.3_1 connected to TorguardVPN service and have had http transparent proxy (squid) running fine for some time. I've recently decided to add https filtering with squid/squidguard on netgate and WPAD on a separate http server. I can access the wpad files just fine using both the wpad.localdomain address or ip address. But the SSL filtering doesn't seem to work.
                              It seems on my system, transparent proxy only works on http sites. In order to get to https/ssl sites I have to explicitly setup my proxy address and port in windows.

                              Any help or hint is appreciated

                              A 1 Reply Last reply Jun 23, 2018, 4:28 AM Reply Quote 0
                              • A
                                aGeekhere @stuck
                                last edited by Jun 23, 2018, 4:28 AM

                                @stuck In squid SSL - Man In the Middle Filtering - HTTPS/SSL Interception tic Enable SSL filtering and SSL/MITM Mode set to splice all

                                Never Fear, A Geek is Here!

                                1 Reply Last reply Reply Quote 0
                                • Z
                                  zonda
                                  last edited by Jul 3, 2018, 12:00 AM

                                  Thank you for this guide aGeekHere!

                                  Would you be able provide instructions on how to exclude websites from being proxied please? I have transparent proxy setup, no WPAD, SSL/MITM mode set to Splice All. Filtering web content is working great except that Microsoft Office 365 applications seem to be having connectivity issues. When user traffic is no longer being proxied, issue is resolved. Would like to exclude all Office 365 related domains/sites from the proxy.

                                  Thank you again for any help or advice in advance.

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    aGeekhere
                                    last edited by Jul 3, 2018, 10:27 AM

                                    You need to set up the WPAD for windows updates and office for connections to work.

                                    This is how it works
                                    The WPAD directs traffic to port 80 and 443 if the programs supports it (a lot of programs do not).
                                    Transparent proxy catches any traffic going to port 80 which is not supported by the WPAD.
                                    SSL/MITM Mode set to splice all catches any traffic going to port 443 which is not supported by the WPAD, However this may not always work with things like windows updates and word, this is meant to be like a safety net that catches any missed connections from the WPAD.

                                    So you need all 3.

                                    Never Fear, A Geek is Here!

                                    1 Reply Last reply Reply Quote 0
                                    • Z
                                      zonda
                                      last edited by Jul 5, 2018, 12:12 PM

                                      Is it still possible to create exceptions to not proxy trusted sites with transparent proxy?
                                      There are some specific sites that will need to bypass the proxy for some of our application to function. WPAD was used initially in our environment but some traveling users had issues with persistent proxy settings so would prefer to only use the transparent method.

                                      A 1 Reply Last reply Jul 5, 2018, 12:42 PM Reply Quote 0
                                      • A
                                        aGeekhere @zonda
                                        last edited by Jul 5, 2018, 12:42 PM

                                        @zonda
                                        Sure, the option is under ACLs in squid. If users do not want to use the WPAD (auto config proxy) then they should manually enter in the proxy settings in internet options (if using windows). Do not rely on SSL/MITM Mode splice all for all 443 traffic or else you will get connection errors. That is if you are blocking port 80 and 443 and forcing users to use the proxy.

                                        Never Fear, A Geek is Here!

                                        1 Reply Last reply Reply Quote 0
                                        • J
                                          Jopeme2000
                                          last edited by Sep 5, 2018, 7:04 AM

                                          Hello. I use your guide http and https to filter traffic but i have problems with youtube videos

                                          Restricts many videos in an erroneous way

                                          Do you know how i can solve it?

                                          A 1 Reply Last reply Sep 13, 2018, 5:01 AM Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received