Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Guide to filtering web content (http and https) with pfsense 2.3

    Scheduled Pinned Locked Moved Documentation
    190 Posts 54 Posters 222.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      aGeekhere
      last edited by

      lets try without the wpad and use just the transparent, set proxy direct does it work then?
      If you still cannot access https sites then post you squid config.

      Never Fear, A Geek is Here!

      1 Reply Last reply Reply Quote 0
      • T
        techbee
        last edited by

        @aGeekHere:

        lets try without the wpad and use just the transparent, set proxy direct does it work then?
        If you still cannot access https sites then post you squid config.

        If I use only the http proxy without the ssl mitm and set browser to auto, then the http proxy filter works.

        If you are saying to setup http transparent proxy and ssl mitm filtering and install cert to client device and set browser proxy, then I have not done this yet.

        1 Reply Last reply Reply Quote 0
        • A
          aGeekhere
          last edited by

          and install cert to client device and set browser proxy, then I have not done this yet.

          Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
          Set browser to direct.
          This will use transparent proxy for both http and https.

          If this works then there is an issue with your WPAD.
          If this does not work then something is broken.

          Never Fear, A Geek is Here!

          1 Reply Last reply Reply Quote 0
          • T
            techbee
            last edited by

            @aGeekHere:

            and install cert to client device and set browser proxy, then I have not done this yet.

            Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
            Set browser to direct.
            This will use transparent proxy for both http and https.

            If this works then there is an issue with your WPAD.
            If this does not work then something is broken.

            What do you mean for "Set browser to direct", is this the automatic detect settings ?

            On the other hand, my wpad only have the following inside:

            function FindProxyForURL(url,host)
            {
            return "PROXY 192.168.1.1:3128";
            }

            I am not sure what you example below means so I use the example from pfsense wiki.

            function FindProxyForURL(url, host)
            {
                if (isPlainHostName(host) ||
                    shExpMatch(host, "*.local") ||
                    isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
                    return "DIRECT";

            return "PROXY 192.168.1.1:3128";
            }

            1 Reply Last reply Reply Quote 0
            • A
              aGeekhere
              last edited by

              yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.

              Never Fear, A Geek is Here!

              1 Reply Last reply Reply Quote 0
              • T
                techbee
                last edited by

                @aGeekHere:

                yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.

                By turn off, you mean uncheck the automatic detect settings and others options as well ?

                1 Reply Last reply Reply Quote 0
                • A
                  aGeekhere
                  last edited by

                  yes

                  Never Fear, A Geek is Here!

                  1 Reply Last reply Reply Quote 0
                  • T
                    techbee
                    last edited by

                    May I ask though, what is the exact version of pfsense that you are using just for comparison.  If this will not work, then I will need to reinstall my pfsense box, maybe its broken.

                    I am using:

                    2.3.4-RELEASE (amd64)
                    built on Wed May 03 15:13:29 CDT 2017
                    FreeBSD 10.3-RELEASE-p19

                    On the other hand, below is the screen shot of my testing NOT USING WPAD but setup the browser proxy.

                    Also, I tried direct approach that you said, and the result the same error as before.

                    What exact pfsense version are you using ?

                    error1.jpg_thumb
                    error1.jpg

                    1 Reply Last reply Reply Quote 0
                    • A
                      aGeekhere
                      last edited by

                      2.3.4-RELEASE (amd64)
                      Something is broken.

                      Never Fear, A Geek is Here!

                      1 Reply Last reply Reply Quote 0
                      • T
                        techbee
                        last edited by

                        @aGeekHere:

                        2.3.4-RELEASE (amd64)
                        Something is broken.

                        what is in your wpad file?

                        did you install your pfsense or upgraded to 2.3.4 release?

                        1 Reply Last reply Reply Quote 0
                        • A
                          aGeekhere
                          last edited by

                          wpad is in the guide.

                          I have been upgrading since i joined.

                          Never Fear, A Geek is Here!

                          1 Reply Last reply Reply Quote 0
                          • 4
                            4Qman
                            last edited by

                            Hi,

                            I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

                            ![pfsense Nat PortForward 1.png](/public/imported_attachments/1/pfsense Nat PortForward 1.png)
                            ![pfsense Nat PortForward 1.png_thumb](/public/imported_attachments/1/pfsense Nat PortForward 1.png_thumb)

                            1 Reply Last reply Reply Quote 0
                            • 4
                              4Qman
                              last edited by

                              I have followed this guide.

                              When I run proxy test it shows that I am under proxy. Google works fine, and so do most websites. But such websites as Amazon won't load, gives error.

                              Any advice would be greatly appreciated.

                              1 Reply Last reply Reply Quote 0
                              • A
                                aGeekhere
                                last edited by

                                Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

                                The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

                                Never Fear, A Geek is Here!

                                1 Reply Last reply Reply Quote 0
                                • 4
                                  4Qman
                                  last edited by

                                  @aGeekHere:

                                  Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

                                  The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

                                  Hi thanks for your reply.

                                  I followed the guide, page 1.  I copied and pasted logs from PackageSquidGuardLogs

                                  10.07.2017 17:06:47 [squid_reconfigure] Remove old redirector options from Squid config.
                                  10.07.2017 17:05:35 [squid_reconfigure] Add new redirector options to Squid config.
                                  10.07.2017 17:05:35 [squid_reconfigure] Remove old redirector options from Squid config.
                                  10.07.2017 17:05:35 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
                                  10.07.2017 17:05:35 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
                                  10.07.2017 17:05:35 [sg_create_config] Add Default
                                  10.07.2017 17:05:35 [sg_create_config] Add rewrites: safesearch;
                                  10.07.2017 17:05:35 [sg_create_config] Add destinations: whitelist; blacklist;
                                  10.07.2017 17:05:35 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
                                  10.07.2017 17:05:35 [sg_create_config] Add blacklist entries
                                  10.07.2017 17:05:35 [squidguard_rebuild_db] Start rebuild DB.
                                  10.07.2017 17:05:24 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
                                  10.07.2017 17:05:24 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
                                  10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
                                  10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
                                  10.07.2017 17:05:24 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
                                  10.07.2017 17:05:24 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
                                  10.07.2017 17:05:24 [sg_reconfigure_user_db] Add user entries
                                  10.07.2017 17:05:24 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
                                  10.07.2017 16:36:15 [squid_reconfigure] Add new redirector options to Squid config.
                                  10.07.2017 16:36:15 [squid_reconfigure] Remove old redirector options from Squid config.
                                  10.07.2017 16:36:15 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
                                  10.07.2017 16:36:15 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
                                  10.07.2017 16:36:15 [sg_create_config] Add Default
                                  10.07.2017 16:36:15 [sg_create_config] Add rewrites: safesearch;
                                  10.07.2017 16:36:15 [sg_create_config] Add destinations: whitelist; blacklist;
                                  10.07.2017 16:36:15 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
                                  10.07.2017 16:36:15 [sg_create_config] Add blacklist entries
                                  10.07.2017 16:36:15 [squidguard_rebuild_db] Start rebuild DB.
                                  10.07.2017 16:36:05 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
                                  10.07.2017 16:36:05 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
                                  10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
                                  10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
                                  10.07.2017 16:36:05 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
                                  10.07.2017 16:36:05 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
                                  10.07.2017 16:36:05 [sg_reconfigure_user_db] Add user entries
                                  10.07.2017 16:36:05 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
                                  09.07.2017 18:01:06 [squid_reconfigure] Add new redirector options to Squid config.
                                  09.07.2017 18:01:06 [squid_reconfigure] Remove old redirector options from Squid config.
                                  09.07.2017 18:01:06 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
                                  09.07.2017 18:01:06 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
                                  09.07.2017 18:01:06 [sg_create_config] Add Default
                                  09.07.2017 18:01:06 [sg_create_config] Add rewrites: safesearch;
                                  09.07.2017 18:01:06 [sg_create_config] Add destinations: whitelist; blacklist;
                                  09.07.2017 18:01:06 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
                                  09.07.2017 18:01:06 [sg_create_config] Add blacklist entries
                                  09.07.2017 18:01:06 [squidguard_rebuild_db] Start rebuild DB.
                                  09.07.2017 18:00:56 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
                                  09.07.2017 18:00:56 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
                                  09.07.2017 18:00:56 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.

                                  Proxy Config # This file is automatically generated by pfSense

                                  Do not edit manually !

                                  http_port 192.168.1.1:3128
                                  http_port 127.0.0.1:3128
                                  icp_port 0
                                  digest_generation off
                                  dns_v4_first off
                                  pid_filename /var/run/squid/squid.pid
                                  cache_effective_user squid
                                  cache_effective_group proxy
                                  error_default_language en
                                  icon_directory /usr/local/etc/squid/icons
                                  visible_hostname localhost
                                  cache_mgr drobinson@brightsure.co.uk
                                  access_log /var/squid/logs/access.log
                                  cache_log /var/squid/logs/cache.log
                                  cache_store_log none
                                  netdb_filename /var/squid/logs/netdb.state
                                  pinger_enable off
                                  pinger_program /usr/local/libexec/squid/pinger

                                  logfile_rotate 2
                                  debug_options rotate=2
                                  shutdown_lifetime 3 seconds

                                  Allow local network(s) on interface(s)

                                  acl localnet src  192.168.1.0/24 127.0.0.0/8
                                  forwarded_for on
                                  httpd_suppress_version_string on
                                  uri_whitespace strip

                                  cache_mem 1000 MB
                                  maximum_object_size_in_memory 256 KB
                                  memory_replacement_policy heap GDSF
                                  cache_replacement_policy heap LFUDA
                                  minimum_object_size 0 KB
                                  maximum_object_size 1000 MB
                                  cache_dir ufs /var/squid/cache 3000 16 256
                                  offline_mode off
                                  cache_swap_low 90
                                  cache_swap_high 95
                                  cache allow all

                                  Add any of your own refresh_pattern entries above these.

                                  refresh_pattern ^ftp:    1440  20%  10080
                                  refresh_pattern ^gopher:  1440  0%  1440
                                  refresh_pattern -i (/cgi-bin/|?) 0  0%  0
                                  refresh_pattern .    0  20%  4320

                                  #Remote proxies

                                  Setup some default acls

                                  ACLs all, manager, localhost, and to_localhost are predefined.

                                  acl allsrc src all
                                  acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
                                  acl sslports port 443 563

                                  acl purge method PURGE
                                  acl connect method CONNECT

                                  Define protocols used for redirects

                                  acl HTTP proto HTTP
                                  acl HTTPS proto HTTPS
                                  http_access allow manager localhost

                                  http_access deny manager
                                  http_access allow purge localhost
                                  http_access deny purge
                                  http_access deny !safeports
                                  http_access deny CONNECT !sslports

                                  Always allow localhost connections

                                  http_access allow localhost

                                  request_body_max_size 0 KB
                                  delay_pools 1
                                  delay_class 1 2
                                  delay_parameters 1 -1/-1 -1/-1
                                  delay_initial_bucket_level 100
                                  delay_access 1 allow allsrc

                                  Reverse Proxy settings

                                  Custom options before auth

                                  Setup allowed ACLs

                                  Allow local network(s) on interface(s)

                                  http_access allow localnet

                                  Default block all to be sure

                                  http_access deny allsrc

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    aGeekhere
                                    last edited by

                                    if you set squidguard to allow all do still get the issue?

                                    Never Fear, A Geek is Here!

                                    1 Reply Last reply Reply Quote 0
                                    • 4
                                      4Qman
                                      last edited by

                                      @aGeekHere:

                                      if you set squidguard to allow all do still get the issue?

                                      Hi, thank you for the reply.

                                      I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

                                      At the moment I have Wpad.dat working, I also have MSSL MITM working on splice all, I have android working now (manually entered wpad.dat address).

                                      Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?

                                      1 Reply Last reply Reply Quote 0
                                      • A
                                        aGeekhere
                                        last edited by

                                        Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?

                                        Not sure, maybe if you had a very large number of users you could use another webserver for the wpad.

                                        I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

                                        Even if you left that unticked the transparent proxy should have still connected you, verify that the transparent proxy is working (e.g block port 80 and 443 and use a program that gets updates form the net to check).

                                        Never Fear, A Geek is Here!

                                        1 Reply Last reply Reply Quote 0
                                        • R
                                          rebytr
                                          last edited by

                                          @4Qman:

                                          Hi,

                                          I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

                                          Was looking at this guide, https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense and comparing to your screenshot.  What are the implications of unchecking the "invert match" box and not disabling NAT reflection?

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            bwlinux
                                            last edited by

                                            Thanks aGeekHere!  This guide encouraged me to revisit HTTPS filtering for several of my sites.

                                            The WPAD options seems great for PCs but didn't work well on Android devices.  Things get really difficult when you have 2 or more network interfaces you want to proxy/filter using WPAD.

                                            In the end, I just set up transparent proxy for both http and https. Creating a CA on the firewall and then assigning that in Squid for the CA along with "Splice All" for SSL/MITM Mode works great!

                                            I think it's been mentioned before but your guide would really benefit from having 2 completely separate sets of instructions for transparent only and WPAD only.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.