• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Guide to filtering web content (http and https) with pfsense 2.3

Scheduled Pinned Locked Moved Documentation
190 Posts 54 Posters 224.5k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • 4
    4Qman
    last edited by Jul 11, 2017, 7:39 PM

    I have followed this guide.

    When I run proxy test it shows that I am under proxy. Google works fine, and so do most websites. But such websites as Amazon won't load, gives error.

    Any advice would be greatly appreciated.

    1 Reply Last reply Reply Quote 0
    • A
      aGeekhere
      last edited by Jul 11, 2017, 11:22 PM

      Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

      The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

      Never Fear, A Geek is Here!

      1 Reply Last reply Reply Quote 0
      • 4
        4Qman
        last edited by Jul 12, 2017, 5:07 PM

        @aGeekHere:

        Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

        The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

        Hi thanks for your reply.

        I followed the guide, page 1.  I copied and pasted logs from PackageSquidGuardLogs

        10.07.2017 17:06:47 [squid_reconfigure] Remove old redirector options from Squid config.
        10.07.2017 17:05:35 [squid_reconfigure] Add new redirector options to Squid config.
        10.07.2017 17:05:35 [squid_reconfigure] Remove old redirector options from Squid config.
        10.07.2017 17:05:35 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
        10.07.2017 17:05:35 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
        10.07.2017 17:05:35 [sg_create_config] Add Default
        10.07.2017 17:05:35 [sg_create_config] Add rewrites: safesearch;
        10.07.2017 17:05:35 [sg_create_config] Add destinations: whitelist; blacklist;
        10.07.2017 17:05:35 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
        10.07.2017 17:05:35 [sg_create_config] Add blacklist entries
        10.07.2017 17:05:35 [squidguard_rebuild_db] Start rebuild DB.
        10.07.2017 17:05:24 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
        10.07.2017 17:05:24 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
        10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
        10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
        10.07.2017 17:05:24 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
        10.07.2017 17:05:24 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
        10.07.2017 17:05:24 [sg_reconfigure_user_db] Add user entries
        10.07.2017 17:05:24 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
        10.07.2017 16:36:15 [squid_reconfigure] Add new redirector options to Squid config.
        10.07.2017 16:36:15 [squid_reconfigure] Remove old redirector options from Squid config.
        10.07.2017 16:36:15 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
        10.07.2017 16:36:15 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
        10.07.2017 16:36:15 [sg_create_config] Add Default
        10.07.2017 16:36:15 [sg_create_config] Add rewrites: safesearch;
        10.07.2017 16:36:15 [sg_create_config] Add destinations: whitelist; blacklist;
        10.07.2017 16:36:15 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
        10.07.2017 16:36:15 [sg_create_config] Add blacklist entries
        10.07.2017 16:36:15 [squidguard_rebuild_db] Start rebuild DB.
        10.07.2017 16:36:05 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
        10.07.2017 16:36:05 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
        10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
        10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
        10.07.2017 16:36:05 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
        10.07.2017 16:36:05 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
        10.07.2017 16:36:05 [sg_reconfigure_user_db] Add user entries
        10.07.2017 16:36:05 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
        09.07.2017 18:01:06 [squid_reconfigure] Add new redirector options to Squid config.
        09.07.2017 18:01:06 [squid_reconfigure] Remove old redirector options from Squid config.
        09.07.2017 18:01:06 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
        09.07.2017 18:01:06 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
        09.07.2017 18:01:06 [sg_create_config] Add Default
        09.07.2017 18:01:06 [sg_create_config] Add rewrites: safesearch;
        09.07.2017 18:01:06 [sg_create_config] Add destinations: whitelist; blacklist;
        09.07.2017 18:01:06 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
        09.07.2017 18:01:06 [sg_create_config] Add blacklist entries
        09.07.2017 18:01:06 [squidguard_rebuild_db] Start rebuild DB.
        09.07.2017 18:00:56 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
        09.07.2017 18:00:56 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
        09.07.2017 18:00:56 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.

        Proxy Config # This file is automatically generated by pfSense

        Do not edit manually !

        http_port 192.168.1.1:3128
        http_port 127.0.0.1:3128
        icp_port 0
        digest_generation off
        dns_v4_first off
        pid_filename /var/run/squid/squid.pid
        cache_effective_user squid
        cache_effective_group proxy
        error_default_language en
        icon_directory /usr/local/etc/squid/icons
        visible_hostname localhost
        cache_mgr drobinson@brightsure.co.uk
        access_log /var/squid/logs/access.log
        cache_log /var/squid/logs/cache.log
        cache_store_log none
        netdb_filename /var/squid/logs/netdb.state
        pinger_enable off
        pinger_program /usr/local/libexec/squid/pinger

        logfile_rotate 2
        debug_options rotate=2
        shutdown_lifetime 3 seconds

        Allow local network(s) on interface(s)

        acl localnet src  192.168.1.0/24 127.0.0.0/8
        forwarded_for on
        httpd_suppress_version_string on
        uri_whitespace strip

        cache_mem 1000 MB
        maximum_object_size_in_memory 256 KB
        memory_replacement_policy heap GDSF
        cache_replacement_policy heap LFUDA
        minimum_object_size 0 KB
        maximum_object_size 1000 MB
        cache_dir ufs /var/squid/cache 3000 16 256
        offline_mode off
        cache_swap_low 90
        cache_swap_high 95
        cache allow all

        Add any of your own refresh_pattern entries above these.

        refresh_pattern ^ftp:    1440  20%  10080
        refresh_pattern ^gopher:  1440  0%  1440
        refresh_pattern -i (/cgi-bin/|?) 0  0%  0
        refresh_pattern .    0  20%  4320

        #Remote proxies

        Setup some default acls

        ACLs all, manager, localhost, and to_localhost are predefined.

        acl allsrc src all
        acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
        acl sslports port 443 563

        acl purge method PURGE
        acl connect method CONNECT

        Define protocols used for redirects

        acl HTTP proto HTTP
        acl HTTPS proto HTTPS
        http_access allow manager localhost

        http_access deny manager
        http_access allow purge localhost
        http_access deny purge
        http_access deny !safeports
        http_access deny CONNECT !sslports

        Always allow localhost connections

        http_access allow localhost

        request_body_max_size 0 KB
        delay_pools 1
        delay_class 1 2
        delay_parameters 1 -1/-1 -1/-1
        delay_initial_bucket_level 100
        delay_access 1 allow allsrc

        Reverse Proxy settings

        Custom options before auth

        Setup allowed ACLs

        Allow local network(s) on interface(s)

        http_access allow localnet

        Default block all to be sure

        http_access deny allsrc

        1 Reply Last reply Reply Quote 0
        • A
          aGeekhere
          last edited by Jul 14, 2017, 12:46 AM

          if you set squidguard to allow all do still get the issue?

          Never Fear, A Geek is Here!

          1 Reply Last reply Reply Quote 0
          • 4
            4Qman
            last edited by Jul 14, 2017, 8:30 PM

            @aGeekHere:

            if you set squidguard to allow all do still get the issue?

            Hi, thank you for the reply.

            I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

            At the moment I have Wpad.dat working, I also have MSSL MITM working on splice all, I have android working now (manually entered wpad.dat address).

            Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?

            1 Reply Last reply Reply Quote 0
            • A
              aGeekhere
              last edited by Jul 17, 2017, 11:07 PM

              Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?

              Not sure, maybe if you had a very large number of users you could use another webserver for the wpad.

              I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

              Even if you left that unticked the transparent proxy should have still connected you, verify that the transparent proxy is working (e.g block port 80 and 443 and use a program that gets updates form the net to check).

              Never Fear, A Geek is Here!

              1 Reply Last reply Reply Quote 0
              • R
                rebytr
                last edited by Sep 9, 2017, 2:32 PM

                @4Qman:

                Hi,

                I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

                Was looking at this guide, https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense and comparing to your screenshot.  What are the implications of unchecking the "invert match" box and not disabling NAT reflection?

                1 Reply Last reply Reply Quote 0
                • B
                  bwlinux
                  last edited by Sep 30, 2017, 6:05 PM

                  Thanks aGeekHere!  This guide encouraged me to revisit HTTPS filtering for several of my sites.

                  The WPAD options seems great for PCs but didn't work well on Android devices.  Things get really difficult when you have 2 or more network interfaces you want to proxy/filter using WPAD.

                  In the end, I just set up transparent proxy for both http and https. Creating a CA on the firewall and then assigning that in Squid for the CA along with "Splice All" for SSL/MITM Mode works great!

                  I think it's been mentioned before but your guide would really benefit from having 2 completely separate sets of instructions for transparent only and WPAD only.

                  1 Reply Last reply Reply Quote 0
                  • A
                    aGeekhere
                    last edited by Sep 30, 2017, 11:16 PM

                    I now recommend that you use both, WPAC as the default and transparent with splice all as backup. So software that has no proxy settings still get redirected to the proxy in stead of getting blocked by the firewall rule 80 443. For android you can manually set the proxy, sometimes splice all can show SSL errors when web browsing. I will clean up the guide when I have time.

                    Never Fear, A Geek is Here!

                    1 Reply Last reply Reply Quote 0
                    • M
                      minhgi
                      last edited by Nov 29, 2017, 8:29 PM

                      After using the guide and having everything setup correctly, I noticed pfblockerng easy_list stop blocking ads.  Have anyone have this issue and know how to fix it?

                      1 Reply Last reply Reply Quote 0
                      • V
                        vielfede
                        last edited by Dec 15, 2017, 8:34 AM Dec 5, 2017, 5:20 PM

                        @aGeekHere:

                        sometimes splice all can show SSL errors when web browsing.

                        I'd say "often" a  lot of SSL errors!  ;)
                        In my opinion Splice All is not usable in the way it is now (without WPAD).

                        1 Reply Last reply Reply Quote 0
                        • O
                          originalsin
                          last edited by Dec 21, 2017, 2:34 AM

                          guess it is way too late for asking abbration.

                          But what does "Tic" stands for?

                          I can not find out the Tic in Pfsense and can not figure it out.

                          Plz somebody help me!!!

                          1 Reply Last reply Reply Quote 0
                          • A
                            aGeekhere
                            last edited by Dec 21, 2017, 11:12 PM

                            Tick, select.

                            Never Fear, A Geek is Here!

                            1 Reply Last reply Reply Quote 0
                            • Y
                              yaska
                              last edited by Dec 22, 2017, 5:51 AM

                              I have multiple vlans with different IP addresses

                              VLAN_50 = 192.168.50.0
                              VLAN_60 = 192.168.60.0

                              How do I add all the IPs to the below section

                              function FindProxyForURL(url, host)
                              {
                                  if (isPlainHostName(host) ||
                                      shExpMatch(host, "*.local") ||
                                      isInNet(dnsResolve(host), "192.168.60.0",  "255.255.255.0"))
                                      return "DIRECT";

                              return "PROXY 192.168.60.1:3128";
                              }

                              1 Reply Last reply Reply Quote 0
                              • S
                                shan52
                                last edited by Jan 5, 2018, 7:43 PM

                                Why is the difference between Squiguard and OpenDNS?

                                1 Reply Last reply Reply Quote 0
                                • V
                                  vielfede
                                  last edited by Jan 8, 2018, 5:09 PM

                                  Just one thing I can not get working:

                                  Windows 7 updates, W7 detects them, but it's not able to download. I read something about here, but I do not know how to manage it.
                                  https://wiki.squid-cache.org/SquidFaq/WindowsUpdate#Squid_with_SSL-Bump_and_Windows_Updates

                                  Anyone with the same problem?
                                  Solutions? Workarounds?
                                  Thanks in advance

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    shan52
                                    last edited by Jan 8, 2018, 8:13 PM

                                    We have lots of laptop and pc, is there an easier way to install the cert for all the devices?

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      aGeekhere
                                      last edited by Jan 8, 2018, 10:28 PM

                                      I have multiple vlans with different IP addresses

                                      I do not use vlans so I do not know, sorry

                                      Why is the difference between Squiguard and OpenDNS?

                                      Squiguard block content on rules you set on your router (totally private).
                                      OpenDNS block content on rules you set on their servers (they see all traffic, also can be slower)

                                      Just one thing I can not get working

                                      Windows updates (as well as Mac and linux) should all be working, if it is not that you may have missed a step. Check that the WPAD is working and your pc is set to auto configure.
                                      If it is still not working read though the guide again, it should work.

                                      We have lots of laptop and pc, is there an easier way to install the cert for all the devices?

                                      You only have to create a cert of squid when using splice all not for all devices

                                      Hope this helps

                                      Never Fear, A Geek is Here!

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        DerHeldvomErdbeerfeld
                                        last edited by Jan 23, 2018, 10:24 AM

                                        Hi there,

                                        I just read this guide and also tried a little bit an the most I tried worked. Now I just have some
                                        questions for better understanding.

                                        @aGeekHere:

                                        Guide to filtering web content (http and https) with pfsense 2.3 updated 09 June 2017

                                        To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
                                        IPv4 TCP * * * 80 - 443 * none.
                                        Save

                                        If I set it up like this FROM 80 TO 443 (this is what it mean to me 80-443) didnt I block also all ports
                                        between 80 and 443? Wouldnt it be better to set up two rules or an alias (Firewall/Aliases/Ports)
                                        for the pot 80 and 443 or did I understood it wrong?

                                        I also want to use a dual wan configuration, is there something to take care of?

                                        –-
                                        greets from Germany

                                        DerHeldvomErdbeerfeld

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          aGeekhere
                                          last edited by Jan 24, 2018, 6:16 AM

                                          hi
                                          1. I created two rules, one for port 80 and one for 443 (I will update this part of the guide)

                                          2. I do not think squid works on dual wan

                                          Never Fear, A Geek is Here!

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received