Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Guide to filtering web content (http and https) with pfsense 2.3

    Scheduled Pinned Locked Moved Documentation
    190 Posts 54 Posters 223.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      stuck
      last edited by stuck

      @aGeekhere
      First thank you for the detail guide. I'm running netgate 2.4.3_1 connected to TorguardVPN service and have had http transparent proxy (squid) running fine for some time. I've recently decided to add https filtering with squid/squidguard on netgate and WPAD on a separate http server. I can access the wpad files just fine using both the wpad.localdomain address or ip address. But the SSL filtering doesn't seem to work.
      It seems on my system, transparent proxy only works on http sites. In order to get to https/ssl sites I have to explicitly setup my proxy address and port in windows.

      Any help or hint is appreciated

      A 1 Reply Last reply Reply Quote 0
      • A
        aGeekhere @stuck
        last edited by

        @stuck In squid SSL - Man In the Middle Filtering - HTTPS/SSL Interception tic Enable SSL filtering and SSL/MITM Mode set to splice all

        Never Fear, A Geek is Here!

        1 Reply Last reply Reply Quote 0
        • Z
          zonda
          last edited by

          Thank you for this guide aGeekHere!

          Would you be able provide instructions on how to exclude websites from being proxied please? I have transparent proxy setup, no WPAD, SSL/MITM mode set to Splice All. Filtering web content is working great except that Microsoft Office 365 applications seem to be having connectivity issues. When user traffic is no longer being proxied, issue is resolved. Would like to exclude all Office 365 related domains/sites from the proxy.

          Thank you again for any help or advice in advance.

          1 Reply Last reply Reply Quote 0
          • A
            aGeekhere
            last edited by

            You need to set up the WPAD for windows updates and office for connections to work.

            This is how it works
            The WPAD directs traffic to port 80 and 443 if the programs supports it (a lot of programs do not).
            Transparent proxy catches any traffic going to port 80 which is not supported by the WPAD.
            SSL/MITM Mode set to splice all catches any traffic going to port 443 which is not supported by the WPAD, However this may not always work with things like windows updates and word, this is meant to be like a safety net that catches any missed connections from the WPAD.

            So you need all 3.

            Never Fear, A Geek is Here!

            1 Reply Last reply Reply Quote 0
            • Z
              zonda
              last edited by

              Is it still possible to create exceptions to not proxy trusted sites with transparent proxy?
              There are some specific sites that will need to bypass the proxy for some of our application to function. WPAD was used initially in our environment but some traveling users had issues with persistent proxy settings so would prefer to only use the transparent method.

              A 1 Reply Last reply Reply Quote 0
              • A
                aGeekhere @zonda
                last edited by

                @zonda
                Sure, the option is under ACLs in squid. If users do not want to use the WPAD (auto config proxy) then they should manually enter in the proxy settings in internet options (if using windows). Do not rely on SSL/MITM Mode splice all for all 443 traffic or else you will get connection errors. That is if you are blocking port 80 and 443 and forcing users to use the proxy.

                Never Fear, A Geek is Here!

                1 Reply Last reply Reply Quote 0
                • J
                  Jopeme2000
                  last edited by

                  Hello. I use your guide http and https to filter traffic but i have problems with youtube videos

                  Restricts many videos in an erroneous way

                  Do you know how i can solve it?

                  A 1 Reply Last reply Reply Quote 0
                  • A
                    aGeekhere @Jopeme2000
                    last edited by

                    @jopeme2000
                    "NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out."

                    Google has to improve it, nothing you can do about it

                    Never Fear, A Geek is Here!

                    1 Reply Last reply Reply Quote 0
                    • R
                      roberz
                      last edited by

                      @aGeekhere
                      Hi,
                      A great thank you for this guide. It is the most complete guide for traffic filtering on pfSense I was able to find.
                      I made traffic filtering work for a customer who wants porn filtering and it works. But doesn't work perfect. It works great on desktop machines while on Android phones it blocks more than required. For example: it also blocks all Skype and Facebook Messenger traffic and also blocks my RSS feed reader to connect to Internet on Android phones.
                      Can you explain a bit what you thought with this in june 2016 edit:
                      "I have found that if you have connection issues using auto config for android or other smart phones try manually setting the proxy, now opening port 80 and 443 is not needed."
                      Thank you.

                      A 1 Reply Last reply Reply Quote 0
                      • A
                        aGeekhere @roberz
                        last edited by

                        @roberz
                        For android phones (depending on with android version) auto config has issues so you have to manually enter in the proxy settings.

                        If things like Skype and Facebook Messenger are being blocked make sure you also have set up transparent proxy with splice all for SSL Man In the Middle Filtering.

                        The idea is
                        First use the WPAD to connect to the proxy (if the device has auto config issues you have to manually add it). If a program does not have proxy settings the transparent proxy and SSL Man In the Middle should catch the rest. The reason why to first use the WPAD is that some programs/software (like windows updates) have connection issues with transparent proxy and SSL Man In the Middle, splice all.

                        So first use the WPAD as a shortcut so you do not have to configure all devices on your network to use your proxy.
                        Then for devices that have auto conf issues manually config them.
                        Finally have the transparent proxy and SSL Man In the Middle splice all to catch any traffic that cannot use the proxy .

                        Hope this was helpful

                        Never Fear, A Geek is Here!

                        1 Reply Last reply Reply Quote 0
                        • M
                          marksantos
                          last edited by

                          f3ae6cac-c635-4451-9111-21aa2f13f337-image.png

                          Hi sir, all step are follow, search engine n safe search is working, but some of the images in whitelist website can't load.

                          1 Reply Last reply Reply Quote 0
                          • A
                            aGeekhere
                            last edited by

                            Just tested and it loads fine for me. Check your blocklist in your web filter (e.g squidguard).

                            Never Fear, A Geek is Here!

                            1 Reply Last reply Reply Quote 0
                            • M
                              marksantos
                              last edited by

                              the problem is default acl. when i try do deny all, the whitelist website can't load some images, when i try to allow default acl the whitelist website, can load images but the browsing of the whitelist are very slow.

                              1 Reply Last reply Reply Quote 0
                              • M
                                marksantos
                                last edited by

                                i'll try the whitelist website can only access and the other can block.

                                1 Reply Last reply Reply Quote 0
                                • M
                                  marksantos
                                  last edited by

                                  hi sir, how about gmail, how to allow gmail in whitelist, i tried add gmail.com in whitelist but still block the access.

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    aGeekhere
                                    last edited by

                                    In squidguard set Default access [all] to allow. Now only block the categories you want. You can use shallalist for categories.

                                    Never Fear, A Geek is Here!

                                    1 Reply Last reply Reply Quote 1
                                    • 4
                                      4o4rh
                                      last edited by

                                      dns resolver doesn't start when i include the forcegoogle.conf in the custom options.

                                      if i remove it, it works just fine.

                                      1 Reply Last reply Reply Quote 0
                                      • A
                                        aGeekhere
                                        last edited by

                                        did you make the forecegoogle.conf file?

                                        Never Fear, A Geek is Here!

                                        4 1 Reply Last reply Reply Quote 0
                                        • 4
                                          4o4rh @aGeekhere
                                          last edited by

                                          @aGeekhere problem was, i followed the initial instructions by putting the youtube and bing into the host overrides, then copied the whole forcegoogle section without checking the contents.

                                          In has youtube with a different address and bing. Removed those and it works.

                                          1 Reply Last reply Reply Quote 0
                                          • J
                                            johnchon
                                            last edited by

                                            i have follow the topic to do the filter but when i just activate transparent mode only i get this message " 400 Bad Request
                                            The plain HTTP request was sent to HTTPS port
                                            nginx"

                                            May i know what happen for this ?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.