Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Guide to filtering web content (http and https) with pfsense 2.3

    Scheduled Pinned Locked Moved Documentation
    190 Posts 54 Posters 236.6k Views 15 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • O Offline
      oddworld19
      last edited by

      Thanks. Right. My only thought there was:

      I understand SSL decryption and inspection gets…. Nasty.

      For example, if browsers require HSTS or whatever other protocols check certificates, then it might get annoying to keep installing certs on all on my computers on the LAN. For example... A small FTP server has no need to access Reddit, but I don't know if SSL bumping will cause issues with package updates in the future (without a cert).

      Maybe it will... Maybe it won't. I don't know... Do you think it would potentially be an annoyance if I don't have certs installed?

      Supermicro SYS-5018A-FTN4 (Atom c2758)
      pfSense 2.3.2

      1 Reply Last reply Reply Quote 0
      • M Offline
        maverik1
        last edited by

        Lots of good info! Thanks!!

        Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.

        1 Reply Last reply Reply Quote 0
        • C Online
          chris4916
          last edited by

          @maverik1:

          Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.

          What "notification" are you speaking about?
          I'm confused  :-[

          Jah Olela Wembo: Les mots se muent en maux quand ils indisposent, agressent ou blessent.

          1 Reply Last reply Reply Quote 0
          • M Offline
            maverik1
            last edited by

            I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

            1 Reply Last reply Reply Quote 0
            • M Offline
              maverik1
              last edited by

              @aGeekHere:

              Update Youtube safe mode
              Click add under Host overrides
              Host = www
              Domain = youtube.com
              IP =  216.239.38.120
              Description = youtube
              Save
              NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

              How can we get this working with mobile devices Android and iOS that use the youtube mobile app or m.youtube.com?

              1 Reply Last reply Reply Quote 0
              • C Online
                chris4916
                last edited by

                @maverik1:

                I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

                Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)

                Jah Olela Wembo: Les mots se muent en maux quand ils indisposent, agressent ou blessent.

                1 Reply Last reply Reply Quote 0
                • M Offline
                  maverik1
                  last edited by

                  @chris4916:

                  Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)

                  Push notifications are alerts that are received by cellular phones;mostly smart phones, for different types of apps such as social media, email, games and whatnot. I have no issues receiving these alerts when using data only. However, when I was on my wifi network they were not coming through. I spend a few weeks working with the firewall rules and proxy settings trying everything I could think of to troubleshoot the issue. Turns out the problem was with the phone and not pfsense.

                  1 Reply Last reply Reply Quote 0
                  • K Offline
                    kpa
                    last edited by

                    I'd be very surprised if all those "push" system weren't all emulated by polling a server periodically, that kind of set up needs absolutely nothing else but an outgoing connection from the device to the server. The other option would be a service running on the device and the notifications would be then really pushed on the device by an incoming connection from the server to the device. I don't think you'd want implement it like that however  ;)

                    1 Reply Last reply Reply Quote 0
                    • L Offline
                      laurenzzo
                      last edited by

                      @aGeekHere:

                      […]

                      First we are going to setup squidguard
                      Update
                      In squidguard under General settings
                      Tic enable
                      Tic Enable log
                      Tic Enable log rotation
                      Tic enable blacklist
                      Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
                      Save
                      apply (you must always hit apply for any changes you made to squidguard).

                      In Package/Proxy filter SquidGuard: General settings/General settings
                      click blacklist
                      enter http://www.shallalist.de/Downloads/shallalist.tar.gz
                      download
                      wait to finish

                      […]

                      Hi aGeekHere,

                      Thanks a lot for your guide !
                      ~~I tried to follow your steps to get my blacklist downloaded but without success.

                      Could you please have a look on my post for more details ?~~
                      https://forum.pfsense.org/index.php?topic=117314.msg649961#msg649961

                      I really don't understand why the download doesn't work…

                      Thanks in advance for any help !

                      EDIT : SOLVED !
                      this was an issue with Firefox…
                      Could you please update your nice guide with a small note ? (e.g. "don't use Firefox to download the blacklist, IE do the job correctly")

                      Thanks !

                      Kind regards,
                      Laurenzzo

                      1 Reply Last reply Reply Quote 0
                      • J Offline
                        jadog
                        last edited by

                        Thanks for taking the time to post this excellent guide! Just a few additions I would recommend adding.

                        YouTube mobile still allows unrestricted access. Including the below corrected it for me.

                        
                        Click add under Host overrides
                        Host = m
                        Domain = youtube.com
                        IP =  216.239.38.120
                        Description = youtube mobile
                        Save
                        

                        Also when using vi after you ssh into your router, using wq does not allow saving. You need to use ":wq" (without the quotes). All working perfectly for me!

                        1 Reply Last reply Reply Quote 1
                        • E Offline
                          eixel05
                          last edited by

                          Hi,

                          First of all thank you so much for this guide, just what I've been looking for!

                          Before this guide I was using Transparent+MITM approach but because some HTTPS sites doesn't load as they should (I'm not sure if the cause was indeed MITM but after turning it off the sites loads fine)

                          I followed this guide thoroughly, HTTP filtering works fine but I don't know if HTTPS filtering is working? Before, I am using the tail command (/var/squid/logs/access.log) and while MITM is on, https site access is being logged but now with this approach, not a single HTTPS is being logged.

                          Is there another way to check if indeed this approach is working for HTTPS?

                          Out-of-topic question, since my main purpose for this Proxy server is local cache, Are HTTPS objects being cache as well? or only HTTP?

                          Update (09/07/2016): NVM, I figured it out :D all is working great, thank you so much for this guide!

                          I'M REALLY SORRY FOR BEING NOOB, I just started my way to "Networking", please be patient with me.

                          1 Reply Last reply Reply Quote 0
                          • M Offline
                            Maxim_Al
                            last edited by

                            Can anybody help me?
                            I test  pfsense for gateway/proxy for a company. But I have one strange trouble:
                            The pfsense is installed and work but if I use it for access (through proxy) to https://code.getmdl.io/1.2.1/material.grey-orange.min.css  I have in proxy log:
                            Date IP Status Address User Destination
                            16.09.2016 14:37:28 192.168.1.222 TAG_NONE/503 code.getmdl.io:443 - -
                            What is it an how I must to do that resolve it?

                            UPDATE 28/10/2016 - This was provider :) not pfsense!

                            1 Reply Last reply Reply Quote 0
                            • K Offline
                              klmiciano
                              last edited by

                              Which rule should be above between these two?

                              FIRST – 
                              "Now we are going to create a rule that will force the network to use our route as the DNS server.
                              In Firewall/NAT/Port forward
                              add a new rule

                              Interface = LAN
                              Protocol = TCP/UDP
                              Source ports = *
                              Dest address = *
                              Dest ports = 53
                              NAT IP = 127.0.0.1
                              NAT Ports = 53
                              Description = Redirect DNS
                              LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
                              Save"

                              SECOND --
                              "To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
                              IPv4 TCP * * * 80 - 443 * none.
                              Save"

                              1 Reply Last reply Reply Quote 0
                              • K Offline
                                klmiciano
                                last edited by

                                @aGeekHere:

                                I found this post and I am researching to see how well it will work.

                                https://forum.pfsense.org/index.php?topic=106016.0

                                The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

                                Update
                                OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

                                So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.

                                How to do this?

                                1 Reply Last reply Reply Quote 0
                                • A Offline
                                  aGeekhere
                                  last edited by

                                  @klmiciano:

                                  Which rule should be above between these two?

                                  FIRST – 
                                  "Now we are going to create a rule that will force the network to use our route as the DNS server.
                                  In Firewall/NAT/Port forward
                                  add a new rule

                                  Interface = LAN
                                  Protocol = TCP/UDP
                                  Source ports = *
                                  Dest address = *
                                  Dest ports = 53
                                  NAT IP = 127.0.0.1
                                  NAT Ports = 53
                                  Description = Redirect DNS
                                  LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
                                  Save"

                                  SECOND --
                                  "To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
                                  IPv4 TCP * * * 80 - 443 * none.
                                  Save"

                                  first one on top

                                  @klmiciano:

                                  @aGeekHere:

                                  I found this post and I am researching to see how well it will work.

                                  https://forum.pfsense.org/index.php?topic=106016.0

                                  The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

                                  Update
                                  OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

                                  So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.

                                  How to do this?

                                  after you have the wpad setup and working enable transperrent proxy in squid

                                  Never Fear, A Geek is Here!

                                  1 Reply Last reply Reply Quote 0
                                  • P Offline
                                    phunni
                                    last edited by

                                    If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?

                                    I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there…

                                    1 Reply Last reply Reply Quote 0
                                    • J Offline
                                      jbourn1907
                                      last edited by

                                      Sorry I'm new in pfsense.

                                      I have an internal DNS server windows server 2008. How can I blocked internal computers to direct access to http. I want to filter it through pfsense firewall. Can you give me some screen shots or sample of rules and where can I put this?

                                      I want all computers to go first in pfsense proxy before connecting to internet. I already configure transparent proxy.

                                      Thanks for the help.

                                      1 Reply Last reply Reply Quote 0
                                      • A Offline
                                        aGeekhere
                                        last edited by

                                        "How can I blocked internal computers to direct access to http"
                                        read the part in the guide about blocking port 80 and 443 and instead of using pfsense as your DNS server use your internal one, (for more help on this issue ask in the Cache/Proxy forum).

                                        "If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?….I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there..."
                                        If the transparent proxy has issue with the vpn you may need to bypass it.

                                        Never Fear, A Geek is Here!

                                        1 Reply Last reply Reply Quote 0
                                        • G Offline
                                          genesislubrigas
                                          last edited by

                                          thanks for this tutorial.  i am not using dns resolver but bind.  can you also show how to configure bind.

                                          1 Reply Last reply Reply Quote 0
                                          • A Offline
                                            aGeekhere
                                            last edited by

                                            Have not used bind, but after a quick google try http://blog.muhammadattique.com/configuring-bind-dns-server-on-pfsense-firewall/

                                            If it does not help ask in the main forum

                                            Never Fear, A Geek is Here!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.