• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Simplest way to LOG all URL that users browse to

Scheduled Pinned Locked Moved General pfSense Questions
11 Posts 4 Posters 7.1k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • C
    cmdias
    last edited by Jul 20, 2018, 3:06 PM

    Hello Guys,

    I don't want to do any web-filtering (for now) but would love a simple way to LOG all HTTP and HTTPS requests (i dont care about the content on the site.... just need the URLs) that all users browse on my LAN (computers, tablets, phones etc...)

    I know about SQUID and all but HTTPS becomes an issue as i don't want to have to push certificates.

    Other devices we have like watchguard do this very easily and now we are considering PFSENSE but this is the first issue i hit.

    Any suggestions ?

    1 Reply Last reply Reply Quote 0
    • J
      johnpoz LAYER 8 Global Moderator
      last edited by johnpoz Jul 20, 2018, 3:43 PM Jul 20, 2018, 3:42 PM

      Not sure where people get the idea that you have to push certs to log/filter https traffic? When set to use a proxy client will send connect with the url of the parent domain in the clear via the connect

      You can see in simple sniff I took on my work computer sending to proxy for https site. Now you see talking to the proxy at 10.56.226.130 on port 8080, sending the connect command to the fqdn of the where I want to go for https..

      0_1532101409352_connect.png

      So while you see I tried to go to some host.domain.tld/path/whatever the thing that is logged is the just the parent domain, ie the fqdn want to connect to. The full uri would not be sent until the https connection has been made to the dest server.

      To filter "paths" inside the url of https then yes you have to do MITM on the certs. But if all you want to do is log or filter on the parent domain your fine... so you will see host.domain.tld in the connect command sent to the proxy, but you will not see host.domain.tld/PATH

      Are you saying watchguard gives you the full URI?

      An intelligent man is sometimes forced to be drunk to spend time with his fools
      If you get confused: Listen to the Music Play
      Please don't Chat/PM me for help, unless mod related
      SG-4860 24.11 | Lab VMs 2.7.2, 24.11

      1 Reply Last reply Reply Quote 0
      • C
        cmdias
        last edited by Jul 20, 2018, 4:54 PM

        Wow thx for the quick reply.

        Good point on capturing the info. So in this case i should setup squid and force all devices (windows, android and iOS) with something like WPAD to use the proxy i guess ?

        Or is there any chance this can be done in transparent mode also?

        Thx again!

        -carlos

        1 Reply Last reply Reply Quote 0
        • J
          johnpoz LAYER 8 Global Moderator
          last edited by Jul 20, 2018, 6:28 PM

          problem with transparent mode is the client doesn't send the connect command. Hard to do filtering of https in transparent.. really need to implicit where client knows there is a proxy - but sure you can use auto discovery/wpad to tell the client use this proxy.

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 24.11 | Lab VMs 2.7.2, 24.11

          1 Reply Last reply Reply Quote 0
          • C
            cmdias
            last edited by Jul 25, 2018, 5:17 PM

            Ok will give it a try.

            thx!

            1 Reply Last reply Reply Quote 0
            • K
              KOM
              last edited by Jul 25, 2018, 5:27 PM

              HTTPS with transparent mode is a hassle because you have to install the trusted certificate on every client or else you will get MitM warnings from their browsers.

              C 1 Reply Last reply Jul 25, 2018, 10:52 PM Reply Quote 0
              • C
                cmdias @KOM
                last edited by Jul 25, 2018, 10:52 PM

                @kom Did you find a better way?

                K 1 Reply Last reply Jul 26, 2018, 1:55 PM Reply Quote 0
                • M
                  marvosa
                  last edited by Jul 26, 2018, 6:36 AM

                  The problem is PFsense is a firewall, not a UTM. If you try to use tools for one purpose that are really intended for something else but have some side benefits... you typically see mixed results at best. Plus, it will be a management nightmare. Meaning if you're looking to spend all your time combing thru Squid logs... then I guess go for it.

                  However, If you're looking for UTM features, but want to keep PFsense as the edge device, another option is to install a UTM product in bridge mode inline with your network. For example, this is what I have implemented at a high level:

                  PFsense -> Untangle (in bridge mode) -> Switch -> LAN

                  The web filter app logs all web events, so you will have access to exactly what you are requesting. Also, everything will be saved in a database, displayed in a useful format and the info can be easily filtered and exported. Not to mention, you will have dozens of other web related metrics to leverage.

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmdias
                    last edited by Jul 26, 2018, 1:51 PM

                    I was actually just downloading untangle last night ... can you give me more information about the "bridge mode" between PF and Untangle ?

                    Back in the days i was using SONIC WALL + WEBSENSE and it as super simple to setup..... miss those days! lol

                    M 1 Reply Last reply Jul 26, 2018, 3:04 PM Reply Quote 0
                    • K
                      KOM @cmdias
                      last edited by Jul 26, 2018, 1:55 PM

                      @cmdias Yes, it's called explicit mode plus WPAD. I don't waste time fighting with transparent proxy & certs.

                      1 Reply Last reply Reply Quote 0
                      • M
                        marvosa @cmdias
                        last edited by Jul 26, 2018, 3:04 PM

                        @cmdias said in Simplest way to LOG all URL that users browse to:

                        I was actually just downloading untangle last night ... can you give me more information about the "bridge mode" between PF and Untangle ?

                        Back in the days i was using SONIC WALL + WEBSENSE and it as super simple to setup..... miss those days! lol

                        Take a look at step 3 here -> https://www.untangle.com/untangle-ng-firewall/resources/how-to-deploy/

                        Here's some info on a bridge mode deployment:

                        In Bridge mode, NG Firewall is set between your existing firewall and main switch. When in Bridge mode NG Firewall is transparent, meaning you won’t need to change the default gateway of the computers on your network or the routes on your firewall – just put NG Firewall between your firewall and main switch and… that’s it! You’ll need to give NG Firewall’s External interface an IP in the subnet of the firewall, set the Internal interface to bridge and bridge it to External.

                        To get a better idea of what you'll have access to, check out their live demo here -> http://demo.untangle.com

                        1 Reply Last reply Reply Quote 0
                        11 out of 11
                        • First post
                          11/11
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                          This community forum collects and processes your personal information.
                          consent.not_received