Simplest way to LOG all URL that users browse to
-
Hello Guys,
I don't want to do any web-filtering (for now) but would love a simple way to LOG all HTTP and HTTPS requests (i dont care about the content on the site.... just need the URLs) that all users browse on my LAN (computers, tablets, phones etc...)
I know about SQUID and all but HTTPS becomes an issue as i don't want to have to push certificates.
Other devices we have like watchguard do this very easily and now we are considering PFSENSE but this is the first issue i hit.
Any suggestions ?
-
Not sure where people get the idea that you have to push certs to log/filter https traffic? When set to use a proxy client will send connect with the url of the parent domain in the clear via the connect
You can see in simple sniff I took on my work computer sending to proxy for https site. Now you see talking to the proxy at 10.56.226.130 on port 8080, sending the connect command to the fqdn of the where I want to go for https..
So while you see I tried to go to some host.domain.tld/path/whatever the thing that is logged is the just the parent domain, ie the fqdn want to connect to. The full uri would not be sent until the https connection has been made to the dest server.
To filter "paths" inside the url of https then yes you have to do MITM on the certs. But if all you want to do is log or filter on the parent domain your fine... so you will see host.domain.tld in the connect command sent to the proxy, but you will not see host.domain.tld/PATH
Are you saying watchguard gives you the full URI?
-
Wow thx for the quick reply.
Good point on capturing the info. So in this case i should setup squid and force all devices (windows, android and iOS) with something like WPAD to use the proxy i guess ?
Or is there any chance this can be done in transparent mode also?
Thx again!
-carlos
-
problem with transparent mode is the client doesn't send the connect command. Hard to do filtering of https in transparent.. really need to implicit where client knows there is a proxy - but sure you can use auto discovery/wpad to tell the client use this proxy.
-
Ok will give it a try.
thx!
-
HTTPS with transparent mode is a hassle because you have to install the trusted certificate on every client or else you will get MitM warnings from their browsers.
-
@kom Did you find a better way?
-
The problem is PFsense is a firewall, not a UTM. If you try to use tools for one purpose that are really intended for something else but have some side benefits... you typically see mixed results at best. Plus, it will be a management nightmare. Meaning if you're looking to spend all your time combing thru Squid logs... then I guess go for it.
However, If you're looking for UTM features, but want to keep PFsense as the edge device, another option is to install a UTM product in bridge mode inline with your network. For example, this is what I have implemented at a high level:
PFsense -> Untangle (in bridge mode) -> Switch -> LAN
The web filter app logs all web events, so you will have access to exactly what you are requesting. Also, everything will be saved in a database, displayed in a useful format and the info can be easily filtered and exported. Not to mention, you will have dozens of other web related metrics to leverage.
-
I was actually just downloading untangle last night ... can you give me more information about the "bridge mode" between PF and Untangle ?
Back in the days i was using SONIC WALL + WEBSENSE and it as super simple to setup..... miss those days! lol
-
@cmdias Yes, it's called explicit mode plus WPAD. I don't waste time fighting with transparent proxy & certs.
-
@cmdias said in Simplest way to LOG all URL that users browse to:
I was actually just downloading untangle last night ... can you give me more information about the "bridge mode" between PF and Untangle ?
Back in the days i was using SONIC WALL + WEBSENSE and it as super simple to setup..... miss those days! lol
Take a look at step 3 here -> https://www.untangle.com/untangle-ng-firewall/resources/how-to-deploy/
Here's some info on a bridge mode deployment:
In Bridge mode, NG Firewall is set between your existing firewall and main switch. When in Bridge mode NG Firewall is transparent, meaning you won’t need to change the default gateway of the computers on your network or the routes on your firewall – just put NG Firewall between your firewall and main switch and… that’s it! You’ll need to give NG Firewall’s External interface an IP in the subnet of the firewall, set the Internal interface to bridge and bridge it to External.
To get a better idea of what you'll have access to, check out their live demo here -> http://demo.untangle.com
