A Defininitive Web Filtering Solution - $400
-
Note the $$$ mark is just a base contribution to grab some traction. Judging from the requests of other pfsense users and some digging around the forum, there's obviously need for such a feature.
The Concept
We have a pfsense acting as a firewall/router(Wireless even) on a given network. The network is separated into different subnets; think private and guest.
The device is now configured so that the internet traffic of each network is filtered based on selected categories and is independent of one another. The guest network could have increased restrictions compared to the private network, etc. But, the user doesn't have to go through the painful process of manually configuring a proxy or jumping through any other hoops. As we all know, even the slightest bit of configuration is over the heads of many, and we want something that works across all devices without any problems.
But, could it go further? With the captive portal, it's possible to associate user names with device IP addresses. So, why not extend this to be applied on a basis of users if need be? New lists defined for each user that would give us user-based web filtering in addition to the "baseline" level associated with each subnet/interface. Even more, squidguard has a pretty well-functioning time function, so we could probably get that implemented as well. Now you can imagine: an Internet Access Schedule that equates to a more "professional" parental-control model; rules based on subnet and/or user accounts themselves.
Implemented, we'd have a comprehensive, no-fuss web filter. Giving administrators the power to control who views what, when, and how. No more trading ease-of-use for complicated initial configurations.
Realities
Firstly, it's not to say that there aren't close alternatives or derivatives. Squid+SquidGuard does amazing when it's explicit, but that requires setup on all the user devices. WPAD, when used, isn't a guaranteed solution. So, we're forced to go transparent. The problem with transparent proxies? We have to implement a MitM SSL Bump to filter any sort of HTTPS traffic and that could be costly. In addition, there's the ethical grey area of getting subordinate root certificates through a trusted CA (I'm looking at you TrustWave….) Similarly, there may again be the requirement of configuring per device as the certificates must be verified, etc. Regardless, SquidGuard does give a lot of hope to the problem. It integrates really nicely with Shallalist and has components for time-based ACLs, and even user-based ACL's. But to get user based rules with the transparent proxy (thinking captive portal), there would need to be a added system to match logged in users and their IP addresses to user-specific ACL's and apply them while also ensuring old unused IP's are discarded.
Next, theres DNS filtering. This, from the side of the end user, works beautifully. Websites are categorized and blocked regardles of the protocol used (HTTP/HTTPS). And lets be honest; does the end user care that theres active content filtering done? Or do they just want to make sure that the ~100 sites they are most likely to visit are being filtered correctly. So, here at least, DNS filtering wins. But it's not without it's faults. Back in our theoretical best case, we have multiple subnets each having their own filtering rules. Even more, we thought of having user based rules as well! How then, could we get DNS filtering rules to be interface/subnet specific? That's a big hitch. OpenDNS lets us point to a WAN gateway....but that's it...one gateway. Configuring your own (https://forum.pfsense.org/index.php?topic=102432.0) you can (kinda) manage to get two separate networks (one connecting to the dns server and the other not). But, it's not perfect, and not nearly robust as we'd like.
The Bounty
Either by means of proxy or dns filtering, develop a web filter to make pfSense not only compete with competitors (ISP parental controls, OpenDNS), but surpass them in terms of functionality.
Requirements
-
No Fuss: We can't expect anything out of the end user. This would have to be a "just works" solution
-
Multiple Networks: If we were dealing with one network, we might as well hop on OpenDNS. Think multiple network, multiple levels of filtering, even VLAN separated wireless networks.
-
Network AND User based: If we're going to go this far, might as well go all out. A solution that would allow user based rules thru the captive portal in addition to baseline network filtering levels
-
HTTP and HTTPS: It's no good if the second the user communicates through HTTPS the filtering goes out the window
-
Time based rules, easy update, etc: Lastly we should think about keeping it updated, how to enforce. Shallalist is great and I think we could work within this. Have the categories be set, and give the option to add websites to a given "blacklist" or "whitelist" wherever shalla messes up.
Existing tools/packages that contain components of the proposed bounty:
-
Squid/SquidGuard: http/https proxy, content filtering, time based filtering, user based filtering, captive portal authentication (if it ever works), transparent proxy (no config), subnet based filtering
-
PfblockerNG/DNSBL: DNS filtering, customizable black/whitelists
-
Shallalist: Comrpehensive categorized domain list, updated regularly
-
OpenDNS: Proof of concept, the standard in DNS and DNS filtering, has time based restrictions and bypass accounts
-
DNS Resolver/Forwarder: necessary components
-
Captive Portal: user login and authentication means. The possibility of user-based filtering within pfSense
-
NXFilter: HUGE tool. Refer to thread: https://forum.pfsense.org/index.php?topic=86923.0 Working installation in pfSense, and the utility has a lot of functions described. Only missing the ability to have different filtering rules based on subnet (by nature, 1 WAN = 1 Filter).
All in all, it's clear that administrators and developers have been looking for this kind of system. Looking at pfSense, the components are available: albeit separated amongst different tools. But if it were working, think of the huge leap forward the system would make in future versions and how large the fanbase could grow. I know plenty of clients/users that just won't configure these web filters because they're too complicated or require outside registration, etc. Even more, there's always some sort of catch or limitation. Creating something a little hearty now would really define the system and give a lot of people the freedom to administer their networks exactly the way they want.
Feel free to add ideas, criticisms, or just bump it for interest. PM's are always welcome and I'm sure this will be picked up quick.
Edit Added thread on NXFilter. Proof of concept installed in pfSense and dns allows for user based and time based rulesets. However, does not address multiple subnets.
Edit 12/23/16 Upped initial contribution
-
-
Try this https://forum.pfsense.org/index.php?topic=112335.0
-
Great post I wish web filtering can be That way. When I get a client that wants filtering especially filtering based on different users/sub and without having to touch the clients devices I will have to use a different firewall, filtering based on user is much better then IP address, and also it would be great if one day captive portal usernames not ip address can be shown in web usage reports example user 1 whent to http://xxxx as right now this only can be done using windows AD with squid
-
Sorry for the delay.
Try this https://forum.pfsense.org/index.php?topic=112335.0
This guide references partial solutions using the DNS resolver, and then using both transparent and explicit proxies to handle the filtering. While it could work, it's a multi-package solution (Unbound + Squid + SquidGuard) and isn't simple by any means.
I'm gonna increase my initial contribution to the bounty.
Also @Chismallia
…and also it would be great if one day captive portal usernames not ip address can be shown in web usage reports example user 1 whent to http://xxxx as right now this only can be done using windows AD with squid
Thanks for the reply, we need as many people as possible in here! Also, with regards to your desired feature: it may not be an official release but I'm working on something along those lines right now so keep your eyes peeled haha
-
Nice be sure to keep us updated :)
-
Relevant Update
Through looking at that old thread I posted about NXfilter, and then talking to some developers, we've gotten two possible solutions. One, exactly what we were talking about. The second….not so much.
The good news
After digging around, it became clear that NXFilter was going to be the clear winner for getting this to work. There's some different installation guides lying around in both english and portuguese. With a working FreeBSD version, I was able to get the software running on our 2.3 test unit. I had to install java manually, among other steps, but it's running without a hitch. Using the NXfilter system, we can:
-
Use Shallalist (or two paid list services) to categorize websites
-
Define filtering rules based on time of day
-
Filter different subnets while having only one WAN address
-
User-Defined Rules: Either actively or passively define user based rules with two different policies (think on/off hours or high/low security hours)
-
Add websites to the blacklist or whitelist them as needed
-
Use the full logging feature of NXfilter to see multiple logs and generate reports based on those logs
-
and probably more? hahahaha
So, this is huge news. I expected there to be some compatibility issues or larger errors when running it on the pfSense unit. However, this is a completely working product. Also, there might be a possibility to pair this with the transparent HTTP content filtering available with Squid+SquidGuard; doubling down by giving you the depth of security with HTTP content filtering in addition to dns filtering for both http and https.
If there's enough of a desire, I could possibly write up some documentation on how to get this working. But, it would be even better if pfSense developers thought about integrating this free, open-source system as a package. This would make pfSense better than Untangle, ipcop, or any free(mium) UTM solution available.
ps. for those asking about the second mentioned method, I thought there might still be some possible solution with squidguard (content filtering thru a transparent proxy). The only means by which you could enable https filtering WITHOUT configuration was an exploit of HSTS by setting up two captive portals: one with a signed certificate and the other with the pfSense CA and cert. Basically, it would install the CA as a trusted CA and avoid the user manually installing it. While it works, it falls too close into an ethical grey area and it would only take one update of the HSTS system and security protocols to render it useless again. If you want simple content filtering, go DNS.
-
-
Hi great it would be great if Pfsense team integrate this some how. So to understand this for nxfilter you have to add your own black list like in squid guard? oh and if you have a guide please share I am interested to try this out thanks for sharing regards
-
+1 for interest in a pfSense package..
-
I wouldn't call pre-installed subordinate wildcard certs a "gray area", that is definitely a black hat type thing to do and don't try to tell yourself that it is OK because you work for a fortune 500 company or something like that.
If they are using company equipment then you can simply install a certificate or have a client side based monitor solution which is entirely legal and ethical otherwise I see no real need for this besides spying on your users.
DNS/IP based filtering is more than good enough for any public AP legitimate use case; by doing this you are selling/using illegal private use surveillance equipment which violates the wiretap act and the stored communications act - you can be prosecuted and sued - it doesn't matter if you had your users agree to a massive click-through EULA that they didn't read any decent lawyer will tell you those are unenforceable.
-
what?? illegal ? once people agree to use my network and In the agreement i write that the network is monitored I have the right to know whats going on if they do not like it they do not accept and leave, if they accept they accepted that I have the right to monitor them, example in win 10 you cant do anything that MS is spying on you, why? cos you agreed. So utm that can monitor should be shutdown by law right? then why are they still around and improving everyday?
-
Dont forget its completly legit for private/home use as well..
-
Sure it is I have a untangle utm at home monitoring my guests , they accept they will be monitored they dont like it? they do not use my network, simple as that, I have all the right to know whats going on in my network, If they do something bad on my network I have the right to know who
-
I wouldn't call pre-installed subordinate wildcard certs a "gray area", that is definitely a black hat type thing to do and don't try to tell yourself that it is OK because you work for a fortune 500 company or something like that.
If they are using company equipment then you can simply install a certificate or have a client side based monitor solution which is entirely legal and ethical otherwise I see no real need for this besides spying on your users.
DNS/IP based filtering is more than good enough for any public AP legitimate use case; by doing this you are selling/using illegal private use surveillance equipment which violates the wiretap act and the stored communications act - you can be prosecuted and sued - it doesn't matter if you had your users agree to a massive click-through EULA that they didn't read any decent lawyer will tell you those are unenforceable.
Look at Trustwave. They happened to see a bit more flak but there wasn't any huge ramifications or press. Regardless, how you monitor your network in a private/enterprise environment is your own prerogative. This is why, at the end of the day, private is private.
BUT
This isn't a thread to discuss ethics, it's to show support for a pfSense bounty to get a better filtering solution.
-
…to show support for a pfSense bounty to get a better filtering solution.
I'm in for $50 if this results in a complete and workable solution that satisfies my needs..
-
This isn't about ethics, this is about not going to jail or being sued.
what?? illegal ? once people agree to use my network and In the agreement i write that the network is monitored I have the right to know whats going on if they do not like it they do not accept and leave, if they accept they accepted that I have the right to monitor them, example in win 10 you cant do anything that MS is spying on you, why? cos you agreed. So utm that can monitor should be shutdown by law right? then why are they still around and improving everyday?
Monitoring a public AP type network you own is entirely legal and ethical this isn't what I am referencing, I am simply letting everyone know that breaking someone elses crypto is illegal.
The way you see things isn't how contract law works in any sane country, if you want to do this you should be talking to a decent lawyer.
-
This is not a thread to break someones crypto, this is support for pfsense to have a good web filtering capability and have guest captive portal reports instead of by ip we can view the reports by username the guest has. So if you do not have anything useful to say please go play Judge Judy some place else
-
+1 for a defininitive web filtering solution
+1 for NXfilter -
+1 for NXfilter
-teddy
