Content Filtering on CF

MrMoo

Well, interpreting the license allows us to make the package available at least. And we may even include it into pfSense. The problems start when people start installing dansGuardian on site.

Its more misleading than anything. The author is saying you cannot download the software from him for commercial use. You can download for non-commercial use. After downloading for non-commercial use it uses the GPL license which means you can now use it for commercial use. So you can download from Debian's ftp size and have no issues.

Strange.

This thread is for posting a bounty and conversation regarding work being done on that bounty. It is NOT for discussion of various content filtering technologies. Please do not pollute this thread with non-relevant discussion.

For reference, if you have not posted a bounty relevant to the topic, or will not post a bounty relevant to the topic, please take your discussion to Packages.

doc_holiday

@submicron:

This thread is for posting a bounty and conversation regarding work being done on that bounty. It is NOT for discussion of various content filtering technologies. Please do not pollute this thread with non-relevant discussion.

For reference, if you have not posted a bounty relevant to the topic, or will not post a bounty relevant to the topic, please take your discussion to Packages.

I have started a thread for those interested… http://forum.pfsense.org/index.php/topic,3660.0.html

guybrush

ok, let´s help a little.

i´ll add $50,- if this feature is included for embedded devices (in my special case: WRAP).
maybe í´ll be able to give the devs something back…

or you´ll wait until i´ve finished my bachelor degree, so i´ll raise my bounty ;D ;D

cheers,
hannes

DignionASP

I raised the bounty with $175 If Content Filtering is seen in a little easier way.. (CF)

http://forum.pfsense.org/index.php/topic,3660.msg23800.html#msg23800

PFsener

@submicron:

It should be noted that squid+squidGuard (or DansGuardian) are very memory intensive. Since most WRAP and similar embedded devices are limited in the amount of memory available, this is going to be a severe limitation. I've used squid+squidGuard for a number of years now, so believe me when I tell you that putting such a solution onto the pfSense embedded platform is going to be no small feat.

Not all embedded hardware has limited RAM. I have a TNet Pro 1000 with 512 MB RAM that is expandable, but has a 256 MB CF card.

Great to see this thread come to life!

Take care -

PfSener

hoba

If we are talking about features for embedded hardware we have to look at the least powerful system (talking about our official minimum specs) as this means the feature has to go into base (unless we provide another way to install addons on embeddeds). You can run the embedded install on a very powerful machine as well but that's not the point here.

DignionASP

:-\ mmm, we do not make so much progress here.
Who has an idea how to make this more sexy so this thread is not gonna die in silence.
I'm realy interested in a good contentfilter as I wrote earlier ( http://forum.pfsense.org/index.php/topic,3660.msg23800.html#msg23800 ). Also the reaction on that post is worth while thinking off.

Maybe an idea is to change the title from the post that started this thread to.. $1100 for a decent content filter (administrator is this possible ?? )
I know if you count the money that we come up to $975 (all contributors) but I'm willing to raise my share of the bounty to mach the $1100..

Maybe that another problem is that there are to many contributers a develloper might think that it is to hard to get all the money from everybody. Lady's and gentleman devellopers, let's hear your pov 's ..

cmb

The biggest problem with this is it would be very difficult to implement on embedded, which is what you're requiring. If it could be a regular package, it'd probably be done by now, but that would only work on full installs. Since it can't be a package, and it's outside the realm of what we would want to add to the base system, it's unlikely this will get done any time soon.

doc_holiday

After hearing here and then understanding the problems by doing a bit of experimenting with DansGuardian on a Linux box, I think Content Filtering on CF is a bridge too far. I think the bounty needs to be re-tabulated on the basis on a full install as a add-in module. Even as a full install, it is going to require a decent machine to do it, so the minimum specs are going to be higher than what you need for PFSense in its bare bones configuration, otherwise it will be slow.

So, I still stand by my contribution to the bounty for a full install version. I wonder how much support there would be from others on this as well?

Ryan

As i Stated earlier, I am more looking into a PC build and i think a package add on would be ideal in my situation. I posted here because it seemed like the most likely place to post. Maybe we should start a new tread for those who dont mind havin a generic pc build. My bounty still stands for whatever type of Content filtering comes to pass, generic or embedded. I do agree that most embedded hardware would probably be too light for a good content filtering.

candengo

Bump.

Ill add $400 to any bounty for a full install module of any content filtering solutions (dansguard or similar). Ability to block sites by keywords contained on pages etc.

patord

I've been using untangle.com's filtering behind a pfsense box to get content filtering at a certain location. In wanting to consolidate this I started evaluating if squidGuard (1.3beta) on a "full install" pfsense with the squid package is viable. My current testing shows promise.

Before I proceed with trying to make a package that may satisfy a good amount of interested folks:

1. I'd like confirmation of parties interested and their current bounties. This thread is pretty old. I suggest someone start a new "squidGuard" package bounty thread and start posting there since the title of this thread indicates it needs to run on the CF-embedded platform.
2. Your absolute minimum requirements to see if they are feasible.

Here's what I envision for the initial releases:
a. this will be an installable package
b. it will require a "full install" pfsense on a box with very decent resources (RAM/processor/diskspace)
c. it will pull and install any needed packages from FreeBSD ports (eg, BerkeleyDB)
d. it will require the most recent squid package (pfsense) to be installed
e. at a minimum will use the MESD blacklists, and allow user to select which specific sublist to use
f. most allowable squidGuard rules expressions (time/dest/src/acl etc) to be definable via GUI

From what I am seeing this is going to take a fair amount of time and effort to do this one.

cybrsrfr

I've started working on a content filter for PFSense before I noticed the last post. I've looked into Dan's Guard which is GPL for home use and costs for commercial use and Squid Guard which is GPL and filters through URL black lists.

In my mind URL black list is good to an extent but impossible to get all the domains. So with that in mind word content filtering is a must.

My planned method is to use the internal web server already included with PFSense and use its proxy extension pointed to code written in PHP. PHP is a fast scripting language, easy to learn and already included on PFSense so it is ideal for this task.

Benefits to this approach:
1. This method will be able to work on any PFSense system including embedded systems.
2. Filter URLs .
3. Content Filter words.
4. Will work on any PFSense systems including embedded.
5. Point to an internal or an external proxy.
6. Will be licensed under the BSD License.
7. Because the filtering will be done with PHP it will be easy to add additional features.

A basic proof of concept is working on my PFSense system.

dhipo

some toughts

i 've used squid+squidguard modified pfsense on embeded, but restrictions apply … my CF life .. was reduced ... and die ... some googling and the answer was about write times to CF .... it's limited ... and now i am using a hard disk based version

patord

A few updates to my post:

1. I am sending my mdmfs package to cmb (one of the principals of pfsense) for an initial eval. This allows one to make mfs (ram disk-ish) mounts via the GUI. I am using this on a few of my pfsense deployed boxes. This will provide users who want to run on CF the ability to run a full pfsense install on a reasonable sized CF. You then create mfs mounts for heavy writable directories (eg, /var/log and /tmp) to minimize writes to the CF.

As such this package with a few user selectable defaults COULD create mfs mounts for squid and squidguard to reduce heavy CF direct writes.

I am currently running a live pfsense box using this setup (proof of concept) with squid+squidguard using URL blacklists from MESD and some content filtering via regular expressions in the squidguard engine.

All this on a Via c3-800 with 256mb of RAM and a 1gb CF card - no HD. Response times seem decent thus far.

2. one problem I am already forseeing is the time it takes to run "squidGuard -C" on updated lists (db file creation of the blacklist files). This is CPU intensive and would make a low end box unresponsive during that time. I see no freely usable blacklists for squidguard that distribute the DB files already created. If someone knows of one, let me know.

3. I am using squidGuard since it is just GPL with out the lovely complications of the DansGuardian license model.

4. I've already coded a super simple package for squidguard for my core needs. More work is needed obviously to make it usable for anyone else… I'll dump screen shots after some cleanup and more testing. When that happens, that may be some time down the road unless I see more interest.

dhipo

hi patord,

very intersting ,,, i am using squidguard package .. with features created by dvserg on russian forum .. but embeded version is awesome…

if you need some help to test, deploy or something ... let me know ..

an ideia to db files .... create an site with compacted db files and uncompress in boot time on device ...

gthornock

I'll add $50 to the pot for DansGuardian as a transparent proxy on the generic PC install. I don't need it for work, but this is pretty much the last piece I need to get working to use pfSense at home.

cybrsrfr

Content filtering can be done using OpenDNS.
Use the following OpenDNS servers:

208.67.222.222
208.67.220.220

Then sign up for an account at OpenDNS define your network IP so that OpenDNS can identify you and then set what categories of sites you want blocked or add in the domains of your choice. Also you have a dynamic IP you can use DNS-O-Matic that is provided from OpenDNS to keep a track of your IP. So that it stays synched with OpenDNS. For additional security block UDP 53 (DNS) for everything but the OpenDNS servers.

DNS-O-Matic will be available in PFSense 1.3. For those that would like to have it now see:
http://forum.pfsense.org/index.php/topic,7311.msg41445.html#msg41445

dingo

Ok question…. do it have to be squid if I could meet your needs.... Theres better out there then squid for this stuff