Content Filtering on CF



  • Hello,

    I need a reliable way of using Content Filtering on PFsense. Bounty is $50. $300 if the PFsense team will include it in their embedded release.

    Thanks!



  • What do you mean by content filtering?  I can make assumptions, but I think whoever takes this bounty on would want more details.

    –Bill



  • Squid module on embedded flash

    Thanks!



  • It would be nice to be able to deny/grant access based on AD and LDAP groups. I have this setup on a squid server using winbind. Having a easy to use interface to set this up would be great. But the minimum requirements would be..

    Runs efficiently on CF
    Ability to deny/grant IP's and/or specified users and groups access to certain IP's and domain's.

    Dansguardian funtionality would be really great for this project!

    Thanks!



  • DansGuardian has a paying license. It cannot be used in a commercial context without paying a fee:

    http://dansguardian.org/?page=pricing

    Of course, the package could always be marked as being 'non-free'. DansGuardian is a very worthy addition to squid, but it costs money.

    -G



  • @gbelanger:

    DansGuardian has a paying license. It cannot be used in a commercial context without paying a fee:

    http://dansguardian.org/?page=pricing

    Of course, the package could always be marked as being 'non-free'. DansGuardian is a very worthy addition to squid, but it costs money.

    -G

    Actually, DansGuardian is licensed under GPL.  See bottom of this page: http://dansguardian.org/?page=copyright2



  • @mrsense:

    Actually, DansGuardian is licensed under GPL.  See bottom of this page: http://dansguardian.org/?page=copyright2

    Apparently you didn't read your own link:

    DansGuardian 2 is:

    **    * free for non-commercial use**
        * not free for installation by 3rd parties charging for installation or support
    **    * not free for commercial use**
        * licensed under the GPL
        * copyright Daniel Barron
        * is a registered trade mark of Daniel Barron



  • Nice conflicting license.  No thanks.



  • Well, interpreting the license allows us to make the package available at least. And we may even include it into pfSense. The problems start when people start installing dansGuardian on site.

    I want to make it part of a unix-like distribution such as RedHat.
    Yes.
    I want to try it out for potential commercial use.
    Yes, but only once.
    I want to use it commercially[2].
    No, you must buy a download licence.
    I want to incorporate it into our product or solution.
    No, you must buy a solution provider download licence.



  • On that note, we have a squid package that can do transparent proxy well now. However we are currently missing a no-cache option iirc.
    And we also have a dansguardian package available from the menu. Although this is currently not tested.



  • How about something with not-so-hinky of a license like http://www.squidguard.org/ ?  I have no experience with it but I know some folks that swear by it.



  • I am on board with the original poster.
    I will contribute $150.00 USD for Squidguard or dansguardian in the embedded release.
    That is the last thing that I was looking for in a firewall/router for light office and home use.  I was using IPCop (WRAPCOP) for this until I found pfSense and found it to be a better solution other than filtering.



  • Very well, I'll need to see what can be done, from the looks of it we need squidguard to make proper filtering work.

    However, with regards to a embedded release I assume you have a large(r) cf to account for the extra size squid and friends bring with them. This would require at least a 256MB CF if you still want to be able to make it firmware upgradable on the CF.

    Also, the squidgard database files need a proper spot where they can live.
    It's not impossible, just hard(er).

    I'll see if a can conjur a working configuration on a normal install first. I have no ETA at this moment but I need a month at least.



  • I just want to chime in and say I'm very interested in a web filter that has a list of categories and block list downloads etc to block certain stuff. It's just for my home network, not commercial at all. I'd like to be able to do filtering by clicking on a category to block such as hate, violence or whatever and be able to use keywords too.



  • The current blacklisting and whitelisting in the squid package version p8 and later should work.



  • I'll contribute $50 to the bounty for a content filtering feature similar to Dans Guardian.



  • It should be noted that squid+squidGuard (or DansGuardian) are very memory intensive.  Since most WRAP and similar embedded devices are limited in the amount of memory available, this is going to be a severe limitation.  I've used squid+squidGuard for a number of years now, so believe me when I tell you that putting such a solution onto the pfSense embedded platform is going to be no small feat.



  • You say embedded, but I gather you refer to something with a harddisk in it like a nexcom.

    A soekris 4801-60 with squid and squidguard might work. But all bets are off.
    A nexcom with 1 GB of ram and disk should be fine. Consider a normal install on a disk (with swap) or CF (without swap) and enable the serial console from the menu. Then transplant the disk into your "embedded" box.

    It also allows you to install packages.



  • @buggs1a:

    I just want to chime in and say I'm very interested in a web filter that has a list of categories and block list downloads etc to block certain stuff. It's just for my home network, not commercial at all. I'd like to be able to do filtering by clicking on a category to block such as hate, violence or whatever and be able to use keywords too.

    This section of the forum is for people contributing money.

    We understand you are interested in this but with all due respect stop posting in the bounty area!



  • Having just deployed my first PFsense box and I really like what I see!  I'd be happy to increase my contribution to the bounty to $100 for a Dan's Guardian type content filtering system for pfsense.  I need to buy a commerical product anyhow for the kiddies and I would rather plow it into a great cause like pfsense.



  • I would contribute $200 to anyone thats gets a dansguardian (or equivilent) content filtering working on PFsense.  I am really only interested in a Generic PC build, so embedded or not is of no concern to me.  I am mainly interested in 3 things.  1: Blocking adult websites.  2: Blocking websites based on custom word search (Searching html for the word "AOL", "garbage" ect) 3: Making it fairly easy for an idiot like me to use.  I mean a stright forward web configurator.

    As a side note.  I would also be willing to pay more to have the ability to email an administrator with a report of users who attempt to go to blocked sites more than x number of times a day.  I really don't even care if this would have to run on another machine.  I will pay at least $50 for this additional feature.  More if i have the money.

    PFsense is a great product.  Lets work together to make it even better.



  • @Ryan:

    I would contribute $200 to anyone thats gets a dansguardian (or equivilent) content filtering working on PFsense.  I am really only interested in a Generic PC build, so embedded or not is of no concern to me.  I am mainly interested in 3 things.  1: Blocking adult websites.  2: Blocking websites based on custom word search (Searching html for the word "AOL", "garbage" ect) 3: Making it fairly easy for an idiot like me to use.  I mean a stright forward web configurator.

    As a side note.  I would also be willing to pay more to have the ability to email an administrator with a report of users who attempt to go to blocked sites more than x number of times a day.  I really don't even care if this would have to run on another machine.  I will pay at least $50 for this additional feature.  More if i have the money.

    PFsense is a great product.  Lets work together to make it even better.

    I should have said that I am only interested in a Generic PC Build as well, especially if this would simplify the deployment.



  • @databeestje:

    Well, interpreting the license allows us to make the package available at least. And we may even include it into pfSense. The problems start when people start installing dansGuardian on site.

    Its more misleading than anything.  The author is saying you cannot download the software from him for commercial use.  You can download for non-commercial use.  After downloading for non-commercial use it uses the GPL license which means you can now use it for commercial use.  So you can download from Debian's ftp size and have no issues.

    Strange.



  • This thread is for posting a bounty and conversation regarding work being done on that bounty.  It is NOT for discussion of various content filtering technologies.  Please do not pollute this thread with non-relevant discussion.

    For reference, if you have not posted a bounty relevant to the topic, or will not post a bounty relevant to the topic, please take your discussion to Packages.



  • @submicron:

    This thread is for posting a bounty and conversation regarding work being done on that bounty.  It is NOT for discussion of various content filtering technologies.  Please do not pollute this thread with non-relevant discussion.

    For reference, if you have not posted a bounty relevant to the topic, or will not post a bounty relevant to the topic, please take your discussion to Packages.

    I have started a thread for those interested… http://forum.pfsense.org/index.php/topic,3660.0.html



  • ok, let´s help a little.

    i´ll add $50,- if this feature is included for embedded devices (in my special case: WRAP).
    maybe í´ll be able to give the devs something back…

    or you´ll wait until i´ve finished my bachelor degree, so i´ll raise my bounty  ;D ;D

    cheers,
    hannes



  • I raised the bounty with $175 If Content Filtering is seen in a little easier way.. (CF)

    http://forum.pfsense.org/index.php/topic,3660.msg23800.html#msg23800



  • @submicron:

    It should be noted that squid+squidGuard (or DansGuardian) are very memory intensive.  Since most WRAP and similar embedded devices are limited in the amount of memory available, this is going to be a severe limitation.  I've used squid+squidGuard for a number of years now, so believe me when I tell you that putting such a solution onto the pfSense embedded platform is going to be no small feat.

    Not all embedded hardware has limited RAM. I have a TNet Pro 1000 with 512 MB RAM that is expandable, but has a 256 MB CF card.

    Great to see this thread come to life!

    Take care -

    PfSener



  • If we  are talking about features for embedded hardware we have to look at the least powerful system (talking about our official minimum specs) as this means the feature has to go into base (unless we provide another way to install addons on embeddeds). You can run the embedded install on a very powerful machine as well but that's not the point here.



  • :-\ mmm, we do not make so much progress here.
    Who has an idea how to make this more sexy so this thread is not gonna die in silence.
    I'm realy interested in a good contentfilter as I wrote earlier ( http://forum.pfsense.org/index.php/topic,3660.msg23800.html#msg23800 ). Also the reaction on that post is worth while thinking off.

    Maybe an idea is to change the title from the post that started this thread to.. $1100 for a decent content filter (administrator is this possible ?? )
    I know if you count the money that we come up to $975 (all contributors)  but I'm willing to raise my share of the bounty to mach the $1100..

    Maybe that another problem is that there are to many contributers a develloper might think that it is to hard to get all the money from everybody.  Lady's and gentleman devellopers, let's hear your  pov 's ..



  • The biggest problem with this is it would be very difficult to implement on embedded, which is what you're requiring. If it could be a regular package, it'd probably be done by now, but that would only work on full installs. Since it can't be a package, and it's outside the realm of what we would want to add to the base system, it's unlikely this will get done any time soon.



  • After hearing here and then understanding the problems by doing a bit of experimenting with DansGuardian on a Linux box, I think Content Filtering on CF is a bridge too far. I think the bounty needs to be re-tabulated on the basis on a full install as a add-in module.  Even as a full install, it is going to require a decent machine to do it, so the minimum specs are going to be higher than what you need for PFSense in its bare bones configuration, otherwise it will be slow.

    So, I still stand by my contribution to the bounty for a full install version. I wonder how much support there would be from others on this as well?



  • As i Stated earlier, I am more looking into a PC build and i think a package add on would be ideal in my situation.  I posted here because it seemed like the most likely place to post.  Maybe we should start a new tread for those who dont mind havin a generic pc build.  My bounty still stands for whatever type of Content filtering comes to pass, generic or embedded. I do agree that most embedded hardware would probably be too light for a good content filtering.



  • Bump.

    Ill add $400 to any bounty for a full install module of any content filtering solutions (dansguard or similar).  Ability to block sites by keywords contained on pages etc.



  • I've been using untangle.com's filtering behind a pfsense box to get content filtering at a certain location.  In wanting to consolidate this I started evaluating if squidGuard (1.3beta) on a "full install" pfsense with the squid package is viable.  My current testing shows promise.

    Before I proceed with trying to make a package that may satisfy a good amount of interested folks:

    1. I'd like confirmation of parties interested and their current bounties.  This thread is pretty old.  I suggest someone start a new "squidGuard" package bounty thread and start posting there since the title of this thread indicates it needs to run on the CF-embedded platform.
    2. Your absolute minimum requirements to see if they are feasible.

    Here's what I envision for the initial releases:
    a. this will be an installable package
    b. it will require a "full install" pfsense on a box with very decent resources (RAM/processor/diskspace)
    c. it will pull and install any needed packages from FreeBSD ports (eg, BerkeleyDB)
    d. it will require the most recent squid package (pfsense) to be installed
    e. at a minimum will use the MESD blacklists, and allow user to select which specific sublist to use
    f. most allowable squidGuard rules expressions (time/dest/src/acl etc) to be definable via GUI

    From what I am seeing this is going to take a fair amount of time and effort to do this one.



  • I've started working on a content filter for PFSense before I noticed the last post. I've looked into Dan's Guard which is GPL for home use and costs for commercial use and Squid Guard which is GPL and filters through URL black lists.

    In my mind URL black list is good to an extent but impossible to get all the domains. So with that in mind word content filtering is a must.

    My planned method is to use the internal web server already included with PFSense and use its proxy extension pointed to code written in PHP. PHP is a fast scripting language, easy to learn and already included on PFSense so it is ideal for this task.

    Benefits to this approach:
    1. This method will be able to work on any PFSense system including embedded systems.
    2. Filter URLs .
    3. Content Filter words.
    4. Will work on any PFSense systems including embedded.
    5. Point to an internal or an external proxy.
    6. Will be licensed under the BSD License.
    7. Because the filtering will be done with PHP it will be easy to add additional features.

    A basic proof of concept is working on my PFSense system.



  • some toughts

    i 've used squid+squidguard modified pfsense on embeded, but restrictions apply … my CF life .. was reduced ... and die ... some googling and the answer was about write times to CF .... it's limited ... and now i am using a hard disk based version



  • A few updates to my post:

    1. I am sending my mdmfs package to cmb (one of the principals of pfsense) for an initial eval.  This allows one to make mfs (ram disk-ish) mounts via the GUI.  I am using this on a few of my pfsense deployed boxes.  This will provide users who want to run on CF the ability to run a full pfsense install on a reasonable sized CF.  You then create mfs mounts for heavy writable directories (eg, /var/log and /tmp) to minimize writes to the CF.

    As such this package with a few user selectable defaults COULD create mfs mounts for squid and squidguard to reduce heavy CF direct writes.

    I am currently running a live pfsense box using this setup (proof of concept) with squid+squidguard using URL blacklists from MESD and some content filtering via regular expressions in the squidguard engine.

    All this on a Via c3-800 with 256mb of RAM and a 1gb CF card - no HD.  Response times seem decent thus far.

    2. one problem I am already forseeing is the time it takes to run "squidGuard -C" on updated lists (db file creation of the blacklist files).  This is CPU intensive and would make a low end box unresponsive during that time.  I see no freely usable blacklists for squidguard that distribute the DB files already created.  If someone knows of one, let me know.

    3. I am using squidGuard since it is just GPL with out the lovely complications of the DansGuardian license model.

    4. I've already coded a super simple package for squidguard for my core needs.  More work is needed obviously to make it usable for anyone else…  I'll dump screen shots after some cleanup and more testing.  When that happens, that may be some time down the road unless I see more interest.



  • hi patord,

    very intersting ,,, i am using squidguard package .. with features created by dvserg on russian forum .. but embeded version is awesome…

    if you need some help to test, deploy or something ... let me know ..

    an ideia to db files .... create an site with compacted db files and uncompress in boot time on device ...



  • I'll add $50 to the pot for DansGuardian as a transparent proxy on the generic PC install.  I don't need it for work, but this is pretty much the last piece I need to get working to use pfSense at home.


Locked