Unofficial E2guardian package for pfSense

marcelloc

Pushed 0.4.2.6 to Unofficial repo.

This version includes:

Categories on blacklist files
Fix file permission regression
Check for missing include files selected from default sample e2guardian config files on acl save.
E2guardian binary version 4.1.2

If you have any issues updating, installing or reinstalling, check on console or system logs what errors you are receiving.

if its related to permissions, try

chmod 755 /usr/local/etc/e2guardian/ssl
chmod 755 /var/log/e2guardian

if you get config dirs missing or openssl libs, try to force a binaries package reinstall from console with

pkg install -f e2guardian

Do not forget to apply the blacklist, check config on gui, save and apply.

pfsensation

@marcelloc:

Pushed 0.4.2.6 to Unofficial repo.

This version includes:

Categories on blacklist files

Fix file permission regression

Check for missing include files selected from default sample e2guardian config files on acl save.

E2guardian binary version 4.1.2

If you have any issues updating, installing or reinstalling, check on console or system logs what errors you are receiving.

if its related to permissions, try
chmod 755 /usr/local/etc/e2guardian/ssl
chmod 755 /var/log/e2guardian
if you get config dirs missing or openssl libs, try to force a binaries package reinstall from console with
pkg install -f e2guardian
Do not forget to apply the blacklist, check config on gui, save and apply.

So I just upgraded to the latest 0.4.2.6, via the upgrade button on package manager. Immediately after the upgrade finished I downloaded the black list again, and re applied the configuration. Then I tried going to a blocked site to see it the black list categories showed… To my surprise nope. So I tried completely uninstalling and installing again, then checked the black listed sites again. Still no joy.

Reinstalled again, tried again, no categories shown on black list still... So then I decided to completely stop E2Guardian. So I unchecked the checkmark which says "Enable or disable E2Guardian service", and turned off the watch dog script. To my surprise, E2 Guardian was still running even after re-applying configuration.

This told me that the process wasn't being properly killed. I confirmed this by going to SSH and typing "top". And yep, E2 Guardian was still running when it shouldn't be, I killed it using the process ID.
Enabled E2Guardian from the GUI and enabled watch dog, went to a black listed sites and categories finally showed. However, very unclearly. It says said "porn" it should say Category: Porn clearly but that is a minor thing and can be fixed in the future.

The problem I faced means a couple of things Marcello:

When upgrading old E2 Guardian isn't killed off before carrying out the update, which may lead to some files not correctly updating because they're in use or cause corruption. (Thought you addressed this before?)
Somehow you must make sure that old files are ALWAYS deleted off while retaining configs, by this I mean old binaries and left over junk (even phrase lists should always reinstall) in case of any updates.
Clean installing E2Guardian for the first time vs upgrading shouldn't cause problems. Because on my old install I upgraded from 3.5.1 and all test versions, many things may have become corrupted and I was getting core dumps. After a clean reinstall it seems better however I am worried I'll be in the same situation again. Therefore please make sure that you can do what you can go make sure files, update correctly and aren't getting corrupted. I will support you as much as I can but I need you to understand that by the word "Consistency" I mean old installs upgrading should work just as well as clean fresh installs on a pfsense box which never had E2Guardian before.

I hope I've been clear, and I'd like to clarify that I really want E2Guardian to work and be a real solution to filtering on pfSense, but I hope you also understand that we can't still have basic problems like permission issues. When something is reported and supposedly fixed, it should stay fixed. :p

Despite everything, you deserve thanks and appreciation for all your hard work. I know it's a big project, and I've even contributed some commits to help stop some overblocking yesterday.
I'm willing to help you as much as I can, however could you please try to make sure that consistency is maintained as much as possible. So that people upgrading from older versions of E2 Guardian get the same experience as people installing it fresh (without any problems or having to keep reinstalling).

marcelloc

Just updated on one of the e2guardian installations I have and blacklist category was applied to files after clicking "Re-apply current blacklist" without any other hack or pkg changes.

tail -1 /usr/local/etc/e2guardian/lists/blacklists/*/domains


==> /usr/local/etc/e2guardian/lists/blacklists/adv/domains <==
#listcategory: "adv"

==> /usr/local/etc/e2guardian/lists/blacklists/aggressive/domains <==
#listcategory: "aggressive"

==> /usr/local/etc/e2guardian/lists/blacklists/alcohol/domains <==
#listcategory: "alcohol"

==> /usr/local/etc/e2guardian/lists/blacklists/anonvpn/domains <==
#listcategory: "anonvpn"

==> /usr/local/etc/e2guardian/lists/blacklists/chat/domains <==
#listcategory: "chat"

==> /usr/local/etc/e2guardian/lists/blacklists/costtraps/domains <==
#listcategory: "costtraps"

==> /usr/local/etc/e2guardian/lists/blacklists/dating/domains <==
#listcategory: "dating"

==> /usr/local/etc/e2guardian/lists/blacklists/downloads/domains <==
#listcategory: "downloads"

==> /usr/local/etc/e2guardian/lists/blacklists/drugs/domains <==
#listcategory: "drugs"

==> /usr/local/etc/e2guardian/lists/blacklists/dynamic/domains <==
#listcategory: "dynamic"

==> /usr/local/etc/e2guardian/lists/blacklists/fortunetelling/domains <==
#listcategory: "fortunetelling"

==> /usr/local/etc/e2guardian/lists/blacklists/forum/domains <==
#listcategory: "forum"

==> /usr/local/etc/e2guardian/lists/blacklists/gamble/domains <==
#listcategory: "gamble"

==> /usr/local/etc/e2guardian/lists/blacklists/government/domains <==
#listcategory: "government"

==> /usr/local/etc/e2guardian/lists/blacklists/hacking/domains <==
#listcategory: "hacking"

==> /usr/local/etc/e2guardian/lists/blacklists/homestyle/domains <==
#listcategory: "homestyle"

==> /usr/local/etc/e2guardian/lists/blacklists/hospitals/domains <==
#listcategory: "hospitals"

==> /usr/local/etc/e2guardian/lists/blacklists/imagehosting/domains <==
#listcategory: "imagehosting"

==> /usr/local/etc/e2guardian/lists/blacklists/isp/domains <==
#listcategory: "isp"

==> /usr/local/etc/e2guardian/lists/blacklists/jobsearch/domains <==
#listcategory: "jobsearch"

==> /usr/local/etc/e2guardian/lists/blacklists/library/domains <==
#listcategory: "library"

==> /usr/local/etc/e2guardian/lists/blacklists/military/domains <==
#listcategory: "military"

==> /usr/local/etc/e2guardian/lists/blacklists/models/domains <==
#listcategory: "models"

==> /usr/local/etc/e2guardian/lists/blacklists/movies/domains <==
#listcategory: "movies"

==> /usr/local/etc/e2guardian/lists/blacklists/music/domains <==
#listcategory: "music"

==> /usr/local/etc/e2guardian/lists/blacklists/news/domains <==
#listcategory: "news"

==> /usr/local/etc/e2guardian/lists/blacklists/podcasts/domains <==
#listcategory: "podcasts"

==> /usr/local/etc/e2guardian/lists/blacklists/politics/domains <==
#listcategory: "politics"

==> /usr/local/etc/e2guardian/lists/blacklists/porn/domains <==
#listcategory: "porn"

==> /usr/local/etc/e2guardian/lists/blacklists/radiotv/domains <==
#listcategory: "radiotv"

==> /usr/local/etc/e2guardian/lists/blacklists/redirector/domains <==
#listcategory: "redirector"

==> /usr/local/etc/e2guardian/lists/blacklists/religion/domains <==
#listcategory: "religion"

==> /usr/local/etc/e2guardian/lists/blacklists/remotecontrol/domains <==
#listcategory: "remotecontrol"

==> /usr/local/etc/e2guardian/lists/blacklists/ringtones/domains <==
#listcategory: "ringtones"

==> /usr/local/etc/e2guardian/lists/blacklists/searchengines/domains <==
#listcategory: "searchengines"

==> /usr/local/etc/e2guardian/lists/blacklists/shopping/domains <==
#listcategory: "shopping"

==> /usr/local/etc/e2guardian/lists/blacklists/socialnet/domains <==
#listcategory: "socialnet"

==> /usr/local/etc/e2guardian/lists/blacklists/spyware/domains <==
#listcategory: "spyware"

==> /usr/local/etc/e2guardian/lists/blacklists/tracker/domains <==
#listcategory: "tracker"

==> /usr/local/etc/e2guardian/lists/blacklists/updatesites/domains <==
#listcategory: "updatesites"

==> /usr/local/etc/e2guardian/lists/blacklists/urlshortener/domains <==
#listcategory: "urlshortener"

==> /usr/local/etc/e2guardian/lists/blacklists/violence/domains <==
#listcategory: "violence"

==> /usr/local/etc/e2guardian/lists/blacklists/warez/domains <==
#listcategory: "warez"

==> /usr/local/etc/e2guardian/lists/blacklists/weapons/domains <==
#listcategory: "weapons"

==> /usr/local/etc/e2guardian/lists/blacklists/webmail/domains <==
#listcategory: "webmail"

==> /usr/local/etc/e2guardian/lists/blacklists/webphone/domains <==
#listcategory: "webphone"

==> /usr/local/etc/e2guardian/lists/blacklists/webradio/domains <==
#listcategory: "webradio"

==> /usr/local/etc/e2guardian/lists/blacklists/webtv/domains <==
#listcategory: "webtv"

@pfsensation:

Enabled E2Guardian from the GUI and enabled watch dog, went to a black listed sites and categories finally showed. However, very unclearly. It says said "porn" it should say Category: Porn clearly but that is a minor thing and can be fixed in the future.

I guess it's better to include the 'Category:' on html template.

pfsensation

@marcelloc:

Just updated on one of the e2guardian installations I have and blacklist category was applied to files after clicking "Re-apply current blacklist" without any other hack or pkg changes.

Being applied is one thing, and it actually taking effect when you go to a blacklisted website is another. In my case it seemed like the categories were in fact applied, however, since the E2Guardian daemon wasn't killed and the old daemon was running from memory. I could still visit black listed domains and not get the categories to show.

Therefore, what I am trying to say is, why did I have to manually actually kill E2Guardian via SSH to make it show the categories? Why wasn't unticking the "Enable E2Guardian" check-mark or any of the GUI stuff enough to make E2Guardian process stop? Even reinstallation, and uninstallation of E2Guardian didn't stop it.

@marcelloc:

@pfsensation:

Enabled E2Guardian from the GUI and enabled watch dog, went to a black listed sites and categories finally showed. However, very unclearly. It says said "porn" it should say Category: Porn clearly but that is a minor thing and can be fixed in the future.

I guess it's better to include the 'Category:' on html template.

Oops my bad, I will do that. However, I will push out some commits to your repo to make things a little more professional. :)
By the way, so far everything has been far more stable. I've pushed over 120GB through the proxy without issues so far, hope everything lasts. Even after I upgrade E2Guardian.

If this works well, I will definitely start deploying it out. The only problems I see with E2Guardian now is those small errors with permissions and whatnot, which you say are now fixed.
And the other problem I see is that the phraselists are pretty old, and haven't been updated since years… Maybe those need a little checking up on by E2Guardian devs. I will try bring this to their attention. It works well still, however in some small cases it can overblock, but I've pushed a commit out to official repo which should help resolve those issues with problematic phraselists.

I have two questions for you Marcello:

So as you know, I've fully clean installed my pfSense production box. And upgraded E2Guardian from 12 to 13. If I keep upgrading, do you think you've got things to a state where I won't have random corruption issues? Also the fact that E2Guardian after my upgrade was running, is some cause for concern which shouldn't be dismissed. If E2Guardian is updating, all instances of the old process should be killed, then the upgrade should take place.

The second question I have is:
Could you explain what each sections mean in the E2Guardian widget? And how many HTTP workers is 'enough' and how can I actually tell. Because the requests column sometimes shows over 1k. I don't know if that means all the workers are busy or not. I know there's a 'busy' column but it's a bit daunting and confusing just to look at and understand.

marcelloc

@pfsensation:

Being applied is one thing, and it actually taking effect when you go to a blacklisted website is another. In my case it seemed like the categories were in fact applied, however, since the E2Guardian daemon wasn't killed and the old daemon was running from memory. I could still visit black listed domains and not get the categories to show.

I did not had to kill or restart, just apply config as binaries was already ok to show categories. The missing part was to adapt shallist to have the category line inside it. The documentation says that -Q arg will kill(not kill -9) all current instances and start new one. This is the default options and I'm using it in 4 different networks without strange behaviors.

pfsensation

@marcelloc:

@pfsensation:

Being applied is one thing, and it actually taking effect when you go to a blacklisted website is another. In my case it seemed like the categories were in fact applied, however, since the E2Guardian daemon wasn't killed and the old daemon was running from memory. I could still visit black listed domains and not get the categories to show.

I did not had to kill or restart, just apply config as binaries was already ok to show categories. The missing part was to adapt shallist to have the category line inside it. The documentation says that -Q arg will kill(not kill -9) all current instances and start new one. This is the default options and I'm using it in 4 different networks without strange behaviors.

I think it is safe to say that if people uninstall E2Guardian, they actually want it to stop running. So in that case why did the daemon stay running? -Q didn't seem to be enough to actually apply changes to the blacklist. Which is weird, I had to forcefully kill the client.

Anyhow, everything seems fine for me now. However, I am only bringing it up to stop newer people installing E2Guardian running into these same issues, and to avoid problems when upgrading.

EDIT: It's really amazing to see the categories finally working. However they need a few changes, such as capital letters. Below it should say "Gambling" not "gamble".

PS: Ignore the IP being shown twice. After device it is meant to show hostname, it doesn't seem to work with my PC, but works well with IOS devices for example.

jetberrocal

Marcelloc

Maybe your hardware is faster than the one of pfsensation making the kill fast enough to finish on time before the file replacements. Pfsensation maybe is getting a race condition in his harware and the kill is to slow overlaping some instances with the file replacement.

Can you add a step in rhe update script to make sure all instances were killed or aborted before doing the file replacement? Thus avoiding any race condition no matter the hardware.

pfsensation

@jetberrocal:

Marcelloc

Maybe your hardware is faster than the one of pfsensation making the kill fast enough to finish on time before the file replacements. Pfsensation maybe is getting a race condition in his harware and the kill is to slow overlaping some instances with the file replacement.

Can you add a step in rhe update script to make sure all instances were killed or aborted before doing the file replacement? Thus avoiding any race condition no matter the hardware.

I am suggesting exactly that! For a command when upgrading or uninstalling to actually kill the client and to have it force restarted. As regards to hardware… I am rocking a Dual Core at 3.1GHz, since I am using this at home for a handful of devices. It seems to handle everything just fine, and usage usually stays quite low. Load averages are usually around 0.20 max, unless downloading and caching massive content with Squid caching in realtime which it can go up slightly. I don't exactly know what it is, but I know for sure, E2Guardian must be gone when uninstalled.

marcelloc

I got a report from e2guardian on freebsd11 ( not pfSense) of some zumbie process that keeps running after the stop call.

This is something that will be really hard to identify as e2guardian developers do it on Linux.

I'm started studying c++ but will take a lot of time to ve good enough to debug threads.

Try changing the apply action on daemon tab to use stop and start instead o -G call.

pfsensation

@marcelloc:

I got a report from e2guardian on freebsd11 ( not pfSense) of some zumbie process that keeps running after the stop call.

This is something that will be really hard to identify as e2guardian developers do it on Linux.

I'm started studying c++ but will take a lot of time to ve good enough to debug threads.

Try changing the apply action on daemon tab to use stop and start instead o -G call.

It's already on stop and start by default. However I don't think that's working. For now I think one workaround you could add is a script to run and kill off all E2Guardian processes when "enabled E2Guardian" is unchecked. And also to kill all processes before upgrades. But yeah processing not being killed causes some issues, sometimes configuration updates are not taking effect, despite the stop and start option being selected on apply.

marcelloc

@pfsensation:

It's already on stop and start by default. However I don't think that's working.

How many time did you used the -Q option? This is the option I'm using since this field was added to gui.

It's a demon option to kill all running instances and start new ones.

Try it on console and see what it returns to you.

/usr/local/sbin/e2guardian -Q

techbee

if i may say, this package is not much really that helpful without its installation and setup guide.

marcelloc

@techbee:

if i may say, this package is not much really that helpful without its installation and setup guide.

@marcelloc:

This is a tutorial in Portuguese with basic setup instructions and some screenshots

https://eliasmoraispereira.wordpress.com/2017/06/21/pfsense-proxy-transparente-mitm-no-modo-splice-all-com-squid-e2guardian/

https://github.com/e2guardian/e2guardian/wiki

As almost all pfSense packages, configure it checking tabs and options from left to right.

The package installs a blacklist to make things easier, all you have to do is understand what a content filter do and follow the tabs.

suggestion: start with ip based authentication.

pfsensation

@marcelloc:

@pfsensation:

It's already on stop and start by default. However I don't think that's working.

How many time did you used the -Q option? This is the option I'm using since this field was added to gui.

It's a demon option to kill all running instances and start new ones.

Try it on console and see what it returns to you.

/usr/local/sbin/e2guardian -Q

I'm confused. My apply action is set up "Kill all running copies and start a new one". Isn't this what you're referring to? There's no -Q option in the GUI.

Edit: will e2 guardian work fine on the new pfsense 2.3.1_1? Everything has been stable since I've reinstalled. Don't want to screw it up again without knowing for sure any problems that arise are dealt with before hand.

marcelloc

@pfsensation:

I'm confused. My apply action is set up "Kill all running copies and start a new one". Isn't this what you're referring to? There's no -Q option in the GUI.

Restart by SO rc.d script is the stop and start script you told me you're using
the kill any running copy and start a new one is the -Q arg passed to the e2guradian process that will do all internally
Reload list and groups with a HUP is the -r options that softly restart the damon without killing any active connection.

@pfsensation:

Edit: will e2 guardian work fine on the new pfsense 2.3.1_1? Everything has been stable since I've reinstalled. Don't want to screw it up again without knowing for sure any problems that arise are dealt with before hand.

Did not tested. I'll do always checking service, return erros, file permissions and commands at console. Some time pfSense pkg reinstall the binaries, sometimes not. This is an internal pfSense pkg processes that I cannot change.

If the update keeps the Unofficial repo, then the update may be painless.

![appy option.png](/public/imported_attachments/1/appy option.png)
![appy option.png_thumb](/public/imported_attachments/1/appy option.png_thumb)

pfsensation

@marcelloc:

@pfsensation:

I'm confused. My apply action is set up "Kill all running copies and start a new one". Isn't this what you're referring to? There's no -Q option in the GUI.

Restart by SO rc.d script is the stop and start script you told me you're using
the kill any running copy and start a new one is the -Q arg passed to the e2guradian process that will do all internally
Reload list and groups with a HUP is the -r options that softly restart the damon without killing any active connection.

@pfsensation:

Edit: will e2 guardian work fine on the new pfsense 2.3.1_1? Everything has been stable since I've reinstalled. Don't want to screw it up again without knowing for sure any problems that arise are dealt with before hand.

Did not tested. I'll do always checking service, return erros, file permissions and commands at console. Some time pfSense pkg reinstall the binaries, sometimes not. This is an internal pfSense pkg processes that I cannot change.

If the update keeps the Unofficial repo, then the update may be painless.

I thought you were talking about "kill all running copies and start new ones". Because that is what you've been effectively describing. That phrase. XD

My point is, if I can change configuration on E2 Guardian, without killing the process and have everything working fine. Then I'm happy with that.
The problem I have is those zombie processes running which are able to survive reinstall and upgrades, and they run the old configuration. This creates confusion, conflict and many other issues. Because the zombie process is pretty much in the RAM and isn't being killed.

In regards to the update, I updated pfSense E2 Guardian wasn't started. I applied config and it started, weird the pfSense update didn't touch E2 Guardian files as far as I know. Why did it have to have the config applied again?

I really hope that E2 Guardian continues to run smoothly. The last two days since I reinstalled pfSense and re did my setup. Everything has been very smooth and stable. No unintentional crashes or anything. I've pushed over 150GB through it too lol.

techbee

pfsense wiki has a setup/how to guide for squid and wpad. although not that extensive but it tackled setting up wpad. and in youtube, you can see guide for squid/squidguard and wpad. however, e2guardian do not have public documentation aside from github wiki link you shown.

so it is really hard for novices to follow. yes, setup options are there from left to right but it doesnt say how to really setup this e2guardian with https and http filtering with wpad. it just shows you options to click.

you say e2guardian is a daemon which may work alone without squid or may work also with squid.

its like selling a sophisticated television but no exacting manual how to go with it and your on your own to figure out how to go with it.

marcelloc

@techbee:

its like you sell a sophisticated television but no exacting manual how to go with it.

The gui package helps a lot with e2guardian configuration but you need to know how content filter works, network, proxy and tcp-ip base.

You have the option to install only the e2guardian binaries from freebsd and configure all under console following wiki or any e2guardian step by step configuration for linux. Maybe this way you get more comfortable with the content filtering solution implemented by e2guardian.

This topic is also full of information about the package and configuration scenarios with e2guardian + tinyproxy, e2guardian+ squid, squid + e2guardian + squid, e2guardian + user authentication, e2guardian + ip authentication, config trouble shooting, etc…

techbee

this topic ? so you mean, to know all those information, we need to read this whole topic thread of 36 pages and running and connect the dots by ourselves.

marcelloc

You must be kidding me.