Unofficial E2guardian package for pfSense

pfsensation · Jun 7, 2017, 9:04 PM

I realised the sudden memory spikes too. I had 350MB, didn't worry too much as I assumed it was for cache + I have 2GB total.

I had another crash today after 2 days, it's better than the nearly instant crash every 30 minites - 2 hours I was getting before.

Jun 7 02:56:44	check_reload_status		Syncing firewall
Jun 7 02:56:46	php-fpm	18746	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 02:56:46	php-fpm	19894	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 02:56:48	php-fpm	20300	/status_services.php: The command '/usr/local/etc/rc.d/e2guardian.sh stop' returned exit code '1', the output was 'kern.ipc.somaxconn: 16384 -> 16384 kern.maxfiles: 131072 -> 131072 kern.maxfilesperproc: 104856 -> 104856 kern.threads.max_threads_per_proc: 4096 -> 4096 e2guardian not running? (check /var/run/e2guardian.pid).'
Jun 7 02:56:54	e2guardian	20294	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 02:56:54	e2guardian	19797	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 02:56:55	e2guardian	23430	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 02:56:55	e2guardian	19797	I seem to be running already!
Jun 7 02:56:55	e2guardian	23430	I seem to be running already!
Jun 7 02:56:55	root		/usr/local/etc/rc.d/e2guardian.sh: WARNING: failed to start e2guardian
Jun 7 02:57:52	check_reload_status		Syncing firewall
Jun 7 02:57:52	php-fpm	56014	/pkg_edit.php: [E2guardian] - Save settings package call pr:1 bp: rpc:no
Jun 7 02:57:52	check_reload_status		Syncing firewall
Jun 7 02:57:55	php-fpm	78906	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 02:57:55	php-fpm	80648	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 02:58:00	e2guardian	81133	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 02:58:00	e2guardian	80510	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 02:58:01	e2guardian	80510	I seem to be running already!
Jun 7 03:01:00	root		rc.update_bogons.sh is starting up.
Jun 7 03:01:00	root		rc.update_bogons.sh is sleeping for 31584
Jun 7 03:14:47	check_reload_status		Syncing firewall
Jun 7 03:14:47	php-fpm	9104	/pkg_edit.php: [E2guardian] - Save settings package call pr:1 bp: rpc:no
Jun 7 03:14:47	check_reload_status		Syncing firewall
Jun 7 03:14:49	php-fpm	19850	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 03:14:55	e2guardian	23043	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:24:23	check_reload_status		Syncing firewall
Jun 7 03:24:23	php-fpm	60117	/pkg_edit.php: [E2guardian] - Save settings package call pr:1 bp: rpc:no
Jun 7 03:24:23	check_reload_status		Syncing firewall
Jun 7 03:24:25	php-fpm	11356	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 03:24:25	php-fpm	15238	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 03:24:30	e2guardian	15501	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:24:31	e2guardian	15228	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:24:31	e2guardian	15228	I seem to be running already!
Jun 7 03:24:39	e2guardian	21488	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:31:24	check_reload_status		Syncing firewall
Jun 7 03:31:24	php-fpm	64424	/pkg_edit.php: [E2guardian] - Save settings package call pr:1 bp: rpc:no
Jun 7 03:31:24	check_reload_status		Syncing firewall
Jun 7 03:31:26	php-fpm	41060	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 03:31:32	e2guardian	44437	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:31:46	e2guardian	49314	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 03:31:46	php-fpm	46997	/pkg.php: Restarting e2g by sending -Q action to e2g binaries
Jun 7 03:31:52	e2guardian	49618	Reporting_level is : 0 file /usr/local/etc/e2guardian/e2guardianf3.conf
Jun 7 11:47:24	root		rc.update_bogons.sh is beginning the update cycle.
Jun 7 11:47:26	root		Bogons V4 file downloaded: 15 addresses added.
Jun 7 11:47:26	root		Bogons V4 file downloaded: 16 addresses deleted.
Jun 7 11:47:27	root		Bogons V6 file downloaded: 302 addresses added.
Jun 7 11:47:27	root		Bogons V6 file downloaded: 69 addresses deleted.
Jun 7 11:47:27	root		rc.update_bogons.sh is ending the update cycle.
Jun 7 20:39:37	kernel		pid 49681 (e2guardian), uid 106: exited on signal 11

pfsensation · Jun 10, 2017, 12:32 AM

Any updates on this? Is there anyway I can downgrade for now?

4.1 is giving me a lot of hassle with crashing. Maybe you could make a script to automatically restart it if it crashes Marcello?

marcelloc · Jun 10, 2017, 12:48 AM

I have a watchdog script. I'll share here when I get home.

jetberrocal · Jun 10, 2017, 1:00 AM

@marcelloc:

I have a watchdog script. I'll share here when I get home.

Let say the e2g has the watchdog script working.

From the perspective of the clients, is e2g as of now, doing all the features OK and they will not notice the crashes?

I want to know because if the features are not working OK then I will have to wait for the fixes to be able to use it in production. Also if the crashes can be noticed by the clients then I have to wait for the fixes to be able to use it in production. Maybe the crashes can be tolerated if they are noticed but in sporadic instances.

marcelloc · Jun 10, 2017, 6:52 PM

I guess the current crashes are related to a old exit status 11 that happens for a Long Time since dansguandian but with current thread version, the main process dies too. The issue is related with empty headers and connection that doesn't answer.

marcelloc · Jun 10, 2017, 2:42 AM

whatchdog script for e2guardian

#!/bin/sh
for a in 5 10 15 20 25 30 35 40 45 50 55
do
if [ -f /var/run/e2guardian.pid ];then
 cat /var/run/e2guardian.pid | xargs ps
 if [ $? -ne 0 ]; then
   /usr/local/etc/rc.d/e2guardian.sh start
   tail -50 /var/log/e2guardian/access.log >> /var/log/e2guardian/last.log
   echo "`date` start" >> /var/log/e2guardian/start.log
 fi
fi
sleep 5
done

Run it every minute via cron. /var/log/e2guardian/start.log will log the crashes.

pfsensation · Jun 10, 2017, 1:13 PM

@marcelloc:

whatchdog script for e2guardian

#!/bin/sh
for a in 5 10 15 20 25 30 35 40 45 50 55
do
if [ -f /var/run/e2guardian.pid ];then
 cat /var/run/e2guardian.pid | xargs ps
 if [ $? -ne 0 ]; then
   /usr/local/etc/rc.d/e2guardian.sh start
   tail -50 /var/log/e2guardian/access.log >> /var/log/e2guardian/last.log
   echo "`date` start" >> /var/log/e2guardian/start.log
 fi
fi
sleep 5

Run it every minute via cron. /var/log/e2guardian/start.log will log the crashes.

Just tried setting this up now. When I copy and paste your code into Cron, it becomes all jumbled. Here's an example of what happened :

#!/bin/sh
for a in 5 10 15 20 25 30 35 40 45 50 55
do
if [ -f /var/run/e2guardian.pid ];then
 cat /var/run/e2guardian.pid | xargs ps
 if [ $? -ne 0 ]; then
   /usr/local/etc/rc.d/e2guardian.sh start
   tail -50 /var/log/e2guardian/access.log >> /var/log/e2guardian/last.log
   echo "`date` start" >> /var/log/e2guardian/start.log
 fi
fi
sleep 5

Went from the above code, to the below code

#!/bin/shfor a in 5 10 15 20 25 30 35 40 45 50 55doif [ -f /var/run/e2guardian.pid ];then cat /var/run/e2guardian.pid | xargs ps if [ $? -ne 0 ]; then   /usr/local/etc/rc.d/e2guardian.sh start   tail -50 /var/log/e2guardian/access.log >> /var/log/e2guardian/last.log   echo "`date` start" >> /var/log/e2guardian/start.log fifisleep 5

Will this cause any problems?

And also, since this cron job is running every minute. What if the case is that E2Guardian crashes just after the cron job is run, then you have to wait an entire minute before E2Guardian is put back up. It could be incredibly annoying in a production environment.

pfsensation · Jun 10, 2017, 1:23 PM

@jetberrocal:

@marcelloc:

I have a watchdog script. I'll share here when I get home.

Let say the e2g has the watchdog script working.

From the perspective of the clients, is e2g as of now, doing all the features OK and they will not notice the crashes?

I want to know because if the features are not working OK then I will have to wait for the fixes to be able to use it in production. Also if the crashes can be noticed by the clients then I have to wait for the fixes to be able to use it in production. Maybe the crashes can be tolerated if they are noticed but in sporadic instances.

Pretty sure at some point it will be noticed, and it will become annoying. If the cause is actually an empty header, and someone is accessing a site which is causing the empty headers, you may end up with more crashes at once. And then you have to wait a minute for it to come back etc etc… I use this at home, and I can say that 4.1 is extremely fast, and nice to use in general. But the crashes for now are making it more hassle than it's worth. I'm stuck in between for now, 3.5.1 gave me redirect issues, and sometimes some slowness (not a huge deal) but redirect issue was annoying. But 4.1 is very very quick, fixes the redirect issues and sudden lag but is prone to crashes. And I guess it has some sort of memory leak? I hope the watchdog script at least takes some frustration out. D:

@Marcelloc, I have a NAT rule to redirect port 80 to my pfsense box IP and port 8080. Should I do the same for 443? Or will that completely break it? I don't mind being having to install the certificate, but is there a way to just force all the traffic through E2Guardian. In some cases, such as Android, it seems other methods such as WPAD etc, don't work.

jetberrocal · Jun 10, 2017, 1:23 PM

Pfsensation:

You have to add spaces where they stick together.

Example
55doif -> 55 do if

It seems you lost carriege returns. But whats important is spaces between commands and parameters.

pfsensation · Jun 10, 2017, 1:24 PM

@jetberrocal:

Pfsensation:

You have to add spaces where they stick together.

Example
55doif -> 55 do if

It seems you lost carriege returns. But whats important is spaces between commands and parameters.

Thanks for that. I will go edit the cron now

marcelloc · Jun 10, 2017, 1:35 PM

Create a file on filesystem with the script and then call the file on cron.

jetberrocal · Jun 10, 2017, 1:44 PM

@Marcelloc, I have a NAT rule to redirect port 80 to my pfsense box IP and port 8080. Should I do the same for 443? Or will that completely break it? I don't mind being having to install the certificate, but is there a way to just force all the traffic through E2Guardian. In some cases, such as Android, it seems other methods such as WPAD etc, don't work.

That rule will brake wpad retrival. Did you add that before testing Androids?

pfsensation · Jun 10, 2017, 4:12 PM

@jetberrocal:

@Marcelloc, I have a NAT rule to redirect port 80 to my pfsense box IP and port 8080. Should I do the same for 443? Or will that completely break it? I don't mind being having to install the certificate, but is there a way to just force all the traffic through E2Guardian. In some cases, such as Android, it seems other methods such as WPAD etc, don't work.

That rule will brake wpad retrival. Did you add that before testing Androids?

Hmm how can I get that setup with WPAD still working? Somehow WPAD seems to be working. My pc is set to automatically detect proxy settings and it does, however it didn't work on IOS maybe this is it.

I tested with this rule on android, works for http traffic. It'll successfully go to the block page if the website isn't allowed.

marcelloc · Jun 10, 2017, 4:18 PM

Create the rule and try to fetch the wpad file. If it fetches, the is no conflict.

pfsensation · Jun 10, 2017, 5:21 PM

@marcelloc:

Create the rule and try to fetch the wpad file. If it fetches, the is no conflict.

Able to fetch it no problems, as long as I type http, since web config is on https.

pfsensation · Jun 10, 2017, 11:04 PM

@marcelloc:

Create a file on filesystem with the script and then call the file on cron.

Didn't work at first, but added "Done" at the end of the script. Now it seems to be working fine.

marcelloc · Jun 10, 2017, 11:28 PM

@pfsensation:

@marcelloc:

Create a file on filesystem with the script and then call the file on cron.

Didn't work at first, but added "Done" at the end of the script. Now it seems to be working fine.

Bad copy and paste. Sorry. I've fixed the post

marcelloc · Jun 11, 2017, 5:29 AM

Can you test if a full MITM e2guardian setup with no exceptions crashes? current binaries from package are not on the latest version but it's close to it. If you can confirm a crash, I'LL build with current latest code.

marcelloc · Jun 11, 2017, 5:29 AM

I found an workaround util 4.1.1 gets fixed 8)

Configure squid to intercept SSL connections with splice all mode (this checks only remote certificate)
Configure e2guardian Parent proxy Settings with your squid ssl interface configured.

Testing with steps I know that crashes the daemon but it's still alive with and without MITM.

pfsensation · Jun 11, 2017, 9:11 AM

@marcelloc:

I found an workaround util 4.1.1 gets fixed 8)

Configure squid to intercept SSL connections with splice all mode (this checks only remote certificate)

Configure e2guardian Parent proxy Settings with your squid ssl interface configured.

Testing with steps I know that crashes the daemon but it's still alive with and without MITM.

Won't squid and E2Guardian conflict with each other if I configure squid to intercept traffic? Since it creates a NAT rule and I already have a NAT rule for port 80 > 8080 for E2guardian.
Remember squid is completely unrestricted in terms of allowing Web access without E2 Guardian. I also realised you updated the package, what's new in the newer version?

Got the error below on the latest version available in the repo.

PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 1731, Message: Allowed memory size of 262144000 bytes exhausted (tried to allocate 47625514 bytes)	@ 2017-06-11 11:16:25

Then got this crash report…

					Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p19
FreeBSD 10.3-RELEASE-p19 #0 bbfdb9a1d(RELENG_2_3_4): Wed May  3 16:09:14 CDT 2017     root@ce23-amd64-builder:/builder/pfsense-234/tmp/obj/builder/pfsense-234/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[11-Jun-2017 11:16:25 Europe/London] PHP Fatal error:  Allowed memory size of 262144000 bytes exhausted (tried to allocate 47625514 bytes) in /etc/inc/util.inc on line 1731
[11-Jun-2017 11:16:25 Europe/London] PHP Stack trace:
[11-Jun-2017 11:16:25 Europe/London] PHP   1\. {main}() /usr/local/www/pkg_edit.php:0
[11-Jun-2017 11:16:25 Europe/London] PHP   2\. eval() /usr/local/www/pkg_edit.php:141
[11-Jun-2017 11:16:25 Europe/London] PHP   3\. e2guardian_check_config() /usr/local/www/pkg_edit.php(141) : eval()'d code:1
[11-Jun-2017 11:16:25 Europe/London] PHP   4\. e2guardian_start() /usr/local/pkg/e2guardian.inc:1309
[11-Jun-2017 11:16:25 Europe/London] PHP   5\. mwexec() /usr/local/pkg/e2guardian.inc:1357
[11-Jun-2017 11:16:25 Europe/London] PHP   6\. sprintf() /etc/inc/util.inc:1731

No FreeBSD crash data found.