Unofficial E2guardian package for pfSense

pfsensation · Jul 22, 2017, 8:21 PM

@techbee:

its hard playing something that gets broken and you need to start from scratch again. everytime, it does not work and seems broken, you need to reformat and reinstall pfsense and such waste of time while the maker most probably knows the right way putting into clear and exacting documentation.

If you don't know what you're doing, then it's probably what you will need to do with everything you try to play without a minimum knowledge.

Hope you improve your IT skills and spend more energy helping collaborative opensource projects than complaining, blaming, depreciating or whatever you're trying to do here.

Marcello dropped the fire xD

Techbee, even I get frustrated and complain sometimes. However I contribute to the project as much as possible and the source code, even if it's small things I discover or big overhauls like the more modernised friendly block page. If you have issues at least post some logs or something so we can understand the problem better. But first of all I recommend you do a clean fresh pfsense install, this helped me massively even though it's annoying. Backup your settings and restore only rules, don't restore packages. And clean install E2 Guardian.

PS: @Marcello A certain closed source inferior alternative to E2 Guardian on pfSense is taking a hit due to E2 Guardian being free and open source and not requiring a subscription being charged at ultra stupid levels (not gonna name and shame). You should be happy lol xD

pfsensation · Jul 22, 2017, 11:42 PM

Is it just me or after around 2 days E2 Guardian goes haywire with RAM usage. This problem is certainly new, usage for E2 Guardian becomes really high and RAM gets swapped. This is the second time it's happened to me, reinstall fixes it for another two days.

Weird… Especially since now there's only one user, there should be nothing really using up that much RAM. Seems like there could be a memory leak somewhere.

marcelloc · Jul 22, 2017, 11:47 PM

Report it on e2guardian git issues. I'm using in the package the latest code from 4.1.2

Does a restart om the service is enough to free some memory?

pfsensation · Jul 23, 2017, 12:04 AM

@marcelloc:

Report it on e2guardian git issues. I'm using in the package the latest code from 4.1.2

Does a restart om the service is enough to free some memory?

Yep a restart pretty much instantly clears the RAM. So I guess it's a source issue? E2 Guardian ever only used 300mb ish RAM now it's going past 2gb and using swap lol. However restarting doesn't completely clean the swap. It's gone from 7% swap to 3%. If I leave the problem to develop it'll end up eating all the swap.

marcelloc · Jul 23, 2017, 12:06 AM

Squis was doing same thing. There is a contribution script that restarts it on 95% memory usage. With minor changes, it can be applied to e2guardian.

https://forum.pfsense.org/index.php?topic=126309.msg727239#msg727239

pfsensation · Jul 23, 2017, 12:09 AM

@marcelloc:

Squis was doing same thing. There is a contribution script that restarts it on 95% memory usage. With minor changes, it can be applied to e2guardian.

Shouldn't the issue be fixed instead? Especially if there is a memory leak, instead of finding workarounds?

Could you report it on Github instead? I'm currently on my phone. XD

Thanks Bud

marcelloc · Jul 23, 2017, 12:12 AM

@pfsensation:

I'm currently on my phone. XD

Me too.

I'll check my boxes. I have one stable at 2g on previous binaries. Will how it goes with 4.1.2

pfsensation · Jul 23, 2017, 12:23 AM

@marcelloc:

@pfsensation:

I'm currently on my phone. XD

Me too.

I'll check my boxes. I have one stable at 2g on previous binaries. Will how it goes with 4.1.2

Don't update for now, I guess until this is solved. Eventually I guess it'll cause the system to crash.

How many users does it have with 2gb RAM usage? With 15 or so devices at home usually I have around 300mb usage, now it's going well over 4gb. If you can't report it then I'll have to tomorrow, I have to edit my pull request anyways to get some fixes added to V5. I've been slightly busy recently.

By the way, what kind of organisation are you running your E2 Guardian box in? Is it a school? Have you realised over blocking? I pushed out some commits to disable alpha stage phrase lists.

techbee · Jul 23, 2017, 12:54 AM

ofcourse it a complain when you purchase something without manual and able to break it doing it and going over again.

its like you burn down your house because it does not say that the device will explode above 110 voltage. and your happy burning down your house then build it to burn again. lol

pfsensation · Jul 23, 2017, 12:59 AM

@techbee:

ofcourse it a complain when you purchase something without manual and able to break it doing it and going over again.

its like you burn down your house because it does not say that the device will explode above 110 voltage. and your happy burning down your house then build it to burn again. lol

First of all, you didn't pay for E2 Guardian.

Second of all, stop using those examples, it really doesn't work. Are you trying to say that you need a manual for everything to learn how it works? Like I said, if you want something that works and works really well, check out Smoothwall. It's a commercial product that works similar to this but nearly all the work has been taken out. That will be something you purchase and it comes with a manual and support. I'm not even trying to be rude to you here, I'm being serious.

However, if you decide to continue to use E2 Guardian, take 3 deep breaths. And explain the issue the best you can with some logs ideally. (Please don't tell me you need a manual for that too.)

pfsensation · Jul 24, 2017, 7:18 AM

Hi Marcello,

I reported the issue to E2 Guardian devs : https://github.com/e2guardian/e2guardian/issues/266

Could you please provide your input as you made the package and know exactly what changed since the last update?

Thanks

marcelloc · Jul 24, 2017, 5:11 PM

@pfsensation:

Hi Marcello,

I reported the issue to E2 Guardian devs : https://github.com/e2guardian/e2guardian/issues/266

Could you please provide your input as you made the package and know exactly what changed since the last update?

Thanks

I'm trying to reproduce de issue first on my boxes. try to include on git issue a screenshot of top showing e2guardian memory usage.

pfsensation · Jul 28, 2017, 7:36 AM

That issue caused my pfsense box to restart (after nearly a week) … Don't you get this issue Marcelloc?

marcelloc · Aug 1, 2017, 3:47 PM

@pfsensation:

That issue caused my pfsense box to restart (after nearly a week) … Don't you get this issue Marcelloc?

I'm really busy these days on my main job. I'll test it since I have some time.

Don't you think it's better to restart the process every two days to avoid these memory's problems until a new binary version?

pfsensation · Aug 3, 2017, 8:39 PM

@marcelloc:

@pfsensation:

That issue caused my pfsense box to restart (after nearly a week) … Don't you get this issue Marcelloc?

I'm really busy these days on my main job. I'll test it since I have some time.

Don't you think it's better to restart the process every two days to avoid these memory's problems until a new binary version?

Yes I do, but it may also be a log rotation issue. In some cases restarting the process doesn't give the allocated RAM back (from e2g). Clearing the logs seem to help quite significantly. Maybe you should try adding a access.log rotation every 24 hours until the issue is fixed?

It seems this is not even a BSD issue, as people on Ubuntu are also noticing high RAM usage.

Regarding work… I'm on the same boat. But this is annoying, whenever I am home I need to keep checking my pfSense, I absolutely hate logging in and seeing that it crashed...:/

empbilly · Aug 12, 2017, 4:46 PM

I am also using the e2guardian and in a matter of hours the "memory ram" and swap are skyrocketing. I modified the script that marcelloc suggested in some posts above and when it restarts the service, for example 90% and 75% respectively, the values fall to 50% for memory and normalize the swap.

pfsensation · Aug 12, 2017, 4:55 PM

@empbilly:

I am also using the e2guardian and in a matter of hours the "memory ram" and swap are skyrocketing. I modified the script that marcelloc suggested in some posts above and when it restarts the service, for example 90% and 75% respectively, the values fall to 50% for memory and normalize the swap.

Can you share your script? This is driving me insane now…

empbilly · Aug 12, 2017, 6:46 PM

Script by remzej
Original: https://forum.pfsense.org/index.php?topic=126309.msg727239#msg727239

Modified:

#!/usr/local/bin/php-cgi -q
/*
 * monitor_memory_usage.php
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2011-2015 Rubicon Communications, LLC (Netgate)
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
require_once('config.inc');
require_once('util.inc');
require_once('squid.inc');
require_once('e2guardian.inc');
global $config;

	// Monitor memory usage by remzej
	// Get SWAP usage funtion
	function swap_usage() {
		exec("/usr/sbin/swapinfo", $swap_info);
		$swap_used = "";
		foreach ($swap_info as $line) {
			if (preg_match('/(\d+)%$/', $line, $matches)) {
				$swap_used = $matches[1];
				break;
			}
		}
		return $swap_used;
	}

	// Get memory usage function
	function mem_usage() {
		$memory = "";
		exec("/sbin/sysctl -n vm.stats.vm.v_page_count vm.stats.vm.v_inactive_count " .
			"vm.stats.vm.v_cache_count vm.stats.vm.v_free_count", $memory);

		$totalMem = $memory[0];
		$availMem = $memory[1] + $memory[2] + $memory[3];
		$usedMem = $totalMem - $availMem;
		$memUsage = round(($usedMem * 100) / $totalMem, 0);

		return $memUsage;
	}

	// Get memory and SWAP usage value
	$memusage_pct = mem_usage();
	$swapusage_pct = swap_usage();

	// Display memory usage
	echo "Memory Usage: " . $memusage_pct . "%" . PHP_EOL;
	echo "SWAP Usage: " . $swapusage_pct . "%" . PHP_EOL;

	// If memory usage is above 90% and SWAP usage is above 75%, stop and restart squid services.  
	if ((($memusage_pct > 70) && ($swapusage_pct > 50)) || ($memusage_pct > 55) ) {
		// stop e2g service
		exec("/usr/local/sbin/e2guardian stop");
		// stop squid service
		squid_stop_monitor();
		if (is_service_running('squid')) {
			stop_service("squid");
		}
                // start e2g service
		exec("/usr/local/sbin/e2guardian start");
		// start squid service
                squid_restart_services();
		log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent and SWAP usage is $swapusage_pct percent, stopping and restarting services.")));
	}
?>

pfsensation · Aug 12, 2017, 9:40 PM

@empbilly:

Script by remzej
Original: https://forum.pfsense.org/index.php?topic=126309.msg727239#msg727239

Modified:

#!/usr/local/bin/php-cgi -q
/*
 * monitor_memory_usage.php
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2011-2015 Rubicon Communications, LLC (Netgate)
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
require_once('config.inc');
require_once('util.inc');
require_once('squid.inc');
require_once('e2guardian.inc');
global $config;

	// Monitor memory usage by remzej
	// Get SWAP usage funtion
	function swap_usage() {
		exec("/usr/sbin/swapinfo", $swap_info);
		$swap_used = "";
		foreach ($swap_info as $line) {
			if (preg_match('/(\d+)%$/', $line, $matches)) {
				$swap_used = $matches[1];
				break;
			}
		}
		return $swap_used;
	}

	// Get memory usage function
	function mem_usage() {
		$memory = "";
		exec("/sbin/sysctl -n vm.stats.vm.v_page_count vm.stats.vm.v_inactive_count " .
			"vm.stats.vm.v_cache_count vm.stats.vm.v_free_count", $memory);

		$totalMem = $memory[0];
		$availMem = $memory[1] + $memory[2] + $memory[3];
		$usedMem = $totalMem - $availMem;
		$memUsage = round(($usedMem * 100) / $totalMem, 0);

		return $memUsage;
	}
	
	// Get memory and SWAP usage value
	$memusage_pct = mem_usage();
	$swapusage_pct = swap_usage();

	// Display memory usage
	echo "Memory Usage: " . $memusage_pct . "%" . PHP_EOL;
	echo "SWAP Usage: " . $swapusage_pct . "%" . PHP_EOL;

	// If memory usage is above 90% and SWAP usage is above 75%, stop and restart squid services.  
	if ((($memusage_pct > 70) && ($swapusage_pct > 50)) || ($memusage_pct > 55) ) {
		// stop e2g service
		exec("/usr/local/sbin/e2guardian stop");
		// stop squid service
		squid_stop_monitor();
		if (is_service_running('squid')) {
			stop_service("squid");
		}
                // start e2g service
		exec("/usr/local/sbin/e2guardian start");
		// start squid service
                squid_restart_services();
		log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent and SWAP usage is $swapusage_pct percent, stopping and restarting services.")));
	}
?>

Thanks! I see you left the code for Squid there too, is there really a need to kill off squid too? From what I've seen, restarting E2 Guardian works just fine.

empbilly · Aug 12, 2017, 10:14 PM

@pfsensation:

@empbilly:

Script by remzej
Original: https://forum.pfsense.org/index.php?topic=126309.msg727239#msg727239

Modified:

#!/usr/local/bin/php-cgi -q
/*
 * monitor_memory_usage.php
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2011-2015 Rubicon Communications, LLC (Netgate)
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
require_once('config.inc');
require_once('util.inc');
require_once('squid.inc');
require_once('e2guardian.inc');
global $config;

	// Monitor memory usage by remzej
	// Get SWAP usage funtion
	function swap_usage() {
		exec("/usr/sbin/swapinfo", $swap_info);
		$swap_used = "";
		foreach ($swap_info as $line) {
			if (preg_match('/(\d+)%$/', $line, $matches)) {
				$swap_used = $matches[1];
				break;
			}
		}
		return $swap_used;
	}

	// Get memory usage function
	function mem_usage() {
		$memory = "";
		exec("/sbin/sysctl -n vm.stats.vm.v_page_count vm.stats.vm.v_inactive_count " .
			"vm.stats.vm.v_cache_count vm.stats.vm.v_free_count", $memory);

		$totalMem = $memory[0];
		$availMem = $memory[1] + $memory[2] + $memory[3];
		$usedMem = $totalMem - $availMem;
		$memUsage = round(($usedMem * 100) / $totalMem, 0);

		return $memUsage;
	}
	
	// Get memory and SWAP usage value
	$memusage_pct = mem_usage();
	$swapusage_pct = swap_usage();

	// Display memory usage
	echo "Memory Usage: " . $memusage_pct . "%" . PHP_EOL;
	echo "SWAP Usage: " . $swapusage_pct . "%" . PHP_EOL;

	// If memory usage is above 90% and SWAP usage is above 75%, stop and restart squid services.  
	if ((($memusage_pct > 70) && ($swapusage_pct > 50)) || ($memusage_pct > 55) ) {
		// stop e2g service
		exec("/usr/local/sbin/e2guardian stop");
		// stop squid service
		squid_stop_monitor();
		if (is_service_running('squid')) {
			stop_service("squid");
		}
                // start e2g service
		exec("/usr/local/sbin/e2guardian start");
		// start squid service
                squid_restart_services();
		log_error(gettext(sprintf("[squid] Memory usage is $memusage_pct percent and SWAP usage is $swapusage_pct percent, stopping and restarting services.")));
	}
?>

Thanks! I see you left the code for Squid there too, is there really a need to kill off squid too? From what I've seen, restarting E2 Guardian works just fine.

So, I ended up just adding the e2g block. :D