PfBlocker
-
Are you sure you are not reading an old alert?
BTW, if a really large value does not exaust memory, It's an excelent option to prevent errors.
Yes, I was making sure they where not old. I have 4GB in this box. Current memory usage:
Mem: 603M Active, 67M Inact, 1151M Wired, 1048K Cache, 406M Buf, 1991M Free
Swap: 8192M Total, 8192M FreeWhen I initially click on save, it all works fine for a few minutes[~2]. Then after a random amount of time, clicking different menus to check the settings the error pops back up. At first after using 9999999999999999; I was still getting the error. It has now been ~19 minutes since the last error message. Will keep my on it too see if the error comes back. If the error comes back, I will try deleting the offending list and re adding it, to see if that helps.
Thanks for the help!
-
Check on widget if were changes on amount of cidrs applied.
CIDRs are get from applied file and package count from firewall tables.
-
Check on widget if were changes on amount of cidrs applied.
CIDRs are get from applied file and package count from firewall tables.
IP-Blocklist injects a large number like that to /tmp/rules.debug at the top to ensure no matter how many lists a user has there won't be any errors. So with pfBlocker, even though it's a manual process, I add that large number.
-
The CIDRs are the same as in the screenshot I posted above:
pfBlockerBluetack 761470
pfBlockerTBG 1262743It has been ~35 minutes and no new errors :) I'm crossing my fingers ;)
-
I need to improve lists size check to reduce this kind of issues or change error message suggesting tommyboy180 large configuration.
It's in my 'todo list when time permits' ;)
-
Needed to restart Squid and the got an alert with the error.
EDIT: Every time Squid is restarted is when the error occurs. I was messing with Squid yesterday, but did not put it together until now.
-
There was a update on squid package today, apply it and see what happens.
-
There was a update on squid package today, apply it and see what happens.
Squid was reinstalled with the latest version, but still the same.
-
I'll try to reproduce it here.
You get this memory alert once or after squid restart alerts are frequent?
-
I'll try to reproduce it here.
You get this memory alert once or after squid restart alerts are frequent?
The memory alerts only shows after a Squid restart. After the initial alerts there are no more alerts. At least that is what is occurring here.
Dec 30 15:11:54 php: : There were error(s) loading the rules: /tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [19]: table <pfblockertbg>persist file "/var/db/aliastables/pfBlockerTBG.txt"
Dec 30 15:11:54 php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded The line in question reads [19]: table <pfblockertbg>persist file "/var/db/aliastables/pfBlockerTBG.txt"
Dec 30 15:11:46 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was '/tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded'
Dec 30 15:11:16 php: : There were error(s) loading the rules: /tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [19]: table <pfblockertbg>persist file "/var/db/aliastables/pfBlockerTBG.txt"
Dec 30 15:11:16 php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded The line in question reads [19]: table <pfblockertbg>persist file "/var/db/aliastables/pfBlockerTBG.txt"
Dec 30 15:11:07 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was '/tmp/rules.debug:19: cannot define table pfBlockerTBG: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded'
Dec 30 15:10:32 php: /pkg_edit.php: Reloading Squid for configuration sync
Dec 30 15:10:32 check_reload_status: Reloading filter
Dec 30 15:10:32 check_reload_status: Syncing firewall
Dec 30 15:10:32 check_reload_status: Reloading filter
Dec 30 15:10:31 php: /pkg_edit.php: Reloading Squid for configuration sync</pfblockertbg></pfblockertbg></pfblockertbg></pfblockertbg> -
When that error occurs, Squid fails to work as a transparent proxy. Squid still works if you manually set the proxy settings in a browser for example. I disabled pfBlocker for now. Will try a few things.
-
Increase even more Firewall Maximum Table Entries in system->advanced -> firewall/NAT
not!!!! Firewall Maximum States ;)
-
PfBlocker use url tables, so it is firewall maximum table entries for sure.
Firewall maximum states are related to active connections. When you reach this value, you can't do anything on firewall.
-
After trying out a few things all is fine for now. Disabled the TBG List and left the Bluetack enabled. Also in System -> Advanced -> Firewall and NAT; Firewall Maximum Table Entries, removed that large number[9999999999999999] from there. Restarted pfBlocker and now there are no error messages and Squid works as transparent proxy, even with pfBlocker enabled. So it seems the TBG list is the cause of all these issues I was having.
TBG list:
http://list.iblocklist.com/?list=ijfqtofzixtwayqovmxn&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=ecqbsykllnadihkdirsh&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=tbnuqfclfkemqivekikv&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=ewqglwibdgjttwttrinl&fileformat=p2p&archiveformat=gzBluetack list:
http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_level2&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_level3&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_spyware&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_proxy&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_templist&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_hijacked&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_dshield&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=bt_bogon&fileformat=p2p&archiveformat=gz -
You may try if you want add each TBG list until you find what list get errors.
-
You may try if you want add each TBG list until you find what list get errors.
Yes, I'm going to try that. I might spring for the i-block lists, need to do some research on that. Are the i-block lists any better than the public lists?
-
Did some testing.
With the following in the TBG list, whenever I try to enable them[one at a time], get the Fatal error below.
http://list.iblocklist.com/?list=ijfqtofzixtwayqovmxn&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=ecqbsykllnadihkdirsh&fileformat=p2p&archiveformat=gzFatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8650655 bytes) in /usr/local/pkg/pfblocker.inc on line 268
Note, the Bluetack list is still enabled. When testing the above.
-
the total download exceeds php memory limit.
try to apply one list at time
-
the total download exceeds php memory limit.
try to apply one list at time
whenever I try to enable them[one at a time]
I did, or do you mean something else?
-
I mean that. Sorry, I didn't saw the "[one at a time]"
-
I mean that. Sorry, I didn't saw the "[one at a time]"
No problem and guess no more lists for me :(
-
The are some file hacks to increase php memory limit, but it is not supported or recommended by core team.
DO IT AT YOUR OWN RISK
edit ini_set("memory_limit","128M"); to 500M in /etc/inc/config.inc file
and include ini_set('memory_limit', '490M'); on pfblocker.inc
Do not increase it to a value greater then 510M, suhosin.memory_limit is set to 512M
-
edit ini_set("memory_limit","128M"); to 500M in /etc/inc/config.inc file
and include ini_set('memory_limit', '490M'); on pfblocker.inc
Do not increase it to a value greater then 510M, suhosin.memory_limit is set to 512M
I take it a reboot is needed for these changes to take affect?
-
Not at all. There is no changes on php.ini
-
Hmm, I edited the following to test:
/etc/inc/config.inc and changed ini_set("memory_limit","128M"); to 500M
/usr/local/pkg/pfblocker.inc and added ini_set('memory_limit', '490M');
Then tried to enable the TBG list, but still get the Fatal error. Did I do something wrong?
-
Not sure, I did it on a x64 virtual machine and I do not get php limit errors.
All files are uploaded.
EDIT
With all these large files, my virtual machine gets 100% cpu for more then a minute and even with a very large table size, it reach virtual machine memory limit (384Mb).
-
Something interesting about table-entries limit:
I've set up maximum table entries to a very high value 99999999999 at gui and could see it reflected on /tmp/rules.debug
BUT
on pfctl -sa I get this:
LIMITS:
states hard limit 1000000
src-nodes hard limit 1000000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 1215752191I saw my virtual machine freezing console with 99999999999 value, so I reduced it on gui to 5000000
EDIT
After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot
I get no erros with both large lists applied
as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc
-
Something interesting about table-entries limit:
I've set up maximum table entries to a very high value 99999999999 at gui and could see it reflected on /tmp/rules.debug
BUT
on pfctl -sa I get this:
LIMITS:
states hard limit 1000000
src-nodes hard limit 1000000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 1215752191I saw my virtual machine freezing console with 99999999999 value, so I reduced it on gui to 5000000
EDIT
After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot
I get no erros with both large lists applied
as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc
Interesting, and yes when I edited /etc/inc/config.inc I saw it already had ini_set("memory_limit","256M"); . I have a x64 install. Nevertheless when I get a chance I will try your last settings again and reboot. Thanks for the info.
-
EDIT
After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot
I get no erros with both large lists applied
as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc
Yes! :) I tested the above and no more errors, sort of; I'll explain below.
I did the following:
Edited /usr/local/pkg/pfblocker.inc adding the following on the first line where the comments end[from the top]. Right after this line:
*/ <- this is line number 31
ini_set("memory_limit", "250M"); <- this would be line number 32Then set the Firewall Maximum Table Entries: 5000000 and did a reboot. But received these errors:
Dec 31 18:13:44 php: : There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.Dec 31 18:13:44 php: : The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was 'no IP address found for 550.164.3.166/32 /tmp/rules.test.packages:17: file "/var/db/aliastables/pfBlockerBluetack.txt" contains bad data'
After some testing those errors are due to using the following list:
http://list.iblocklist.com/?list=bt_proxy&fileformat=p2p&archiveformat=gzNOTE: I tried several times removing and adding that list, with the same result.
Once I removed that list I got this:
Dec 31 18:46:28 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was 'no IP address found for 75.35.185. /tmp/rules.debug:19: file "/var/db/aliastables/pfBlockerTBG.txt" contains bad data no IP address found for ÿ0.0.0.0/8 /tmp/rules.debug:95: file "/etc/bogons" contains bad data pfctl: Syntax error in config file: pf rules not loaded'More testing and the error above was due to this list:
http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gzBut this time after removing the list above and re-adding the error went away. I guess there are some issues with those lists. Will try to contact the author to see if that can be fixed.
So in the end, no more errors and Squid works as a transparent proxy using pfBlocker as well now :)
For reference these are the lists that I'm currently using error free:
<pfblockerlists><config><aliasname>Bluetack</aliasname>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_level2&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_level3&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_spyware&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_templist&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_hijacked&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_dshield&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=bt_bogon&fileformat=p2p&archiveformat=gz</url></row>
<action>Deny_Inbound</action>
<cron>EveryDay</cron></config>
<config><aliasname>TBG</aliasname><row><format>gz</format>
<url>http://list.iblocklist.com/?list=ijfqtofzixtwayqovmxn&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=ecqbsykllnadihkdirsh&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=tbnuqfclfkemqivekikv&fileformat=p2p&archiveformat=gz</url></row>
<row><format>gz</format>
<url>http://list.iblocklist.com/?list=ewqglwibdgjttwttrinl&fileformat=p2p&archiveformat=gz</url></row>
<action>Deny_Inbound</action>
<cron>EveryDay</cron></config></pfblockerlists>I would like to Thank marcelloc for his time looking into this. A donation was made[yesterday, before marcelloc posted the fix]; will donate again when I can. Hope this change is added to the package for all to benefit, after a few more days of testing. To make sure this change does not affect anything else.
EDIT: Happy New Year :)
-
Thanks ;D. Did you mentioned these packages developers on donation?
-
Did you mentioned these packages developers on donation?
Yes, I broke it down on how the donation should broken up. Like so: $XX for the X package maintainer(s).
-
I'll include this ini_set('memory_limit', '250M') patch when pfblocker detects a x64 version.
There are other places to edit it, maybe a redmine for that as diagnostics-> tables also has a 128Mb limit even on x64.
-
I'll include this ini_set('memory_limit', '250M') patch when pfblocker detects a x64 version.
There are other places to edit it, maybe a redmine for that as diagnostics-> tables also has a 128Mb limit even on x64.
Great and yes good point, since Diagnostics -> Tables gives a similar: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/www/diag_tables.php on line 89
But at least that does not affect things like the previous "Fatal error: Allowed memory size…" did.
-
-
I ran into an issue with pfBlocker when trying to copy the configuration onto a new install.
I am using aliases only, a Block in rule on WAN, a Reject out rule on LAN.
I have the following packages: Backup 0.1.5, Cron 0.1.5, Filemanager 0.1.1, pfBlocker 1.0.1, Snort 2.9.1 v 2.0.2
Running pfsense 2.0.1 x86-
I saved the configuration including RRD data
-
I restored the configuration to a fresh 2.0.1 install on the same machine
It rebooted and started to install packages, but pfsense could not access the internet because of the pfBlocker was not running and the missing aliases break the firewall.
I could not recover, removing rules didn' work. I was spending too much time trying things and failing, I ended up cloning the old disk instead.What is the way to recover from the pfBlocker aliases problem?
just removing the rules in the GUI didn't work. I didn't try to remove the table or alias.
I remove the rules in the config.xml, to no available.
Today I see that I should have removed the alias. Is this enough?
I do not have a spare machine to do testing.What is the best way to copy the configuration ?
Was I supposed to install all the packages before restoring the configuration?
This might have worked if the Restore config script did the package install before the reboot.
(or download the package now before the reboot to prepare for package install)I guess removing the rules and disabling pfBlocker before saving the configuration should work.
-
-
I restored the configuration to a fresh 2.0.1 install on the same machine
You may need to install packages before config restore
I did few upgrades with pfblocker installed with no erros, but I did not tried a system restore.
I'll try to reproduce it.
Are you using pfBlocker on full install or nanobsd?
I guess removing the rules and disabling pfBlocker before saving the configuration should work.
This should be the best option
-
Are you using pfBlocker on full install or nanobsd?
Full install from CD pfSense-2.0.1-RELEASE-i386.iso.gz
2.0.1-RELEASE (i386)
built on Mon Dec 12 18:24:17 EST 2011
FreeBSD 8.1-RELEASE-p6Platform pfSense
CPU Type Intel(R) Pentium(R) 4 CPU 1.80GHz -
I'll include this ini_set('memory_limit', '250M') patch when pfblocker detects a x64 version.
marcelloc,
Just wanted to report back. After using the above now for the last 6 days. I don't see that affecting anything.
-
countryipblocks.net released a new bogon network list on their site,
check this out. :)
http://forum.pfsense.org/index.php/topic,44867.msg233407.html#msg233407
-
I had to do a configuration restore today and I'm not getting any php memory error(s) so far. This includes reinstalling all packages. So, ini_set("memory_limit", "250M") in /usr/local/pkg/pfblocker.inc seems to not be needed here anymore, well at least so far. Will report back if I start to get the php memory error(s). But I wonder why it is working fine now without editing /usr/local/pkg/pfblocker.inc ?