PfBlocker
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268 -
I was just going through some notes on pfblocker. This brought back some memories. I thought I would share with you guys since it's apart of the pfblocker past.
This note is dated 2010!
If you guys are interested I have a working BETA for pfBlocker that incorporates features that lots of us would like to see in a new release of pfBlocker. :)
Will this Beta be released as a new package or as an update to the existing pfBlocker?
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000. -
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
Yea, I saw that and was just coming back to edit my post.
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
Yea, I saw that and was just coming back to edit my post.
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
Which files have I to delete? I'd like to be sure not to make a mess…
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
I use a mix of local files, Dropbox, hosting on a local IIS server but as often as not I'll push to pfSense using the pscp utility.
-
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
Which files have I to delete? I'd like to be sure not to make a mess…
Correction: /usr/local/pkg/pfblocker directory.
The custom files have names like a8a234f32e30a362cbebb7c0606b5f01.txtAnything else you delete in there can be put back by reinstalling the pfblocker package.
-
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268Hi Panz,
The file can be local to pfsense, or a link to a local file server that pfSense has access to.
The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.
Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.
262 # Network range 192.168.0.0-192.168.0.254 263 if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){ 264 $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]); 265 if (is_array($a_cidr)) { 266 foreach ($a_cidr as $cidr) { 267 ${$alias}.= $cidr."\n"; 268 $new_file.= $cidr."\n"; 269 } 270 } 271 } -
Which files have I to delete? I'd like to be sure not to make a mess…
Take a look at this link where I described a method to find out which file belongs to which URL list.
https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698
-
Which files have I to delete? I'd like to be sure not to make a mess…
Take a look at this link where I described a method to find out which file belongs to which URL list.
https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698
This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?
-
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268Hi Panz,
The file can be local to pfsense, or a link to a local file server that pfSense has access to.
The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.
Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.
262 # Network range 192.168.0.0-192.168.0.254 263 if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){ 264 $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]); 265 if (is_array($a_cidr)) { 266 foreach ($a_cidr as $cidr) { 267 ${$alias}.= $cidr."\n"; 268 $new_file.= $cidr."\n"; 269 } 270 } 271 }My "big" list is already in CIDR… and it is loaded FAST in PeerBlock (which, BTW, works flawlessly with that list, blocking and notifying like a charm) ... the same list crashes pfBlocker! This doesn't sound good to me (but I'm not an expert, so my opinion could be horribly wrong).
I don't agree that having a big list = more difficult to find why an IP is blocked. I constantly monitor my "recipe", my "compilation" of lists and I'm sure that those are the IPs I want to block.
Otherwise, it is less safer (and error prone IMHO) to build a firewall rule for each different list... I always prefer to keep things as simple as possible.
-
The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:
ini_set('memory_limit', '250M');You could just increase it since you have RAM to spare. However it begs the question what else has used RAM since it trying to allocate only 5MB? Memory leak?
Steve
-
This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?
The md5 is based upon the URL name. Does your URL change? My guess is that it is consistent, so you should be able to run that command and it will calculate the md5. This will allow you to see which filename in the directory matches your URL.
This is not related to your problem as its a PHP Memory Error.
The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:
Steve, what limits would you expect would be appropriate for pfBlocker for various Hardware?
-
I have no idea. I only spotted that because the limit is 250MB rather than 256 which I might have expected. There appears to be another overall limit in the suhosin config in php.inc of 512MB.
I know pretty much nothing about php. ;) 250MB seems like it should be fine. My home box only has 512MB anyway. A better question might be does the fact that it's used 250MB seem reasonable?Steve
-
I have "splitted" my (big) list: now each I-Block list matches one pfBlocker list (with Alias_only) selected (see attached screenshot). Lists are set on I-Block URL to be in CIDR.
I'm still receiving this error message when I try to start pfBlocker
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 18 bytes) in /usr/local/pkg/pfblocker.inc on line 253
 -
This is 2.1.4 right? 2.2 Alpha will fail loading/updating these lists, that has been reported and they are working on it. Another thing is a lot of those lists duplicate each other, ie. have the same IP's in the lists.
Read this thread:https://forum.pfsense.org/index.php?topic=78062.0
Remove Prime, it is a combination of several the lists you already have as aliases and is not needed, read the description of Prime. Prime might be the problem since it itself is absolutely huge… If removing Prime doesn't help then:
Back up pfBlocker.inc then Look for a line in the pfBlocker.inc that looks something like this:```
if ($uname['machine'] == 'amd64')
ini_set('memory_limit', '256M');Change the 256M to 512M and see if that helps. @panz: > I have "splitted" my (big) list: now each I-Block list matches one pfBlocker list (with Alias_only) selected (see attached screenshot). Lists are set on I-Block URL to be in CIDR. > > I'm still receiving this error message when I try to start pfBlocker > > ``` > Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 18 bytes) in /usr/local/pkg/pfblocker.inc on line 253 > ``` -
Thank you @wcrowder for the fast response. I don't like any list in particular: I wanted to test each of them with a firewall rule for each list so - by analyzing the logs - I'll be able to keep the lists that hit the blocking rule the most of times.
I can't still believe that a Unix OS can't handle a simple task like this: as I wrote in this thread, my stupid Windows 7 machine with the same amount of RAM can load ALL those lists in PeerBlock FAST! And it works flawlessly, blocking and notifying 24/7.
-
You can not compare pfSense to Peer-block, in comparison Peer-block is truly useless. How may Windows 7 machines act as firewalls to large networks, do you wonder why? pfSense is. Did you follow through the link to the thread I sent you above? Follow the instructions, remove the duplication? Here, I'll give it to you again: Read this thread:https://forum.pfsense.org/index.php?topic=78062.0
Thank you @wcrowder for the fast response. I don't like any list in particular: I wanted to test each of them with a firewall rule for each list so - by analyzing the logs - I'll be able to keep the lists that hit the blocking rule the most of times.
I can't still believe that a Unix OS can't handle a simple task like this: as I wrote in this thread, my stupid Windows 7 machine with the same amount of RAM can load ALL those lists in PeerBlock FAST! And it works flawlessly, blocking and notifying 24/7.
-
Yes, I removed the duplicates: I logged into pfSense with WinSCP and manually deleted those jsjdjdjjdjjddkkakirtitjjf.txt files. Then I deleted the pfBlocker package and all the files in the upper pkg dir that contained the word pfBlocker.
Strangely enough, after pfBlocker reinstallation, all the lists were still there, listed in the GUI, so maybe do I need to purge other files?
-
The xx.txt files need to be there. They are the county code files for country blocking. The duplicates I was talking about are IP's in the lists. Example: On the iBlockList website the iBlock Prime list already includes the US Government, Malicious, Piracy Related, Pornography, Ads and Trackers and Proxies lists so when you add Prime AND those lists you have DUPLICATION that you don't need…
Yes, I removed the duplicates: I logged into pfSense with WinSCP and manually deleted those jsjdjdjjdjjddkkakirtitjjf.txt files. Then I deleted the pfBlocker package and all the files in the upper pkg dir that contained the word pfBlocker.
Strangely enough, after pfBlocker reinstallation, all the lists were still there, listed in the GUI, so maybe do I need to purge other files?
-
Ok, I'm going to dedupe the lists. Where is the lists' list stored? Even after a package reinstall, the list is there.
-
Could we make a pfblocker community so we easily can share files and lists?
Maybe from within the package itself?
-
I'm emailing to I-Blocklist maintainer to ask him if he could implement a deduplication feature in the customized lists section. For now, I'll be happy if the pfBlocker package could handle a memory increase by itself. I don't want to mess with file editing in an appliance.
-
Not a bad idea.
I'm emailing to I-Blocklist maintainer to ask him if he could implement a deduplication feature in the customized lists section. For now, I'll be happy if the pfBlocker package could handle a memory increase by itself. I don't want to mess with file editing in an appliance.
-
Just wait for a pfBlocker update… ::)
-
-
It is being very actively forked, we will just have to wait for an announcement from the author doing the update… Or, you could read this thread more carefully. ;D Did you read the thread I posted to you Panz? :-*
Is MArcello still developing this?
Just wait for a pfBlocker update… ::)
-
Thanks! We need forks :)
-
<shaking head="">:-\ and thinking out loud… In the 30yrs I've done this every good Forum has a Troll.
Thanks! We need forks :)</shaking>
-
And youre doing it better than ANY out there ;)
<shaking head="">:-\ and thinking out loud… In the 30yrs I've done this every good Forum has a Troll.
Thanks! We need forks :)</shaking>
-
Wasnt me…dont use that at all
-
-
I really like the idea of blocking entire countries that my network has no business connecting to and put together a script to update pfBlockers country CIDR lists with the lists available from IPdeny.com
Edit - The script didn't work as intended and has been taken down
The download is at the bottom of http://legoclan.com/pfsense/ all feedback is welcome
-
Hi!
How can i fix this on PF 2.2 BETA?
Crash report begins. Anonymous machine information:
amd64
10.1-PRERELEASE
FreeBSD 10.1-PRERELEASE #28 30e366f(HEAD)-dirty: Fri Sep 19 23:30:48 CDT 2014 root@pf22-amd64-snap:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10Crash report details:
PHP Errors:
in /usr/local/pkg/pfblocker.inc on line 256 -
I really like the idea of blocking entire countries that my network has no business connecting to and put together a script to update pfBlockers country CIDR lists with the lists available from IPdeny.com
The download is at the bottom of http://legoclan.com/pfsense/ all feedback is welcome
Hi f34rinc,
This will update the ISO files for pfBlocker but it doesn't update the XML Continent Files (ie - Africa) with the changes made to the Country Names. So there could be some inconsistencies.
There are files named
Africa_cidr.txt that have this type of header:
Continent: Africa
Country: Reunion
ISO Code: RE
Total Networks: 245
These files are used by this command to rebuild the XML files based upon the 6 Continent Files.
/usr/local/bin/php /usr/local/www/pfblocker.php uc
-
Hi!
How can i fix this on PF 2.2 BETA?
Crash report begins. Anonymous machine information:
amd64
10.1-PRERELEASE
FreeBSD 10.1-PRERELEASE #28 30e366f(HEAD)-dirty: Fri Sep 19 23:30:48 CDT 2014 root@pf22-amd64-snap:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10Crash report details:
PHP Errors:
in /usr/local/pkg/pfblocker.inc on line 256This error doesn't seem to be related to Country Blocking. Maybe its one of your Lists that is failing? Which lists are you using?
-
Thank you BBcan177 I don't believe I have the skill set to accomplish this
Edit - I took the country list of top spammers from http://www.projecthoneypot.org/spam_server_top_countries.php and made a script to make my own replacement for the pfBlocker top spammers list. Please use the link from my previous post above.
