PfBlocker
-
I can't load any list into pfBlocker that is bigger than 10 MB: it crashes with an out if memory error, even if I choose the Alias creation.
Are you pasting it on pfbloker gui or linking it to a url?
I've setup an I-BlockList account. They have an option to create a custom blocklist: you choose which lists you'd like to merge and they provide you an URL to download a .gz package, containing a .txt file with all the lists you choose, merged.
In pfBlocker I'm pasting this URL into the GUI interface and then I select the "create alias".
Before creating this "big list" you have the option to select the format: p2p, dat, CIDR and the compression: gz or zip.
-
I think the Maxmind GeoIP database is more Stable. I also don't expect them to close the Country Code Lists. They have a paid subscription for Cities and Streets, but the Country Listings have been free for a long time now.
The Data is provided in a Range format.I'm with BBcan177. The Maxmind GeoIP lists are a much better long-term solution than the list I was using.
I wrote a script to d/l -> unzip -> extract the Maxmind IP ranges into their respective country files (ie: /usr/local/pkg/pfblocker/US.txt) but couldn't figure out how to convert range to CIDR.
(I tried these scripts https://forum.pfsense.org/index.php?topic=73244.msg400025#msg400025 but couldn't get them working.)However, if someone is already hosting the Maxmind data in CIDR format - that'd be a lot simpler update method, than the script I wrote.
Edit: My script, if anyone is interested (missing range to CIDR conversion)
https://gist.github.com/LinuxTracker/8332e0d620ab5f6df1dc -
I was just going through some notes on pfblocker. This brought back some memories. I thought I would share with you guys since it's apart of the pfblocker past.
This note is dated 2010!
If you guys are interested I have a working BETA for pfBlocker that incorporates features that lots of us would like to see in a new release of pfBlocker. :)
I want! :)
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268 -
I was just going through some notes on pfblocker. This brought back some memories. I thought I would share with you guys since it's apart of the pfblocker past.
This note is dated 2010!
If you guys are interested I have a working BETA for pfBlocker that incorporates features that lots of us would like to see in a new release of pfBlocker. :)
Will this Beta be released as a new package or as an update to the existing pfBlocker?
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000. -
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
Yea, I saw that and was just coming back to edit my post.
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
-
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268System: Advanced: Firewall and NAT:Firewall Maximum Table Entries -> Set this to at least 400000.
I have several huge lists and mine is 900000.They are all at 12000000 :(
Yea, I saw that and was just coming back to edit my post.
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
Which files have I to delete? I'd like to be sure not to make a mess…
-
Is it possible we can host our own lists centrally so we dont rely on any outsiders?
You can use localfiles with pfBlocker currently. Instead of a URL link just enter the path to the local file.
I use a mix of local files, Dropbox, hosting on a local IIS server but as often as not I'll push to pfSense using the pscp utility.
-
The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.
Which files have I to delete? I'd like to be sure not to make a mess…
Correction: /usr/local/pkg/pfblocker directory.
The custom files have names like a8a234f32e30a362cbebb7c0606b5f01.txtAnything else you delete in there can be put back by reinstalling the pfblocker package.
-
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268Hi Panz,
The file can be local to pfsense, or a link to a local file server that pfSense has access to.
The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.
Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.
262 # Network range 192.168.0.0-192.168.0.254 263 if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){ 264 $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]); 265 if (is_array($a_cidr)) { 266 foreach ($a_cidr as $cidr) { 267 ${$alias}.= $cidr."\n"; 268 $new_file.= $cidr."\n"; 269 } 270 } 271 } -
Which files have I to delete? I'd like to be sure not to make a mess…
Take a look at this link where I described a method to find out which file belongs to which URL list.
https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698
-
Which files have I to delete? I'd like to be sure not to make a mess…
Take a look at this link where I described a method to find out which file belongs to which URL list.
https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698
This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?
-
What do you mean by "local" file? Do I have to upload the file to pfSense?
This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268Hi Panz,
The file can be local to pfsense, or a link to a local file server that pfSense has access to.
The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.
Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.
262 # Network range 192.168.0.0-192.168.0.254 263 if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){ 264 $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]); 265 if (is_array($a_cidr)) { 266 foreach ($a_cidr as $cidr) { 267 ${$alias}.= $cidr."\n"; 268 $new_file.= $cidr."\n"; 269 } 270 } 271 }My "big" list is already in CIDR… and it is loaded FAST in PeerBlock (which, BTW, works flawlessly with that list, blocking and notifying like a charm) ... the same list crashes pfBlocker! This doesn't sound good to me (but I'm not an expert, so my opinion could be horribly wrong).
I don't agree that having a big list = more difficult to find why an IP is blocked. I constantly monitor my "recipe", my "compilation" of lists and I'm sure that those are the IPs I want to block.
Otherwise, it is less safer (and error prone IMHO) to build a firewall rule for each different list... I always prefer to keep things as simple as possible.
-
The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:
ini_set('memory_limit', '250M');You could just increase it since you have RAM to spare. However it begs the question what else has used RAM since it trying to allocate only 5MB? Memory leak?
Steve
-
This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?
The md5 is based upon the URL name. Does your URL change? My guess is that it is consistent, so you should be able to run that command and it will calculate the md5. This will allow you to see which filename in the directory matches your URL.
This is not related to your problem as its a PHP Memory Error.
The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:
Steve, what limits would you expect would be appropriate for pfBlocker for various Hardware?
-
I have no idea. I only spotted that because the limit is 250MB rather than 256 which I might have expected. There appears to be another overall limit in the suhosin config in php.inc of 512MB.
I know pretty much nothing about php. ;) 250MB seems like it should be fine. My home box only has 512MB anyway. A better question might be does the fact that it's used 250MB seem reasonable?Steve
