PfBlocker

LinuxTracker

@LinuxTracker:

The other time I got that I had a list with bad data. I had to delete all my custom lists in /usr/local/pkg, then re-introduce them one at a time.

Which files have I to delete? I'd like to be sure not to make a mess…

Correction: /usr/local/pkg/pfblocker directory.
The custom files have names like a8a234f32e30a362cbebb7c0606b5f01.txt

Anything else you delete in there can be put back by reinstalling the pfblocker package.

BBcan177

@panz:

What do you mean by "local" file? Do I have to upload the file to pfSense?

This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268 

Hi Panz,

The file can be local to pfsense, or a link to a local file server that pfSense has access to.

The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.

Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.


262   # Network range 192.168.0.0-192.168.0.254
263        if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){
264               $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]);
265                if (is_array($a_cidr)) {
266                        foreach ($a_cidr as $cidr) {
267                              ${$alias}.= $cidr."\n";
268                              $new_file.= $cidr."\n";
269                        }
270                }
271        }

BBcan177

@panz:

Which files have I to delete? I'd like to be sure not to make a mess…

Take a look at this link where I described a method to find out which file belongs to which URL list.

https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698

panz

@BBcan177:

@panz:

Which files have I to delete? I'd like to be sure not to make a mess…

Take a look at this link where I described a method to find out which file belongs to which URL list.

https://forum.pfsense.org/index.php?topic=42543.msg436698#msg436698

This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?

panz

@BBcan177:

@panz:
What do you mean by "local" file? Do I have to upload the file to pfSense?

This is the error I'm getting when pasting a link to an I-BlockList which is of 10,565,669 bytes; it contains a .txt file of 710,772 lines
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 4980644 bytes) in /usr/local/pkg/pfblocker.inc on line 268 
Hi Panz,

The file can be local to pfsense, or a link to a local file server that pfSense has access to.

The memory is due to PHP failing on such a large list. I still don't recommend one large list as its hard to see why an IP is being Blocked. You should break down the One List into Main Categories and apply smaller Lists.

Line 268 is where the pfBlocker code is trying to convert IP Ranges to CIDRs and is crashing due to running out of memory.
262   # Network range 192.168.0.0-192.168.0.254
263        if (preg_match("/(\d+\.\d+\.\d+\.\d+)-(\d+\.\d+\.\d+\.\d+)/",$line,$matches)){
264               $a_cidr = ip_range_to_subnet_array($matches[1],$matches[2]);
265                if (is_array($a_cidr)) {
266                        foreach ($a_cidr as $cidr) {
267                              ${$alias}.= $cidr."\n";
268                              $new_file.= $cidr."\n";
269                        }
270                }
271        }

My "big" list is already in CIDR… and it is loaded FAST in PeerBlock (which, BTW, works flawlessly with that list, blocking and notifying like a charm) ... the same list crashes pfBlocker! This doesn't sound good to me (but I'm not an expert, so my opinion could be horribly wrong).

I don't agree that having a big list = more difficult to find why an IP is blocked. I constantly monitor my "recipe", my "compilation" of lists and I'm sure that those are the IPs I want to block.

Otherwise, it is less safer (and error prone IMHO) to build a firewall rule for each different list... I always prefer to keep things as simple as possible.

stephenw10

The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:

ini_set('memory_limit', '250M');

You could just increase it since you have RAM to spare. However it begs the question what else has used RAM since it trying to allocate only 5MB? Memory leak?

Steve

BBcan177

@panz:

This doesn't work, because I can't get an "md5 -s" of a list that changes every day (or more often) and compare it to an older version. Am I missing something?

The md5 is based upon the URL name. Does your URL change? My guess is that it is consistent, so you should be able to run that command and it will calculate the md5. This will allow you to see which filename in the directory matches your URL.

This is not related to your problem as its a PHP Memory Error.

@stephenw10:

The limit you are hitting '262144000 bytes' is that imposed on the pfblocker process by the line at the start of the pfblocker.php file:

Steve, what limits would you expect would be appropriate for pfBlocker for various Hardware?

stephenw10

I have no idea. I only spotted that because the limit is 250MB rather than 256 which I might have expected. There appears to be another overall limit in the suhosin config in php.inc of 512MB.
I know pretty much nothing about php. ;) 250MB seems like it should be fine. My home box only has 512MB anyway. A better question might be does the fact that it's used 250MB seem reasonable?

Steve

panz

I have "splitted" my (big) list: now each I-Block list matches one pfBlocker list (with Alias_only) selected (see attached screenshot). Lists are set on I-Block URL to be in CIDR.

I'm still receiving this error message when I try to start pfBlocker

Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 18 bytes) in /usr/local/pkg/pfblocker.inc on line 253

![pfBlocker lists.jpg](/public/imported_attachments/1/pfBlocker lists.jpg)
![pfBlocker lists.jpg_thumb](/public/imported_attachments/1/pfBlocker lists.jpg_thumb)

wcrowder

This is 2.1.4 right? 2.2 Alpha will fail loading/updating these lists, that has been reported and they are working on it. Another thing is a lot of those lists duplicate each other, ie. have the same IP's in the lists.

Read this thread:https://forum.pfsense.org/index.php?topic=78062.0

Remove Prime, it is a combination of several the lists you already have as aliases and is not needed, read the description of Prime. Prime might be the problem since it itself is absolutely huge… If removing Prime doesn't help then:

Back up pfBlocker.inc then Look for a line in the pfBlocker.inc that looks something like this:```
if ($uname['machine'] == 'amd64')
ini_set('memory_limit', '256M');


Change the 256M to 512M and see if that helps.

@panz:

> I have "splitted" my (big) list: now each I-Block list matches one pfBlocker list (with Alias_only) selected (see attached screenshot). Lists are set on I-Block URL to be in CIDR.
> 
> I'm still receiving this error message when I try to start pfBlocker
> 
> ```
> Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 18 bytes) in /usr/local/pkg/pfblocker.inc on line 253 
> ```

panz

Thank you @wcrowder for the fast response. I don't like any list in particular: I wanted to test each of them with a firewall rule for each list so - by analyzing the logs - I'll be able to keep the lists that hit the blocking rule the most of times.

I can't still believe that a Unix OS can't handle a simple task like this: as I wrote in this thread, my stupid Windows 7 machine with the same amount of RAM can load ALL those lists in PeerBlock FAST! And it works flawlessly, blocking and notifying 24/7.

wcrowder

You can not compare pfSense to Peer-block, in comparison Peer-block is truly useless. How may Windows 7 machines act as firewalls to large networks, do you wonder why? pfSense is. Did you follow through the link to the thread I sent you above? Follow the instructions, remove the duplication? Here, I'll give it to you again: Read this thread:https://forum.pfsense.org/index.php?topic=78062.0

@panz:

Thank you @wcrowder for the fast response. I don't like any list in particular: I wanted to test each of them with a firewall rule for each list so - by analyzing the logs - I'll be able to keep the lists that hit the blocking rule the most of times.

I can't still believe that a Unix OS can't handle a simple task like this: as I wrote in this thread, my stupid Windows 7 machine with the same amount of RAM can load ALL those lists in PeerBlock FAST! And it works flawlessly, blocking and notifying 24/7.

panz

Yes, I removed the duplicates: I logged into pfSense with WinSCP and manually deleted those jsjdjdjjdjjddkkakirtitjjf.txt files. Then I deleted the pfBlocker package and all the files in the upper pkg dir that contained the word pfBlocker.

Strangely enough, after pfBlocker reinstallation, all the lists were still there, listed in the GUI, so maybe do I need to purge other files?

wcrowder

The xx.txt files need to be there. They are the county code files for country blocking. The duplicates I was talking about are IP's in the lists. Example: On the iBlockList website the iBlock Prime list already includes the US Government, Malicious, Piracy Related, Pornography, Ads and Trackers and Proxies lists so when you add Prime AND those lists you have DUPLICATION that you don't need…

@panz:

Yes, I removed the duplicates: I logged into pfSense with WinSCP and manually deleted those jsjdjdjjdjjddkkakirtitjjf.txt files. Then I deleted the pfBlocker package and all the files in the upper pkg dir that contained the word pfBlocker.

Strangely enough, after pfBlocker reinstallation, all the lists were still there, listed in the GUI, so maybe do I need to purge other files?

panz

Ok, I'm going to dedupe the lists. Where is the lists' list stored? Even after a package reinstall, the list is there.

Supermule

Could we make a pfblocker community so we easily can share files and lists?

Maybe from within the package itself?

panz

I'm emailing to I-Blocklist maintainer to ask him if he could implement a deduplication feature in the customized lists section. For now, I'll be happy if the pfBlocker package could handle a memory increase by itself. I don't want to mess with file editing in an appliance.

Supermule

Not a bad idea.

@panz:

I'm emailing to I-Blocklist maintainer to ask him if he could implement a deduplication feature in the customized lists section. For now, I'll be happy if the pfBlocker package could handle a memory increase by itself. I don't want to mess with file editing in an appliance.

digdug3

Just wait for a pfBlocker update… ::)

Supermule

Is MArcello still developing this?

@digdug3:

Just wait for a pfBlocker update… ::)