PfBlockerNG v2.0 w/DNSBL
-
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3874 Call Stack: 0.0000 250096 1. {main}() /usr/local/www/pkg_edit.php:0 0.1150 2023904 2. eval('global $pfb; $pfb['save'] = TRUE; sync_package_pfblockerng();') /usr/local/www/pkg_edit.php:253 0.1150 2024816 3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3 9.7837 279349272 4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3874 PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3874, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes)
-
Seems like the code is bugged, reinstalling the package doesnt even work, the post install script fails and the pfblocker interface doesnt show up in the firewall tab.
-
@haleakalas:
Follow up on my own earlier post.
It turns out that the php command is not executed at all, that's probably why my browser times out after a few minutes.When I run the same command (php /usr/local/www/pfblockerng/pfblockerng.php dc) in a "php command window" (without the php part) I get the following message:
Parse error: syntax error, unexpected '/' in /tmp/V7fSSN on line 5I reinstalled pfblockerng once again thinking that maybe i damaged something during my file edit earlier, but same things have happened.
The install seems to go through. The package doesn't show up in the menu.If only pfsense would allow to go back to an earlier version of a package…
Going back to an earlier will not work as the MaxMind GeoIP database changes since.
Before the db was in range format, now it is in CIDR format. BBcan177 will fix this in a week or so. In the meantime we are try to get thing going by raising the php memory limit.Can you retry the php /usr/local/www/pfblockerng/pfblockerng.php dc command from the shell (option 8 or SSH) to see if it succeeds.
Once this works, there is no need to re-run it. The next change to the DB will in a month.Did you read https://forum.pfsense.org/index.php?topic=116307 ? Please reply in that post for the memory issue.
-
Seems like the code is bugged, reinstalling the package doesnt even work, the post install script fails and the pfblocker interface doesnt show up in the firewall tab.
Installation might fail at the MaxMind GeoIP step, the memory issue is discussed here https://forum.pfsense.org/index.php?topic=116307
-
@haleakalas:
Follow up on my own earlier post.
It turns out that the php command is not executed at all, that's probably why my browser times out after a few minutes.When I run the same command (php /usr/local/www/pfblockerng/pfblockerng.php dc) in a "php command window" (without the php part) I get the following message:
Parse error: syntax error, unexpected '/' in /tmp/V7fSSN on line 5I reinstalled pfblockerng once again thinking that maybe i damaged something during my file edit earlier, but same things have happened.
The install seems to go through. The package doesn't show up in the menu.If only pfsense would allow to go back to an earlier version of a package…
Hi did you look at RonpfS remark #26 in the thread below?
https://forum.pfsense.org/index.php?topic=116307.15
-
One consequence of the MaxMind GeoIP change is that IPV6 table are much bigger than before.
So if you are blocking loads of Country, you might ran out of memory processing the feeds and loading the rules.For some users, you may have to fix the pfblockerng.inc file.
You may also have to change the Firewall Maximum Table Entries.There is a note under GeoIP
Note: pfSense by default implicitly blocks all unsolicited inbound traffic to the WAN interface.
Therefore adding GeoIP based firewall rules to the WAN will not provide any benefit, unless there are open WAN ports.It's also not recommended to block the 'world', instead consider rules to 'Permit' traffic from selected Countries only.
Also consider protecting just the specific open WAN ports and it's just as important to protect the outbound LAN traffic.Some users change from blocking the "World" to allowing specific countries and that fixes the memory issues.
You can read the following topic Taming the beasts… aka suricata blueprint. It explain the concept of setting up a Firewall. Some information is outdated, but the concepts are still actual.
-
@haleakalas:
Follow up on my own earlier post.
It turns out that the php command is not executed at all, that's probably why my browser times out after a few minutes.When I run the same command (php /usr/local/www/pfblockerng/pfblockerng.php dc) in a "php command window" (without the php part) I get the following message:
Parse error: syntax error, unexpected '/' in /tmp/V7fSSN on line 5I reinstalled pfblockerng once again thinking that maybe i damaged something during my file edit earlier, but same things have happened.
The install seems to go through. The package doesn't show up in the menu.If only pfsense would allow to go back to an earlier version of a package…
Hi did you look at RonpfS remark #26 in the thread below?
https://forum.pfsense.org/index.php?topic=116307.15
I did but it wasn't the problem or the solution in my case. What I did finally to make it work was:
-
I reinstalled the package manager of pfsense using a remote ssh access. Several attempts to run it through the GUI Command Prompt miserably failed with a variety or error messages. Literally every time with a different message (real twilight zone…)
-
I reinstalled pfBlockerNG again. It showed up on the Firewall menu.
-
I forced a cron update - everything went ok and the package is up and running.
So, finally I did not increase the memory allocation to 300MB or more.
Clearly, I don't quite understand why what has been working well for over a month suddenly started to fail. Neither do I understand why reinstalling pfBlockerNG made it disappear from the Firewall Menu and from there everything fell apart.
The only thing I can tell for sure is that this system was left alone and in good working condition for over a month. No update or upgrade was necessary nor any such action was initiated by admin.
I am quite concerned now that pfSense is actually running internal updates without confirmation by admin, because the package manager was not broken. And I do not buy the idea that an ill behaving package caused the failure of the package manager as it was suggested elsewhere in the forums.That said, I fail to understand why pfBlockerNG needs to go into an hourly update spree! I changed it to a daily update.
-
-
There is an issue with the pkg manager that is fixed and will be made availabe at some point
https://redmine.pfsense.org/issues/6665 -
@haleakalas:
Follow up on my own earlier post.
It turns out that the php command is not executed at all, that's probably why my browser times out after a few minutes.When I run the same command (php /usr/local/www/pfblockerng/pfblockerng.php dc) in a "php command window" (without the php part) I get the following message:
Parse error: syntax error, unexpected '/' in /tmp/V7fSSN on line 5I reinstalled pfblockerng once again thinking that maybe i damaged something during my file edit earlier, but same things have happened.
The install seems to go through. The package doesn't show up in the menu.If only pfsense would allow to go back to an earlier version of a package…
Going back to an earlier will not work as the MaxMind GeoIP database changes since.
Before the db was in range format, now it is in CIDR format. BBcan177 will fix this in a week or so. In the meantime we are try to get thing going by raising the php memory limit.Can you retry the php /usr/local/www/pfblockerng/pfblockerng.php dc command from the shell (option 8 or SSH) to see if it succeeds.
Once this works, there is no need to re-run it. The next change to the DB will in a month.Did you read https://forum.pfsense.org/index.php?topic=116307 ? Please reply in that post for the memory issue.
Ron, I was able to fix my system per the actions in my reply to Qinn. So, everything is well now except for the many hours wasted time.
But since you diligently intervened in this saga a short summary about the sequence of events in my case.- pfSense was upgraded to 2.3.2 the day after it was released. The packages were upgraded in the vacuum. I don't recall ever doing a pfBlockerNG specific package upgrade.
- After the upgrade my system has been running flawlessly for many days.
- Couple of days ago, out of the blue I got error messages pointing at pfBlockerNG running out of memory with the 128MB limit thing.
- Prior to consulting the forums, I attempted to reinstall pfBlockerNG thinking that a fresh install would solve the problem with the most up to date code and tables etc. Huge mistake!, pfBlockerNG disappears from the radar or the Firewall menu, although every installation attempt results in "Success". (The coders should maybe check their understanding about what success is…)
- During that time all the advice about editing the php config file and increasing the memory limit to 300MB or more was futile at least in my case. Further, I doubt that the php command actually worked on a broken or semi broken pfBlockerNG (I mean when pfBlocker is in such a state that it doesn't show up in the Firewall menu)
- Glancing through other threads I suspected that my system's problem was about a broken package installer. Unfortunately attempts to reinstall it through the GUI did not work either. Eventually remote ssh allowed for a clean reinstall.
- Then GUI based reinstall of pfBlocker and forced cron table updates to verify that all was well.
- But then, what was that memory limit thing that never showed up again. Btw, my system uses only 600,000 of the 2,000,000 firewall table entries. So that whole discussion was also irrelevant in my system's case.
Final thought: It would be truly fantastic if pfsense implemented a button called "Leave My System Alone" so that the admin could freeze a well behaving system in that shape and form. No structural changes of internal tables, no new code, no updates, nothing - not even virus table updates nor country code table updates.
-
Well things change over time. IP changes so we need to keep updating, MaxMind changed the format, then pfBlockerNG had to adapt.
At the same time pfsense changes. So things can break from time to time.Keep in mind that Open Source projects like pfsense are developed by people on their free time. So time and resources are limited.
In the case of pfBlockerNG 2.1, there was a dozen users without issue till this month's MaxMind update. Users ranged from 1gb to 16gb (32-64bit) boxes.
Even then, in a few hours, BBcan177 (traveling) was finding solutions while busting his data plan :o.
-
@haleakalas good that you solved it. On the part of the "leave this system alone" part I cannot agree, security requires maintenance. That's why I prefer open source software, because:
1. It's open.
2. It's maintained (thumbs up to everyone here!!!)The problem IMHO with (commercial) software development is that you never get the chance to do it right, things like cleaning up code, education etc. cost money and thus are not allowed or have a limited lifespan, which invokes insecure systems. That's why I never ever used an router/WiFi provided by the ISP, but millions do and I think that they contribute highly to the Internets instability, but that's just my two cents ;)
-
I have a huge issue with the last update as well. It eats memory like a banshee in a matter of minutes!
Are you using the new DNSBL TLD feature?
I believe I had that turned on, I can't check now because after trying to re-install the package it doesn't show on the menu anymore even though it installs successfully. Apparently there is bad code in this update since it went from working fine to a lot of issues. I'd think 16Gb of RAM is more than enough.
-
The is an issue with MaxMind GeoIP mostly on i386 https://forum.pfsense.org/index.php?topic=116307
There is also an issue with the pkg manager.
What do you see in pfblockerng.log system.log, what does the installation log looks like ?
-
Hi there I could use some help on the following settings:
Looking at the setup here => https://forum.pfsense.org/index.php?topic=102470.15 I still don't grasp what I should select at "DNSBL Firewall Rule" I have 2 NIC's; WAN and LAN with 2 VLAN's should I select te VLAN's also now only LAN is selected here and what about loopback and what will it bring?
Second is there any use when using DNSBL to ad list on the IP4 tab?
and thirdly with the "DNSBL IP Firewall Rule Settings:"setting here select outbound is suggested as default it's disabled, why?
Thanks for any explanation and your time!
Cheers Qinn
-
Add me to the list of users with new problems using pfBlockerNG. They also appeared after the pfSense update to the newest version and involve the same memory errors. I increased the number of table entries allowed by a factor of 2. It worked for a while, then the problem reappeared. I'm usjng a J1900 based supermicro motherboard with 8GB ram. I've been using pfSense about 1.5 years. Errorlogs were forwarded to pfsense.
I also suspect it has to do with geoblocking. I have now disabled all country blocking and use only a few malware lists. I like pfBlockerNG for the outbound protection.
Edit: this may be helpful: CPU usage was much higher with country blocking on the updated version with geoblocking. With geoblocking removed, cpu usage is very low again.
For my education, I would appreciate it if someone could offer answers to a few questions below.
1. I use OpenVPN and have a couple of servers available. One is bridged so I can access my home network from out of the house and feel secure. I also use remote desktop through it. It is safer than remote TeamViewer (using their servers) this way. The other is routed for simple pass through over public wifi.
Question: Does OpenVPN protect the ports well enough? I use certificates, multiple passwords, and each device uses it's own certificate, and the name on the certificate match match the userid. the config file names have been changed to hide the device name. I assume this is about as safe as you can get. Is pfBlockerNG country blocking overkill in this instance?
- Regarding SPI, assuming someone tries to hack my ip 123.44.55.12:4321 which happens to be open because a browser just sent out a request on port 4321. Am I at risk and in need on country blocking, or will SPI protect that port because it is not a specific response to the browser request? I see hundreds of attempted hacks daily using pfBlockerng geoblocking.
I also use Snort and have managed to get all the false positives out of the way. It seems to catch a lot. I have no idea if it is serious stuff it stops or just noise. I purchased a one year oinkcode.
If the answer to both indicate safety, then all I need concern myself with is malware site list blocking. I can only assume country blocking uses a tremendous amount of resources
-
Hi there I could use some help on the following settings:
Looking at the setup here => https://forum.pfsense.org/index.php?topic=102470.15 I still don't grasp what I should select at "DNSBL Firewall Rule" I have 2 NIC's; WAN and LAN with 2 VLAN's should I select te VLAN's also now only LAN is selected here and what about loopback and what will it bring?
I don't have VLAN, but 2 LANs, I selected both and loopback.
Second is there any use when using DNSBL to ad list on the IP4 tab?
and thirdly with the "DNSBL IP Firewall Rule Settings:"setting here select outbound is suggested as default it's disabled, why?
Thanks for any explanation and your time!
Cheers Qinn
The DNSBL feeds contain IPs, so it put those in an Alias. If you don't want to block IP, you can keep the Firewall Rule "disabled" and no rules will be created.
-
-
Is there a patch comming anytime soon to fix this?
-
Hi Everyone,
I just wanted to share another source of feeds that you can pull in. Register for free at https://otx.alienvault.com. Once you get logged in start searching the pulses for ipv4 or domain etc to pull the lists you want. When you hit the download page click download, right click CSV; this is the feed you want to load into your DNSBL.
Also note that IOCs can contain file hashes and other types of data not useful to pfblockerng. In that case you can click "create pulse", copy the CSV feed you want to include, and follow the wizard through to remove any of the IOCs that cant be read by pfblockerng.
Edit: Ah nevermind, these don't parse clean
local-data: "'domain','akamaitechnologysupport.com','' 60 IN A 10.10.10.1"
-
-
Hi Everyone,
I just wanted to share another source of feeds that you can pull in. Register for free at https://otx.alienvault.com. Once you get logged in start searching the pulses for ipv4 or domain etc to pull the lists you want. When you hit the download page click download, right click CSV; this is the feed you want to load into your DNSBL.
Also note that IOCs can contain file hashes and other types of data not useful to pfblockerng. In that case you can click "create pulse", copy the CSV feed you want to include, and follow the wizard through to remove any of the IOCs that cant be read by pfblockerng.
Edit: Ah nevermind, these don't parse clean
local-data: "'domain','akamaitechnologysupport.com','' 60 IN A 10.10.10.1"
When BBcan177 return from vacation, he will look at this.
-
Is it possible to use DNSBL on a transparent firewall (none of the NIC's has an IP except for the Bridge itself (10.1.0.10))? I am asking because I tried and I can't seem to access the VIP at 172.16.15.14. I guess NAT does not work with bridges.. hmm, any help appreciated. I'd really like to use DNSBL! Thanks!
-
Is it possible to use DNSBL on a transparent firewall (none of the NIC's has an IP except for the Bridge itself (10.1.0.10))? I am asking because I tried and I can't seem to access the VIP at 172.16.15.14. I guess NAT does not work with bridges.. hmm, any help appreciated. I'd really like to use DNSBL! Thanks!
What are the choice available DNSBL Listening Interface ?
Did you look at the Firewall / NAT / Port Forward tab? -
What are the choice available DNSBL Listening Interface ?
Did you look at the Firewall / NAT / Port Forward tab?My choice for the listening interface is LAN and BRIDGE. Tried both but no dice. The Port Forward tab lookes similar to yours (technically the same) when I enable DNSBL.
-
No clue, no bridge here.
Did you look at all posts in this thread and the pfBlockerNG thread to see if people had the same issues? -
Can you ping the VIP ?
Can you access http://172.16.15.14
Are all clients DNS using pfsense with DNSBL ?
Anything in the Systems Log or Firewall Logs, pfblockerNG log files ? -
No clue, no bridge here.
Did you look at all posts in this thread and the pfBlockerNG thread to see if people had the same issues?No, not yet.
Can you ping the VIP ?
Can you access http://172.16.15.14
Are all clients DNS using pfsense with DNSBL ?
Anything in the Systems Log or Firewall Logs, pfblockerNG log files ?Can't ping or access it. I think it's because bridges usually don't do NAT and I don't think that there is a way to create the functionality DNSBL needs (services on port 80/443). but without using NAT. Please tell me I am wrong. :)
-
I found a way to use DNSBL on my transparent bridge! :D
I configure DNSBL to listen on LAN at 10.10.10.1 as usual. But then (because I never get a reply from 10.10.10.1) I just reject all connections to "This Firewall" on ports 8081 and 8443.
I won't get any stats or 1x1 gifs but web sites load fast and without ads. :) -
Hey all,
I'm trying to set up PFBlockerNG with GeoIP to block everything apart from my country by only PERMITTING my country, rather than selecting everything apart from my country and blocking inbound.So under GeoIP i have selected <mycountry>under Top 20 and then Permit Inbound
This creates a permit rule under floating rules but i think im confusing myself as to what is happening here.
I have a web server running on the WAN, for which i have a WAN rule that permits any source to port 80 and 443. If there is a floating rule that allows UK IP's is that not allowing all UK IP's / Ports into my network? At what point does the float rule come into it?
What i want is for the firewall to check if the source is from the UK, if not block, then if its destined for port 80 or 443, if not block.
So under General - Rule Order should i select:-
pfB_Pass/Match | pfSense Pass/Match | pfB_Block/Reject | pfSense Block/Reject
or (leaning to this one)
pfSense Pass/Match | pfB_Pass/Match | pfB_Block/Reject | pfSense Block/Reject
Thinking it will match on 80 or 443 then match on UK.
Or something else, am i going about this the correct way? I kind of expected a blocking rule somewhere for ! <mycountry>Sorry - getting my head in a right pickle here. ;D</mycountry></mycountry>
-
https://forum.pfsense.org/index.php?topic=102470.msg647719#msg647719
Then you can install or re-install the pfBlockerNG 2.1.1_2.
The install should update the MaxMind (this take 5+ minutes) and proceed to completion. -
hi all,
i have performed this change, still the pfBlockerNG is not seen anymore in gui.
anyone had this before? -
Where does it fails when you install it ? There must be some hints in the installation logs, system log or pfblockerng log.
-
php /usr/local/www/pfblockerng/pfblockerng.php dc
and as it's shown in the logs :
Download Process Starting [ 08/15/16 21:18:57 ]
/usr/local/share/GeoIP/GeoIP.dat.gz 200 OK
/usr/local/share/GeoIP/GeoIPv6.dat.gz 200 OK
/usr/local/share/GeoIP/GeoLite2-Country-CSV.zip 200 OK
/var/db/pfblockerng/top-1m.csv.zip 200 OK
Download Process Ended [ 08/15/16 21:19:31 ]Country code update Start
Converting MaxMind Country databases for pfBlockerNG.
Processing ISO IPv4 Continent/Country Data
Processing ISO IPv6 Continent/Country Data
Creating pfBlockerNG Continent XML files
IPv4 Africa
IPv6 Africa
IPv4 Antarctica
IPv6 Antarctica
IPv4 Asia
IPv6 Asia
IPv4 Europe
IPv6 Europe
IPv4 North America
IPv6 North America
IPv4 Oceania
IPv6 Oceania
IPv4 South America
IPv6 South America
IPv4 Proxy and Satellite
IPv6 Proxy and Satellite
IPv4 TOP 20
IPv6 TOP 20
Saving pfBlockerNG Reputation TAB
Country Code Update Ended - [ 08/15/16 21:24:37 ]–-----
tail -f pfblockerng.log
DNSBL - Adding Unbound custom 'include' option
Saving new DNSBL web server configuration to port [ 8081 and 8443 ]
Saving pfSense config…
VIP address configured. Widget Packet statistics reset.
Restarting Service DNSBL...**Saving configuration [ 08/15/16 19:16:28 ] …
Restarting Service DNSBL...and stays like this for 20 minutes.
-
Looks like the MaxMind database creation went ok.
Can you try to reinstall pfBlockerNG? -
Just did this:
install log looks like :
Upgrading pfSense-pkg-pfBlockerNG…
Updating pfSense-core repository catalogue...
pfSense-core repository is up-to-date.
Updating pfSense repository catalogue...
pfSense repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):Installed packages to be REINSTALLED:
pfSense-pkg-pfBlockerNG-2.1.1_2 [pfSense]Number of packages to be reinstalled: 1
[1/1] Reinstalling pfSense-pkg-pfBlockerNG-2.1.1_2…
[1/1] Extracting pfSense-pkg-pfBlockerNG-2.1.1_2: …....... done
Removing pfBlockerNG components...
Menu items... done.
Services... done.
Loading package instructions...
Removing pfBlockerNG...cat: /var/db/pfblockerng/dnsbl/.txt: No such file or directory
All customizations/data will be retained... done.
Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...
MaxMind GeoIP databases previously downloaded.
Adding pfBlockerNG Widget to the Dashboard... done.
Remove any existing and create link for DNSBL lighttpd executable... done.
Creating DNSBL web server start-up script... done.
Creating DNSBL web server config ... done.
Starting DNSBL Service... done.
Upgrading Adv. Inbound firewall rule settings ... no changes required ... done.
Custom commands completed ... done.
Executing custom_php_resync_config_command()...1 table created.
1/1 addresses added.
cat: /var/db/pfblockerng/dnsbl/.txt: No such file or directory
pkg: POST-INSTALL script failedCleaning up cache... done.
Success
just after the refresh i got a crash report:
Crash report begins. Anonymous machine information:
amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 24 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP Stack trace:
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP 1. {main}() /etc/rc.start_packages:0
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP 2. sync_package() /etc/rc.start_packages:90
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP 3. eval() /etc/inc/pkg-utils.inc:631
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP 4. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
[15-Aug-2016 21:57:31 Europe/Bucharest] PHP 5. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 -
Allowed memory size of 104857600 bytes exhausted (tried to allocate 24 bytes)
This tells me you only have 100M for memory limit.
What are the fixes you applied up to now to the system? -
/usr/local/pkg/pfblockerng/pfblockerng.inc
pfb_global();
// DNSBL Lighttpd HTTPS Daemon (Scans Lighttpd dnsbl_error.log for requested https domain names)
if (isset($argv[1]) && $argv[1] == 'dnsbl') {
set_time_limit(0);
pfb_livetail($pfb['dnserrlog'], 'dnsbl');
exit;
}and i guess is default.
and also usr/local/www/pfblockerng/pfblockerng.php - not modified.
-
And no modifications to /etc/inc/config.inc or /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
-
well based on my comments i did one to the first file:
// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
ini_set("memory_limit", "512M");
} else {
ini_set("memory_limit", "256M"); # increased from 128M
}–--
-
This change only affect i386 systems, you are on a amd64.
Did you raise the System / Advanced / Firewall & NAT Firewall Maximum Table Entries ?