PfBlockerNG v2.1 w/TLD
-
I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher. But it seems itself to contain some second level domains?
Hi Andrew453,
If I only used the TLD, it would be a simple process of looking at any listed Domain and seeing if it had only a second-level Domain (SLD) then block the entire Domain. However, there are suffixes like "uk.com" which is what I would call the TLD that is used to determine if there is one more level. So all of the TLDs (suffixes) in that file are known TLDs which is used in the determination process. Most of the file was taken from the "Public Suffix Registry".
-
Hi BBcan177,
Is there any good install/setup/configure instruction (video or guide) for the last version op pfblockerNG, that you could/would recommend?
Thanks for your advice, cheers Qinn
There is a pfSense Hangout that I did which can be used for an overview of the pkg functionality. However, apart from the three main pfBlockerNG threads in this forum, there isn't any other documentation.
-
Thanks for the quick reply. Darn :( I found this one can you can agree to this one?
https://www.youtube.com/watch?v=YLhDOaH0q5U
-
I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher. But it seems itself to contain some second level domains?
Hi Andrew453,
If I only used the TLD, it would be a simple process of looking at any listed Domain and seeing if it had only a second-level Domain (SLD) then block the entire Domain. However, there are suffixes like "uk.com" which is what I would call the TLD that is used to determine if there is one more level. So all of the TLDs (suffixes) in that file are known TLDs which is used in the determination process. Most of the file was taken from the "Public Suffix Registry".
Yes ok. That's exactly what I thought the file was for. (i.e. some eTLDs are longer than others, so you need a list e.g. .com vs .co.uk to work out what to treat as an eTLD)
The thing that was confusing me was there were some domains in the list that looked a bit odd, e.g.
myactivedirectory.com
mydrobo.com
mysecuritycamera.com
myshopblocks.com
myvnc.comI think all you're saying is that pfblockerng will treat those as eTLDs even though, strictly speaking, they aren't … which is fine.
p.s. a big thank you for implementing this. It was on my wish-list as I recall - https://forum.pfsense.org/index.php?topic=106534
-
Hi BBcan177,
I can't update h3x feed from available feeds list in pfBlockerNG v2.1.
It show below.
[ h3x ] Downloading update .. 200 OK Remote timestamp missing No Domains Found
Same here
And I can't let TLD Exclusion List working. Can you give a example or check it works?
Did you do a Force Reload after changing the list ?
I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.
I tried with Update, Cron , Reload.
[ malw_corpus ] Downloading update .. 200 OK
Remote timestamp missing
No Domains Found -
Hi,
Thank you for your hard work on this package :)After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packets![Desktop 31-07-2016 17.00.06-358.png_thumb](/public/imported_attachments/1/Desktop 31-07-2016 17.00.06-358.png_thumb)
![Desktop 31-07-2016 17.00.06-358.png](/public/imported_attachments/1/Desktop 31-07-2016 17.00.06-358.png)
[Video 31-07-2016 16.54.57.zip](/public/imported_attachments/1/Video 31-07-2016 16.54.57.zip) -
I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.
I tried with Update, Cron , Reload.
[ malw_corpus ] Downloading update .. 200 OK
Remote timestamp missing
No Domains FoundEach URL contains sites that were active in the last period (month, week, day or hour).
If you look at the 1hour or the 1day csv file, they only have one comment. The 1week and 1month have entries.
You should only choose one of the feeds according to your need. I guess most will pick the 1month URL.
-
Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package. It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information". This happens for the "Installed Packages as well as "Available Packages" tabs!
On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".
Tried disabling both pfblockerNG and DNSBL to no avail. Snort is disabled and the blocked hosts list is empty.
Now I cannot update, install or uninstall packages…. How do I remedy to this?
-
I've had that trouble before too. It happened when I was trying to change from the development thread for updates to the stable thread. I couldn't update anything. I eventually found some instructions to reinstall the main pfsense components from the command line. I ended up still on the development thread and didn't venture to try to change it back after that.
-
I've had that trouble before too. It happened when I was trying to change from the development thread for updates to the stable thread. I couldn't update anything. I eventually found some instructions to reinstall the main pfsense components from the command line. I ended up still on the development thread and didn't venture to try to change it back after that.
Not sure I understand that. I am not playing with development stuff, nor that I am configured to retrieve packages from development repos.. Just a vanilla pfsense install with pfblockerNG, snort and thats it. Not normal all of a sudden I lose connection to repos..
Also after a reboot I see these warnings in the main page:
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00 There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11 There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20
-
@lpallard:
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00 There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11 There were error(s) loading the rules: /tap/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20
These all seem to be related to the MaxMind IPv6 database. Looks like you will need to bump the pfSense max aliastable entries limit from 2M to 4M. If you enable aggregation in the general tab, it should condense the CIDRs and reduce the overall IP count. This changed due to using the new MaxMind Geolite2 database which seems to have smaller subsets of the data listed causing more IP entries to be added.
-
@CiscoX:
After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packetsI am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.
-
@lpallard:
Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package. It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information". This happens for the "Installed Packages as well as "Available Packages" tabs!
On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".
Tried disabling both pfblockerNG and DNSBL to no avail. Snort is disabled and the blocked hosts list is empty.
Now I cannot update, install or uninstall packages…. How do I remedy to this?
From the following thread:
https://forum.pfsense.org/index.php?topic=116019.0
I followed the ssh command line execution steps:
pkg update -f
pkg upgrade -fand the same problem was resolved.
-
@CiscoX:
After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packetsI am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.
Hi, No problem. Have a nice Holiday :)
-
Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.
-
Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.
Here are the original pfBlockerNG thread https://forum.pfsense.org/index.php?topic=86212.0
and the pfBlockerNG v2.0 w/DNSBL thread https://forum.pfsense.org/index.php?topic=102470 -
I am getting this error when I try to use the Spamhaus list in this tread.
===[ DNSBL Process ]================================================
[ EasywoElements ] exists.
[ SpamHouse_TLDS ] Downloading update .. 200 OK
Remote timestamp missing .
–--------------------------------------------------------------------
Orig. Unique # Dups # White # Alexa Final
----------------------------------------------------------------------
3 3 0 0 0 3
----------------------------------------------------------------------[ DNSBL FAIL ] [ Skipping : SpamHouse_TLDS ]
[1470071701] unbound-checkconf[87654:0] error: error parsing local-data at 38 '(xmlhttp.readystate 60 IN A 10.10.10.1': Syntax error, could not parse the RR
[1470071701] unbound-checkconf[87654:0] error: Bad local-data RR (xmlhttp.readystate 60 IN A 10.10.10.1
[1470071701] unbound-checkconf[87654:0] fatal error: failed local-zone, local-data configuration
[ Malware_1month ] Downloading update [ 08/01/16 12:15:01 ] .. 200 OK
Remote timestamp missing .
–--------------------------------------------------------------------
Orig. Unique # Dups # White # Alexa Final
----------------------------------------------------------------------
1221 956 0 0 0 956
----------------------------------------------------------------------[ Malware_1week ] Downloading update [ 08/01/16 12:15:04 ] .. 200 OK
Remote timestamp missing .
–--------------------------------------------------------------------
Orig. Unique # Dups # White # Alexa Final
----------------------------------------------------------------------
526 487 487 0 0 0
----------------------------------------------------------------------[ Malware_1day ] Downloading update [ 08/01/16 12:15:05 ] .. 200 OK
Remote timestamp missing .
–--------------------------------------------------------------------
Orig. Unique # Dups # White # Alexa Final
----------------------------------------------------------------------
48 47 47 0 0 0
----------------------------------------------------------------------[ Malware_1hour ] Downloading update .. 200 OK
Remote timestamp missing
No Domains Found–----------------------------------------
Assembling database... completed
Executing TLD
TLD analysis. completed
Finalizing TLD... completedOriginal Matches Removed Final
6062 5530 1 6061
Validating database... completed [ 08/01/16 12:15:08 ]
Reloading Unbound…. completed
DNSBL update [ 6061 | PASSED ]… completed -
Which Spamhaus URL are you using ?
this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.as for the H3X, only one is needed
https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896And do a Force Reload after making the modifications.
-
Which Spamhouse URL are you using ?
this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.as for the H3X, only one is needed
https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896And do a Force Reload after making the modifications.
Thank you.
i see my mistake now.
I was certain I had 2 feeds that contained data but I must have misplaced it? -
Read the first posts (or more ;)) of each of these threads:
pfBlockerNG
pfBlockerNG v2.0 w/DNSBL
pfBlockerNG v2.1 w/TLDYou will find some posts about IP and DNSBL Feed.
-
First of all thank you very much for your hard work and this awesome package!
I was just wondering is it possible to somehow change the Rule Order setting to something like:
pfB_Pass/Match | pfB_Block/Reject | All other Rules | (original format)
so the first IP-list would be the whitelist?Right now I can't seem to figure out how to make custom LAN IPv4 whitelist (Permit_Outbound) rule to be the first in the rule list of the LAN interface. If I manually move it first. Next list update puts it bellow the blocklists (Deny_Outbound) again. Right now only the default setting | pfB_Block/Reject | All other Rules | (Original format) is partly usable for me (whitelist won't work) and all other rule order settings just mess my original LAN rules.
I use Traffic Shaper queues in the floating rules so prefer not to move pfBlockerNG's rules in there too.
Is this somehow possible or what am I missing, thanks?
-
Which version are you using ?
with pfBlockerNG 2.1.1_2 I have these choices.
And you can still use the Floating Rules, it won't affect the Traffic Shaper rules.
-
Which Spamhouse URL are you using ?
this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.as for the H3X, only one is needed
https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896And do a Force Reload after making the modifications.
Thank you.
i see my mistake now.
I was certain I had 2 feeds that contained data but I must have misplaced it?The https://www.spamhaus.org/statistics/tlds/ page can be useful to find TLD to put in the TLD Blacklist.
-
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00 There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11 There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20
Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.
Rob
-
-
When I try to add a new TLD Blacklist i.e. "Google.com", I get the following error:
Clearing all DNSBL Feeds… completed
Executing TLD
Blocking full TLD/Sub-Domain(s)... |google.com| completed
TLD analysis completed
Finalizing TLD... head: 1: No such file or directory
tail: 1: No such file or directory
completedOriginal Matches Removed Final
0 0 -1 1
Validating database... completed
DNSBL enabled FAIL - restoring Unbound conf
/var/unbound/pfb_dnsbl.conf:1: error: unknown keyword '.google.com'
/var/unbound/pfb_dnsbl.conf:1: error: unknown keyword '60'
read /var/unbound/unbound.tmp failed: 2 errors in configuration fileAny ideas why DNSBL is failing to add the TLD blacklist entries?
Thanks.
-
Do you have any DNSBL feeds defined and enabled?
I did put Google.com in the DNSBL TLD Blacklist, ran a Force Reload and things looks ok.This is the part of pfblockerNG log after the last DNSBL feed
[ BBC_C2 ] Reload [ 08/08/16 15:25:16 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 332 332 331 0 0 1 ---------------------------------------------------------------------- [ DNSBL_IP ] Updating aliastable [ 08/08/16 15:25:22 ]... no changes. Total IP count = 280 ------------------------------------------ Assembling database... completed Executing TLD Blocking full TLD/Sub-Domain(s)... |google.com| completed TLD analysis...xxxxxxxxxxx completed ** TLD Domain count exceeded. [ 250000 ] All subsequent Domains listed as-is ** Finalizing TLD... completed ---------------------------------------- Original Matches Removed Final ---------------------------------------- 1323464 87716 169286 1154178 ----------------------------------------- Validating database... completed [ 08/08/16 15:31:20 ] Reloading Unbound.... completed DNSBL update [ 1154178 | PASSED ]... completed [ 08/08/16 15:32:02 ] ------------------------------------------ ===[ Continent Process ]============================================
-
Do you have any DNSBL feeds defined and enabled?
I did put Google.com in the DNSBL TLD Blacklist, ran a Force Reload and things looks ok.No, I only want to block a couple domains and not use any DNSBL lists.
Must I have a DNSBL list for TLD to work?
-
Do you have any DNSBL feeds defined and enabled?
I did put Google.com in the DNSBL TLD Blacklist, ran a Force Reload and things looks ok.No, I only want to block a couple domains and not use any DNSBL lists.
I solved the issue by create a dummy feed, the inside the feed add the "Custom Block List" this seems to allow the domains to be blocked.
Is this the expected behaviour?
Well maybe nobody tested a setup without any DNSBL feed. Using a Custom Block List does create a feed and seems to remove the issue.
-
Well maybe nobody tested a setup without any DNSBL feed. Using a Custom Block List does create a feed and seems to remove the issue.
BBCan177 got back to me even though he was on vacation (thanks!).
Basically create a dummy DNSBL feed, in the bottom section called Custom Domains, add the subdomains there. This will block the domains correctly.
-
Hello BBcan177 and pfsense users,
Great work on pfblockerng. I have one question. I have DNSBL listening port 8081 and when I type 10.10.10.1:8081 I get the gif image. Now when I try the DNSBL SSL listening port 8443 10.10.10.1:8443 I get the connection was reset. So it doesn't work.
I have been doing some reading on why I was getting the "googleads.g.doubleclick.net" and in one post someone talked about limiters causing problem. I don't have any limiters setup. I think it's because DNSBL SSL isn't working.
Anyone have an idea why DNSBL SSL isn't working for me ?
Thanks
-
http://10.10.10.1:8443 return a gif
It should be https://10.10.10.1:443 but that doesn't return and doesn't it log to dnsbl.log either.
-
I tried https://10.10.10.1:443 and it returned a gif so that works. Anyone else have the google ads certificate popup? I get the popup in Safari and in Firefox I see the error message where the ads used to be.
It would be nice to have just empty space without the error.
Thanks Ronpfs for your reply.
-
You have the URL that generate the errors so I can reproduce here?
-
I have been surfing the web to find one. Just cause i'm trying I am having a hard time.
This site did it once on my desktop but didn't do it on my phone.
https://www.instantssl.com/ssl-certificate-products/https.html
-
Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.
Rob
Hi Rob,
The memory issue will be fixed with v2.1.1_3, however, you don't want to reverse the "Registered" vs the "Represented" entries. Please refer to the link in the GeoIP tabs "Whats new in GeoIP2" to help you understand the difference between those two types.
-
Here is a link to PR # 175 for pfBlockerNG v2.1.1_3 (This PR first needs to be reviewed and merged by the pfSense Devs)
The 2.1 release was beta tested for several months with approx a dozen testers with varying hardware (1GB-16GB, i386-AMD64). The MaxMind database is updated the first Tuesday of each month.
After reviewing the latest MaxMind IPv6 database, you can see below that the IPv6 line count increased 5 fold vs the previous month. This is a significant increase and as such the package required more PHP memory to be able to process the updated MaxMind database. The two Countries that changed significantly are US and DE, so until MaxMind has resolved this issue, you might consider not using those two IPv6 GeoIP lists.
This month:
1,147,813 US_v6.txt
1,137,159 DE_v6.txtLast Month:
222,937 US_v6.txt
205,571 DE_v6.txtI have contacted MaxMind support, to get some clarity on this issue, with the following response:
Thank you for contacting support. We did also observe a significant increase in IPv6 mappings, due to more specific blocks being mapped, starting with the 2016-07-05 release, and we are currently investigating what may be causing such an increase in the recent releases.
We do indeed aim to list the IP networks as efficiently as possible to help keep CSV file sizes down, so ideally the file sizes should not continue to increase dramatically once a fix is deployed. However, for the time being, the additional mappings shouldn't adversely affect the lookup results.
Thank you for the additional information; I've passed along your observations to our developers. At this time, we unfortunately do not have an ETA on a fix, but when I do receive any news, I'll be in touch.
I have re-factored the code to be able to handle this change in database size. This will reduce the overall PHP memory required. Its not recommended to "Block the world"; however, should your configuration follow this approach, then you may need to increase the pfSense Advanced "Firewall Maximum Table Entries" to 4M (or higher depending on the other Table entry size).
In my absense (vacation), forum user RonpfS steped up and helped convey some temporary workarounds and help users who were affected by this issue. I would personally like to extend my appreciation for all of his efforts. Its what "Open Source" is all about, and I encourage more people to get involved.
Everyone needs to bump his Karma! Thanks again!
Additional Changes:
-
Added a 'placeholder' for undefined MaxMind 'Represented Countries'. This is necessary as month-to-month MaxMind Updates seem to have fluctuations that can cause a list to become undefined/redefined.
-
Improved DNSBL Firewall Permit Rule options (Added OpenVPN, IPsec and Interface group options)
-
Improved removal of DNSBL VIP address mapping when DNSBL is disabled.
-
Added DNSBL parser for Alienvault OTX pulses. This will only collect "Domains". You can add the same feed into the IPv4 tab to collect "IPs".
-
Added a "Disabled" option to the CRON update options.
-
Additions to the DNSBL TLD suffixes
-
Fixed issue with widget not clearing DNSBL packet counts
-
-
Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.
Rob
Hi Rob,
The memory issue will be fixed with v2.1.1_3, however, you don't want to reverse the "Registered" vs the "Represented" entries. Please refer to the link in the GeoIP tabs "Whats new in GeoIP2" to help you understand the difference between those two types.
Not sure if we are talking about the same thing there, i was referring to the North American IPV6 tab which my selection from a few countries which i picked changed to unselecting those ones and selecting all the ones i had not picked.
This is what i meant by it had reversed the selection and seemed to be the cause of all my memory use as you have pointed out its is now several million.
Rob
-
pfBlockerNG v2.1.1_3 has been approved and merged and is now available to be installed/Upgraded.
I have noticed that some of the installation log messages are not appearing in the pkg install window. I am investigating that; however, the installation is still occurring in the background.
The MaxMind conversion will take a few mins to process, so wait for it to complete.
UPDATE
I pushed a fix for this just now. The pfBlockerNG version is now 2.1.1_4
-
pfBlockerNG v2.1.1_3 has been approved and merged and is now available to be installed/Upgraded.
I have noticed that some of the installation log messages are not appearing in the pkg install window. I am investigating that; however, the installation is still occurring in the background.
The MaxMind conversion will take a few mins to process, so wait for it to complete.
UPDATE
I pushed a fix for this just now. The pfBlockerNG version is now 2.1.1_4
Thanks I noticed that the log while updating to 2.1.1_3 didn't gave sign it finished, after updating to 2.1.1_4 all seems well ;)
btw I would like to test a php /usr/local/www/pfblockerng/pfblockerng.php dc but as I have dramatically changed the hardware I cannot compare it to when the memory issues occurred (see https://forum.pfsense.org/index.php?topic=102470.750 )!
Cheers Qinn