PfBlocker
-
Marcello or anyone with a good idea,
I am pulling my hair out over one new (to me) LIST.http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt
I've created, combined, deleted and used dozens of lists, both gz and txt, and had no issues. I can get THIS list built in pfBlocker but it never populates in rules, alias or in the dashboard widget. I've looked around the logs and not seen anything (good chance I'm looking in the wrong place). I know its a big list but some of yours are sizeable too. Per a discussion around snort on another thread I even bumped up my "Firewall Max Tables" and "Firewall Max Tables Entries". I'm running a 64bit copy of 2.1 with 4G memory that is only about 24% used.
I've made it a standalone rule, I've tried it in "deny" and "alias only" modes; neither way works. I've included it in another list under a single alias. Random selection of IPs from this list never make it into the tables. I've not had any problems with any other lists. I can get to THIS list from any browser behind the router/UTM so access through the network doesn't seem to be an issue. Any ideas? Am I missing something?
TIA,
Rick -
Bump!
OK, so I've tried other avenues and just can't get this list to build correctly. It shows in the list of "Lists" but just doesn't seem to go beyond that. I've killed off every other "added" list and started new with this one. All others build accordingly, this one just does nothing.
-
I"ve checked every log I can find but don't see any errors relating to this. Does anyone know exactly where an error would post if it is being logged?
-
could there be a corrupt marker somewhere that I need to kill? where would that be?
Thanks for any help or ideas?
Rick
Edit: had the time today so on the advice of two others I completely removed pfBlocker. Removed its files/directories. Reinstalled the whole package. Rebuilt all the lists I wanted under the lists tab… some txt, some gz. Every list rebuilt fine except this @#$$&#^ Russian Business Networks list. Unless someone has an idea, I'm going to throw in the towel on this one and just go with the RBN rules under SNORT. Marcello? Tommyboy?
Add: Does anyone have a confirmed instance of this list actually working?? -
-
I for one can't get the mentioned list to work. I've posted a php error related to the list on this thread when pfSense 2.1 was still in beta and it was showing the php error that occurs with said list.
-
Hi, got exactly the same problem, since a long ago, tried everything but this list http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt never loaded. This morning found this list http://rules.emergingthreats.net/blockrules/rbn-ips.txt that loads flawlessy. In my opinion it is quite the same listing. Try it.
-
I just made a URL Table Alias with http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt on a 2.1-RELEASE Alix nanoBSD. The table loads into pf with 9252 entries. Used it as source in a block rule on WAN and it works. So the list itself is not bad. And you can easily use it anyway without pfBlocker, just using the ordinary URL Table Alias and Firewall Rules features of pfSense.
-
I just made a URL Table Alias with http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt on a 2.1-RELEASE Alix nanoBSD. The table loads into pf with 9252 entries. Used it as source in a block rule on WAN and it works. So the list itself is not bad. And you can easily use it anyway without pfBlocker, just using the ordinary URL Table Alias and Firewall Rules features of pfSense.
Probably a stupid question (you are used to that coming from me by now ;D) but what actually is the added value of pfBlocker, given that obviously you can do the same that does without it, given your comment.
-
I can run this to see what happens getting the files:
$url_list1 = file("http://rules.emergingthreats.net/blockrules/rbn-ips.txt"); var_dump(count($url_list1)); $url_list2 = file("http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt"); var_dump(count($url_list2));
and I get:
int(9194) Warning: file(http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden in /usr/local/www/exec.php(246) : eval()'d code on line 7 int(1)
rbn-ips.txt is fetched OK by PHP into an array of 9194 entries.
RussianBusinessNetworkIPs.txt does not come, something blocking it from being read with PHP file()Additional info: If I change pfblocker.inc sync_package_pfblocker() to use the function download_file() from pfsense-utils.inc, downloading it first to a local file then letting the rest of the code parse a local copy, then it works. So the code in /etc/inc/pfsense-utils.inc:download_file() is able to download the list OK, getting 9251 entries.
Maybe pfblocker should use download_file() rather than PHP file()?
(That would need testing against a bunch of things people are using - some other list download might break???) -
Hi, got exactly the same problem, since a long ago, tried everything but this list http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt never loaded. This morning found this list http://rules.emergingthreats.net/blockrules/rbn-ips.txt that loads flawlessy. In my opinion it is quite the same listing. Try it.
Thanks! had no problem getting this one to load either. Looking at the site, at least this one seems more current as well. It's just a couple weeks old, not 2 years. I had brought the offend file local so I randomly picked about 30 addresses and they were in both so I'll hope the slightly lower line item count is because of newer data.
Rick
-
using pfblocker for the list management allows you to enter all the lists in a single alias. This is not possible for the regular aliases (url + url table). It's either many small lists, or one huge list, with those.
I'll start using http://rules.emergingthreats.net/blockrules/rbn-ips.txt since it causes fewer problems with people. Thanks for the info. Expect the update to come with the next blueprint update.
EDIT: Just checked and http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/emerging-rbn-malvertisers.txt should be http://rules.emergingthreats.net/blockrules/rbn-malvertisers-ips.txt, so it's 2 upcoming updates to the blueprint.
As far as I can remember those 2 were chosen because of lack of the rules.emergingthreats.net lists. -
@jflsakfja:
using pfblocker for the list management allows you to enter all the lists in a single alias. This is not possible for the regular aliases (url + url table). It's either many small lists, or one huge list, with those.
I'll start using http://rules.emergingthreats.net/blockrules/rbn-ips.txt since it causes fewer problems with people. Thanks for the info. Expect the update to come with the next blueprint update.
EDIT: Just checked and http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/emerging-rbn-malvertisers.txt should be http://rules.emergingthreats.net/blockrules/rbn-malvertisers-ips.txt, so it's 2 upcoming updates to the blueprint.
As far as I can remember those 2 were chosen because of lack of the rules.emergingthreats.net lists.Don't know your name so I'll just use the first two initials and say thanks JF!! (significant if you're a Phillip Dick fan… and how can you follow BOHP and not be a Dick fan)
One request; If possible and not too much hassle, could you somehow highlight the changes from your last blueprint?
I must say, since switching over to your method and using the rules to do pfblocker's work, even with more rules active under SNORT, system is much faster AND using much less memory. Which, made it possible to commit more memory to Squid which helps even more!
Thanks,
Rick -
I'm interested in the country-blocking abilities of pfBlocker.
I've got assets that are 99.99% of the time only accessed from within my country. So, I've added a rule with my country as the block list, then inverted the match so any traffic from OUTSIDE the country is dropped. Seems to work well enough, but can someone comment as to:
-
Where does pfBlocker gets its IPs from?
-
How often does pfBlocker update its IP list?
-
What is the likelihood that an IP range will be assigned to a country but won't be picked up by pfBlocker?
-
-
I'm interested in the country-blocking abilities of pfBlocker.
I've got assets that are 99.99% of the time only accessed from within my country. So, I've added a rule with my country as the block list, then inverted the match so any traffic from OUTSIDE the country is dropped. Seems to work well enough, but can someone comment as to:
-
Where does pfBlocker gets its IPs from?
-
How often does pfBlocker update its IP list?
-
What is the likelihood that an IP range will be assigned to a country but won't be picked up by pfBlocker?
1/ Here - The lists are 2 years old. ::)
2/ Never, the lists have gone commercial quite some time ago.
3/ Pretty high, given the above.All the country-based stuff should have been removed altogether from the package quite some time ago, useless.
-
-
I agree or the lists should be updated.
-
That is most unfortunate. I don't suppose anyone knows if an up-to-date country list is provided somewhere?
-
That is most unfortunate. I don't suppose anyone knows if an up-to-date country list is provided somewhere?
You can add the Country Block lists from IBlock Lists.
https://www.iblocklist.com/lists.php?category=country
I haven't tested it, but they are listed there.
-
Good afternoon,
Great thread so far, thanks for all your great detective work. :)
Im running pfs at a minimum with pfBlocker and system patches. I'm using nested alias lists: two url aliases, Evil_Lists_1 and 2, each containing 3-5 localhost list urls. All pfBlocker lists are set to "alias only" as I prefer to create my own rules (this also seems to be a better way according to others as well). I then created two WAN rules for each list, one to block inbound traffic and another to reject outbound traffic, and also two LAN rules for each interface rejecting outbound traffic for each list. They seem to be blocking properly as far as I can tell and the correct CIDR numbers are showing in the widget at all times but the lists are always shown as down in the widget. I have tried renaming my rule descriptions based on what Marcello and others have recommended in earlier posts ("lead with pfblocker* and dont end with rule") in varying ways but it still wont show as up in the widget.
Not a serious problem but I like the widget and I want it to work. Any suggestions?
Also, would I be better off having a rule for each original pfBlocker alias? I prefer steamlined rules and less of them, is there any benefit to individual rules other than the widget working?
Thanks again for an informative thread.
-
-
Not a serious problem but I like the widget and I want it to work. Any suggestions?
Try deleting the underscores from your alias descriptions, e.g., use 'pfBlockerEvilList' rather than 'pfBlockerEvil_List'. The widget doesn't seem to like spaces or special characters.
-
Semi-fix, I added the prefix "pfBlockerBadList1" and "pfBlockerBadList2" to the beginning of the respecive WAN rule for each and now both "pfBlockerBadList1" and "pfBlockerBadList2" show up in the widget and are recognized as being up; while this does not solve the original widget problem it does let me know at a glance that my lists are functioning; and as a nice additional benefit the packets that are blocked by the respective individual nested aliases within the two lists are still recorded correctly under the original widget list names. Sweet! :D
Here is a pictorial example of the lists:
(The blocked packets are not showing now beacause I restarted, but will show up under the individual list names, not BadList1/2, I like this.)
-
All issues we found has already been fixed.
The last thing to code is lists update frequency.
can you add the option to update weekly? it only allows up to daily right now ..which is awesome.. but sometimes iblocklist.com denies updating that frequently. at least it does on peerblock 1.2+ …
???
-
All issues we found has already been fixed.
The last thing to code is lists update frequency.
can you add the option to update weekly? it only allows up to daily right now ..which is awesome.. but sometimes iblocklist.com denies updating that frequently. at least it does on peerblock 1.2+ …
???
Isn't this done in a cron job? I think you can just set the cron job wday from a number 0-6 to set the day (somebody correct me if I am wrong).
-
Isn't this done in a cron job? I think you can just set the cron job wday from a number 0-6 to set the day (somebody correct me if I am wrong).
The pfBlocker frequency is set in the pfBlocker GUI per each individual Blocklist. (Never, 1hour, 4hour, or 24hours)
The Cron job that you see is to Update the URL tables which in most cases shouldn't be played with.
-
Is there a way that we can see what IP's are being currently blocked by PFblocker? The widget shows 85 blocked for a given list, I would like to know which one from that list are being blocked?
Thanks
-
To see what IP's were blocked by pfBlocker, you need to look at the Firewall log.
The widget only displays the total number of ips per blocklist.
-
I'm also having a problem "finding" the IP's that the widget is blocking. I go Status: System logs: Firewall but I see a bunch of traffic blocked. I added some allow rules in pfblocker and I see some IP's allowed but don't know what list they are from or any way to tell they are from pfblocker.
I grepped my syslogs and do not see anything except for stuff like this: /pkg_edit.php: [pfblocker] pfblocker_xmlrpc_sync.php is starting. I can see the see the pfblocker lists in the widget on the dashboard counting packets and I can see the IP lists in the Diagnostics: Tables so I know it's working, but I'm not sure where to be looking for detailed info such as logs that say pfblocker and then the list name and the IP's that were blocked. Any info will help
-
For the blocklists that you created in pfBlocker, did you use "aliases"? This is the preferred way to utilize pfBlocker.
Once the aliases are created by the pfBlocker program, you can add block or reject rules on each interface utilizing the defined aliases.
-
Thanks I did not realize that alias_only was the proper way to implement this.I was originally using the deny drop down. I've modified the lists to use alias_only and added the rules on the interfaces. I'm still unclear as to how to know that pfBlocker was the reason an IP was blocked (or more specifically which alias I would need to check).
Is there a log entry that's created that say something like "pfBlocker [alias name]: log entry"and the alias name or something in the log?
Also is it normal to have the pfBlocker widget show the lists and the "up" status but the packets field is blank? Sorry if I'm missing something obvious.
-
When you create a block or reject rule for an alias, any ip addresses that are listed in a particular alias will shows blocks/rejects in the firewall log.
I like to have separate aliases so i can tell which list/alias is blocking/rejecting.
The widget will show a green up arrow once a rule is associated to an alias name. The packet counter has never worked from what I can tell.
When you look at the firewall logs it will show the block/reject by interface, Rule name, ip address etc..
If you modify a rule, it can mismatch the alerts to the firewall logs. This hopefully will be fixed in the next release (2.2). So before making changes to rules it's a good habit to look at the firewall logs first.
-
I understand the block/reject rules based on an alias. That part is working as expected for the most part and there are now sites that I can't ping. In my logs I do not see any reference to pfBlocker or which alias the IP's belong to. You stated that you have separate aliases (and I do too) but I'm wondering if there is another step I'm missing.
Basically, this isn't the case for me but this is what I'm after:
When you look at the firewall logs it will show the block/reject by interface, Rule name, ip address etc..
All I get is "rule <number>(match): block in <interface>: and then some protocol stuff, ttl, id, offset, and flags. Maybe something needs to be cleared or reset?</interface></number>
-
When you add the rules, make sure you click "log packets that are handled by this rule" so that they show up in the Firewall logs
-
Under Firewall: pfBlocker, I have the "Enable Logging" checked. Also for each pfBlocker alias (one firewall rule per alias) I also have "Log packets that are handled by this rule" checked. I had that enabled logging checked for almost every single rule previously. Would someone be able to give me a string to grep for to see if I have any reference to that in my logs? I grepped for pfBlocker (with -i) and nothing except status messages come up. That's why I wonder if I needed to reset or reconfigure something to get it to work as expected.
-
That's odd.. did you check the firewall settings tab settings? If you don't see any alerts from pfblocker in the firewall log than greping from a shell won't get you anywhere.
When you hover over a rule that holds a pfblocker alias, does it show a list of ip addresses? Or check to see if in the Tables menu that the aliases have addresses?
If you have a block or reject rule on the LAN interface for a pfBlocker list, try to ping one of those addresses and see if it generates an alert in the firewall tab.
Maybe a reinstall of pfBlocker or start a new thread for this issue?
EDIT - if you want to grep the logs, grep for "block in on". If they alerts were there, they would also have the pfBlocker alias name (rule name)
grep -i "block in on" /var/log/*
-
Yes, the alias do have a list of IPs (with the exception of 2 lists that were massive). I've removed the 2 lists that were acting up and confirmed both in the tables menu and by hovering over the aliases. I do have the LAN rules and while I cannot ping the IP's, no logs are generated from it. I have reinstalled pfBlocker before I started this thread but I'm going to wipe the settings and remove pfBlocker and then start fresh. I figured something isn't working properly since it seems everything is in order. Thanks for the pointers though.
-
If the blacklists are large, it can take a little longer to view the list as you hover over the rule. They are most likely working correctly.
-
I'm using pfblocker with a simple configuration and it might be the culprit for some wierd behavior i get in my network…
Configuration:
Inbound Interfaces: WAN
Inbound deny action: BlockOutbound Interfaces: LAN
Outbound deny action: RejectEvery list & country blocked is with "deny inbound".
Is the setting for the outbound interfaces needed if all the actual lists and countries are deny inbound only?...
i suspect pfblocked is blocking some valid outgoing traffic for a specific lan service (some outgoing sendmail) but i cant find anything in the logs.
This behavior only gets fixed after i restart the whole firewall, and works fine for about 10-15 days.After that, no outgoing mails are possible, till i restart the machine. Everything else seems to be working fine, only connections to the specific mail relay server.
Please give me some insight on what i should be looking for to fix the problem permanently.
-
The Country Codes is not being supported any longer and should not be used as it is Out of Date this functionality should be removed from the package. Hopefully pfBlocker will be getting a face lift in pfSense v 2.2.?
-
Hello,
So I just installed pfBlocker and I am trying to find some good ad blocking lists. My comparison list(s) are from AdAway on Android which pulls from the below sites.
http://hosts-file.net/.%5Cad_servers.txt
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts$showintro=0&mimetype=plaintext
http://winhelp2002.mvps.org/hosts.txtThe test site I am using to check the ad blocking is http://extreme.outervision.com/psucalculatorlite.jsp
When using AdAway on Android I don't get any ads along the sides/top of the above site. I tried using the Bluetrack ad list from here http://list.iblocklist.com/?list=dgxtneitpuvgqqcpfulq&fileformat=p2p&archiveformat=gz but I still get ads on the site.
Can pfblocker use a list of URLs or does it only do IP address? If it only does IP address I guess that is fine, I have made a bash script that will pull the lists from the AdAway sites and convert the URLs into IP address. My only issue with the script I have is I don't know of a way to convert all of the IP addresses into a smaller list of CDIR networks so I have just made each IP into its own network by adding a /32 which I am sure is an awful idea. So if anyone knows how to convert a huge amount of IP addresses into CDIR notation or if there is a way to use URLs with pfblocker can you please let me know. Thanks.
-
Hi w1ll1am,
These lists are all URL and can't be used by pfBlocker.
pfBlocker only accepts the following formats:
Note:
Compressed lists must be in gz format.
Downloaded or local file must have only one network per line and could follows PeerBlock syntax or this below:
Network ranges: 172.16.1.0-172.16.1.255
IP Address: 172.16.1.10
CIDR: 172.16.1.0/24pfBlocker already can accept the IBlock lists without any conversion (as they are in a range format already), but the following will also convert the file which you could link pfBlocker to a local txt file.
The following will download and convert that file to a range acceptable by pfBlocker.
fetch -v -o infile.gz -T 20 ?list=dgxtneitpuvgqqcpfulq&fileformat=p2p&archiveformat=gz
gunzip infile.gzcat infile | cut -d':' -f2
I posted several lists here:
https://forum.pfsense.org/index.php?topic=73353.msg402927#msg402927
-
Thanks BBcan17. Those lists look good I will use some of them. I managed to get my script to work with the AdAway lists.
I have posted it on paste bin http://pastebin.com/xJYz9qgW
I have also posted the cleaned up list of URLs from those three sites http://pastebin.com/vFbA20CG which will make the script much quicker if you pass it into the script at run time, it will only update newly added sites. Which you could then add the new IP address to the IP list for those URLs as of 5/23 http://pastebin.com/f3kUW3arThere are about 10000 IP address and they seem to be blocking a ton of traffic
I have not tested it a lot but I have noticed some sites that are blocked that shouldn't be like unix.stackexchange.com ( already removed from the pastebin IP list )
please excuse my poor bash code.
![Screenshot - 05232014 - 11:10:29 AM.png](/public/imported_attachments/1/Screenshot - 05232014 - 11:10:29 AM.png)
![Screenshot - 05232014 - 11:10:29 AM.png_thumb](/public/imported_attachments/1/Screenshot - 05232014 - 11:10:29 AM.png_thumb) -
Hi w1ll1am,
Spammers must die! There is also a website called "Phishtank" that has Phishing URL's posted also.
I think you might get some false positives with this method. As some of these hosts are actually hosted by Big ISP's.
These URLs would be better suited for DNS lookup blocking methods (hosts file). Need to find a way to weed out the false positives.
I would like to have this accomplished on my MS Active Directory DNS server so that it can handle all of the workstations it manages and filter out DNS requests to these URLS. It would be a good addition to the DNS Forwarder/BIND etc packages in pfSense also.
I pulled some of the hosts from the first link and ran a Dig command. It shows as Yahoo.
[1]
dig +short ypn-js.overture.com
us.srv.ysm.yahoodns.net.
98.139.225.108[2]
dig +short 13.ptp22.com
50.62.128.39IPVOID shows it being Registered by GoDaddy.
http://www.ipvoid.com/scan/50.62.128.39/[3]
dig +short admeld.comGoogle
173.194.43.97
173.194.43.103
173.194.43.98
173.194.43.105
173.194.43.110
173.194.43.101
173.194.43.99
173.194.43.100
173.194.43.102
173.194.43.96
173.194.43.104http://kb.bothunter.net/ipInfo/nowait.php?IP=173.194.43.97