PfBlockerNG

tonymorella

Has anyone else looked at Critical Shiack's Free Threat Intel Marketplace and thought how it could be used in pfsense?? They already have 104 feeds, and doing normalization on them. The dash board allows you to create collections from the differnet feeds which could in theroy be used with pfBlockerNG, seems like a good match.

https://intel.criticalstack.com/

Edit: Just set this up on a test system and used the command line client to export the final file which combined sources from 10 different lists, example of the output.


#fields	indicator	indicator_type	meta.source	meta.do_notice
62.210.72.158	Intel::ADDR	from http://rules.emergingthreats.net/blockrules/emerging-botcc.rules via intel.criticalstack.com	F
62.245.45.132	Intel::ADDR	from http://rules.emergingthreats.net/open/suricata/rules/compromised.rules via intel.criticalstack.com	F

Next step of getting it into a format pfBlockerNG can import is simple, so we can make this work. But a plugin that hooks directly into criticalstack.com via an API would be a powerful tool, could then be uses in any package via aliases.

Thoughts?
Tony

BBcan177

I've looked at C.S. before, but I don't really see any current benefit to integrate it into pfBNG.

All of the feeds are free and can be used directly in pfBNG 'as is'. This allows greater control over the feeds and on the frequency of the updates to the Source Feeds. The de-duplication (normalization) in CS is done on a "Class" basis, versus as a whole with pfBNG. Not to mention that pfBNG can take into consideration Country/Continent blocking and further de-duplicate redundant IPs that are already being blocking by a Country/Continent blocklist. I also assume that de-duplication in CS is not done on a CIDR basis as in pfBNG and instead is de-duplicated on a 1:1 basis.

It should also be noted that CS formats the Feeds to be used primarily with BRO. There are also limitations/efficiency issues with large lists as is noted on the CS site. Also having CS process these free Feeds, add extra text per line, and then have pfBNG strip away the extra text to get the IP/Domain is a bit redundant when the Free Feed can be downloaded directly with pfBNG. Some of the Domain based Feeds also mix some IPs which pfBNG can process into an IP Blocklist automatically.

pfBNG can also make use of "Reputation" to further analyse the maliciousness of an IP range.

pfBNG v2.0 (expected to post a Pull Request in the next few weeks), will have DNSBL Domain name blocking via Unbound DNS.

Of the 104 Lists currently available in CS, most of those are listed in the Script that I have provided in this Thread previously. Some of the Sources (BBC, hpHosts for example) have listed all of their feeds individually (some Feeds provide a combined Feed URL) which exaggerates the number of Feeds available. Most of the Domain based Feeds provided in CS are also being beta tested in the DEV version of pfBNG by several beta testers. I have written a Domain parser for the 'Original' Source Feeds and to use the CS Domain based feeds, I would have to re-write the code to allow for the change in format.

I also prefer to deal with the 'Original' provider of Feeds instead of using a 'middleman' approach. If anything, CS can be used as a listing of Feeds that are freely available, and these can be used directly in pfBNG without the need to have the Feeds processed by CS.

Next steps for pfBNG will be to implement an improved method to add Source Feed URLs, instead of manually adding these one by one. I am considering some scenarios for upcoming releases.

TMiland

@BBcan177:

@marcus556:

Yes, I've got it all worked out now very good work sir on this package.

As far as the AdBlock on iblock, you are right it isn't the greatest. I have a question since your doing most of the design it seems, when 2.0 expected for one and two are the list going to be limited to only IP lists or could we say use the AdBlock plus list aka easylist?

Yes I am the Developer of the package .. I haven't put much development into it for the past couple weeks, since v1.0 was released.. Maybe another month or so… I want to make sure that its stable and all the tires kicked every which way by the beta testers that helped me with v1.0 :)

You can see some more Features in this link:
https://forum.pfsense.org/index.php?topic=78356.msg477682#msg477682

I have all of the features in the link working and will be working on Easylist next...

Hi, are there any progress with the easylist integration? :)

Regards.

abujammy

I just had a huuuuuge issue with pfBlocker and I think I'm missing a few things and I was hoping someone could help me out.

Firstly, how do you completely wipe out all cached lists and force pfBlocker to redownload everything? I was getting a super weird error in the update logs about not being able to match a cidr, then it wouldn't update, then the dashboard widget showed "pf Error" where it usually shows how many firewall rules are attached to an alias list.

I didn't do anything weird. All I did was update some lists. Then all hell broke loose. It was even reverting my firewall configuration to a previous state. It was a complete clusterfuck and I've been trying to fix it for the last 3 hours.

Any help would be appreciated. Thanks!

abujammy

Oh yeah, and where does pfBlocker keep the cached versions of the lists on disk? I was looking for them in hopes that I could delete them and have it redownload all of them.

BBcan177

@abujammy:

Oh yeah, and where does pfBlocker keep the cached versions of the lists on disk? I was looking for them in hopes that I could delete them and have it redownload all of them.

Goto the "General tab", unclick "keep settings", then unclick "Enable"… Click "Save"... all previous files are wiped. Reverse the above to re-enable...

tonymorella

@BBcan177:

I've looked at C.S. before, but I don't really see any current benefit to integrate it into pfBNG.

All of the feeds are free and can be used directly in pfBNG 'as is'. This allows greater control over the feeds and on the frequency of the updates to the Source Feeds. The de-duplication (normalization) in CS is done on a "Class" basis, versus as a whole with pfBNG. Not to mention that pfBNG can take into consideration Country/Continent blocking and further de-duplicate redundant IPs that are already being blocking by a Country/Continent blocklist. I also assume that de-duplication in CS is not done on a CIDR basis as in pfBNG and instead is de-duplicated on a 1:1 basis.

It should also be noted that CS formats the Feeds to be used primarily with BRO. There are also limitations/efficiency issues with large lists as is noted on the CS site. Also having CS process these free Feeds, add extra text per line, and then have pfBNG strip away the extra text to get the IP/Domain is a bit redundant when the Free Feed can be downloaded directly with pfBNG. Some of the Domain based Feeds also mix some IPs which pfBNG can process into an IP Blocklist automatically.

pfBNG can also make use of "Reputation" to further analyse the maliciousness of an IP range.

pfBNG v2.0 (expected to post a Pull Request in the next few weeks), will have DNSBL Domain name blocking via Unbound DNS.

Of the 104 Lists currently available in CS, most of those are listed in the Script that I have provided in this Thread previously. Some of the Sources (BBC, hpHosts for example) have listed all of their feeds individually (some Feeds provide a combined Feed URL) which exaggerates the number of Feeds available. Most of the Domain based Feeds provided in CS are also being beta tested in the DEV version of pfBNG by several beta testers. I have written a Domain parser for the 'Original' Source Feeds and to use the CS Domain based feeds, I would have to re-write the code to allow for the change in format.

I also prefer to deal with the 'Original' provider of Feeds instead of using a 'middleman' approach. If anything, CS can be used as a listing of Feeds that are freely available, and these can be used directly in pfBNG without the need to have the Feeds processed by CS.

Next steps for pfBNG will be to implement an improved method to add Source Feed URLs, instead of manually adding these one by one. I am considering some scenarios for upcoming releases.

Thanks for the great feedback! Missed the script post, maybe put that on the first post??

Tony

cyberbot

I am using pfblockerng and blocking all country which I don't do business with .
Europe is not blocked . This weekend I am away to Germany " which is not blocked "
Right now I am on the hotel but the connection to the server behind pfsense is blocked .
Why it's blocking Germany IP even i don't block Germany country on pfblockerng?
IP where I am connecting from is
62.143.80.249

Is this ip on a different categories ?

musicwizard

i also had something strange.

i don't have country blocking enabled at all. But PFsense did block the forums.pfsense IP. which is on the United States list from i-blocklist

United States:208.123.64.0-208.123.95.255

but i don't use that list nor do i use country block.

When i disabled the floating rules in pfblocker settings site worked again. when i put it on floating again it still works the site. maybe after a while it starts blocking again.

Edit:
i used a list before that had a few sites on it that i use a lot. ( probably false positive ) so i removed that list from pfblockerng. i also already cleared the files manually and disabled pfblockerng and re-enabled it but it keeps blocking sites. Like www.nvidia.com etc. But once i disabled the floating rules it works again

edit2:
i did disabled pfblockerng totally. removed all files manually so it has to re-download all files when enabled again. restarted the system. enabled it again ( with floating rules also enabled). so it downloaded all rules again that i have. i checked www.pfsense.org and it didnt work same for nvidia.com. When i disabled floating rules it instantly worked.

i have reputation also enabled. But i haven't checked any of the countries on that page and left the standard settings.

f34rinc

@cyberbot:

IP where I am connecting from is
62.143.80.249

Is this ip on a different categories ?

Edit - Different version ignore previous post

cyberbot

unfortunately have to disable the package
its not working fine in my case,
its blocking all Europe countries every its enabled to block those countries .

doktornotor

@cyberbot:

its blocking all Europe countries every its enabled to block those countries .

That's extremely surprising that it's doing what you told it to do… ::) ;D If you don't want to block those, then don't block those. (Not to mention the endless explanations on this thread about defaulty deny and how NOT to (ab)use the country lists.)

musicwizard

@Music:

i also had something strange.

i don't have country blocking enabled at all. But PFsense did block the forums.pfsense IP. which is on the United States list from i-blocklist

United States:208.123.64.0-208.123.95.255

but i don't use that list nor do i use country block.

When i disabled the floating rules in pfblocker settings site worked again. when i put it on floating again it still works the site. maybe after a while it starts blocking again.

Edit:
i used a list before that had a few sites on it that i use a lot. ( probably false positive ) so i removed that list from pfblockerng. i also already cleared the files manually and disabled pfblockerng and re-enabled it but it keeps blocking sites. Like www.nvidia.com etc. But once i disabled the floating rules it works again

edit2:
i did disabled pfblockerng totally. removed all files manually so it has to re-download all files when enabled again. restarted the system. enabled it again ( with floating rules also enabled). so it downloaded all rules again that i have. i checked www.pfsense.org and it didnt work same for nvidia.com. When i disabled floating rules it instantly worked.

i have reputation also enabled. But i haven't checked any of the countries on that page and left the standard settings.

What i trying to do what to see if i can get the pfblocker working over VPN. if i connect to the vpn on my phone it also uses the blocklists
i have enabled route all traffic over vpn.

cyberbot

@doktornotor:

@cyberbot:

its blocking all Europe countries every its enabled to block those countries .

That's extremely surprising that it's doing what you told it to do… ::) ;D If you don't want to block those, then don't block those. (Not to mention the endless explanations on this thread about defaulty deny and how NOT to (ab)use the country lists.)

i've configured it to not block europe at all !
but it still blocking countries in europe
Germany, Spain, UK,…... Europe filter is not selected and not enabled.

doktornotor

@cyberbot:

i've configured it to not block europe at all !

You told us the exact opposite earlier. Perhaps, post in some of internation forums in your native language.

BBcan177

@Music:

i don't have country blocking enabled at all. But PFsense did block the forums.pfsense IP. which is on the United States list from i-blocklist

United States:208.123.64.0-208.123.95.255

Several of the IBlock lists should be used with caution. They are intended for specific setups. There is a script that is posted in the thread for other lists that are available. Also you shouldn't use "IBlock/other Country specific blocklists" as Maxmind is a more reliable and reputable Country list to use.

When i disabled the floating rules in pfblocker settings site worked again. when i put it on floating again it still works the site. maybe after a while it starts blocking again.

All blocks are shown in the "Alerts Tab". Please refer to that tab to find out which List is causing the false positives.

i have reputation also enabled. But i haven't checked any of the countries on that page and left the standard settings.

For "Reputation", I would recommend whitelisting USA and Canada at a minimum.

BBcan177

@cyberbot:

I am using pfblockerng and blocking all country which I don't do business with .
Europe is not blocked . This weekend I am away to Germany " which is not blocked "
Right now I am on the hotel but the connection to the server behind pfsense is blocked .
Why it's blocking Germany IP even i don't block Germany country on pfblockerng?
IP where I am connecting from is
62.143.80.249

Is this ip on a different categories ?

If you run this command, it should report that IP as belonging to Germany (DE):
(Change the path to -i386 as required)

grep "^62.143." /usr/pbi/pfblockerng-amd64/share/GeoIP/*
/usr/pbi/pfblockerng-amd64/share/GeoIP/DE_v4.txt:62.143.0.0/16
/usr/pbi/pfblockerng-amd64/share/GeoIP/Europe_v4.txt:62.143.0.0/16

Also check the pfBNG Log tab, and see the Maxmind Log for the last updated timestamp.

tonymorella

I had an issue with how the auto rule order works, my details rules include individual destination along with the ports where the general rule is any destination on a limited port (i.e 80 and 443). None of the rule order options worked so used "Alias Deny" on all the pfBlockerNG rules to could achieve:

pfsense Pass/Match (Detailed rules) > pfBlockerNG Block/Reject > pfsense Pass/Match (General Rules)

It seems logical to have detail allow rules followed by pfBlockerNG block rules before allowing destination ANY rules, did I miss an option to have it work in this order?

Thanks
Tony

musicwizard

@BBcan177:

@Music:

i don't have country blocking enabled at all. But PFsense did block the forums.pfsense IP. which is on the United States list from i-blocklist

United States:208.123.64.0-208.123.95.255

Several of the IBlock lists should be used with caution. They are intended for specific setups. There is a script that is posted in the thread for other lists that are available. Also you shouldn't use "IBlock/other Country specific blocklists" as Maxmind is a more reliable and reputable Country list to use.

When i disabled the floating rules in pfblocker settings site worked again. when i put it on floating again it still works the site. maybe after a while it starts blocking again.

All blocks are shown in the "Alerts Tab". Please refer to that tab to find out which List is causing the false positives.

i have reputation also enabled. But i haven't checked any of the countries on that page and left the standard settings.

For "Reputation", I would recommend whitelisting USA and Canada at a minimum.

The problem when it blocks those ips are when i have the floating rules enabled.

What is the difference between floating rules enabled and disabled. As i see in the rules the list gets moved from wan to floating but are the same.
Do the floating rules use a extra list that i can setup somewhere and/or remove/edit?

Floating Rules can:
Filter traffic in the outbound direction (all other tabs are Inbound processing only)

this might be it when it's enabled.

The reason i had it enabled was to test if i can use the blocklist when im connected with the vpn from my mobile. but i didn't see any change with it enabled or disabled.

cyberbot

@BBcan177:

@cyberbot:

I am using pfblockerng and blocking all country which I don't do business with .
Europe is not blocked . This weekend I am away to Germany " which is not blocked "
Right now I am on the hotel but the connection to the server behind pfsense is blocked .
Why it's blocking Germany IP even i don't block Germany country on pfblockerng?
IP where I am connecting from is
62.143.80.249

Is this ip on a different categories ?

If you run this command, it should report that IP as belonging to Germany (DE):
(Change the path to -i386 as required)

grep "^62.143." /usr/pbi/pfblockerng-amd64/share/GeoIP/*
/usr/pbi/pfblockerng-amd64/share/GeoIP/DE_v4.txt:62.143.0.0/16
/usr/pbi/pfblockerng-amd64/share/GeoIP/Europe_v4.txt:62.143.0.0/16

Also check the pfBNG Log tab, and see the Maxmind Log for the last updated timestamp.

Thank you for your answer.
yes exactly the IP is from Germany, and i didn't block Germany at all.
when i was last weekend in Germany i couldn't log using VPN to the server. neither the OWA.
also on the same weekend users were on Spain and they couldn't VPN to the server.
after i called my wife and she started Teamviewer for me i've disabled the Pfblockerng the connections start working.

this the Maxmind log i can find

MaxMind GeoLite Date/Time Stamps

MaxMind_v4 Last-Modified: Wed, 05 Aug 2015 02:35:49 GMT
Local_v4 Last-Modified: Wed, 05 Aug 2015 02:34:55 GMT

MaxMind_v6 Last-Modified: Wed, 05 Aug 2015 02:35:47 GMT
Local_v6 Last-Modified: Wed, 05 Aug 2015 02:35:47 GMT

These Timestamps should match