Firehol level 1 list blocking LAN resources

[davidology]

The FireHol level 1 list (https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level1.netset)

has all local IP addresses (like 10.0.0.0/8 and 192.168.0.0/24 etc) listed. When I subscribe to it in pfBlockerNG under IPv4, it obviously blocks my access to my LAN resources. Is there a way to have it disregard those entries without compromising overall security?

Sorry if this is a basic question. I've been searching. I have a couple ideas, but don't want to lock myself out again so some advice from someone who has implemented this successfully is welcome!

Thanks in advance..

[BBcan177]

Lvl1 contains bogons and shouldn't be used to block Outbound…. You can also manually add the same feeds that comprise the Lvl1 feed directly..... also can enable "Suppression" in the General Tab which will filter out RFC1918 and loopback addresses...

[davidology]

Great idea on recreating the feed without the local addresses. Worked flawlessly.

Thank you!

[frankvh]

This is interesting because I recently had the same experience. I lost access to my LAN because I'd used the level 1 list. I switched to Firehol level 3 but it's a shorter list.

As a "lazy man's" alternative, am I safe if I use the Firehol level 1 list, but also enable "supression" in the pfBlockNG general tab?

Thanks.

[charvey]

I had similar issues with firehol lvl 1, but with broadcast packets.

See https://forum.pfsense.org/index.php?topic=138877.0

[frankvh]

I ended up taking BBcan's advice (good advice it always is) and simply put the lists that comprise firehol1 into pfblocker (minus the bogons list of course).

[BBcan177]

@frankvh:

I ended up taking BBcan's advice (good advice it always is) and simply put the lists that comprise firehol1 into pfblocker (minus the bogons list of course).

Using the Lvl 1 Feed is going to cause grief… Just don't do it  ;) ;)

[iorx]

The level1 list looks like this, excluding the bogons.
(I really hope that I understood the firehol list content correctly. If that is the case, now someone else don't have to recreate this wheel again.)

Here goes:

http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txt
http://feeds.dshield.org/block.txt
https://feodotracker.abuse.ch/blocklist/?download=ipblocklist
https://sslbl.abuse.ch/blacklist/sslipblacklist.csv
https://zeustracker.abuse.ch/blocklist.php?download=badips
http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt

[BBcan177]

Try to use HTTPS for all sites that support it…

[adoucette]

@iorx:

The level1 list looks like this, excluding the bogons.
(I really hope that I understood the firehol list content correctly. If that is the case, now someone else don't have to recreate this wheel again.)

Here goes:

http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txt
http://feeds.dshield.org/block.txt
https://feodotracker.abuse.ch/blocklist/?download=ipblocklist
https://sslbl.abuse.ch/blacklist/sslipblacklist.csv
https://zeustracker.abuse.ch/blocklist.php?download=badips
http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt

iorx, thank you for compiling and posting that.
Is the above list sufficient for the residential user? Or are there additional sources I should add?

On the other hand, given the great work BBcan has done with the "suppression" option etc, is it safe to just use the firehol level 1 list just in case the sources it pulls from change over time?

(just one small piece of 1) protecting the kids online and 2) protecting our networked computers from what the kids may inadvertently do online!)

Thank you all.
BBCan – donation coming your way.

Ari

[seanr22a]

This is the IP lists I’m using. I started with a few and over time added more and more lists I’ve found. I use a script that download all lists with wget, unpack those that need and put all of them in one file. The resulting file is filtered leaving only ip addresses and checked and cleaned for duplicates and sorted. Finally, the file is cleaned from RFC 1918 and RFC 4193 addresses and some common important ip addresses such as 8.8.8.8  8.8.4.4 and so on. The finished file is put on a small internal webserver. Pfblocker get the file from there. I do the same with DNSBL lists, I can post all DNSBL lists I have if someone is interested. The ones here is only IP lists.
I put the resulting IP list and DNSBL list on an external webserver as well currently updated once a week. You have them here: http://dnsbl.dyndns.org:9080/MyBlocklist.txt and http://dnsbl.dyndns.org:9080/mydnsblfeed.txt  The DNSBL file is big so it can take a while to download.

https://gist.githubusercontent.com/BBcan177/d7105c242f17f4498f81/raw/90eb2ac8bdc01af3008d728b7c0f10dc7b2506b4/MS-3
https://gist.githubusercontent.com/BBcan177/bf29d47ea04391cb3eb0/raw/b344ebc9475acdea1fae38a12c4ea9332838a184/MS-1
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
https://www.spamhaus.org/drop/drop.txt
https://pfblockerlists.smallbusinesstech.net/hackerlist.txt
https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
https://rules.emergingthreats.net/blockrules/compromised-ips.txt
http://www.abuseat.org/iotcc.txt
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
http://cinsscore.com/list/ci-badguys.txt
https://ransomwaretracker.abuse.ch/downloads/LY_PS_IPBL.txt
https://ransomwaretracker.abuse.ch/downloads/TL_C2_IPBL.txt
https://zeustracker.abuse.ch/blocklist.php?download=badips
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
http://danger.rulez.sk/projects/bruteforceblocker/blist.php
https://lists.blocklist.de/lists/all.txt
http://malc0de.com/bl/IP_Blacklist.txt
https://feodotracker.abuse.ch/blocklist/?download=ipblocklist
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
http://feeds.dshield.org/top10-2.txt
https://feeds.dshield.org/block.txt
http://list.iblocklist.com/?list=dgxtneitpuvgqqcpfulq&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=dgxtneitpuvgqqcpfulq&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=dgxtneitpuvgqqcpfulq&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=llvtlsjyoyiczbkjsxpf&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=usrcshglbiilevmyfhse&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=xpbqleszmajjesnzddhv&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=xpbqleszmajjesnzddhv&fileformat=p2p&archiveformat=gz
http://list.iblocklist.com/?list=xpbqleszmajjesnzddhv&fileformat=p2p&archiveformat=gz
http://www.spamhaus.org/drop/edrop.txt
https://sslbl.abuse.ch/blacklist/sslipblacklist.csv
http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt

If you want to use the Firehol level 1 only you could use the same approach with a script to filter out those addresses you don't want.

[EDIT]
added three more lists to my script from this thread
Forgot that the webserver is on port 9080 Fixed now

[Presbuteros]

@seanr22a:

I put the resulting IP list and DNSBL list on an external webserver as well currently updated once a week. You have them here: https://dnsbl.dyndns.org/modules/mymod/MyBlocklist.txt and https://dnsbl.dyndns.org/modules/mymod/mydnsblfeed.txt  The DNSBL file is big so it can take a while to download.

The links are not working.

Connection timed out after 15039 milliseconds Retry in 5 seconds...
. cURL Error: 28
Connection timed out after 15015 milliseconds Retry in 5 seconds...
. cURL Error: 28
Connection timed out after 15021 milliseconds Retry in 5 seconds...
.. unknown http status code
Download FAIL [ 11/20/17 08:12:54 ]
  Firewall and/or IDS are not blocking download.

The Following list has been REMOVED 

Thanks for sharing. Let us know when the webserver is working again.

[seanr22a]

@Presbuteros:

The links are not working.

Sorry, forgot that the server is on port 9080. Changed in the original post.

[edit]
Changed to http port 9080

[Presbuteros]

Tried again with updated URL

https://dnsbl.dyndns.org:9443/modules/mymod/MyBlocklist.txt

[ Comp ]		 Downloading update .
**Saving configuration [ 11/20/17 09:21:40 ] ...
 cURL Error: 51
SSL: no alternative certificate subject name matches target host name 'dnsbl.dyndns.org' Retry in 5 seconds...
. cURL Error: 51
SSL: no alternative certificate subject name matches target host name 'dnsbl.dyndns.org' Retry in 5 seconds...
. cURL Error: 51
SSL: no alternative certificate subject name matches target host name 'dnsbl.dyndns.org' Retry in 5 seconds...
.. unknown http status code 

[seanr22a]

This got far more complicated than I thought ….. Now I've put up a new small webserver for external access so I don't have to mess around with our production systems for this.

http://dnsbl.dyndns.org:9080/MyBlocklist.txt
http://dnsbl.dyndns.org:9080/mydnsblfeed.txt

I hope everything ok now. I modified my previous posts  :P

Info about all the public available IP and DNSBL lists I'm using
http://dnsbl.dyndns.org:9080/info.txt

[adoucette]

Thank you seanr22a – that's very generous of you to compile, filter, host and post. Very generous of iorx as well, above.

I put together a comparison in Excel of the three lists (FireHol lvl 1, iorx's compilation above, and seanr22a's compilation above) and the initial conclusions I draw are:
37% of FireHol Lvl 1's IPs are in iorx's list and 100% are in seanr22a's list
99% of iorx's values are reflected in FireHol's and 65% of the ips are reflected in seanr22a's.
Seanr22a's list may be more comprehensive. It is a 3MB download and has almost 200,000 IPs (the dbsbl list is a 45MB download  :o - may reflect lots of effort on his part).

Excel file attached.

My questions, as a layman, would be:
With the larger list, is there a substantially increased potential for false-positives?
Will this larger list slow down the pfSense box?

Ari

pfb-list-compare.zip

[Presbuteros]

Yes, thanks to all who are contributing here.

@adoucette:

With the larger list, is there a substantially increased potential for false-positives?
Will this larger list slow down the pfSense box?
Ari

I loaded the list from seanr22a. I took a while to download and compile. It did not "appear" to slow down the pfSense box. My mobo is a Gigabyte GA-J1900N-D3V so a Celeron quad-core 2Ghz and 8 GB of RAM. What I did notice was a lot of issues loading news sites like cnn, foznews, drudgereport, etc. Videos were stalling out and certain essential elements of the page would not load. However, I would like to hear from others on their use of the list and if they had any obvious issues. I simply disabled the list until I have more time to test later.

[seanr22a]

@Presbuteros:

Yes, thanks to all who are contributing here.

I loaded the list from seanr22a. I took a while to download and compile. It did not "appear" to slow down the pfSense box. My mobo is a Gigabyte GA-J1900N-D3V so a Celeron quad-core 2Ghz and 8 GB of RAM. What I did notice was a lot of issues loading news sites like cnn, foznews, drudgereport, etc. Videos were stalling out and certain essential elements of the page would not load. However, I would like to hear from others on their use of the list and if they had any obvious issues. I simply disabled the list until I have more time to test later.

Check the Pfblocker logs for what is blocked related to the sites you visit. You have logs for both DNSBL and IP lists.
As you can see in the info.txt file the lists are made from public available lists maintained by many different people and organizations. Unfortunately there is no list that fits everyone. Simply whitelist those sites that causes you problem, I've done that for many ip's and domains to make it work for me so start dig in to the logs :)

[iorx]

@BBcan177:

@frankvh:

I ended up taking BBcan's advice (good advice it always is) and simply put the lists that comprise firehol1 into pfblocker (minus the bogons list of course).

Using the Lvl 1 Feed is going to cause grief… Just don't do it  ;) ;)

Even with Suppression enabled?

[ui5-5e]

@seanr22a:

http://dnsbl.dyndns.org

@seanr22a: i give a "thank you"
and by the way: before importing your banlist, I checked the IP provenience behind your dyndns address (sorry for that).
Interstingly I did note a Spamhaus ZEN blocklist entry for that IP  ;) (just for the record: I imported and I will thankfully update your list weekly).