How I add custom Pi-Hole hosts files or others to pfBlocker

[FrankM]

Folks,

There has been some discussions on how to import hosts files into pfSense/pfBlocker. Especially from those of us who have come from using Pi-Holes. Of which I am one.

Having read the discussions;

here;
https://forum.netgate.com/topic/80156/import-a-local-hosts-file-into-pfsense

and here;
https://forum.pfsense.org/index.php?topic=86986.0

I felt there is probably a different, maybe even easier method than those posts described. Did I find an easier method? I believe so.

I am working with the Community Edition v2.6.0 of pfSense, and v3.1.0_6 of pfBlockerNG-devel. I use the DNS Resolver (unbound). I do not use the DNS Forwarder (dnsmasq). pfS/pfB are run on an older embedded J4355 Intel Celeron processor system.

Some background. When I built my Pi-Hole many years ago, I did it by "Setting It Up Manually (Building It From Scratch)" as described about half way down this link from 2015;
https://jacobsalmela.com/2015/06/16/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0/

And it is/was time to upgrade the Pi-Hole, though I didn't want to go through that process again, and I don't like what the new Pi-Hole has turned into. That's just me.

Yes my Pi-Hole set up is that old! It was built in an old Lenovo small form factor box. Very much a pain to update block lists, some of which are custom, though it has worked excellently through out. Enough about my aged equipment.

So now I tried different ways of getting the Pi-Hole hosts into pfS/pfB - all came up short.

I realized that all the feeds that pfB uses were loaded via the web/http/https. Hmm... what about running a simple web server on one of my machines? A simpler web server than this I could not find.

https://iqcode.com/code/shell/linux-start-simple-http-server

The first line is for python 2. Now when you search for more info (python simple http server) you will find how to define the port the http server uses and so forth. Docs here;

https://docs.python.org/3/library/http.server.html

CAUTION! Security of this python http server is questionable. I would NOT let it see the internet - only internal on the LAN. Did it work? YES!

I did not want to use a cloud based solution like DropBox or other to store my host files, I wanted a local internal solution on my LAN.

So how did I set it up?

Note I am using Linux on all my machines, so how I did it is based only on Linux. Windows and Mac users, will need to find out what is similar for their machines.

My bash script;

#!/bin/bash
#
# Set directory/folder to where you want the server to serve from
cd /home/me/Data/Network/Pi-Hosts/
#
# Start server with port desired
python3 -m http.server 9000
#
# End of script

It is important to set the permissions of the script to be executable.

Be sure to set the correct path to the files in the cd line of the script and to copy the hosts files you want to use into the directory/folder Pi-Hosts

Once the script starts it will display;

Serving HTTP on 0.0.0.0 port 9000 (http://0.0.0.0:9000/) ...

To access the server on that machine go to http://0.0.0.0:9000/ in a browser. Now if the http server machine has been given an IP of 192.168.1.101 by the dhcp server, then to access the http server from another machine do so by going to http://192.168.1.101:9000/ on the other machine. Note to allow it in by setting a rule in the firewall of the http server machine to allow access. This was easy for me as I run linux on my machines and used UFW (Uncomplicated FireWall) to set a rule to allow access.

I then created in pfB groups in both the IP4 and DNSBL areas of pfB. I named them both MyList with a description of My List. Surprisingly naming them both the same was not a problem for my install of pfS and pfB.

Then I needed to add the feed/source of the host files. So I found the name of the Pi-Hole hosts file I wanted to add to the IPv4/DNSBL Source Definitions. It's name is Pi-Hole-Hosts-01.txt and added as;

http://192.168.1.101:9000/Pi-Hole-Hosts-01.txt

Note that the IP is the one that the host files are on, the port 9000 is needed, then the file name. Be sure to use the proper syntax as shown. Note the colon between the IP and port number. To be sure it is correct, put that into a browser and the host file should display.

I made sure to turn it's State on and other settings before I told pfS/pfB to save the changes. And it saved correctly! Happy moment.

Now the moment of truth. I ran Update in pfB, forced Reload, on All, watched the process window ... it finishes ... reviewed ... SUCCESS! Yay!

Checked the Dashboard Status of pfBlockerNG and it shows pfB_MyList_V4 and DNSBL_MyList with Counts! Yay!

To use this I plan on keeping the HTTP server active on one of my machines, probably on a file server. I checked top and found python and the script take minimal resources - about 40k, and zero percent CPU usage.

Another important note about my Pi-Hole hosts files is they are formatted as;

0.0.0.0 some.adserver-01.com
0.0.0.0 some.adserver-02.com
0.0.0.0 some.adserver-03.com
0.0.0.0 some.adserver-04.com

No regex expressions.

I also found the requirements for IPv4 host files and DNSBL host files are just a bit different.

The IPv4 hosts may contain lines like;

0.0.0.0 8.8.8.8
0.0.0.0 64.233.191.201

The DNSBL hosts may not contain IP numbers to block. The DNSBL hosts need a domain name to block. If a DNSBL hosts file contains an IP number to block, it WILL fail to load properly. So editing as appropriate may be needed.

In conclusion.

Will this work for you? I hope so. Though keep in mind I am just one of thousands of pfS/pfB users.

Was this worth it? For me, Yes!

It is a solution that is external to pfS/pfB, that pfS/pfB can use.

I can still use and edit my Pi-Hole hosts as needed, I may add other plain text host files as I feel are needed and pfS/pfB will update the feeds/hosts via cron. So much easier to keep updated.

And now I can retire my old Pi-Hole box along with my con-sumer router. WiFi already provided by business class access point.

Take Care & Enjoy!

[FrankM]

I've done some more looking into how pfBlocker (pfB) uses feeds/block lists. And yes IPv4 and DNSBL requirements are different. No surprise there. What I found is feeds/lists that contain both IP's and domain names to be blocked are interpreted differently.

My understanding is this;

If a feed/list is in an IPv4 group, pfB will go through the feed/list and extract only the IPv4 addresses to block and discard the rest.

If a feed/list is in a DNSBL group, pfB will go through the feed/list and extract only the domain names to block and discard the rest.

Please correct me if I'm wrong.

So this means if I select an external list that is not part of the pfB feeds that contains both IP's and domain names to block, the list needs to be added to both the IPv4 group and the DNSBL group in order for the whole list to be added to pfB.

To find this out I created a Test group in both the IPv4 and DNSBL groups of pfB. I only put one feed/list in each Test group. Then checked in the Update details to see the ===[ Deny List IP Counts ]=== and the ===[ DNSBL Domain/IP Counts ]===. Another place to look would be at the Dashboard in the pfB statistics and check the count to see how many were blocked.

Though reviewing the results in the Update details is much more informative as it will show how many are blocked from each feed/list - after - removing the duplicates.

Why is this important? Some of us came from using Pi-Holes, and in Pi-Hole a block list could be, just IPs, just domain names, or a mixture of both. Also keep in mind that 'unbound' may be added to Pi-Hole while Pi-Hole uses its existing block lists to block without needing to separate IPv4 and domain names into separate lists.

So coming from Pi-Hole with its way of blocking to pfB's way may be a challenge for some. It was for me, until, I understood pfB's way.

Adding to the confusion is how pfB shows ===[ DNSBL Domain/IP Counts ]===. Which in my way of thinking says it adds IPs to the count. Maybe it could read as ===[ Deny List: DNSBL Domain Name Counts ]===. Just my thoughts, though I'm quite sure the development team is already more than busy. I do thank them for their work.

Just my observations.

Take Care and Enjoy!

[pfsjap]

@frankm Have you experimented with this option:

[FrankM]

@pfsjap

No I haven't and I don't recall seeing that. I must be blind in one eye and can't see out of the other. Will look for it and try it.

Thank you for the tip.

Take Care and Enjoy!

Edit: I found it, I set it, now to play around and see what it does.