I am having a similar issue, pdfBlockerNGon my dashboard widget says "[ pfB_PRI1_v4 - Talos_BL_v4 ] Download FAIL [ 01/24/24 07:00:15 ]".

If I go to Firewall->pdfBlockerNG->IP->IPV4 edit, I see 'Talos_BL' header/label highlighted in yellow.

If I manually open the source link, it opens just fine in a browser tab (https://talosintelligence.com/documents/ip-blacklist)

Below the main table, it says "Failed download(s) highlighted in yellow. Click here for Guidelines --->".

When click on Guidelines, the only listed for Header/Label is "Header/Label: This field must be unique. This names the file and is referenced in the widget.  (ie: Spamhaus_drop, Spamhaus_edrop) "

As far as I can see, this field is unique. Any ideas what could be causing this failed download?

@Uglybrian I think I answered my own question. I looked at a couple of the ipv4 lists that had comments embedded.

In my case, I am creating a custom IPv4 list and want to include comments to identify what/why I'm blocking, rather than create a lot of small lists. I am building a custom list in the pfBlocker rules rather than include a file from a URL.

Note that you're example is using DNSBL, though a similar approach could be used per your example for IPv4.

Thanks!

@jrey @Gertjan
I wanted to provide update to these issues so you know what happened and in case it might help someone else. I left this thread partly because I didn't want to take up anyone else's time on what was becoming an increasingly complex and consuming problem.

It turns out this was 2 main problems:

After encountering what seemed like a cascading number of failures and problems, including a boot loop, I was able to verify bad hardware. Consequently the MB, CPU and RAM were replaced with new. The SSD and NIC PCI card did not seem affected. There was indeed a misconfiguration with my account at the VPN provider. Specifically, any filtering at the VPN level was not compatible with DNSSEC. This included any filtering, including of malware, tracking or social sites. This filtering was turned off.

Some notes:
Before wiping out the SSD and in the old installation of pfSense, I did not use any backup files to transfer settings because it seemed likely to transfer misconfigurations as well. Instead I took screens shots of every page I may have modified from a default installation and used those to transfer most of my setup to the new default install of pfsense on the new hardware. Almost every setting that was to be changed from default was questioned and re-researched before doing so. In this process I came across VPN setup instructions from a competing provider that had more specific and complete instructions which included the warning that VPN filtering was not compatible with DNSSEC. Also, I was able to identify and eliminate a few crazy settings made in an apparently futile attempt to address the problems.

Yeah, WAY more than a couple of hours, LOL. But the bad hardware was a far bigger problem than I expected.

Everything now works as expected and DNS resolution, and everything else, is faster than it's ever been. In pfBlockerNG, error.log and py_error.log remain empty after a week or two of continuous use.

I want to thank @jrey and @Gertjan for your help which I have tried to heed. I do have fewer pfBlockerNG feeds thanks to @Gertjan .

And thanks and gratitude to @BBcan177 for creating such an awesome package.

Best Regards,

@mooncaptain in general never install or upgrade packages without being on the same version as the Update Branch setting, or it can pull in later libraries.

@Home-User That is WAD (works as designed).. If you try to send a https request to some block page, yeah your browser is going to complain..

It tried to go to say www.somethingblocked.com, and it gets sent to the VIP ip of pfblocker serving up a page saying this is blocked, this cert is not for www.somethingblocked.com - so yeah your browser going to complain.

See the part of the error where it says "the name on the cert does not match"

https://forum.netgate.com/post/873559

@jrey thank you for the reply.

It think I may have fixed the issue, I am using a ram disk and it was completely full. I have fixed this by encreasing the size and so far no issues.

Thank you

@planedrop

So at least that's setup right in the config

yup that was a long shot that something was corrupt and not showing the correct value.

Sure thing.

@jrey said in Unified Log Growing Infinitely (24 gigabytes and counting):

ls -l /var/log/pfblockerng/unified.log; ls -l /var/unbound/var/log/pfblockerng/unified.log

you could try ^ and see if the files are exactly the same.
Actually so could I -- I'll just break it on my test box and see what happens.

I've actually went a little further on the logging of the logs being trimmed
T(Target) B(efore) A(After)
and at least now I have confirmation that logs are being trimmed.

Also discovered another annoyance (to me) -- there is no reason at all to go through the steps of making a temp, moving it back etc for a file that only has say 6 or 1 line in it. So I might change that some day as well. 😇

UPDATE PROCESS ENDED [ 01/8/24 14:31:00 ] Log trimmed(2): '/var/log/pfblockerng/pfblockerng.log' Lines: T:10000 B:11110 A:10000 Log trimmed(2): '/var/log/pfblockerng/error.log' Lines: T:10000 B:6 A:6 Log trimmed(2): '/var/log/pfblockerng/ip_block.log' Lines: T:20000 B:20878 A:20000 Log trimmed(2): '/var/log/pfblockerng/ip_permit.log' Lines: T:20000 B:20031 A:20000 Log trimmed(1): '/var/unbound/var/log/pfblockerng/dnsbl.log' Lines: T:20000 B:20388 A:20000 Log trimmed(2): '/var/log/pfblockerng/dnsbl_parsed_error.log' Lines: T:10000 B:1 A:1 Log trimmed(1): '/var/unbound/var/log/pfblockerng/dns_reply.log' Lines: T:20000 B:24884 A:20000 Log trimmed(1): '/var/unbound/var/log/pfblockerng/unified.log' Lines: T:20000 B:26126 A:20000

(I liked the logging change enough to patch my production box - let that spin for a while)

@korgua

Just as a proof of concept that the 3.2.0_7 package would run on 2.7.0

I spun fresh 2.7.0 instance
installed 3.2.0_7 pfBlocker
Screen Shot 2024-01-06 at 2.52.23 PM.png

Screen Shot 2024-01-06 at 2.52.59 PM.png

DNS Resolver
Screen Shot 2024-01-06 at 2.53.37 PM.png

DNSBL

Screen Shot 2024-01-06 at 2.54.36 PM.png

The script is there,

total 6146 -rw-r--r-- 1 root unbound 176 Jan 6 19:46 access_lists.conf drwxr-xr-x 2 unbound unbound 2 Jun 28 2023 conf.d dr-xr-xr-x 7 root wheel 512 Jan 6 19:49 dev -rw-r--r-- 1 root unbound 0 Jan 6 19:46 dhcpleases_entries.conf -rw-r--r-- 1 root unbound 3408 Jan 6 19:46 dnsbl_cert.pem -rw-r--r-- 1 root unbound 0 Jan 6 19:46 domainoverrides.conf -rw-r--r-- 1 root unbound 388 Jan 6 19:46 host_entries.conf drwxr-xr-x 4 root wheel 68 Jun 28 2023 lib -rw-r--r-- 1 root unbound 1271 Jan 6 19:49 pfb_dnsbl_lighty.conf -rw-r--r-- 1 root unbound 8429809 Jan 6 19:49 pfb_py_data.txt -rw-r--r-- 1 unbound unbound 8192 Jan 6 19:49 pfb_py_dnsbl.sqlite -rw-r--r-- 1 root unbound 1687428 Jan 6 19:46 pfb_py_hsts.txt -rw-r--r-- 1 unbound unbound 12288 Jan 6 19:58 pfb_py_resolver.sqlite -rw-r--r-- 1 root unbound 1043 Jan 6 19:49 pfb_py_whitelist.txt -r-xr-xr-x 1 root unbound 5534 Jan 6 19:46 pfb_unbound_include.inc -rw-r--r-- 1 root unbound 358 Jan 6 19:49 pfb_unbound.ini -r-xr-xr-x 1 root unbound 68158 Jan 6 19:46 pfb_unbound.py -rw-r--r-- 1 root unbound 300 Jan 6 07:29 remotecontrol.conf -rw-r--r-- 1 unbound unbound 83 Jan 6 19:46 root.key -rw------- 1 unbound unbound 2455 Jan 6 07:29 unbound_control.key -rw-r----- 1 unbound unbound 1411 Jan 6 07:29 unbound_control.pem -rw------- 1 unbound unbound 2455 Jan 6 07:29 unbound_server.key -rw-r----- 1 unbound unbound 1549 Jan 6 07:29 unbound_server.pem -rw-r--r-- 1 unbound unbound 1996 Jan 6 19:49 unbound.conf drwxr-xr-x 3 root unbound 3 Jan 6 19:49 usr drwxr-xr-x 3 root unbound 3 Jan 6 19:49 var

Confirmed the DNSBL is blocking
Screen Shot 2024-01-06 at 3.11.55 PM.png

And the system is running with no issues -- but the recommendation that 2.7.2 is available still stands.

@Popolou

I think i figured it out. In my click -> click -> next in setting up the config so that I can run tests, I inadvertently overlooked a setting that needed to be changed. Running more tests, but the limited testing shows logging happening.

Thank you so much for that assistance.

The Alias native is working! Thank you very much!

@mzaknoen

Looking to block malicious sites on the network

generally the issue regardless of what you are trying to do will be based on the list effectiveness

installed pfBlockerNG DNSBL, did not install the "devel" version

There is currently no difference and the "non - devel" version is the way to go for most users.

Looking into setting up category filtering, when checking the "adult" section, I am prompted with the memory warning.

Memory warning specifically saying?

What list specifically ? Guessing UT1 -> Adult ? that is something like 4.5 million "domains" with a file size of 122mb but well less than 8gb RAM
That said the list is also full of bloat.

is there a way for me to download that list of IP's on a computer and copy is over to my Pfsense device to avoid RAM usage of downloading that big file.

yes, but downloading and using ram based on what was downloaded are different things.

Any alternative options for blocking specific categories?

yes,
everything from education of users
to block everything and move to allow certain sites only
and anything in between.

there is a balance and that will be different for every use case.

consider the following sample from the UT1 adult list as I suspect that is what you are running into problems with.

Ask yourself, do I have a need for anything on blogspot.com?
Yes, write it down (you will want to whitelist that website(s) if you do)
No, nice, just continue

UT1 - Adult domains (raw file) 4,511,799 122mb
remove all blogspot lines (raw file) is now 891,692 domains and 18mb
add 1 line containing blogspot.com to the TLD list
( a quick DNS scan to others for example blogspot.hr are all cname or redirects to the .com)

Okay, i have not even tried to load the list, it is not a list I would ever consider using. There are other ways with far less impact.

However, for the purpose of this example I added only blogspot.com to a TLD for testing. I grabbed one of the URLs from the list (bad me)

then over to a browser. Don't try this at home kids🤣

Screen Shot 2023-12-28 at 3.40.32 PM.png

immediately gets the redirect

Screen Shot 2023-12-28 at 3.42.44 PM.png

and in the log we see the original request getting the cname reply
and the website being blocked by the 1 line added to the TLD
Screen Shot 2023-12-28 at 3.44.57 PM.png

A second scan of the original domain list, shows that many of the names don't even resolve, so those are just old and could also be removed.

Size of list does not equate to "effectiveness" of the list -- also applies to any/all of the available lists.

If the math is correct I've removed 3,620,107 lines from the file, and effectively have the same blocking with the addition of 1 line. Of course I'm not going to try every single one, although it would be easy enough to script a test.
This is where the user education can come into play, why on earth would you be going to a website like

zxaswdserdwokgkmbjnhntbftherhbfokmlplfnvhrfdx.(some TLD)

certainly not by typing that address in.

Often it is better, to determine what needs to be blocked specifically by reviewing logs. Do I use lists, certainly do. But certainly no need to hit the finishing nail with a sledge hammer.

Memory is pretty flat lined here - holding at
Screen Shot 2023-12-28 at 4.28.09 PM.png

@jrey

Thank you

It seems after restarting pfsense a couple of times, the issue seems to have been resolved.

I will monitor it for some time and will update here.

I see. Thanks for the info. I already solve it via PFblockerNG IPv4 and adjust it to firewall rules.

@jrey Thanks a lot for your input! That helps a lot as for example I had not seen the smetrics.cadenaser.com before. It did not appear in pfsense nor the browser dev tools.
At the moment I allowed only pagead2.googlesyndication.com and with this, it takes 2-3 seconds more to start the streaming but I got this without any Ads. I will test some days, but if it does not work I will try whitelisting only your URLs.

Much appreciated! Happy holidays and new year!

@jrey Thanks. I do confirm that I deleted the log entries for the error and when the cron job for the feeds ran again (0315, once daily) no further errors appeared.

With your explanation I now understand what I am seeing and how to resolve it.