DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed)

smolka_J

@smolka_J Gracias, will keep that in mind as I test n tune re-mapping. Just acquired a set of Grandstream APs to first start endulging down that path, still have a few more fiber optic and POE drops to get in place first to LAGG each AP at 5Gb onto 10G backplane, moving through rafters of the attic is a little slow on medical leave

Yoe777

@spinner I am having the same issue... have you been able to resolve this?

smolka_J

@Yoe777 #1, the FTP site does have its time periods of downtime which might fall in line with your current CRON update schedule.
#2 If you had upgraded in the past from a previous version of pfSense and/or with a config.xml imported/restored from a previous installlation, you may have an invalid or non-compatible UT1 feed link that isn't loading properly, pfBlockerNG is programmed in multiple areas to parse only "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz", if that feed URL is different for any reason like mine was from me previously trying to mitigate fixing my concern with FTP failing randomly, I had changed it to the https url which did download but did not parse into the files needed.
Steps that fixed my UT1 feed URL to the correct one in my config and restored full parsing/download:

Make sure pfBlockerNG is enabled on the general tab.
Go to the DNSBL tab and disable onle DNSBL, save
Reboot pfSense
Go back to the DNSBL tab and re-enable DNSBL
Run a Force Update>Reload>All

provels

Just throwing this in, that you may want to set your update time to some random time and not on the hour, and not too often either. These lists don't change much in a day's time.

Yoe777

I followed these steps and still getting same error.
I removed and reinstalled entire package and still am getting the same error.

This is a fresh install of pfSense only 2 weeks old Version 2.7.2-RELEASE (amd64).

smolka_J

@Yoe777 Do you get a valid IP back doing a DNS lookup to ftp.ut-capitole.fr?

Yoe777

@smolka_J

PING heimdall.ut-capitole.fr (193.49.48.249): 56 data bytes
64 bytes from 193.49.48.249: icmp_seq=0 ttl=50 time=119.248 ms
64 bytes from 193.49.48.249: icmp_seq=1 ttl=50 time=118.943 ms
64 bytes from 193.49.48.249: icmp_seq=2 ttl=50 time=118.840 ms

--- heimdall.ut-capitole.fr ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 118.840/119.010/119.248/0.173 ms

smolka_J

@Yoe777 Do you have either Snort or Suricata running? Either could be monitoring/scanning the FTP port keeping pfBlockerNG from being able to process the download timely, may need the IP or domains whitelisted in a passlist there to keep Snort/Suricata from scanning it

Yoe777

@smolka_J No I dont use either currently.

smolka_J

@Yoe777 Im at a loss otherwise then. Have you manually checked your config.xml to verify which URL your UT1 feed is set to currently? It should read as "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz"

Yoe777

@smolka_J That is what it is:

	<item>
		<title>UT1</title>
		<xml>ut1</xml>
		<feed>ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz</feed>

smolka_J

@Yoe777 Not certain if you have that domain whitelisted or not even though it seems to be passing for you otherwise but could be worth trying with it add if its not. Thats maybe the only thing I have different, if it is, that would be letting it work as far as I can tell, I do have ftp.ut-capitole.fr in my whitelist, being FTP it may be working better when whitelisted so there isn't an added delay waiting for the DNS query to pass through python blacklist processing first, FTP connections can be finicky like that when you don't have a full FTP client interface to tune timeout settings or have a retry/re-connect button to use

Yoe777

@smolka_J Where do I check what is whitelisted?

tinfoilmatt

@Yoe777 To check if it's blocklisted, run this command from either a shell or via shell command (Diagnostics / Command Prompt):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

smolka_J

@tinfoilmatt That will check if its being blocked but the theory I'm having is UT1 ftp may be failing to download if its not specifically whitelisted because of the time it take for a non-blocked domain to be passed through Python and all blacklists before it is validated as not being blocked, FTP connections are very time sensitive, depending on the specific FTP client in question, in this case pfBlockerNG being the FTP client, if an attempted FTP connection does not establish within so many milliseconds that the client is configured for then the FTP connection is deemed FAILED. If a domain is whitelisted, it does not have that wasted time being processed through several different other modules first like a non-blocked non-whitelisted domain does. As I noted, thats the only part my configuration has different than many other people's, UT1 ftp for me was NOT being blocked prior to me adding it to my whitelist and presently for me does not have any issue downloading and processing

Yoe777

@tinfoilmatt said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/.txt /var/db/pfblockerng/dnsblorig/.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

grep: /var/unbound/pfb_py_data.txt: No such file or directory
grep: /var/unbound/pfb_py_hsts.txt: No such file or directory
grep: /var/unbound/pfb_py_ss.txt: No such file or directory
grep: /var/unbound/pfb_py_zone.txt: No such file or directory

tinfoilmatt

@Yoe777 To check if the IP address that ftp.ut-capitole.fr resolves to, 193.49.48.249, is listed anywhere:

grep "193.49.48.249" /var/db/pfblockerng/DNSBLIP_v4.txt /var/db/pfblockerng/deny/*.txt /var/db/pfblockerng/original/*.orig /var/unbound/pfb_py_ss.txt

If no output is returned, that means the IP is not potentially being filtered anywhere by pfBlockerNG. (The "No such file or directory" output should be ignored.)

I've also noticed just now that the domain heimdall.ut-capitole.fr is a CNAME of ftp.ut-capitole.fr. You should ensure that heimdall.ut-capitole.fr is also either not listed and/or whitelisted.

smolka_J

@tinfoilmatt @Yoe777
Update from what I found on my end, FTP site is down again at least for me saying connection refused when it was working fine over the past week, likely meaning my IP is blacklisted temporarily from doing too many updates/reloads in too short of time period as I was throwing together a replacement for Shallalist I may try to get up on GitHub. I got my UT1 downloading and processing again by changing the feed URL for UT1 in two files:

/usr/local/pkg/pfblockerng/ut1_global_usage
``` as well as in ```
/usr/local/www/pfblockerng/pfblockerng.php

changed both to the https URL

https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz

followed with then going to the DNSBL Category tab to save settings so that it updates the config.xml. Then run a force reload all. On update/re-install of pfBlockerNG those two files will need updated again because they will be overwritten

vilion

Hi everyone,

I’ve seen quite a few threads here about issues with blacklist feeds (UT1 download errors, Shalla being discontinued, some community feeds being outdated or filled with dead domains). I’m curious how you are currently handling DNSBL filtering in production:
• Which feeds are you relying on today?
• Do you run into problems with overblocking or too many dead entries?
• Do you prefer sticking to a few well-known sources (like Spamhaus, OISD), or do you combine many community lists?
• Are there specific categories (Adult, Gambling, Phishing, Malware, Social Media) where you’d like to see more reliable/curated data?

It seems to me there’s a growing need for “clean, curated lists” – feeds that are actively checked, cleaned up, and categorized, rather than just large dumps.

Gertjan

@vilion said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

It seems to me there’s a growing need for “clean, curated lists” – feeds that are actively checked, cleaned up, and categorized, rather than just large dumps.

The good news is the bad news : these lists exist.
As you've said : “clean, curated lists – feeds that are actively checked, cleaned up, and categorized” so some one is doing the works for us.

So, I'm sure these lists exist. But not for free. This 'some one' wants to get paid ...

The result might be what Maximind is already doing : you (= pfBlockerng) has to use an API to access the files to downloads; The API will check if you are authorized to do so = you've paid the bill.

If there was a good quality maintained etc list avaible for free, then it would be proposed by pfBlockerng tomorrow ^^