DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed)

smolka_J

@Yoe777 Do you get a valid IP back doing a DNS lookup to ftp.ut-capitole.fr?

Yoe777

PING heimdall.ut-capitole.fr (193.49.48.249): 56 data bytes
64 bytes from 193.49.48.249: icmp_seq=0 ttl=50 time=119.248 ms
64 bytes from 193.49.48.249: icmp_seq=1 ttl=50 time=118.943 ms
64 bytes from 193.49.48.249: icmp_seq=2 ttl=50 time=118.840 ms

--- heimdall.ut-capitole.fr ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 118.840/119.010/119.248/0.173 ms

smolka_J

@Yoe777 Do you have either Snort or Suricata running? Either could be monitoring/scanning the FTP port keeping pfBlockerNG from being able to process the download timely, may need the IP or domains whitelisted in a passlist there to keep Snort/Suricata from scanning it

Yoe777

@smolka_J No I dont use either currently.

smolka_J

@Yoe777 Im at a loss otherwise then. Have you manually checked your config.xml to verify which URL your UT1 feed is set to currently? It should read as "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz"

Yoe777

@smolka_J That is what it is:

	<item>
		<title>UT1</title>
		<xml>ut1</xml>
		<feed>ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz</feed>

smolka_J

@Yoe777 Not certain if you have that domain whitelisted or not even though it seems to be passing for you otherwise but could be worth trying with it add if its not. Thats maybe the only thing I have different, if it is, that would be letting it work as far as I can tell, I do have ftp.ut-capitole.fr in my whitelist, being FTP it may be working better when whitelisted so there isn't an added delay waiting for the DNS query to pass through python blacklist processing first, FTP connections can be finicky like that when you don't have a full FTP client interface to tune timeout settings or have a retry/re-connect button to use

Yoe777

@smolka_J Where do I check what is whitelisted?

tinfoilmatt

@Yoe777 To check if it's blocklisted, run this command from either a shell or via shell command (Diagnostics / Command Prompt):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

smolka_J

@tinfoilmatt That will check if its being blocked but the theory I'm having is UT1 ftp may be failing to download if its not specifically whitelisted because of the time it take for a non-blocked domain to be passed through Python and all blacklists before it is validated as not being blocked, FTP connections are very time sensitive, depending on the specific FTP client in question, in this case pfBlockerNG being the FTP client, if an attempted FTP connection does not establish within so many milliseconds that the client is configured for then the FTP connection is deemed FAILED. If a domain is whitelisted, it does not have that wasted time being processed through several different other modules first like a non-blocked non-whitelisted domain does. As I noted, thats the only part my configuration has different than many other people's, UT1 ftp for me was NOT being blocked prior to me adding it to my whitelist and presently for me does not have any issue downloading and processing

Yoe777

@tinfoilmatt said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/.txt /var/db/pfblockerng/dnsblorig/.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

grep: /var/unbound/pfb_py_data.txt: No such file or directory
grep: /var/unbound/pfb_py_hsts.txt: No such file or directory
grep: /var/unbound/pfb_py_ss.txt: No such file or directory
grep: /var/unbound/pfb_py_zone.txt: No such file or directory

tinfoilmatt

@Yoe777 To check if the IP address that ftp.ut-capitole.fr resolves to, 193.49.48.249, is listed anywhere:

grep "193.49.48.249" /var/db/pfblockerng/DNSBLIP_v4.txt /var/db/pfblockerng/deny/*.txt /var/db/pfblockerng/original/*.orig /var/unbound/pfb_py_ss.txt

If no output is returned, that means the IP is not potentially being filtered anywhere by pfBlockerNG. (The "No such file or directory" output should be ignored.)

I've also noticed just now that the domain heimdall.ut-capitole.fr is a CNAME of ftp.ut-capitole.fr. You should ensure that heimdall.ut-capitole.fr is also either not listed and/or whitelisted.

smolka_J

@tinfoilmatt @Yoe777
Update from what I found on my end, FTP site is down again at least for me saying connection refused when it was working fine over the past week, likely meaning my IP is blacklisted temporarily from doing too many updates/reloads in too short of time period as I was throwing together a replacement for Shallalist I may try to get up on GitHub. I got my UT1 downloading and processing again by changing the feed URL for UT1 in two files:

/usr/local/pkg/pfblockerng/ut1_global_usage
``` as well as in ```
/usr/local/www/pfblockerng/pfblockerng.php

changed both to the https URL

https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz

followed with then going to the DNSBL Category tab to save settings so that it updates the config.xml. Then run a force reload all. On update/re-install of pfBlockerNG those two files will need updated again because they will be overwritten

vilion

Hi everyone,

I’ve seen quite a few threads here about issues with blacklist feeds (UT1 download errors, Shalla being discontinued, some community feeds being outdated or filled with dead domains). I’m curious how you are currently handling DNSBL filtering in production:
• Which feeds are you relying on today?
• Do you run into problems with overblocking or too many dead entries?
• Do you prefer sticking to a few well-known sources (like Spamhaus, OISD), or do you combine many community lists?
• Are there specific categories (Adult, Gambling, Phishing, Malware, Social Media) where you’d like to see more reliable/curated data?

It seems to me there’s a growing need for “clean, curated lists” – feeds that are actively checked, cleaned up, and categorized, rather than just large dumps.

Gertjan

@vilion said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

It seems to me there’s a growing need for “clean, curated lists” – feeds that are actively checked, cleaned up, and categorized, rather than just large dumps.

The good news is the bad news : these lists exist.
As you've said : “clean, curated lists – feeds that are actively checked, cleaned up, and categorized” so some one is doing the works for us.

So, I'm sure these lists exist. But not for free. This 'some one' wants to get paid ...

The result might be what Maximind is already doing : you (= pfBlockerng) has to use an API to access the files to downloads; The API will check if you are authorized to do so = you've paid the bill.

If there was a good quality maintained etc list avaible for free, then it would be proposed by pfBlockerng tomorrow ^^

vilion

@Gertjan

Thanks for your reply – that’s also my impression.
The point is: I don’t really see any lists right now that are actually “maintained” in the sense of being actively cleaned up, checked for dead domains, categorized, etc.

That’s why my main interest is more about the demand:
Would curated lists really be a game changer for admins? Would they be more helpful than what’s available today, or are most people already using other alternatives? If so, which ones?

And from your perspective, what would be your expectation towards “community lists”?
(e.g. reliability, update frequency, categories, fewer false positives?)