DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed)

tinfoilmatt

@smolka_J said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

both directories /cf/conf/ and /conf/ are both present in both version, they are symbolically linked

You're right about this, and all my previous comments about the XML config file storage location are incorrect. I was looking at my SSH file browser in that moment, which apparently doesn't display the /conf symlink. Shows you how familiar I am with the file!

(Output of ls -l for a given filesystem location displays any/all symlinks.)

OP's issue with the UT1 feed (and possibly yours too) is almost certainly with either DNSBL or IP filtering. Although in your case, Suricata might've been the more likely culprit. Was your instance in fact running on the WAN interface? And you might've disabled it but did the instance actually stop? You might've ensured it did by rebooting the box, but still—lots of variables there.

You should set your config.xml file back and try troubleshooting whatever the issue is further.

smolka_J

@tinfoilmatt I deleted my other posts to avoid confusion. It may have added to things, Suricata I only run on LAN, VLAN and VPN interfaces but have the same whitelist configured in an alias for it. Currently, the FTP site loads perfectly fine in a ftp or file manager also whether Suricata is disabled or enabled. Reading from why others changed their URL in the past also was due to the FTP link being intermittent at times, changing the feed URL in config.xml when FTP feed is down does allow the file to download and clear the download error itself. After further looking into though, doing this causes the processing of it to stop there and not update the UT1 orig files or populate them if they didn't already previously before changing the feed URL. Having pfBlocker enabled while disabling DNSBL followed with a reboot, re-enable DNSBL, reload restored my "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz" feed URL into my config.xml without needing to manually edit it back, downloaded and fully processed updates.

Lesson learned, if and when UT1's ftp goes down temporarily, leave things as is, it will come back up. More would have to be re-wrote to completely change over to a different URL feed by default but FTP maybe is preferred. Thank you for guiding me to dig further, thought I fixed one issue but led myself to turning it into another

SteveITS

@smolka_J
Side note re: Suricata, because of how it runs the instance in the native interface will see the VLAN packets i.e. it ignores the VLAN tags. So no need/point running a second instance on the VLAN interface if it’s on the parent.

smolka_J

@smolka_J Gracias, will keep that in mind as I test n tune re-mapping. Just acquired a set of Grandstream APs to first start endulging down that path, still have a few more fiber optic and POE drops to get in place first to LAGG each AP at 5Gb onto 10G backplane, moving through rafters of the attic is a little slow on medical leave

Yoe777

@spinner I am having the same issue... have you been able to resolve this?

smolka_J

@Yoe777 #1, the FTP site does have its time periods of downtime which might fall in line with your current CRON update schedule.
#2 If you had upgraded in the past from a previous version of pfSense and/or with a config.xml imported/restored from a previous installlation, you may have an invalid or non-compatible UT1 feed link that isn't loading properly, pfBlockerNG is programmed in multiple areas to parse only "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz", if that feed URL is different for any reason like mine was from me previously trying to mitigate fixing my concern with FTP failing randomly, I had changed it to the https url which did download but did not parse into the files needed.
Steps that fixed my UT1 feed URL to the correct one in my config and restored full parsing/download:

Make sure pfBlockerNG is enabled on the general tab.
Go to the DNSBL tab and disable onle DNSBL, save
Reboot pfSense
Go back to the DNSBL tab and re-enable DNSBL
Run a Force Update>Reload>All

provels

Just throwing this in, that you may want to set your update time to some random time and not on the hour, and not too often either. These lists don't change much in a day's time.

Yoe777

I followed these steps and still getting same error.
I removed and reinstalled entire package and still am getting the same error.

This is a fresh install of pfSense only 2 weeks old Version 2.7.2-RELEASE (amd64).

smolka_J

@Yoe777 Do you get a valid IP back doing a DNS lookup to ftp.ut-capitole.fr?

Yoe777

@smolka_J

PING heimdall.ut-capitole.fr (193.49.48.249): 56 data bytes
64 bytes from 193.49.48.249: icmp_seq=0 ttl=50 time=119.248 ms
64 bytes from 193.49.48.249: icmp_seq=1 ttl=50 time=118.943 ms
64 bytes from 193.49.48.249: icmp_seq=2 ttl=50 time=118.840 ms

--- heimdall.ut-capitole.fr ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 118.840/119.010/119.248/0.173 ms

smolka_J

@Yoe777 Do you have either Snort or Suricata running? Either could be monitoring/scanning the FTP port keeping pfBlockerNG from being able to process the download timely, may need the IP or domains whitelisted in a passlist there to keep Snort/Suricata from scanning it

Yoe777

@smolka_J No I dont use either currently.

smolka_J

@Yoe777 Im at a loss otherwise then. Have you manually checked your config.xml to verify which URL your UT1 feed is set to currently? It should read as "ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz"

Yoe777

@smolka_J That is what it is:

	<item>
		<title>UT1</title>
		<xml>ut1</xml>
		<feed>ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz</feed>

smolka_J

@Yoe777 Not certain if you have that domain whitelisted or not even though it seems to be passing for you otherwise but could be worth trying with it add if its not. Thats maybe the only thing I have different, if it is, that would be letting it work as far as I can tell, I do have ftp.ut-capitole.fr in my whitelist, being FTP it may be working better when whitelisted so there isn't an added delay waiting for the DNS query to pass through python blacklist processing first, FTP connections can be finicky like that when you don't have a full FTP client interface to tune timeout settings or have a retry/re-connect button to use

Yoe777

@smolka_J Where do I check what is whitelisted?

tinfoilmatt

@Yoe777 To check if it's blocklisted, run this command from either a shell or via shell command (Diagnostics / Command Prompt):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

smolka_J

@tinfoilmatt That will check if its being blocked but the theory I'm having is UT1 ftp may be failing to download if its not specifically whitelisted because of the time it take for a non-blocked domain to be passed through Python and all blacklists before it is validated as not being blocked, FTP connections are very time sensitive, depending on the specific FTP client in question, in this case pfBlockerNG being the FTP client, if an attempted FTP connection does not establish within so many milliseconds that the client is configured for then the FTP connection is deemed FAILED. If a domain is whitelisted, it does not have that wasted time being processed through several different other modules first like a non-blocked non-whitelisted domain does. As I noted, thats the only part my configuration has different than many other people's, UT1 ftp for me was NOT being blocked prior to me adding it to my whitelist and presently for me does not have any issue downloading and processing

Yoe777

@tinfoilmatt said in DNSBL Category (Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ... Failed UT1 ... Failed):

grep "ftp.ut-capitole.fr" /var/db/pfblockerng/dnsbl/.txt /var/db/pfblockerng/dnsblorig/.orig /var/unbound/pfb_py_data.txt /var/unbound/pfb_py_hsts.txt /var/unbound/pfb_py_ss.txt /var/unbound/pfb_py_zone.txt /usr/local/pkg/pfblockerng/dnsbl_tld /usr/local/pkg/pfblockerng/pfb_py_hsts.txt

grep: /var/unbound/pfb_py_data.txt: No such file or directory
grep: /var/unbound/pfb_py_hsts.txt: No such file or directory
grep: /var/unbound/pfb_py_ss.txt: No such file or directory
grep: /var/unbound/pfb_py_zone.txt: No such file or directory

tinfoilmatt

@Yoe777 To check if the IP address that ftp.ut-capitole.fr resolves to, 193.49.48.249, is listed anywhere:

grep "193.49.48.249" /var/db/pfblockerng/DNSBLIP_v4.txt /var/db/pfblockerng/deny/*.txt /var/db/pfblockerng/original/*.orig /var/unbound/pfb_py_ss.txt

If no output is returned, that means the IP is not potentially being filtered anywhere by pfBlockerNG. (The "No such file or directory" output should be ignored.)

I've also noticed just now that the domain heimdall.ut-capitole.fr is a CNAME of ftp.ut-capitole.fr. You should ensure that heimdall.ut-capitole.fr is also either not listed and/or whitelisted.