Squidguard blacklist possible update issue
-
SquidGuard meltdown cause: The reduced blacklist from UT Capitole had a broken nested folder structure that recursively doubled every category, creating hundreds of duplicate database entries instead of 144. This caused massive RAM consumption and the "too many queued redirectors" crash.
Fix: Switching to the full blacklists_for_pfsense.tar.gz which has the correct flat folder structure resolved the SquidGuard instability completely.
some of the errors you might see:
FATAL: Too many queued redirector requests; see on-persistent-overload WARNING: All 48/48 redirector processes are busy WARNING: 96 pending requests queued WARNING: Consider increasing the number of redirector processes Squid Cache (Version 5.8): Terminated abnormallyOthers once you find it and reload the list are... this was attempted load with everything turned off
kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem full Jun 9 08:24:40 kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem full Jun 9 08:24:39 kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem full Jun 9 08:24:38 kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem full Jun 9 08:24:37 kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem full Jun 9 08:24:36 kernel pid 99405 (squidGuard), uid 0 inumber 9875 on /tmp/squidGuard: filesystem fullReason:
This is because the list is doubled for some reason light now this is what loads....[blk_blacklists_adult] access ---- [blk_blacklists_agressif] access ---- [blk_blacklists_ai] access ---- [blk_blacklists_arjel] access ---- [blk_blacklists_associations_religieuses] access ---- [blk_blacklists_astrology] access ---- [blk_blacklists_audio-video] access ---- [blk_blacklists_bank] access ---- [blk_blacklists_bitcoin] access ---- [blk_blacklists_blacklists_adult] access ---- [blk_blacklists_blacklists_agressif] access ---- [blk_blacklists_blacklists_ai] access ---- [blk_blacklists_blacklists_arjel] access ---- [blk_blacklists_blacklists_associations_religieuses] access ---- [blk_blacklists_blacklists_astrology] access ---- [blk_blacklists_blacklists_audio-video] access ---- [blk_blacklists_blacklists_bank] access ---- [blk_blacklists_blacklists_bitcoin] access ---- [blk_blacklists_blacklists_blog] access ---- [blk_blacklists_blacklists_catalogue-biu-toulouse] access ---- [blk_blacklists_blacklists_celebrity] access ---- [blk_blacklists_blacklists_chat] access ---- [blk_blacklists_blacklists_child] access ---- [blk_blacklists_blacklists_cleaning] access ---- [blk_blacklists_blacklists_cooking] access ---- [blk_blacklists_blacklists_cryptojacking] access ---- [blk_blacklists_blacklists_dangerous_material] access ---- [blk_blacklists_blacklists_dating] access ---- [blk_blacklists_blacklists_ddos] access ---- [blk_blacklists_blacklists_dialer] access ---- [blk_blacklists_blacklists_doh] access ---- [blk_blacklists_blacklists_download] access ---- [blk_blacklists_blacklists_drogue] access ---- [blk_blacklists_blacklists_dynamic-dns] access ---- [blk_blacklists_blacklists_educational_games] access ---- [blk_blacklists_blacklists_examen_pix] access ---- [blk_blacklists_blacklists_exceptions_liste_bu] access ---- [blk_blacklists_blacklists_fakenews] access ---- [blk_blacklists_blacklists_filehosting] access ---- [blk_blacklists_blacklists_financial] access ---- [blk_blacklists_blacklists_forums] access ---- [blk_blacklists_blacklists_gambling] access ---- [blk_blacklists_blacklists_games] access ---- [blk_blacklists_blacklists_good] access ---- [blk_blacklists_blacklists_hacking] access ---- [blk_blacklists_blacklists_indisponible] access ---- [blk_blacklists_blacklists_jobsearch] access ---- [blk_blacklists_blacklists_jstor] access ---- [blk_blacklists_blacklists_lingerie] access ---- [blk_blacklists_blacklists_liste_blanche] access ---- [blk_blacklists_blacklists_liste_bu] access ---- [blk_blacklists_blacklists_local] access ---- [blk_blacklists_blacklists_malware] access ---- [blk_blacklists_blacklists_manga] access ---- [blk_blacklists_blacklists_marketingware] access ---- [blk_blacklists_blacklists_mixed_adult] access ---- [blk_blacklists_blacklists_mobile-phone] access ---- [blk_blacklists_blacklists_phishing] access ---- [blk_blacklists_blacklists_press] access ---- [blk_blacklists_blacklists_publicite] access ---- [blk_blacklists_blacklists_radio] access ---- [blk_blacklists_blacklists_reaffected] access ---- [blk_blacklists_blacklists_redirector] access ---- [blk_blacklists_blacklists_remote-control] access ---- [blk_blacklists_blacklists_residential-proxies] access ---- [blk_blacklists_blacklists_sect] access ---- [blk_blacklists_blacklists_sexual_education] access ---- [blk_blacklists_blacklists_shopping] access ---- [blk_blacklists_blacklists_shortener] access ---- [blk_blacklists_blacklists_social_networks] access ---- [blk_blacklists_blacklists_special] access ---- [blk_blacklists_blacklists_sports] access ---- [blk_blacklists_blacklists_stalkerware] access ---- [blk_blacklists_blacklists_strict_redirector] access ---- [blk_blacklists_blacklists_strong_redirector] access ---- [blk_blacklists_blacklists_translation] access ---- [blk_blacklists_blacklists_tricheur] access ---- [blk_blacklists_blacklists_tricheur_pix] access ---- [blk_blacklists_blacklists_update] access ---- [blk_blacklists_blacklists_verisign] access ---- [blk_blacklists_blacklists_vpn] access ---- [blk_blacklists_blacklists_warez] access ---- [blk_blacklists_blacklists_webhosting] access ---- [blk_blacklists_blacklists_webmail] access ---- [blk_blacklists_blog] access ---- [blk_blacklists_celebrity] access ---- [blk_blacklists_chat] access ---- [blk_blacklists_child] access ---- [blk_blacklists_cleaning] access ---- [blk_blacklists_cooking] access ---- [blk_blacklists_cryptojacking] access ---- [blk_blacklists_dangerous_material] access ---- [blk_blacklists_dating] access ---- [blk_blacklists_ddos] access ---- [blk_blacklists_dialer] access ---- [blk_blacklists_doh] access ---- [blk_blacklists_download] access ---- [blk_blacklists_drogue] access ---- [blk_blacklists_dynamic-dns] access ---- [blk_blacklists_educational_games] access ---- [blk_blacklists_examen_pix] access ---- [blk_blacklists_exceptions_liste_bu] access ---- [blk_blacklists_fakenews] access ---- [blk_blacklists_filehosting] access ---- [blk_blacklists_financial] access ---- [blk_blacklists_forums] access ---- [blk_blacklists_gambling] access ---- [blk_blacklists_games] access ---- [blk_blacklists_hacking] access ---- [blk_blacklists_jobsearch] access ---- [blk_blacklists_lingerie] access ---- [blk_blacklists_liste_blanche] access ---- [blk_blacklists_liste_bu] access ---- [blk_blacklists_malware] access ---- [blk_blacklists_manga] access ---- [blk_blacklists_marketingware] access ---- [blk_blacklists_mixed_adult] access ---- [blk_blacklists_mobile-phone] access ---- [blk_blacklists_phishing] access ---- [blk_blacklists_press] access ---- [blk_blacklists_publicite] access ---- [blk_blacklists_radio] access ---- [blk_blacklists_reaffected] access ---- [blk_blacklists_redirector] access ---- [blk_blacklists_remote-control] access ---- [blk_blacklists_residential-proxies] access ---- [blk_blacklists_sect] access ---- [blk_blacklists_sexual_education] access ---- [blk_blacklists_shopping] access ---- [blk_blacklists_shortener] access ---- [blk_blacklists_social_networks] access ---- [blk_blacklists_special] access ---- [blk_blacklists_sports] access ---- [blk_blacklists_stalkerware] access ---- [blk_blacklists_strict_redirector] access ---- [blk_blacklists_strong_redirector] access ---- [blk_blacklists_translation] access ---- [blk_blacklists_tricheur] access ---- [blk_blacklists_tricheur_pix] access ---- [blk_blacklists_update] access ---- [blk_blacklists_vpn] access ---- [blk_blacklists_warez] access ---- [blk_blacklists_webhosting] access ---- [blk_blacklists_webmail] accessAnd if you check the list manually you can see what occurs with the reduced file
inside of blacklist for this tar file right now is another blacklist a recursive list

And it has all the same files:

The other list does not have this issue
USE THIS ONE!http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense.tar.gzUse this one :

there is just a bad folder in the smaller list right now. Save yourself a headache and swap to the other list, because it is hard to pinpoint this issue it looks like redirector issues
-
YEAH!!!
The list is officially fixed! A massive thank you to Fabrice Prigent for generously donating his time and expertise to get this sorted for us. We really appreciate your hard work, Fabrice!
Both work now