Problems downloading UT1
-
So yes I am having problems download UT1. The log just times out and fails. There is an older post that clams that the link for the UT1 including squidguard is no longer active.
https://forum.netgate.com/topic/172338/pfblockerng-shallalist-and-ut1-failed/2
They say:
"This is the link that pfBlockerNG should use:
https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz"
So I made the change in
ut1_global_usage and in config.xml of pfSense.
After the change in the config.xml file I did start to get downloads but I got other errors then,
Downloading Blacklist Database(s) [ ut1 (~8.5MB) ] ... Please wait ...
UT1 ... Completed (download completed with new link. Before the new link, this failed.)[ UT1_adult ] Downloading update [ 01/29/24 :19:26:48 ] .
[ UT1_adult] file_get_contents([/var/db/pfblocker/ut1/ut1_adult): Failed to open stream: No such file or directory[ DNSBL UT1 - UT1_adult ] Download FAIL
Local File Failureetc with all the rest of the categories.
It is of course interesting that in my other systems the squidguard link remains and they seem to function fine, which tells me that I am looking in the wrong place. It is, however, puzzling to see that link there when it indeed does not seem to work.
Thanks for any help you might offer.
Roy Eberhardt
Hmm. I'm guessing I am using the wrong link ... ftp://ftp.ut-capitole.fr/blacklist/ will probably be better as you won't have wget in your code with ftp.
-
@reberhar So I had two similar machines acting exactly the sameway. They only shared the config file. I am preparing for HA.
Trying lots of links for the UT1 list and changing many settings, one started to work with the squidguard list. Which means that my config file is corrupted. I am not looking forward to doing a pfblocker clean install, but it seems that that is what it needs.
SIGH.
-
@reberhar said in Problems downloading UT1:
"This is the link that pfBlockerNG should use:
https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz"
Have a look at that zipped file :
It contains files and sub folders with the actual dnsbl files ... (I'm not sure if pfBlockerng can handle such file / resource )
And keep in mind : when the file is 'unzipped' you wind up with 176 Mbytes.
pfBlockerng uses PHP, world's worst ASCI file text handler : the process of sorting, removing doubles, and then 'cat' all these DNSBL together in one big file, and then remove the 'known white list dns' file ..... pfBlockerng will probably bail out, or jusrt show a PHP OOM error, thus the PHP-FPM will core dump.Before, when using UT, you had a make a (wise) selection between the categories. Taking them all might be possible if you have a "big system".
I've a Netgate 4100 : it's a no-go. -
@Gertjan Hi Gertjan,
Thanks for your answer and suggestions. Just FYI, Google sent your reply to the SPAM folder. Hmm, I just saw it today.
I had noted the structure of UT1. I have a couple of comments.
Yes indeed, using this link "https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz" fails when put in the ut1_global_usage in /usr/local/pkg/pfblockerng.
I have download and used the individual *.tar.gz files individually and that works.
On my other units where I am still using the SquidGuard link successfully, it leaves me wondering what is happening there. Perhaps the down load fails and pfBlockerng is using what it had originally download. This is the behavior that I expect from what I read. My problem did not start until I had to make a clean install on pfBlocker. I know, I should look at the logs, but on my large to do list, pfblockerng is pretty far down for now. However, if the SquidGuard list is failing, then I need to move to groups on my other installs.
As I fuss around with all this stuff, I am impressed with how stable Freebsd, and pfSense are. I know I said that I was going to reinstall, but as I looked at the problem I realized that the code had its integrity. It was doing what it was supposed to do. I very rarely reinstall now, and certainly NEVER from scratch. You guys got some really good software.
I wonder if BBcan is going to fix this? Free smut lists are sometimes not very well maintatined. What an awful job anyway.
(What I meant about BBcan "Fixing this" was not that he would fix UT1, but that pfBlockerng would be able to properly parse the profered link and install the categories.)
Roy
-
@reberhar said in Problems downloading UT1:
I wonder if BBcan is going to fix this?
All BB can do is proposing pre build list with 'known' working sites that propose lists to down.
He is not making these lists, doesn't own them, doesn't maintain them ....Image this situation :
You go here and you put a high quality DNSBL on it.
It becomes popular.
People like BB includes it in its pfBlockerng so now many (100 of thousands) can download the list "with one click" - and have the list updates every xx hours ( !! ).
Your list has a reasonable size, just a couple of ten of megabits,It won't take long before your host company contacts you with a huge bill (xxx $ added to the monthly 5 $ renting of the VPS) : your small vps server traffic isn't 'unlimited' : it is metered !!
But you've probably noticed that these 'thousands' of clients trying to download the same file over and over again bring your small VPS to its knees ... So, Ok, go here. Things look better now : you have now at least the 1 Gbit/sec, and more traffic/month before you ruin yourself financially.UT1 is a known list as it was probably a university project : there was a "question" : how to protect the Internet access of schools and other 'government" instances ?
There was the equipment : the university had the system resources (== servers).
So, some students launched a project, probably part of their exam.
It became a success - world wide.
Then the biggest motivation killer kicked in : students graduate, which means that 'everything' was paid by the 'society' (universities are government paid in France) before, became now their own financial burden. Hosting and maintaining a big list like "UT1" costs a lot. And worse, the better the DNSBL is, the more it will cost you.
UT1 will have the same destination as the shallaist : it will go "out of business" if they do not add some financial model to it.
My advise : use UT1, but select only a small sub part of it.
Don't take to grab it all, as this will help to 'kill' them.
Adapt your update frequency :
Another aspect is : pfBlockerng isn't a binary (machine language) but written very in efficient PHP - that's like BASIC, the language you learned at school, but more adapted to web servers.
PHP isn't the best way to handle text files which contains millions of entries.
With my 4 Gbytes 4100, I can't select the entire UT1 list without major issues : it will run out of memory, bringing the firewall to its knees. All of UT1 needs probably 8 or 16 Gbytes, and not a
but a much bigger processor.
So, you also start to add $$ as your monthly electrify bill will rise while feeding this system. -
@Gertjan Hi Gertjan,
(What I meant about BBcan "Fixing this" was not that he would fix UT1, but that pfBlockerng would be able to properly parse the profered link and install the categories.)
Yes the way the UT1 list was formed had occurred to me, mostly, except the details about the actual origin of the lists. I have taken your suggested steps really quite at the inception of our using pfBlocker, after abandoning Dan's Guardian. There are only a handful of sublists that I download, mostly the smut list with a couple of others. These I do individuall through groups, and that has reduced the size of the download from the way BBcan has it programmed in pfBlocker. That was to download the entire list and parse it.
I certainly suspected that UT1's server was being overwhelmed. It is often a response of providers to block those who do too many downloads. This was what I was suspecting on my end. I do indeed download once a week. However one my techs had this server at every hour. It is now at once a week too. Since the web administrator there isn´t known to me I really don´t know what is happening.
As far as php is concerned, I have read the pfBlockerng code and it does indeed remind me of basic. I have written in different basics, C, C++, C#, and older languages still, like Pascal, and some of the other offerings we learned in survey of computer languages, and some in an in house language that we have. I have written exes, libraries, drivers, and applications in machine code and assembly. Yada yada. What they stick in Docker and other like platforms I have not experienced, spending most of my time now trying to dominate Internet infrasture, and different firewalls, but mostly on pfSense.
As far as hardware is concerned, we are a small group so I have been buying used Dell Optiplexes with maybe 6th generation or earlier CPUs, I-7s and I put 16 gigs of ram, an SSD, and appropriate NICs, usually Intel based. The units themselves cost me 120 dollars each. Then there is the cost of the upgrades which is really quite cheap. This is overkill I realize, but we have had quite good service with units like these. I am maintaing 5 locations, with 4 of them using HA. There are other Linux servers involved too. It keeps me quite busy.
I apologize for not being more forthcoming with detail in what equipment I drive and what I do. I appreciate greatly your support, and the support of the forum. I have tried to contribute where I can: where there have been questions that I have answered for myself, so that I can share with others. Sometimes it works and people are helped.
God bless you Gertjan,
Roy
