Shallalist: Failed to open stream: No such file or directory

dmitri · Nov 3, 2023, 3:37 PM

I've recently migrated from one hardware with 23.05 to another non-Netgate appliance (Sophos XG 210 rev3) on 2.7, and moved over all my configuration. Everything seems to work as expected. The only problem I have noticed is that pfblockerNG is having issues loading Shallalist. I see the following lines in the log:

Downloading Blacklist Database(s) [ shallalist (~10MB) | ut1 (~8.5MB) ] ... Please wait ...
	Shallalist ... Completed
	UT1 ... Completed

[ Shallalist_adv ]		 Downloading update [ 11/1/23 23:26:33 ] .
[ Shallalist_adv ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_adv): Failed to open stream: No such file or directory


 [ DNSBL_Shallalist - Shallalist_adv ] Download FAIL
   Local File Failure

[ Shallalist_spyware ]		 Downloading update .
[ Shallalist_spyware ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_spyware): Failed to open stream: No such file or directory


 [ DNSBL_Shallalist - Shallalist_spyware ] Download FAIL
   Local File Failure

[ Shallalist_tracker ]		 Downloading update .
[ Shallalist_tracker ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_tracker): Failed to open stream: No such file or directory


 [ DNSBL_Shallalist - Shallalist_tracker ] Download FAIL
   Local File Failure

I've searched forums and found some old posts, asking to make manual changes to the pfblockerNG code at a specific line number, but I couldn't find that specific entry in my file. I think the versions have changed multiple times and it's no longer applicable. (https://www.reddit.com/r/pfBlockerNG/comments/fqjdc5/comment/flqzkgp/)

I've tried reinstalling the package, and forcing reload, but the problem persists.

Any help would be appreciated! Thanks!

dmitri · Nov 3, 2023, 4:05 PM

Ah, well, never mind. Looks like Shallalist isn't available any longer: https://www.reddit.com/r/pfBlockerNG/comments/sddop5/shallalist_has_closed_its_doors/

I guess on my old box, I had files already loaded, they just weren't being updated, and thus I never saw the error. Maybe this option should be removed from pfBlockerNG, to avoid confusion.

jrey · Nov 3, 2023, 4:06 PM

@dmitri

So did you also notice if the cron job then becomes "hung" or "seems stuck" after the failed download?

https://forum.netgate.com/topic/183728/cron-job-once-day/27?_=1699024553731

dmitri · Nov 3, 2023, 4:16 PM

@jrey No, nothing hangs. I just replied to my own post ... Shallalist is no longer in business, it seems. It's a bit confusing because there is a line in the log that says:

Downloading Blacklist Database(s) [ shallalist (~10MB) | ut1 (~8.5MB) ] ... Please wait ...
	Shallalist ... Completed

It makes it seem as if it's actually successfully downloaded something. I checked the RAW file, and it's only about 45 KiB. Opening the file actually reveals that it's just an HTML file that says the domain is no longer available.

I suppose this could be viewed as a security issue (however slight), if a malicious actor actually buys that domain and millions of users have their firewalls configured to download files from it, it could pose unknown risks.

jrey · Nov 3, 2023, 4:41 PM

@dmitri

Thanks that might be a clue for @NollipfSense

there are likely strange things left in the downloads areas that need to be clean up (perhaps manually)

indeed strange that a download says one size, but the file is another.
would be curious to know what files are in the following

ls -l /var/db/pfblockerng/dnsbl*

dmitri · Nov 3, 2023, 4:56 PM

@jrey I think the statement "shallalist (~10MB)" is hardcoded in the pfBlockerNG code, just a guess. It doesn't actually check the true size. It's just a note so that you know how long it should take to download.

I checked /var/db/pfblockerng/dnsbl*, and there are files for Shallalist there, but they are all in the format of "Shallalist_*.fail" with the size of 0 KiB.

NollipfSense · Nov 3, 2023, 5:45 PM

@jrey said in Shallalist: Failed to open stream: No such file or directory:

@dmitri

Thanks that might be a clue for @NollipfSense

there are likely strange things left in the downloads areas that need to be clean up (perhaps manually)

indeed strange that a download says one size, but the file is another.
would be curious to know what files are in the following

ls -l /var/db/pfblockerng/dnsbl*

That's why I removed it, but the hanging continues...

jdeloach · Nov 3, 2023, 6:21 PM

@NollipfSense said in Shallalist: Failed to open stream: No such file or directory:

@jrey said in Shallalist: Failed to open stream: No such file or directory:

@dmitri

Thanks that might be a clue for @NollipfSense

there are likely strange things left in the downloads areas that need to be clean up (perhaps manually)

indeed strange that a download says one size, but the file is another.
would be curious to know what files are in the following

ls -l /var/db/pfblockerng/dnsbl*

That's why I removed it, but the hanging continues...

Since it's not updated anymore but still catches a lot bad actors, just turn off updates for that list and it will work just fine. I turned off updates for Shallalist a couple years ago but still use it and it causes no issues with the Cron app or anything else.

NollipfSense · Nov 3, 2023, 7:32 PM

@jdeloach said in Shallalist: Failed to open stream: No such file or directory:

I turned off updates for Shallalist

Interesting...making me think, since I should still have data.

Patch · Nov 4, 2023, 8:21 AM

@dmitri said in Shallalist: Failed to open stream: No such file or directory:

Maybe this option should be removed from pfBlockerNG, to avoid confusion.

not sure what is cleanest. There does appear to be some instability in the available data sources.
Similar story for VPN list https://forum.netgate.com/topic/177253/vpn-ip-list

jdeloach · Nov 4, 2023, 12:44 PM

@Patch said in Shallalist: Failed to open stream: No such file or directory:

@dmitri said in Shallalist: Failed to open stream: No such file or directory:

Maybe this option should be removed from pfBlockerNG, to avoid confusion.

not sure what is cleanest. There does appear to be some instability in the available data sources.
Similar story for VPN list https://forum.netgate.com/topic/177253/vpn-ip-list

Not sure what Shallalist and UT1 list have to do with vpn-ip-list.

Shallalist and UT1 list are installed as part of the pfBlockerNG package when you install it on pfSense. As I remember they are installed in different directories from where other lists are put when you add them to pfBlockerNG,

jrey · Nov 4, 2023, 1:13 PM

@Patch said in Shallalist: Failed to open stream: No such file or directory:

Maybe this option should be removed from pfBlockerNG, to avoid confusion.

As a temporary work around if you don't want it to be on the web page for selection

a) make sure none of the lists are selected on the Shallalist selection

b) disable the list if it is select Blacklists (not selected)

c) command line -- ssh into

cd /usr/local/pkg/pfblockerng
mv shallalist_global_usage shallalist_global_usage.notused

d) reload the webpage

Shallalist can no longer be selected.

deftconcern · Dec 5, 2023, 3:34 AM

@jdeloach said in Shallalist: Failed to open stream: No such file or directory:

Since it's not updated anymore but still catches a lot bad actors, just turn off updates for that list and it will work just fine. I turned off updates for Shallalist a couple years ago but still use it and it causes no issues with the Cron app or anything else.

It's really as you said. As long as you control your account, your information will be well protected. Maybe this update will stay the same this time. Usually updates happen when they want to upgrade a feature.