Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29

[provels]

Appears to be the same issue as this post re. 2.4.5RC
https://forum.netgate.com/topic/150239/shallalist-and-ut1-do-not-work-pfblckerng-devel-2-2-5-28-on-pfsense-2-4-5-rc

Downloading Blacklist Database(s) [ shallalist (~10MB) | ut1 (~8.5MB) ] ... Please wait ...
	Shallalist ... Completed
	UT1 ... Completed

[ Shallalist_adv ]		 Downloading update [ 03/26/20 18:09:29 ] .
[ Shallalist_adv ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_adv): failed to open stream: No such file or directory


 [ DNSBL_Shallalist - Shallalist_adv ] Download FAIL
   Local File Failure

[ Shallalist_costtraps ]	 Downloading update .
[ Shallalist_costtraps ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_costtraps): failed to open stream: No such file or directory

.raw files are present in /var/db/pfblockerng.

EDIT - So I built a new 2.4.5 VM and restored my config, same result (plus OpenVPN servers would not start). I rolled back my upgraded 2.4.5 to 2.4.4-RELEASE-p3 VM and all works fine,
I see the Shallalist and UT1 files appear as below on 2.4.4-RELEASE-p3

Not whining. Just telling you what I see.

[provels]

So I fixed my OpenVPN no start, my error. I had 2 scripts I failed to copy over to the virgin 2.4.5 VM.
But pfBNG Shallalist/UT1 still borked.

[riften]

Same here about Shallalist and UT1. Installed fresh and then restored config, issue persists. And on the REPORTS/ALERTS, I have the Unk issue, GEOIP not working.

[provels]

@riften said in Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29:

I have the Unk issue

What is that?

[BBcan177]

See this thread:

https://www.reddit.com/r/pfBlockerNG/comments/fqjdc5/pfblockerngdevel_downloading_lists_but_not_able/flqzkgp/

[BBcan177]

@riften said in Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29:

And on the REPORTS/ALERTS, I have the Unk issue, GEOIP not working.

What version of pfBlockerNG are you using? First make sure that MaxMind Key has been properly entered and that MaxMind was downloaded. Then restart the "pfb_filter" service (pfBlockerNG-devel).

"unk" will show the the Alerts tab - GeoIP column when MaxMind database is not found, so it can't report the correct Country.

[provels]

@BBcan177 said in Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29:

See this thread:

https://www.reddit.com/r/pfBlockerNG/comments/fqjdc5/pfblockerngdevel_downloading_lists_but_not_able/flqzkgp/

Fixed Shallalist and UT1 for me, thank you!

[riften]

@BBcan177 Thanks for the help! Been using PFBlocker for about a year without issue till now. I'm on 2.2.5_29 devel. Yes I put the KEY in, actually went to MAXMIND site and generated new key and am using that. Also went into folder where the files should be and they are there. The 'edit' icons show on the right of the rules on the IP/GEOIP tab when the KEY is listed and they are not there when the key field is blank. It was working fine till I upgraded PFSense. I do NOT get any indication that the page saved or any flag from MAXMIND when I insert the KEY and save. It just does a quick screen refresh. So yesterday I installed fresh from a memory stick (to my SSD) and then imported a prior backup made on the last ver of PFSense. Issue persists. I just found that one of the feeds I just started using, Quidsup_Trackers, was blocking maxmind.com so I disabled it and can now get to it BTW.
From CRON just now:

Downloading Blacklist Database(s) [ shallalist (~10MB) ] ... Please wait ...
Shallalist ... Completed

[ Shallalist_aggressive ] Downloading update [ 03/28/20 09:52:06 ] .
[ Shallalist_aggressive ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_aggressive): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_aggressive ] Download FAIL
Local File Failure

[ Shallalist_costtraps ] Downloading update .
[ Shallalist_costtraps ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_costtraps): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_costtraps ] Download FAIL
Local File Failure

[ Shallalist_spyware ] Downloading update .
[ Shallalist_spyware ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_spyware): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_spyware ] Download FAIL
Local File Failure

[ Shallalist_tracker ] Downloading update .
[ Shallalist_tracker ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_tracker): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_tracker ] Download FAIL
Local File Failure

[ Shallalist_violence ] Downloading update .
[ Shallalist_violence ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_violence): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_violence ] Download FAIL
Local File Failure

[ Shallalist_warez ] Downloading update .
[ Shallalist_warez ] file_get_contents(/var/db/pfblockerng/shallalist/shallalist_warez): failed to open stream: No such file or directory

[ DNSBL_Shallalist - Shallalist_warez ] Download FAIL
Local File Failure

[provels]

@riften do @BBcan177 's edit from the Reddit thread.

49 minutes ago
Ok I think I found the issue.

Can you edit this file: /usr/local/pkg/pfblockerng/pfblockerng.inc

and change Line #2346

From:

if ($file_type == 'application/x-gzip') {

To:

if ($file_type == 'application/x-gzip' || $file_type == 'application/gzip') {

Example here:

https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L2346

Then try a Force Update

[BBcan177]

@provels said in Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29:

@riften do @BBcan177 's edit from the Reddit thread.
49 minutes ago
Ok I think I found the issue.

Can you edit this file: /usr/local/pkg/pfblockerng/pfblockerng.inc

and change Line #2346

From:

if ($file_type == 'application/x-gzip') {

To:

if ($file_type == 'application/x-gzip' || $file_type == 'application/gzip') {

Example here:

https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L2346

Then try a Force Update

Yes there was a change to the file mime-types in pfSense 2.4.5 and the change to fix Shallalist and UT1 is also needed to download one of the MaxMind databases.

Follow those instructions in the reddit post and redownload MaxMind from the shell with this command:

php -f /usr/local/www/pfblockerng/pfblockerng.php dc

I will post a fix for this now.

[riften]

@BBcan177 Thank you again for your help! Yes I followed the Reddit thread and that fixed Shallalist. I can see it working. Unk still shows as GEOIP though it appears to be working.

[BBcan177]

@riften
Did you run the php command above? Restart the pfb_filter Service? Keep in mind that this will only change new IP Block events and not the old events.

[riften]

@BBcan177
Yep I did, but then spent the day doing my taxes. Looking at it again now, yes it is working again! Thank you!

[GregBinSD]

@BBcan177
I just did a clean install of pfSense 2.4.5, and I want to perform the install of pfBlockerNG-devel from the Package Manager. The latest change log entry is from February 8, 2020. I would like to wait until you have pushed the changes to the Package Manager. Can you tell me how long that might be?
Thank you.

[BBcan177]

@GregBinSD said in Shallalist and UT1 lists not working on 2.4.5-RELEASE/pfBlockerNG-devel 2.2.5_29:

Can you tell me how long that might be?

The pfSense devs need to review and approve. Hopefully next week.