pfBlockerNG-devel v3.1.0_19/10

BBcan177

https://www.patreon.com/posts/77420518

Changelog:

Add "application/json" to list of allowed file download mime-types
Remove validation for paths in URL validations.
Add dash as an allowed character in Whois/TLD customlist settings
Add workaround for .ZIP compressed files. Will now validate ZIP files contents after extraction as opposed to beforehand.
Fix regression with ASN/Whois File downloads
Fix regression for the extraction of MaxMind MMDB database file
Increase DNSBL max domain count for 32GB Ram installs.
Fix Advanced Inbound/Outbound Port definition regression not saving properly
Fix DNSBL and IP Counter clearing for the Widget
Fix Alerts Tab - Dark mode colour options missing in some cases
Fix Alerts Tab - Page Refresh on/off not saving correctly

Alejo 0

@bbcan177 Thanks for your hard work and commitment !

fireodo

@bbcan177
Thanks!

On the alerts configuration page is this issue still present:

"Enter 'none' to disable." If in the color field (in GUI) you put "none" it is not getting validated/accepted!

The value is going back to #83791D and disregard the "none" and there is the following error in /var/log/pfblockerng/error.log:

PFB_FILTER - 20 | alerts hex [ 01/19/23 18:18:45 ] Failed validation [ none ]

BBcan177

@fireodo said in pfBlockerNG-devel v3.1.0_19/10:

PFB_FILTER - 20 | alerts hex [ 01/19/23 18:18:45 ] Failed validation [ none ]

Edit: /usr/local/pkg/pfblockerng/pfblockerng.inc
Line: 612
Reference:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L612

From:

if (preg_match("/^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$/", $input)) {

To:

if ($input == 'none' || preg_match("/^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$/", $input)) {

fireodo

@bbcan177 said in pfBlockerNG-devel v3.1.0_19/10:

@fireodo said in pfBlockerNG-devel v3.1.0_19/10:

PFB_FILTER - 20 | alerts hex [ 01/19/23 18:18:45 ] Failed validation [ none ]

Edit: /usr/local/pkg/pfblockerng/pfblockerng.inc
Line: 612
Reference:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L612

From:
if (preg_match("/^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$/", $input)) {
To:
if ($input == 'none' || preg_match("/^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$/", $input)) {

Thank you - now it works as expected!
That was super fast!

jdeloach

@bbcan177
Just updated to latest version v3.1.0_10 on pfSense 2.6.0 with no issues. I am now evaluating things. So far I have not seen any issues with what I use.

Thanks for hard work you put into maintaining/supporting this great package.

fireodo

@bbcan177
Hi again,

( @BBcan177 - please dont shoot )

when I set in the Dashboard Widget pfblocker to reset the IP&DNSBL Counter weekly I get this entry in cron:

after this cronjob is executed:

the DNSBL/IP-Counter Reset is again at Daily:

motivio

@fireodo I‘ve seen this too. I was wondering what is causing it.

fireodo

@motivio said in pfBlockerNG-devel v3.1.0_19/10:

I‘ve seen this too. I was wondering what is causing it.

I think (guess) there are 2 different places where the cron entrys are generated and those 2 places dont talk to each other ... simply said ...

pfT

BBcan177,

At last a chance to thank you for all your work.

Not sure if this is a regression in the latest release or not, since I usually set my ASN aliases to hold.

I noticed that on update of any ASN aliases I am getting the error:

[ AS13076_v4 ]			 Downloading update .
 Failed
 Invalid URL. Terminating Download! [ AS13076 ]

The error log shows basically the same thing:

[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS13076 ] [ 01/19/23 16:31:01 ]
[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS13076 ] [ 01/19/23 16:34:07 ]
 Failed [ 01/19/23 16:34:07 ]
[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS13076 ] [ 01/19/23 16:34:07 ]

I tried several different ASN, so not a specific problem with the example given.

I don't know the format of the URL created so I couldn't test it separately in the browser.

Not a big issue for the current ASN aliases I am using (since they are on HOLD), but does stop me creating new ones and may be more of a problem for others if they need to use them.

Otherwise everything appears to be working well - unbound even started after the update!

Thanks.

fireodo

@pft

Here too. Error thrown is:

[PFB_FILTER - 2] Invalid URL (not allowed2) [ AS47890 ] [ 01/20/23 00:09:28 ]
 Failed [ 01/20/23 00:09:28 ]

emikaadeo

v3.1.0_10 still not available for pfSense+ 22.05 (amd64) ?

manilx

@emikaadeo Wondering about the same.....

fireodo

@manilx said in pfBlockerNG-devel v3.1.0_19/10:

Wondering about the same.....

<guess> Set on hold because of some issues? </guess>

cmcdonald

@fireodo

Try again. Another package was failing and thus blocking the build job for 22.05 packages.

fireodo

@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:

@fireodo

Try again. Another package was failing and thus blocking the build job for 22.05 packages.

Its available now!

manilx

@cmcdonald Installed!

BBcan177

I have pushed another version to fix a couple of issues. So expect to see _20 or _11 depending on what pfSense version you use. Thanks.

manilx

@bbcan177 Just updated to _11.
All OK.

fireodo

@bbcan177
Updated to 3.1.0_11 - all good! Thanks

jdeloach

@bbcan177
Updated from v3.1.0_10 to v3.1.0.11 with no issues.

Thanks for the quick update.

pfT

@bbcan177

I've also updated and can confirm this fixed the error I was seeing with ASN.

I'm now getting:

[ Virgin_Media_UK_AS13076_v4 ]	 Downloading update .
  Downloading ASN: 13076... completed
. completed ..
  Empty file, Adding '127.1.7.7' to avoid download failure.

but I've seen that occasionally before (and why I originally set my other ASN aliases to HOLD when i suddenly found I couldn't connect from my phone externally via Wireguard or OpenVPN).

I suspect this probably just indicates a problem with the download site, rather than any underlying problem within pfBlockerNG-devel v3.1.0_11 on pfSense 2.6.0.

I've leave my test entry in there and see if it properly updates in a few days.

Thanks for such a swift update and resolution.

pfT

@pft said in pfBlockerNG-devel v3.1.0_19/10:

@bbcan177

I've also updated and can confirm this fixed the error I was seeing with ASN.

I'm now getting:
[ Virgin_Media_UK_AS13076_v4 ]	 Downloading update .
  Downloading ASN: 13076... completed
. completed ..
  Empty file, Adding '127.1.7.7' to avoid download failure.
but I've seen that occasionally before (and why I originally set my other ASN aliases to HOLD when i suddenly found I couldn't connect from my phone externally via Wireguard or OpenVPN).

I suspect this probably just indicates a problem with the download site, rather than any underlying problem within pfBlockerNG-devel v3.1.0_11 on pfSense 2.6.0.

I've leave my test entry in there and see if it properly updates in a few days.

Thanks for such a swift update and resolution.

As an update.
I did some more digging and found my test ASN numbers didn't contain any IP ranges, so the result was expected.

Once I chose an ASN containing IP ranges. (AS5089 for what i was expecting from AS13076) then it all worked perfectly.

I am still getting some weirdness though.

I'm getting:

====================[ Empty Lists w/127.1.7.7 ]==================

Spamhaus_Drop_v4.txt

even though the link at https://www.spamhaus.org/drop/drop.txt displays correctly and contains CIDRs.

Not entirely sure what's going on there, but i'm happy to live with it.

BBcan177

@pft said in pfBlockerNG-devel v3.1.0_19/10:

Not entirely sure what's going on there, but i'm happy to live with it

Probably due to deduplication

pfT

@bbcan177 said in pfBlockerNG-devel v3.1.0_19/10:

@pft said in pfBlockerNG-devel v3.1.0_19/10:

Not entirely sure what's going on there, but i'm happy to live with it

Probably due to deduplication

bbcan177,

Thanks.

That's exactly what it was. Coincidentally, I had just finished checking exactly that before seeing your post. I learn something every day.

I feel I have taken this thread completely off topic. Sorry for that.

I'll crawl back into my hole and stop bothering both you and the community. I feel quite abashed at the moment.

BBcan177

@pft read my tagline below....:)

Tigo

@bbcan177

Unfortunately, ver _20 is not showing up for me on pfsense 23.01 - it's still reading ver _16. I have tried updating the repository from shell, and it's reporting that repositories are up to date.

Is there an command that I can run from shell to force the upgrade for it? I also have the portBSD repsositories enabled as well.

Thanks,

yorke

@bbcan177

pfBlockerNG-devel 3.1.0_11 |ERROR| python module 'maxminddb
Pfsense 2.6.0-RELEASE
I upgraded pfBlockerNG-devel to 3.1.0_11 and got some issue before i upgraded everyting was work but now after the upgrade I am geting the errors listed below,
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'
MaxMind GeoIP download the file and GeoLite2-Country.mmdb is in /usr/local/share/GeoIP
Under Report tab Alert country code are listed under GeoIP/ASN.
The report tab showns traffic being pass/block
the dashboard for DNSBL the packets stay at 0 the counter do not move, but the ip counter works
I Referenced these post https://forum.netgate.com/topic/176668/geoip-showing-unk
https://forum.netgate.com/topic/176991/geoip-shows-country-as-unknown
to try and fix it.
their are no other errors but the ones below.

BBcan177

@yorke did you try to reinstall the package? Reboot?

Tigo

@tigo

I had also uninstalled it. Rebooted. Checked the branch updates, - and it’s still v_16. Installed it again - configured - rebooted and yet no v_20.

Perhaps it hasn’t been approved - pushed out yet?

smoke_a_J

I have been getting quite a bit of download/update failures on 3.1.0_11 for any feed trying to update. Going into my previously working feeds lists, when I first enabled a few with pfBlockerng still disabled on the general tab after updating, settings saved fine with no errors. Re-enabled pfBlocker, forced reload, forced update and cron seeing the "Invalid URL. Terminating Download!" for each. Looking into the same DNSBL lists noting failures, attempting to save/edit/disable any while pfBlocker is enabled displays the errors below on both boxes, verified DNS hostnames and lists are all working otherwise except the same couple that were still down prior pending maintenance:

DNSBL Source Definitions, Line 1: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 2: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 3: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 5: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 6: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 7: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 8: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 10: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 11: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 12: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 13: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 14: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 15: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 16: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 18: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 19: Invalid URL or Hostname not resolvable!

yorke

@bbcan177

Yes did a clean fresh install of the PfblockerNG package 3 times with the keep settings uncheck
but the error is still showing up , I notice under the Report unified Geoip is unk but under Alert Geoip/ASN list country,
the packages i have installed are PfblockerNG, Suricata and Cron (memory usage 8% ), (MBUF Usage 3%), (State table size 0%) ( cpu usage 4%) (Swap space 0%) Service Status all green,
did some test clear the Dns Resolver log under( system logs/system/dns resolver/) these 2 lines
unbound 21493 [21493:0] notice: init module 0: python
unbound 21493 [21493:0] info: [pfBlockerNG]: pfb_unbound.py script loaded
reappear go to the dashboard the DNSBL turns yellow and gives the error
|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'

BBcan177

@smoke_a_j said in pfBlockerNG-devel v3.1.0_19/10:

DNSBL Source Definitions, Line 1: Invalid URL or Hostname not resolvable!

Either DNS isn't working on your box or something is blocking those urls.

BBcan177

@yorke I would backup you config and install a fresh copy of pfSense. Followed by a restore of the config.

smoke_a_J

@bbcan177 Gracias, at first I thought it was seeming similar to the inbound permit saving issue. Regardless of having most of these feeds already whitelisted, tracked it down to about 1500 some lines of regex I had came across and added a while back, most of which seemed to not be populating any alerts but invisibly blocking at random until matching the suffix/prefix portions of the code to match known alerting lines started populating the rest. I trimmed out 1300 lines to whats working, I then realized the entire 1500 lines I found were basically a reflection of the DNSBL TLD Group 1 & 2 lists. Went back to just my first 680 lines of regex and no more ghosted double filtering and running smooth

BBcan177

@smoke_a_j If you can pm or email that regex list, I can check it out to see if there is some code improvement required.

nimrod

@yorke said in pfBlockerNG-devel v3.1.0_19/10:

@bbcan177

pfBlockerNG-devel 3.1.0_11 |ERROR| python module 'maxminddb
Pfsense 2.6.0-RELEASE
I upgraded pfBlockerNG-devel to 3.1.0_11 and got some issue before i upgraded everyting was work but now after the upgrade I am geting the errors listed below,
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'

I got the same error on pfSense v2.6.0 since the upgrade to pfBlockerNG v3.1.0_11. I have cleared the error in py_error.log. Lets see if it comes back.

The report tab showns traffic being pass/block
the dashboard for DNSBL the packets stay at 0 the counter do not move, but the ip counter works

Same issue with IP Counter. It shows number of blocked IPs for a while, but when you refresh the page, counter goes to 0. This issue happens if you apply this patch via system patches package.

More details here.

If you revert this change, counter starts working as it should and it doesnt reset to 0 after some time.

Draco

@bbcan177 said in pfBlockerNG-devel v3.1.0_19/10:

Add "application/json" to list of allowed file download mime-types

I had hoped this might let pfBlocker directly download a JSON list like the one found at Microsoft Azure IPs. This is a file I manually download and then use pfSense's GUI CMD interface to upload for pfBlocker (I set the format to AUTO). Ran this on 3.1.0_11 just now.

It didn't work. So what JSON-related things were enabled with this change?

Thanks!

nimrod

It happened again after after update.

This is the content of py_error.log

2023-01-24 16:36:57,206|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
2023-01-24 16:36:57,206|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'

Despite these errors, everything is working fine.

cmcdonald

@nimrod

What is the output of:

pkg info py* unbound