pfBlockerNG-devel v3.1.0_19/10
-
I have been getting quite a bit of download/update failures on 3.1.0_11 for any feed trying to update. Going into my previously working feeds lists, when I first enabled a few with pfBlockerng still disabled on the general tab after updating, settings saved fine with no errors. Re-enabled pfBlocker, forced reload, forced update and cron seeing the "Invalid URL. Terminating Download!" for each. Looking into the same DNSBL lists noting failures, attempting to save/edit/disable any while pfBlocker is enabled displays the errors below on both boxes, verified DNS hostnames and lists are all working otherwise except the same couple that were still down prior pending maintenance:
DNSBL Source Definitions, Line 1: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 2: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 3: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 5: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 6: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 7: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 8: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 10: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 11: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 12: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 13: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 14: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 15: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 16: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 18: Invalid URL or Hostname not resolvable!
DNSBL Source Definitions, Line 19: Invalid URL or Hostname not resolvable! -
Yes did a clean fresh install of the PfblockerNG package 3 times with the keep settings uncheck
but the error is still showing up , I notice under the Report unified Geoip is unk but under Alert Geoip/ASN list country,
the packages i have installed are PfblockerNG, Suricata and Cron (memory usage 8% ), (MBUF Usage 3%), (State table size 0%) ( cpu usage 4%) (Swap space 0%) Service Status all green,
did some test clear the Dns Resolver log under( system logs/system/dns resolver/) these 2 lines
unbound 21493 [21493:0] notice: init module 0: python
unbound 21493 [21493:0] info: [pfBlockerNG]: pfb_unbound.py script loaded
reappear go to the dashboard the DNSBL turns yellow and gives the error
|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3' -
@smoke_a_j said in pfBlockerNG-devel v3.1.0_19/10:
DNSBL Source Definitions, Line 1: Invalid URL or Hostname not resolvable!
Either DNS isn't working on your box or something is blocking those urls.
-
@yorke I would backup you config and install a fresh copy of pfSense. Followed by a restore of the config.
-
@bbcan177 Gracias, at first I thought it was seeming similar to the inbound permit saving issue. Regardless of having most of these feeds already whitelisted, tracked it down to about 1500 some lines of regex I had came across and added a while back, most of which seemed to not be populating any alerts but invisibly blocking at random until matching the suffix/prefix portions of the code to match known alerting lines started populating the rest. I trimmed out 1300 lines to whats working, I then realized the entire 1500 lines I found were basically a reflection of the DNSBL TLD Group 1 & 2 lists. Went back to just my first 680 lines of regex and no more ghosted double filtering and running smooth
-
@smoke_a_j If you can pm or email that regex list, I can check it out to see if there is some code improvement required.
-
@yorke said in pfBlockerNG-devel v3.1.0_19/10:
pfBlockerNG-devel 3.1.0_11 |ERROR| python module 'maxminddb
Pfsense 2.6.0-RELEASE
I upgraded pfBlockerNG-devel to 3.1.0_11 and got some issue before i upgraded everyting was work but now after the upgrade I am geting the errors listed below,
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb'
2023-01-20 18:16:12,627|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'I got the same error on pfSense v2.6.0 since the upgrade to pfBlockerNG v3.1.0_11. I have cleared the error in py_error.log. Lets see if it comes back.
The report tab showns traffic being pass/block
the dashboard for DNSBL the packets stay at 0 the counter do not move, but the ip counter worksSame issue with IP Counter. It shows number of blocked IPs for a while, but when you refresh the page, counter goes to 0. This issue happens if you apply this patch via system patches package.
More details here.
If you revert this change, counter starts working as it should and it doesnt reset to 0 after some time.
-
@bbcan177 said in pfBlockerNG-devel v3.1.0_19/10:
Add "application/json" to list of allowed file download mime-types
I had hoped this might let pfBlocker directly download a JSON list like the one found at Microsoft Azure IPs. This is a file I manually download and then use pfSense's GUI CMD interface to upload for pfBlocker (I set the format to AUTO). Ran this on 3.1.0_11 just now.
It didn't work. So what JSON-related things were enabled with this change?
Thanks!
-
It happened again after after update.
This is the content of py_error.log
2023-01-24 16:36:57,206|ERROR| [pfBlockerNG]: Failed to load python module 'maxminddb': No module named 'maxminddb' 2023-01-24 16:36:57,206|ERROR| [pfBlockerNG]: Failed to load python module 'sqlite3': No module named '_sqlite3'Despite these errors, everything is working fine.
-
-
@cmcdonald
Same problem on my side.[22.05-RELEASE][admin@firewall.home]/root: pkg info py* unbound
pkg: No match. -
-
@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:
Sorry, try this:
pkg info "py*" unboundHere it is:
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: pkg info "py*" unbound py38-ply-3.11 py38-setuptools-57.0.0 py39-maxminddb-2.0.3 py39-setuptools-57.0.0 py39-sqlite3-3.9.9_7 python38-3.8.12_1 python39-3.9.9 unbound-1.13.2 -
@nimrod Thanks. I see the problem. Testing a fix. Standby
-
@nimrod can you also share
pkg info unbound? -
@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:
@nimrod can you also share
pkg info unbound?Of course. Here it is:
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: pkg info unbound unbound-1.13.2 Name : unbound Version : 1.13.2 Installed on : Mon Jan 31 21:24:27 2022 CET Origin : dns/unbound Architecture : FreeBSD:12:amd64 Prefix : /usr/local Categories : dns Licenses : BSD3CLAUSE Maintainer : jaap@NLnetLabs.nl WWW : https://www.nlnetlabs.nl/projects/unbound Comment : Validating, recursive, and caching DNS resolver Options : DEP-RSA1024 : off DNSCRYPT : off DNSTAP : off DOCS : off DOH : on ECDSA : on EVAPI : off FILTER_AAAA : off GOST : on HIREDIS : off LIBEVENT : on MUNIN_PLUGIN : off PYTHON : on SUBNET : off TFOCL : off TFOSE : off THREADS : on Shared Libs required: libexpat.so.1 libnghttp2.so.14 libpython3.8.so.1.0 libevent-2.1.so.7 Shared Libs provided: libunbound.so.8 Annotations : FreeBSD_version: 1203500 build_timestamp: 2022-01-12T15:27:10+0000 built_by : poudriere-git-3.3.99.20211130 cpe : cpe:2.3:a:nlnetlabs:unbound:1.13.2:::::freebsd12:x64 port_checkout_unclean: no port_git_hash : 8df9544dcbab ports_top_checkout_unclean: yes ports_top_git_hash: 7046b65c0d41 repo_type : binary repository : pfSense Flat size : 7.99MiB Description : Unbound is designed as a set of modular components, so that also DNSSEC (secure DNS) validation and stub-resolvers (that do not run as a server, but are linked into an application) are easily possible. Goals: * A validating recursive DNS resolver. * Code diversity in the DNS resolver monoculture. * Drop-in replacement for BIND apart from config. * DNSSEC support. * Fully RFC compliant. * High performance, even with validation enabled. * Used as: stub resolver, full caching name server, resolver library. * Elegant design of validator, resolver, cache modules. o provide the ability to pick and choose modules. * Robust. * In C, open source: The BSD license. * Smallest as possible component that does the job. * Stub-zones can be configured (local data or AS112 zones). Non-goals: * An authoritative name server. * Too many Features. WWW: https://www.nlnetlabs.nl/projects/unbound -
@nimrod Can you now try reinstalling pfBlockerNG-devel on 22.05/2.6, and repeat the above command
pkg info "py*" unbound -
@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:
@nimrod Can you now try reinstalling pfBlockerNG-devel on 22.05/2.6, and repeat the above command
pkg info "py*" unboundI reinstalled it and here is the output:
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: pkg info "py*" unbound py38-maxminddb-2.0.3 py38-ply-3.11 py38-setuptools-57.0.0 py38-sqlite3-3.8.12_7 py39-maxminddb-2.0.3 py39-setuptools-57.0.0 py39-sqlite3-3.9.9_7 python38-3.8.12_1 python39-3.9.9 unbound-1.13.2 -
@nimrod That should be correct now. Clear the unbound errors and try again.
-
@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:
@nimrod That should be correct now. Clear the unbound errors and try again.
Yup. That fixed it. Thank you sir.
-
@draco said in pfBlockerNG-devel v3.1.0_19/10:
I had hoped this might let pfBlocker directly download a JSON list like the one found at Microsoft Azure IPs. This is a file I manually download and then use pfSense's GUI CMD interface to upload for pfBlocker (I set the format to AUTO). Ran this on 3.1.0_11 just now.
The Link you posted is the HTML page. You need to use the direct link:
https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20230123.json
Keep in mind that this will parse all IPs in the json file. You could also create a new shell script to parse this JSON and get more refinement on which IPs to pull ( "Advanced Tunables - Post-Script Script" feature.)
-
I figure out why i was getting those errors some package/feature on pfsense needed to be update (ie unbound and about 4 others ) once I ran the update and reboot and reinstall
PfblockerNG work, no more errors.
Thanks BBcan177 -
@cmcdonald I am seeing the same error about missing python modules on 23.01 RC, was this fixed on that version as well?
-
@bigjohns97 said in pfBlockerNG-devel v3.1.0_19/10:
@cmcdonald I am seeing the same error about missing python modules on 23.01 RC, was this fixed on that version as well?
Yes.
-
@bbcan177 said in pfBlockerNG-devel v3.1.0_19/10:
he Link you posted is the HTML page. You need to use the direct link:
https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20230123.jsonFair enough -- this means I will need to manually update the link each time, but better than copying the file from my computer up to pfSense each time, thanks!
I might have to write a screen-scraper to pull the latest URL off the download page...
-
@nimrod Can you confirm what add-on's I should see because they differ than what is posted above.
pkg info "py*" unbound
py311-maxminddb-2.2.0_2
py311-setuptools-63.1.0
py311-sqlite3-3.11.1_8
py39-libzfs-1.1.2022081600
py39-setuptools-63.1.0
py39-yaml-5.4.1
python311-3.11.1_1
python39-3.9.15
unbound-1.17.0 -
Is this before or after pfblocker reinstall ?
-
@nimrod After
-
I just noticed you are on Plus version of pfsense. The output that i shared is from CE edition.
-
@nimrod That wouldn't matter, the difference between 2.6/22.x and 2.7/23.x is really what I am trying to confirm was fixed.
@BBcan177 builds the pfblockerng code but I believe netgate dev's such as @cmcdonald are who associate package prerequisites and manage how the actual package is presenting in package manager.
This is why my original question was to @cmcdonald as to whether his fix he did in this thread was also applied to the new 2.7/23.x branch.
-
-
@cmcdonald said in pfBlockerNG-devel v3.1.0_19/10:
Report the output of
pkg info unboundunbound-1.17.0
Name : unbound
Version : 1.17.0
Installed on : Sat Jan 14 12:37:18 2023 CST
Origin : dns/unbound
Architecture : FreeBSD:14:amd64
Prefix : /usr/local
Categories : dns
Licenses : BSD3CLAUSE
Maintainer : jaap@NLnetLabs.nl
WWW : https://www.nlnetlabs.nl/projects/unbound
Comment : Validating, recursive, and caching DNS resolver
Options :
DEP-RSA1024 : off
DNSCRYPT : on
DNSTAP : off
DOCS : off
DOH : on
ECDSA : on
EVAPI : off
FILTER_AAAA : off
GOST : on
HIREDIS : off
LIBEVENT : on
MUNIN_PLUGIN : off
PYTHON : on
SUBNET : off
TFOCL : off
TFOSE : off
THREADS : on
Shared Libs required:
libsodium.so.23
libpython3.9.so.1.0
libnghttp2.so.14
libexpat.so.1
libevent-2.1.so.7
Shared Libs provided:
libunbound.so.8
Annotations :
FreeBSD_version: 1400073
build_timestamp: 2022-10-27T06:51:33+0000
built_by : poudriere-git-3.3.99.20220831
cpe : cpe:2.3:a:nlnetlabs:unbound:1.17.0:::::freebsd14:x64
port_checkout_unclean: no
port_git_hash : 7b7b452fb8d5
ports_top_checkout_unclean: yes
ports_top_git_hash: 0c964f08a5cb
repo_type : binary
repository : pfSense
Flat size : 8.36MiB
Description :
Unbound is designed as a set of modular components, so that also
DNSSEC (secure DNS) validation and stub-resolvers (that do not run as
a server, but are linked into an application) are easily possible.Goals:
* A validating recursive DNS resolver.
* Code diversity in the DNS resolver monoculture.
* Drop-in replacement for BIND apart from config.
* DNSSEC support.
* Fully RFC compliant.
* High performance, even with validation enabled.
* Used as: stub resolver, full caching name server, resolver library.
* Elegant design of validator, resolver, cache modules.
o provide the ability to pick and choose modules.
* Robust.
* In C, open source: The BSD license.
* Smallest as possible component that does the job.
* Stub-zones can be configured (local data or AS112 zones).Non-goals:
* An authoritative name server.
* Too many Features.WWW: https://www.nlnetlabs.nl/projects/unbound
ldd `which unbound`/usr/local/sbin/unbound:
libssl.so.111 => /usr/lib/libssl.so.111 (0x822469000)
libsodium.so.23 => /usr/local/lib/libsodium.so.23 (0x8236ec000)
libutil.so.9 => /lib/libutil.so.9 (0x822a37000)
libevent-2.1.so.7 => /usr/local/lib/libevent-2.1.so.7 (0x823fcb000)
libpython3.9.so.1.0 => /usr/local/lib/libpython3.9.so.1.0 (0x824b25000)
libcrypto.so.111 => /lib/libcrypto.so.111 (0x8259f7000)
libnghttp2.so.14 => /usr/local/lib/libnghttp2.so.14 (0x82790a000)
libthr.so.3 => /lib/libthr.so.3 (0x825eff000)
libc.so.7 => /lib/libc.so.7 (0x826edd000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x8284bd000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x829b94000)
libdl.so.1 => /usr/lib/libdl.so.1 (0x828694000)
libm.so.5 => /lib/libm.so.5 (0x828758000)
[vdso] (0x8215a5000)pkg info "py*"py311-maxminddb-2.2.0_2
py311-setuptools-63.1.0
py311-sqlite3-3.11.1_8
py39-libzfs-1.1.2022081600
py39-setuptools-63.1.0
py39-yaml-5.4.1
python311-3.11.1_1
python39-3.9.15 -
@bigjohns97 and this is on 23.01?
-
@cmcdonald Correct, dashboard shows 23.01 RC
Current Base System23.01.r.20230202.1645
Latest Base System23.01.r.20230202.1645
StatusUp to date. -
@bigjohns97 That is very odd.
The problem is you are running older Unbound which is using Python 3.9 and not 3.11
unbound-1.17.1_2 py311-libzfs-1.1.2022081600 py311-maxminddb-2.2.0_2 py311-setuptools-63.1.0 py311-sqlite3-3.11.1_8 py39-libzfs-1.1.2022081600 py39-maxminddb-2.2.0_1 py39-setuptools-63.1.0 python311-3.11.1_1 python39-3.9.16These are the versions that we ship with 23.01-RC
I would try reinstalling unbound:
pkg install -fy unbound -
@cmcdonald That's odd, I also seem to be missing that 311 libzfs which I am using zfs and boot environments.
How would I go about getting these correct packages?
Edit: that worked, I now show the following.
py311-maxminddb-2.2.0_2
py311-setuptools-63.1.0
py311-sqlite3-3.11.1_8
py39-libzfs-1.1.2022081600
py39-setuptools-63.1.0
py39-yaml-5.4.1
python311-3.11.1_1
python39-3.9.15
unbound-1.17.1_2 -
@bigjohns97 what if you just do
pkg upgradewhat does it offer to upgrade? -
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (4 candidates): 100%
Processing candidates (4 candidates): 100%
The following 5 package(s) will be affected (of 0 checked):New packages to be INSTALLED:
whois: 5.5.7 [pfSense]Installed packages to be UPGRADED:
pfSense: 23.01.b.20230106.0600 -> 23.01.r.20230202.1645 [pfSense]
pfSense-Status_Monitoring: 1.7.11_4 -> 1.8 [pfSense]
pfSense-repo: 23.01.b.20230106.0600 -> 23.01.r.20230202.1645 [pfSense]
python39: 3.9.15 -> 3.9.16 [pfSense]Number of packages to be installed: 1
Number of packages to be upgraded: 471 KiB to be downloaded.
Proceed with this action? [y/N]:
-
What repo is set on the update GUI page?
-
