abnormal behavior after upgrade pkg
-
With pfBlockerNG I usually recommend disabling it first from the general tab. Then run the package update and/or first un-install the package (settings will be saved if you have this option enabled) and then re-install pfBlockerNG from fresh install to make sure all other needed package dependencies are installed at their recommended versions instead of being held back on outdated versions from being currently in-use by the system if its still loaded. (in my case, this is the point I edit my pfblockerng.inc to set memory_limit large enough for my use-case of 11.5million in DNSBL to eliminate PHP memory errors at Update/CRON/Reload events parsing a large list. I also up my max domain count numbers to allow this size of list to process) Then re-enable pfBlockerNG in General tab again after re-install/update and wait a few moments for all modules to initialize watching my CPU usage until it returns to idle and out-of-sync triangle displayed, and then run the Force>Reload>All from update tab. I run into similar app update hiccups at my work with their own home-brewn Android app and other complex apps on most any other OS's as well Windows or Linux alike, fresh install is best to eliminate any left-over over-written/amended code. Seeing that your issue is right when TLD is finalizing, you may want to look at editing "/usr/local/pkg/pfblockerng/pfblockerng.inc" and search for two lines that start with
$pfb['pfs_mem'] = arrayits staggered into 1000mb increments, however much physical RAM you have, edit the next number to be higher than the total number of domains in your list. I just add an extra 0 to everything above 7000 since I had 8gb ram, then 16 and now at 32gb. The edit BBcan177 had added a while back I think was just an added field of 32000 for 32gb boxes that wasn't there prior, not certain they'll raise those default "max domain count" numbers in the base configs too too much to avoid excess memory exhaustion issues on lower mem/arm devices and with not knowing each use-case of other application memory demands varying between setups especially if when zero SWAP space is allocated to be available. Also, after editing the pfblockerng.inc file, run the following command to let them apply:
php /usr/local/www/pfblockerng/pfblockerng.php dcSince upgrading to 23.05, I have noticed Unbound stop responding at random shortly after reboots, tracked down on mine mostly to now-unneeded/conflicting custom options set in my DNS Resolver settings that seemed to be working otherwise on prior versions of Unbound/pfSense, running much smoother once these were removed:
outgoing-range: msg-buffer-size: neg-cache-size: key-cache-size: -
@smolka_J Thanks
Could you please explain a bit in details. I have 48Gb of ram but swap is 4Gb only . Blow are details from main Dashboard for Pfblockerng
Alias Count ============== ======== pfB_DNSBLIP_v4 22,253 pfB_PRI1_v4 15,048 pfB_PRI2_v4 594 pfB_PRI3_v4 22 pfB_Proxy_IP_v4 397 pfB_TOR_v4 8,969 pfB_Whitelist_v4 4 DNSBL_UT1 4,664,293 DNSBL_Pi_Hole_list 663,544 DNSBL_DoH 123 DNSBL_TLD 148Total count of Pfblocker = 5328105
Now tell me what values do I have to modify that it will improve long list to parse with out any issue.
Regards
-
I have upgraded to pkg 2.6.0 but unable to update update pkg pflockerng i can see only 3.2.0_4 :( .
-
@scorpoin 48gb ram you "should" be within usable max domain count values unless you happen to notice any messages in your logs similar to what I had noting "TLD Domain count exceeded. [ xx00000 ]" You had noted you had "updated" the package....what is your output of commands:
pkg info "py*"and
pkg info unbound -
@smolka_J said in abnormal behavior after upgrade pkg:
xx00000
I dont see any this kind of message in log or any where I had that in past so I delete some of my list to get rid of it.
PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ login.msa.msidentity.com. ] PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ ph0mgt0101dc002.prdmgt01.prod.exchangelabs.com. msnhst.microsoft.com. 2019445400 300 120 2419200 60 ] PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ client-s.gateway.messenger.geo.msnmessenger.msn.com.akadns.net. ] PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ outlook.office365.com. ] PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ av1.nstld.com. mdnshelp.verisign.com. 1685233134 300 7200 1209600 86400 ] PFB_FILTER - 9 | tld_analysis [ 05/28/23 15:45:19 ] Failed validation [ ns1-39.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300 ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 15:48:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 15:50:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 15:51:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 15:53:15 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 15:55:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:06:51 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:10:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:15:49 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:18:20 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:31:20 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:31:39 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:32:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:34:20 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:34:37 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:47:48 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:51:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:51:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:51:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:51:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:53:58 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:55:24 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:55:39 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:56:27 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:16:07 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:28:35 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:34:30 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:38:37 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:49:01 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:50:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:51:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:53:59 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 17:56:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:07:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:11:35 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:16:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:18:47 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:31:47 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:32:13 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:33:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:35:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:35:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:48:42 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:51:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:52:01 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:52:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:52:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:54:12 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:55:56 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:56:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 18:56:43 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:16:34 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:29:14 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:35:01 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:39:10 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:49:32 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:50:56 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:52:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:54:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 19:56:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:07:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:12:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:16:44 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:18:51 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:32:44 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:33:10 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:33:51 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:35:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:35:43 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:49:09 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:52:12 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:52:12 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:52:32 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:53:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:54:53 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:56:35 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:56:49 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 20:56:56 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:17:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:29:23 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:35:41 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:39:20 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:50:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:51:10 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:52:30 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:54:56 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 21:57:31 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:08:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:12:50 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:17:37 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:19:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:33:14 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:33:55 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:34:51 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:36:01 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:36:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:49:19 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:52:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:52:21 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:53:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:54:11 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:55:39 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:56:53 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:57:35 ] Failed validation [ - ] PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 22:57:40 ] Failed validation [ - ]Currently seeing this in error log of pgblockerng.
output of pkg info unound
pkg info unbound unbound-1.13.2 Name : unbound Version : 1.13.2 Installed on : Sun May 28 12:52:40 2023 PKT Origin : dns/unbound Architecture : FreeBSD:12:amd64 Prefix : /usr/local Categories : dns Licenses : BSD3CLAUSE Maintainer : jaap@NLnetLabs.nl WWW : https://www.nlnetlabs.nl/projects/unbound Comment : Validating, recursive, and caching DNS resolver Options : DEP-RSA1024 : off DNSCRYPT : off DNSTAP : off DOCS : off DOH : on ECDSA : on EVAPI : off FILTER_AAAA : off GOST : on HIREDIS : off LIBEVENT : on MUNIN_PLUGIN : off PYTHON : on SUBNET : off TFOCL : off TFOSE : off THREADS : on Shared Libs required: libexpat.so.1 libnghttp2.so.14 libpython3.8.so.1.0 libevent-2.1.so.7 Shared Libs provided: libunbound.so.8 Annotations : FreeBSD_version: 1203500 build_timestamp: 2023-01-24T16:26:21+0000 built_by : poudriere-git-3.3.99.20220831 cpe : cpe:2.3:a:nlnetlabs:unbound:1.13.2:::::freebsd12:x64 port_checkout_unclean: no port_git_hash : 8df9544dcbab ports_top_checkout_unclean: yes ports_top_git_hash: 3f51c1f85e63 repo_type : binary repository : pfSense Flat size : 7.99MiB Description : Unbound is designed as a set of modular components, so that also DNSSEC (secure DNS) validation and stub-resolvers (that do not run as a server, but are linked into an application) are easily possible. Goals: * A validating recursive DNS resolver. * Code diversity in the DNS resolver monoculture. * Drop-in replacement for BIND apart from config. * DNSSEC support. * Fully RFC compliant. * High performance, even with validation enabled. * Used as: stub resolver, full caching name server, resolver library. * Elegant design of validator, resolver, cache modules. o provide the ability to pick and choose modules. * Robust. * In C, open source: The BSD license. * Smallest as possible component that does the job. * Stub-zones can be configured (local data or AS112 zones). Non-goals: * An authoritative name server. * Too many Features. -
@scorpoin said in abnormal behavior after upgrade pkg:
Could you please explain a bit in details. I have 48Gb of ram but swap is 4Gb only .
In normally cases and hardware you have nothing
to do! (In my opinion only) It is because you
have a sufficient amount of RAM installed and
you may be sorted! In some, rarely or especially
cases let us call it, you may be then on top the
lucky guy that is able to tune, or sort or plain
serve that the entire system will be more smooth
& liquid running that is all.- ZFS ARC problem
Not running out of space - ZFS copies problem
- boot environment space for copies
easy going back to a stable system - mbuf size and amount
tunable for nics - queues amount, size and length
Much CPU cores and threads - state table size amount
Servers in the DMZ - RAM disk for caching
(Squid-SquidGuard-ClamAV)
You will be more able to serve, speed up or
enrich things, services and so on and so on,
without looking on the RAM amount!@scorpoin said in abnormal behavior after upgrade pkg:
I have upgraded to pkg 2.6.0 but unable to update update pkg pflockerng i can see only 3.2.0_4 :( .
Ah, ok this is may be then only available on the
last versions such 23.05 Release and 2.7 Devel. - ZFS ARC problem
-
@Dobby_ Thanks Dobby,
My only concern is to speed up update process when ever it is run for pfblockerng . 3+ hours is not normal behavior.
-
@scorpoin If you have the yellow triangle stating python is out of sync, the lengthier Force>Reload>All 5-10 minutes after a clean re-boot of the device if its still hung is the first option to get it back towards a more speedy "Update" to be able to complete later after. Force>Reload will load each individual list that is already downloaded, not updating them, and the time-consuming step then at that point is to run a de-duplication task comparing each line in each blacklist one-by-one to remove all duplicate entries. Once all lists are in "sync" with each other, later on when you run an "Update" task, that large step for all lists in place is already complete so Update will spend less time then only updating individual blacklists that then have an update. If that de-duplication process is taking too long, you'll want to inspect your update logs for any blacklist feeds you have that show "Final" counts after duplicates were removed stating "0" these feeds are already part of another list you have loaded, disable or remove the extra duplicate blacklists and you'll chop down Reload and Update times. Force>Reload>All can take hours with too many duplicate lists if that yellow out of sync is still present. If the yellow out-of-sync triangle is present and you try running Force>Update>All before having run a Force> Reload>All until it completes, the yellow triangle will not go away and not certain if "Update" can even complete without stating it failed. On long reloads/updates, the Update log viewer sometimes stalls out on a line and I have to toggle the view button a time or two to see where the update actually is in progress. I don't recommend having RAM disk options enabled if by chance for some reason you do, doing so will most of the time entail the NEED to run a Force>Reload>All each and every single reboot. Also good to schedule CRON update task to run overnight during downtime. I'm partially more interested if you find anything more towards the
PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 05/28/23 16:06:51 ] Failed validation [ - ]I've been chasing this same error log message on my box for a while now but doesn't seem to be affecting blocking or performance. Found solutions to similar logs that point to more of a specific feed or incomplete domain name in a domain name blacklist but haven't found anything specific in this ones dialogue
-
@scorpoin said in abnormal behavior after upgrade pkg:
@Dobby_ Thanks Dobby,
My only concern is to speed up update process when ever it is run for pfblockerng . 3+ hours is not normal behaviour.
to small disk space?
to small /tmp folder?
cpu is not strong enough?
a turning hdd is to slow?
your ids is blocking that feed? -
Well all I did for now removed all TLD entries and added it into DNSBL whitelist and DNSBL custom list to block for now. It does not take much time as it was in previous. Yellow triangle is gone as well.
