V 3.2.0 with pfsense 23.01 RC 20230202

pfsjap

@greenflash Ok, hopefully you'll get help with this.

cmcdonald

@greenflash I worked on Unbound quite a bit over the past month tracking down memory-related issues.

What does your DNSBL setup look like?

greenflash

@cmcdonald said in V 3.2.0 with pfsense 23.01 RC 20230202:

What does your DNSBL setup look like?

Do you mean this settings page?

spoiler

Or this one:

spoiler

emikaadeo

@tcw said in V 3.2.0 with pfsense 23.01 RC 20230202:

No change. Confirmed the patch applied. Updated to 23.01.r.20230202.1645 from 23.01.r.20230202.0019 yesterday and confirmed successful pfBlockerNG force reload all, before and after the update, and before and after applying the patch, with success as long as Wildcard Blocking (TLD) is unselected.

The "TLD finalize.." step seemed to take just a couple of seconds on 22.05 with my hardware, so I don't believe it's an issue of my not waiting long enough (especially now since the patch seems to have corrected a typo to enforce timeout in 15 seconds).

Let me know how else I may be able to help.

Finally got time to upgrade to 23.01-RC and can confirm that with Wildcard Blocking (TLD) feature enabled the update/reload process hangs on "TLD finalize..."
There's a Redmine ticket for this issue: https://redmine.pfsense.org/issues/13884

tcw

@jimp's patch just got applied to an updated pfBlockerNG v. 3.2.0_1. (Thanks!) That appears to have been the only change.

Gertjan

Using the latest pfSense RC :

23.01-RC (amd64)
built on Wed Feb 08 06:11:39 UTC 2023

TLD Whitelist selected.
I'm here :

UPDATE PROCESS START [ v3.2.0_1 ] [ 02/8/23 11:13:01 ]

===[  DNSBL Process  ]================================================

Loading DNSBL Statistics... completed
Missing DNSBL stats and/or Unbound DNSBL files - Rebuilding

Loading DNSBL SafeSearch...  enabled
Loading DNSBL Whitelist... completed
Blacklist database(s) ... exists.

[ StevenBlack_ADs ]		 Downloading update .. 200 OK.
 Whitelist: 15.taboola.com|aax-eu.amazon-adsystem.com|adsafeprotected.com|am-match.taboola.com| ..... snipped
 Orig.    Unique     # Dups     # White    # TOP1M    Final                
 ----------------------------------------------------------------------
 177888   177888     0          97         0          177791               
 ----------------------------------------------------------------------

------------------------------------------------------------------------
Assembling DNSBL database...... completed [ 02/8/23 11:13:13 ]
TLD:
TLD analysis.. completed [ 02/8/23 11:13:17 ]
TLD finalize..

and I understand why :

The /tmp/dnsbl_tld_remove file - the list with TLDs to remove is 37000+ lines.
The /var/unbound/pfb_py_data.txt.raw file 133608 lines

[edit]
From what I make of this : each of the 37000+ lines is checked (grepped) with every line in the 133608 file.
So, 37000 times 133608 'greps' to be executed.
That's huge ....

And I have only one dnsbl feed - with "133608" dnsbl entries.
[end edit]

I copied both files to /root/ and repeated the command 'on the command line'.
This command is great to max out one core, 100 %, and it will take minutes if not hours to complete.

pfblockerng-devel does this with PHP handling the return (output). That will make things even worse.

49 degrees and rising. Of to the kitchen, looking for some eggs.

I guess not using (unchecking) Wildcard Blocking (TLD) is the best option right now.

emikaadeo

@gertjan said in V 3.2.0 with pfsense 23.01 RC 20230202:

I guess not using TLD Whitelisting is the best option right now.

I'm not using TLD Whitelist
My DNSBL Mode is set to "Unbound python mode" and as pfBlockerNG states: "TLD Whitelist is not utilized for Unbound python mode! Use DNSBL Whitelist instead."
The main problem is when Wildcard Blocking (TLD) is enabled.

Gertjan

@emikaadeo
You're right :

That's the one :

petrt3522

I believe mine is now having the same issue with my upgrade to 23.01-Final. I manually did a reload and it's at 20 minutes, stuck on "TLD finalize."

I did have an error: On it's first boot I got a banner about this extensive error: https://pastebin.com/aj8q4Mjw than that, It seems to work fine and appears to be passing traffic across 2 VLAN and 1 WAN.

4NVXr3wHBnQYsHwE

This happened to me today as well and likewise disabling Wildcard Blocking (TLD) worked around it. grep was stuck at 100% CPU utilization for several minutes otherwise.

OpIT GmbH

Andy Fix for this? Except disabling Wildcard TLD blocking

SteveITS

@opit-gmbh said in V 3.2.0 with pfsense 23.01 RC 20230202:

Andy Fix for this? Except disabling Wildcard TLD blocking

Not yet: https://www.patreon.com/posts/pfblockerng-v3-2-78781333

emikaadeo

@steveits @jmontleon @OpIT-GmbH
It is now fixed with 3.2.0_3 version :)
https://forum.netgate.com/post/1088962

OpIT GmbH

Yes, working now. THX