pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!

A Former User

@BBcan177

with v3.0.0 load on my Firewall is significant reduced when running large Set of DNSBL Lists (~3 Million Domains).

I don't see the unbound-control process spinning up every couple of seconds anymore.

Well Done

fmroeira86

@BBcan177

Thank you for your effort!

Everytime I do a RELOAD the skew on the backup CARP gets its value set to 0.

This causes the backup machine holding as CARP master when the Master machine comes online again.

I'm running 3.0.0_2

BBcan177

@fmroeira86 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

Thank you for your effort!
Everytime I do a RELOAD the skew on the backup CARP gets its value set to 0.
This causes the backup machine holding as CARP master when the Master machine comes online again.
I'm running 3.0.0_2

Did you update the package on all Nodes?
Did you set the SKEW on all nodes correctly?

The package should not be syncing the SKEW to the other nodes.

See the relevant code here:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L9349-L9352

provels

@bbcan177
I got an alert today that an update from 3.0.0_2 => 3.0.0_3 was available. I had already updated to 3.0.0_3.
But my package manager now shows 3.0.0_2 as current and wants me to revert.
Any problem here? Thanks for reading.

BBcan177

@provels
No Idea. That is outside of the pkg as its part of the pfSense package manager.

asmodeus

@bbcan177 After upgrading to 3.0.0_3 whitelisting domains from the list of blocked domains (e.g. /pfblockerng/pfblockerng_alerts.php?filterdnsbl=DNSBL_Malicious2) no longer worked - getting 504 Gateway Time-out errors after selecting the domain to whitelist and providing the options (wildcard or not etc.)

The entire UI is unresponsive after that and I can get it back up again via ssh with /etc/rc.php-fpm_restart. I have rolled back to 3.0.0_2 but I still have this issue.

Any ideas on what the issue might be?

provels

@bbcan177
Must have been a personal problem. Rebooted the FW, GUI still offered upgrade to _2 when _3 was already installed. Did pkg upgrade from shell, offered _2 to _3. Went ahead and installed, seems fine now. Go figure.

BBcan177

@asmodeus said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

@bbcan177 After upgrading to 3.0.0_3 whitelisting domains from the list of blocked domains (e.g. /pfblockerng/pfblockerng_alerts.php?filterdnsbl=DNSBL_Malicious2) no longer worked - getting 504 Gateway Time-out errors after selecting the domain to whitelist and providing the options (wildcard or not etc.)
The entire UI is unresponsive after that and I can get it back up again via ssh with /etc/rc.php-fpm_restart. I have rolled back to 3.0.0_2 but I still have this issue.
Any ideas on what the issue might be?

There are two different things here.

Whitelisting
Filtering the Alerts tab.

The URL that you posted is for Filtering events, not whitelisting? Write out the steps you took so I can try to reproduce.
It could just be a browser timeout issue if you are trying to view a lot of events? In Alert Settings, what did you set the DNSBL Events number to?

fmroeira86

@bbcan177 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

@fmroeira86 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

Thank you for your effort!
Everytime I do a RELOAD the skew on the backup CARP gets its value set to 0.
This causes the backup machine holding as CARP master when the Master machine comes online again.
I'm running 3.0.0_2

Did you update the package on all Nodes?
Did you set the SKEW on all nodes correctly?

The package should not be syncing the SKEW to the other nodes.

See the relevant code here:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L9349-L9352

Yes.

I'm running the latest version and My main node ovewrites the skew to the secondary one.

BBcan177

@fmroeira86

Ok I see the issue.

As reference:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L9350-L9351

Can you edit the file: /usr/local/pkg/pfblockerng/pfblockerng.inc, Lines 9350 and 9351

From:

// Remove CARP Advskew setting, if found
if (isset($xml['pfblockerngdnsblsettings']['pfb_dnsvip_skew'])) {
    unset($xml['pfblockerngdnsblsettings']['pfb_dnsvip_skew']);
}

To:

// Remove CARP Advskew setting, if found
if (isset($xml['pfblockerngdnsblsettings' ]['config'][0]['pfb_dnsvip_skew'])) {
   unset($xml['pfblockerngdnsblsettings']['config'][0]['pfb_dnsvip_skew']);
}

I missed adding ['config'][0] to each of those two lines... Sorry about that...

asmodeus

@bbcan177 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

The URL that you posted is for Filtering events, not whitelisting? Write out the steps you took so I can try to reproduce.

Sure:

Coming from the pfBlockerNG dashboard on my Status/Dashboard page, I use the alerts link to review blocked domains in a list. The link points to /pfblockerng/pfblockerng_alerts.php?filterdnsbl=DNSBL_Malicious2
Using this list (which takes a while to load) I want to whitelist a domain, so I click on the 'plus' sign to start the process.
I specify wildcard no, add description no.
Page loads for a while and ultimately I am getting the nginx timeout. (Even when this worked it takes 110 seconds or more to complete)
The page I attempted to whitelist is NOT whitelisted.
The entire pfsense UI is unresponsive at this point and I resolve by restarting php-fpm via ssh

The Log Settings are set to default (20000 lines), and the Alert Settings are set to default:1000 Stat Table entries and 100 Filter Limit. Are these the Alert Settings you were asking about?

BBcan177

@asmodeus said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

and the Alert Settings are set to default:1000 Stat Table entries and 100 Filter Limit. Are these the Alert Settings you were asking about?

From the Reports/Alerts tab, click the "+" on the right side of the page to open the "Alert Settings", the default for "Filter Limit Entries" is 100.

If you set it too high, you need to wait for the whole page to finish loading before trying to Whitelist.

There is also a new option in the Reports tab to "Pause" the auto page refresh. Its at the left side of each table heading.

asmodeus

@bbcan177 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

From the Reports/Alerts tab, click the "+" on the right side of the page to open the "Alert Settings", the default for "Filter Limit Entries" is 100.

Got it, that's what Filter Limit Entres is set to for me - 100. Is there a chance to speed up loading of this page before/after whitelisting?

fmroeira86

@bbcan177 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

@fmroeira86

Ok I see the issue.

As reference:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L9350-L9351

Can you edit the file: /usr/local/pkg/pfblockerng/pfblockerng.inc, Lines 9350 and 9351

From:
// Remove CARP Advskew setting, if found
if (isset($xml['pfblockerngdnsblsettings']['pfb_dnsvip_skew'])) {
    unset($xml['pfblockerngdnsblsettings']['pfb_dnsvip_skew']);
}
To:
// Remove CARP Advskew setting, if found
if (isset($xml['pfblockerngdnsblsettings' ]['config'][0]['pfb_dnsvip_skew'])) {
   unset($xml['pfblockerngdnsblsettings']['config'][0]['pfb_dnsvip_skew']);
}
I missed adding ['config'][0] to each of those two lines... Sorry about that...

Thank you.

Is this fix going to be released in the next version?

BBcan177

@fmroeira86 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

Thank you.
Is this fix going to be released in the next version?

Once you confirm that it works :)

fmroeira86

@fmroeira86 said in pfBlockerNG-devel v3.0.0 - No longer bound by Unbound!:

if (isset($xml['pfblockerngdnsblsettings' ]['config'][0]['pfb_dnsvip_skew'])) {
unset($xml['pfblockerngdnsblsettings']['config'][0]['pfb_dnsvip_skew']);

Made the changes you suggested but it is not working.
Skew still gets to 0 on the secondary machine.

Bob.Dig

This post is deleted!

mcury

Since the update to 3.0, cpu usage went down considerable, also the sg-3100 temps.
Really nice work..

axellarsson

Hi BBcan177,

Thanks for the 3.0 release!

I'm currently not using the Python module since I have unbound views set up to only include the DNSBL configuration for certain subnets. However, I also have a custom Python module loaded to filter out AAAA responses from a number of Netflix domains and subdomains (works around an issue with IPv6 tunnelbrokers being blocked by Netflix as proxies).

Presently, when pfBlockerNG-devel is set to Unbound mode it resets my Unbound Python module settings every hour when it refreshes. Is there any way to get it to coexist with my own Python module and remain in Unbound mode? (I understand that this will preclude me from using the pfBlockerNG Python module in the future since pfSense's unbound configuration only supports one Python module, but for the moment I'd rather live with that limitation and use pfBlockerNG in Unbound mode while maintaining my current custom Python configuration).

Thanks!

Gertjan

@axellarsson

This is what you can do : edit : /usr/local/pkg/pfblockerng/pfblockerng.inc : lines 2025 and up :

		// Remove python settings from DNS Resolver configuration
		if (isset($config['unbound']['python'])) {
//			unset($config['unbound']['python']);
//			$config['unbound']['python_order']	= '';
//			$config['unbound']['python_script']	= '';

			$log = 'KEEPING DNSBL Unbound admin added python script';

Add in front of lines 2027 - 2028 - 2029 the "//"
Change line 2031 so it shows a text in the update log what it is doing.

So, 4 lines need minor editing. The last one is even not strictly needed.

Now your unbound python settings like :

persists when pfBlockerNG 'does it things'.

Your own no-aaaa script gets loaded by unbound as before :

This line shows in the log because I modified the original no-aaaa script to show this line at init/start of the script.

When "views" gets integrated into pfBlockerNG, you should be able to switch to the new Python mode, uses views and use the build in 'no-aaaa' facility.

See this as a temporary patch that gives you the possibility to use your own no-aaaa script while you use pfBlockerNG in unbound mode != python mode.
Right now, true, if pfBlockerNG is put in unbound mode, it removes / deactivates unbound own python settings.