pfBlocker not logging after 2.5.2 pfSense upgrade
-
@cefleet When you hover the cursor over the DNSBL / IP numbers, what is the Clear date? Maybe you can clear the counters using the Widget Garbage Icon ?
-
I have this same issue. When on 2.5.1 dashboard was working fine. Showed thousands of requests and counters would keep incrementing ever second due to smart devices.
After upgrading to 2.5.2 dashboard was all 0. I reinstalled PfBlockerNG 3.0.0_16. But that didn't change anything. Looking in dbsbl.log after the upgrade it was all old stuff, nothing new. I did a force reload and didn't change anything. I verified it was in fact blocking ads but just nothing showing up in the logs and therefore not the dashboard or reports.
What is strange is this morning there are now a few things in log and dashboard shows 99 things blocked. But even now, logs have stuff from late last night, nothing from today. So not sure what broke w/ the 2.5.2 update.
-
@bs09 Exactly the same thing happened to me and I tried much the same things as you ... I continue to investigate for a possible solution ...
-
@ronpfs after letting it run for 12hrs. the widget count is 0 for blocked packets (but confirm ads are being blocked). here is the dnsbl log that only shows a handful from yesterday.
-
@dpseattle Maybe the .sqlite files have the wrong ownership ?
ls -al /var/unbound/ total 42831 drwxr-xr-x 7 unbound unbound 39 Jul 9 12:26 . drwxr-xr-x 27 root wheel 27 Jun 2 2020 .. -rw-r--r-- 1 root unbound 176 Jul 5 04:24 access_lists.conf drwxr-xr-x 2 unbound unbound 2 Jun 2 2020 conf.d dr-xr-xr-x 8 root wheel 512 Jul 5 08:20 dev -rw-r--r-- 1 root unbound 0 Jul 5 04:24 dhcpleases_entries.conf -rw-r--r-- 1 root unbound 3371 May 1 00:18 dnsbl_cert.pem -rw-r--r-- 1 root unbound 0 Jul 5 04:24 domainoverrides.conf -rw-r--r-- 1 root unbound 3816 Jul 5 04:24 host_entries.conf drwxr-xr-x 4 root wheel 58 Oct 2 2020 lib -rw-r--r-- 1 root unbound 1697 Mar 22 22:01 pfb_dnsbl_lighty.conf -rw-r--r-- 1 root unbound 0 Jan 8 11:52 pfb_py_cache.dnsbl -rw-r--r-- 1 unbound unbound 8192 Jul 9 12:13 pfb_py_cache.sqlite -rw-r--r-- 1 root unbound 7 Jul 9 08:20 pfb_py_count -rw-r--r-- 1 root unbound 13071812 Jul 9 08:20 pfb_py_data.txt -rw-r--r-- 1 unbound unbound 8192 Jul 9 12:20 pfb_py_dnsbl.sqlite -rwxr-xr-x 1 root wheel 1687428 Jun 28 2020 pfb_py_hsts.txt -rw-r--r-- 1 root unbound 1687428 Jun 28 2020 pfb_py_hsts.txt.pkgsave -rw-r--r-- 1 root unbound 0 Jan 8 11:52 pfb_py_resolver.dnsbl -rw-r--r-- 1 unbound unbound 16384 Jul 9 12:26 pfb_py_resolver.sqlite -rw-r--r-- 1 root unbound 3475 Apr 18 01:16 pfb_py_ss.txt -rw-r--r-- 1 root unbound 2793 Mar 2 2019 pfb_py_whitelist.json -rw-r--r-- 1 root unbound 2750 Mar 22 22:01 pfb_py_whitelist.txt -rw-r--r-- 1 root wheel 52420053 Jul 9 08:20 pfb_py_zone.txt -rw-r--r-- 1 root unbound 782 Feb 28 20:19 pfb_unbound.ini -rwxr-xr-x 1 root wheel 66726 Apr 7 12:46 pfb_unbound.py -rw-r--r-- 1 root unbound 43906 Nov 1 2020 pfb_unbound.py.pkgsave -rwxr-xr-x 1 root wheel 7077 Mar 6 11:44 pfb_unbound_include.inc -rw-r--r-- 1 root unbound 5454 Nov 1 2020 pfb_unbound_include.inc.pkgsave -rw-r--r-- 1 root unbound 300 Dec 8 2018 remotecontrol.conf -rw-r--r-- 1 unbound unbound 758 Jul 9 08:20 root.key -rw-r--r-- 1 unbound unbound 2141 Jul 5 04:24 unbound.conf -rw-r--r-- 1 root unbound 2140 Mar 4 08:19 unbound.conf.error -rw-r----- 1 unbound unbound 2459 Dec 8 2018 unbound_control.key -rw-r----- 1 unbound unbound 1330 Dec 8 2018 unbound_control.pem -rw-r----- 1 unbound unbound 2459 Dec 8 2018 unbound_server.key -rw-r----- 1 unbound unbound 1318 Dec 8 2018 unbound_server.pem drwxr-xr-x 3 root unbound 3 Mar 22 22:01 usr drwxr-xr-x 3 root unbound 3 Mar 22 22:03 var -
@ronpfs looks like .sqlite are set to unbound:unbound/
-
@ronpfs Looks like the sqlite files are correct
-
I'm seeing the same issues with DNSBL. pfSense 2.5.2 upgrade with pfBlocker 3.0.0.16. I just noticed that all blocked HTTP requests are logged fine, however, blocked HTTPS requests are not logged.
-
Looks like mine is logging only HTTP and not HTTPS as well.
-
@cefleet looks like unbound was regressed from 1.13.x to 1.12.x in 2.5.2 due to some other issues... likely related? although IDK when 1.13.x was added to the main tree. Maybe a configuration option available in 1.13.x but not in 1.12.x is borking the logging?
https://docs.netgate.com/pfsense/en/latest/releases/2-5-2.html#dns-resolver
https://redmine.pfsense.org/issues/11915
https://redmine.pfsense.org/issues/11316
-
@nickd-0 said in pfBlocker not logging after 2.5.2 pfSense upgrade:
@cefleet looks like unbound was regressed from 1.13.x to 1.12.x in 2.5.2 due to some other issues... likely related? although IDK when 1.13.x was added to the main tree. Maybe a configuration option available in 1.13.x but not in 1.12.x is borking the logging?
https://docs.netgate.com/pfsense/en/latest/releases/2-5-2.html#dns-resolver
https://redmine.pfsense.org/issues/11915
https://redmine.pfsense.org/issues/11316
oops .. very interesting. It seems a possible cause.
-
@cefleet said in pfBlocker not logging after 2.5.2 pfSense upgrade:
Looks like mine is logging only HTTP and not HTTPS as well.
I have the same problems. Blocking works afer reload, but don't log anymore.
-
I changed from Unbound mode to Unbound Python mode and that has seemed to have fixed things. I thought I did this the other day and it did not work. In any case, it appears to be working now. Thanks for everyone's input.
-
@cefleet Please monitor your disk usage as python mode on 21.05/2.5.2 has a an issue on some systems with slowly consuming all diskspace. The key issue is that no files/logfiles report a size / diskusage that accounts for the space usage - they remain sized like before. So you cannot locate the file/problem that fills the filesystem.
This leads to a situation where the filesystem is full, and you need to stop/start pfBlockerNG completely or reboot pfSense to regain your filesystem space.
-
@keyser Thanks for the heads up. I will keep an eye on the disk usage. So far everything looks good.
-
@cefleet said in pfBlocker not logging after 2.5.2 pfSense upgrade:
I changed from Unbound mode to Unbound Python mode and that has seemed to have fixed things. I thought I did this the other day and it did not work. In any case, it appears to be working now. Thanks for everyone's input.
Thank you very much for the hint. I can confirm, that it is working for me with Unbound Python and enabling Python in pfBlocker DNBL.
-
@dotsch Same here, no issues with the python module and logging.
-
Ok so i also enabled Python Unbound mode and actually the logs are working again. I'll monitor in the coming days the disk occupation
-
Same story for me. Upgraded on the 15th (as clearly visible in the pictures below). Next to not logging of DNSBL there also seems to be a memory leak, unsure if related.
Is it possible to switch to python unbound with DHCP registration? As it still says "Python DNSBL mode is not compatable with the DNS Resolver DHCP Registration option (Unbound will Crash)!" in the information for selecting python unbound.
-
I have faced similar issue, DNSBL stopped blocking after 2.5.2 upgrade. I reinstalled pfsense but it didn't work. Later I changed mode to python unbound and it started working. However post this upgrade, my CPU and memory utilization is very high, earlier I had large list of IP and DNSBL but after this upgrade i cant enable all the list. I am running with only few list b of IP and DNSBL but after this upgrade i cant enable all the list. I am running with only few list but memory is still high. ut memory is still high. Is there any solution for this.
-
@maddy_in65
What is your CPU doing ?
See here Diagnostics > System Activity
Or better : console or SSH access, option 8 and entertopSee here for more info.
-
@gertjan
Here is "Top" output:processes: 4 running, 56 sleeping CPU: 91.3% user, 0.0% nice, 8.7% system, 0.0% interrupt, 0.0% idle Mem: 958M Active, 337M Inact, 750M Wired, 208K Buf, 1698M Free ARC: 278M Total, 161M MFU, 105M MRU, 2654K Anon, 2044K Header, 8178K Other 160M Compressed, 585M Uncompressed, 3.66:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 62378 root 88 20 0 468M 423M nanslp 0 0:37 132.04% ntopng 38952 unbound 1 77 0 91M 79M RUN 1 0:03 32.27% unbound 347 root 1 52 0 102M 39M piperd 0 1:06 25.19% php-fpm 43559 root 1 20 0 13M 3508K CPU1 1 0:00 0.59% top 62111 root 4 20 0 22M 5356K kqread 1 0:00 0.29% redis-server 9748 root 1 20 0 19M 6824K select 1 0:03 0.12% ntpd 36481 root 1 20 0 20M 9220K select 0 0:00 0.10% sshd 85972 root 5 52 0 11M 2592K uwait 0 0:00 0.06% dpinger 47972 dhcpd 1 25 0 23M 12M select 1 0:00 0.02% dhcpd 47822 root 1 20 0 18M 7552K kqread 0 0:00 0.01% lighttpd_pfb 5149 root 3 20 0 18M 7168K select 1 0:06 0.01% pcscd 85427 root 5 52 0 11M 2592K uwait 1 0:00 0.01% dpinger 346 root 1 52 0 102M 39M accept 0 2:03 0.00% php-fpm 48256 root 1 52 0 102M 39M accept 1 1:06 0.00% php-fpm 44666 root 1 52 0 102M 39M accept 1 0:51 0.00% php-fpm 62020 root 1 23 0 102M 38M accept 0 0:35 0.00% php-fpm 19548 root 1 20 0 11M 2648K select 0 0:07 0.00% syslogd 95969 root 2 20 0 229M 191M bpf 0 0:02 0.00% snort 19460 root 1 20 0 104M 36M nanslp 0 0:01 0.00% php-cgi 8574 root 1 20 0 30M 9792K kqread 0 0:01 0.00% nginx 49739 root 1 20 0 61M 39M piperd 0 0:00 0.00% php_pfb 345 root 1 20 0 102M 27M kqread 0 0:00 0.00% php-fpm 31190 root 1 20 0 12M 2956K bpf 1 0:00 0.00% filterlog 8290 root 1 20 0 29M 9248K kqread 1 0:00 0.00% nginx 9162 root 1 45 0 11M 2484K nanslp 1 0:00 0.00% cron 24662 root 1 20 0 21M 8448K select 0 0:00 0.00% mpd5 376 root 1 40 20 11M 2840K kqread 0 0:00 0.00% check_reload_status 49643 root 1 20 0 11M 2212K kqread 0 0:00 0.00% tail_pfb 54837 root 1 20 0 44M 35M bpf 0 0:00 0.00% arpwatch 56501 root 1 20 0 44M 35M bpf 0 0:00 0.00% arpwatch [2.5.2-RELEASE][admin@ -
Well ?
Who is this : ntopng ? I don't know what it is. Is it useful ? It's going at light speed (132 % CPU usage !?!). For me, that alarming. What if that "ntopng" is doing a lot of DNS requests ? That would explain something.
You stopped it ? -
Yes I stopped it. CPU utilization is normal but Memory utilization is still high.
-
Use top again.
It shows the memory usage per process also.
Who is the winner ? -
Isn't high memory usage the nature of the Beastie anyway (file caching)?
I figure I paid for 100% of the memory and CPU and as long as I'm not running out, that's good ROI. -
One thing I noticed post 2.5.2 upgrade is that most clients are loosing internet connection for a while and connects back. I have observed this on few WLAN clients (mobile, tablets), they shows no connectivity . I thought it might be issue with AP (Unifi). I tried rebooting APs but issue still persists. I also check on few LAN clients and they also had similar issue. I have removed pfblocker and the issue didn't appear again. Today I have installed pfblocker again with unbound mode and few feeds. As of now all clients are stable and I am monitoring devices.
Unbound and Snort has major memory utilization.
last pid: 31316; load averages: 0.81, 0.68, 0.72 up 0+06:28:52 15:51:28 65 processes: 1 running, 64 sleeping CPU: 1.4% user, 0.0% nice, 0.6% system, 0.2% interrupt, 97.9% idle Mem: 951M Active, 273M Inact, 531M Wired, 208K Buf, 1988M Free ARC: 214M Total, 126M MFU, 84M MRU, 257K Anon, 1113K Header, 3189K Other 114M Compressed, 368M Uncompressed, 3.22:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 347 root 1 21 0 102M 39M accept 1 0:38 1.56% php-fpm 17542 root 2 20 0 230M 193M bpf 1 2:03 0.49% snort 90137 root 1 20 0 13M 3540K CPU1 1 0:00 0.11% top 70928 root 1 20 0 18M 7924K kqread 0 0:01 0.10% lighttpd_pfb 32031 root 1 20 0 28M 8616K kqread 0 0:03 0.08% nginx 6428 root 3 20 0 23M 11M select 0 0:12 0.05% pcscd 21702 root 3 20 0 238M 197M bpf 1 0:03 0.04% snort 4582 root 1 20 0 104M 36M nanslp 1 0:02 0.03% php-cgi 4447 root 1 20 0 11M 2652K select 1 0:03 0.03% syslogd 71496 root 1 20 0 59M 38M piperd 1 0:01 0.02% php 37225 dhcpd 1 20 0 23M 12M select 0 0:00 0.02% dhcpd 23994 root 5 52 0 11M 2592K uwait 1 0:00 0.02% dpinger 36222 root 1 20 0 12M 2960K bpf 1 0:01 0.02% filterlog 78612 zabbix 1 20 0 19M 9056K nanslp 1 0:00 0.02% zabbix_agentd 24559 root 5 52 0 11M 2592K uwait 1 0:00 0.01% dpinger 33035 root 1 20 0 19M 6900K select 0 0:03 0.01% ntpd 28353 root 1 20 0 20M 9208K select 0 0:00 0.01% sshd 71058 root 1 20 0 10M 2148K kqread 1 0:00 0.01% tail_pfb 71320 root 1 20 0 61M 39M piperd 0 0:00 0.00% php_pfb 345 root 1 20 0 102M 27M kqread 1 0:01 0.00% php-fpm 76619 root 1 20 0 44M 35M bpf 0 0:00 0.00% arpwatch 75582 root 1 20 0 44M 35M bpf 0 0:00 0.00% arpwatch 74827 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 76215 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 75224 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 74047 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 76918 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 75978 root 1 20 0 44M 35M bpf 1 0:00 0.00% arpwatch 28242 unbound 2 20 0 383M 367M kqread 0 0:36 0.00% unbound 46839 root 1 52 0 104M 40M accept 1 0:33 0.00% php-fpm 346 root 1 39 0 101M 39M accept 0 0:32 0.00% php-fpm -
@maddy_in65 said in pfBlocker not logging after 2.5.2 pfSense upgrade:
28242 unbound 2 20 0 383M 367M kqread 0 0:36 0.00% unbound
I have
55688 unbound 2 20 0 105M 82M kqread 1 2:57 0.00% unboundThe difference is probably tour DNSBL usage (pfBlockerNG).
Still, these values are ok.
Btw : i'm not trying to block "everybody", as many and/or huge DNSBL/IP feeds do impact the system.
I'm just using the minimal :
-
Using the latest pfBlockerNG-devel, IP stats won't work unless using auto rules. If using only the "Alias" actions, i.e. custom rules, the package is not able to keep track of the rule matches. I imagine this is the intended behavior.
Note: I've only tested on 21.05.
-
@marcos-ng From the under Action it says :
Note: When manually creating 'Alias' type firewall rules, Prefix the Firewall rule Description with pfb_ . This will ensure that that Dashboard widget reports those statistics correctly. Do not prefix with (pfB_) as those Rules will be auto-removed by package when 'Auto' rules are defined. -
@ronpfs
I missed that, thanks. I suppose then this should be confirmed by whomever reports that it's not working. -
Hello,
today i updated the latest version of pfsense (2.5.2) , and i have the same problem . In the DNSBL configuration , i changed to localhost , i uninstalled pfblockerNG-devel,reinstalled , I have updated the lists , but i still have the same problem .
Someone solved it?
-
@n3xus_x3 said in pfBlocker not logging after 2.5.2 pfSense upgrade:
Someone solved it?
The initial issue was : pfBlockerNG stopped blocking.
Thus is stops logging.
I tend to say that's quiet normal.Here are the 'pfBlockerNG' files :
ls -al /var/unbound/var/log/pfblockerng/You means these files are not growing in size for you ?
Your stats say : it blocks things, as numbers are shown in the "Packet" column.
@n3xus_x3 said in pfBlocker not logging after 2.5.2 pfSense upgrade:
, i changed to localhost
Changed what ? Where ?
'localhost' is the default, and works :
Or I do not understand what you mean by "i changed to localhost".
-
Thanks for replay
in pfblockerng/alerts/Reports , i noticed that after pfsense update, the contents are not blocked like before , in the Deny section it seems that everything works
The problem is on DNSBL Block , it seems to me that the lists work randomly , I'm sure they should be a lot more
these are my DNS settings
I don't use Python mode in DNSBL -
I advise you look up why this so called Python mode was introduced.
There are some reddit posts where the author explains it.To make a long story short :
To make DNSBL work better, info is needed that unbound won't log in detail.
Or worse, won't make available any more.
But, recently, the latest unbound versions, could interface with 'plugins' written in Python.So, thre solution was : this python mode.
Unbound has to be used as a resolver of course - not as a forwarder as you do. -
The initial issue was : pfBlockerNG stopped blocking.
Not sure what you're referring to here. The topic of this thread is 'stopped logging' not stopped blocking.
So, thre solution was : this python mode.
As long as that isn't fully compatible with the rest of pfSense e.g.
Python DNSBL mode is not compatable with the DNS Resolver DHCP Registration option (Unbound will Crash)!that advise doesn't hold up. I'd gladly switch to python mode, however I can't. -
@silliewous Hi! I solved this problem by disabling dnsbl mode in pfblocker and moving that functionality over to a rpi4 with adguard home. Pfblocker with dnsbl was giving me all kinds of crazy behavior and in the end I just need my internet to work while wfh.
-
I changed to python mode in DNSBL settings , it works a little better, I mean more content is blocked ,but i think to go back to the previous version of pfsense,I've never had any problems and all the lists I have are working fine
-
@n3xus_x3 Be aware that older versions of pfsense cannot run the latest version of pfblockerng which have a lot of fixes compared to older versions. For the time being pfblockerng looks death and is not actively being worked on, from what I can see at least.
-
Here is my regex.
It's more test-of-proof sample for me. I remember finding it somewhere on Reddit.^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-] #Regex RGX1 ^(.+[_.-])?telemetry[_.-] #Regex RGX2 ^ad([sxv]?[0-9]*|system)[_.-]([^.[:space:]]+\.){1,}|[_.-]ad([sxv]?[0-9]*|system)[_.-] #test RGX3 ^adim(age|g)s?[0-9]*[_.-] #Regex RGX4 ^adtrack(er|ing)?[0-9]*[_.-] #Regex RGX5 ^advert(s|is(ing|ements?))?[0-9]*[_.-] #Regex RGX6 ^aff(iliat(es?|ion))?[_.-] #Regex RGX7 ^analytics?[_.-] #Regex RGX8 ^banners?[_.-] #Regex RGX9 ^beacons?[0-9]*[_.-] #Regex RGX10 ^count(ers?)?[0-9]*[_.-] #Regex RGX11 ^mads\. #Regex RGX12 ^pixels?[-.] #Regex RGX13 ^stat(s|istics)?[0-9]*[_.-] #Regex RGX14Keep in mind : the ending
"Space # text string" needs to be unique.
