pfBlockerNG/pfBlockerNG-devel v3.2.0_2

[provels]

Am I the only one not seeing this being available? Pulled from Package Manager?

[jrey]

@provels

Lucky you, lol.

I'm really just trying to see how I can confirm if the fix that is suppose to be in it is the potential cause of the "nothing" happening with the feeds being updated that I am seeing in V3.2.0_1 With nothing in the error.log, and only cron entries that report "No Updates required" it is making difficult to see the "real" problem.

[BBcan177]

@jrey disable TLD Wildcard feature as there is a regression with FreeBSD Grep. Might need a reboot to clear the long running grep task. Working on a solution.

[jrey]

@bbcan177

Thanks but this already is and always has been disabled

it is not just the DNSBL lists - custom lists where I am providing a specific url to get IP addresses to block from are it is not obtaining updates there either. (and I can confirm they have in fact been updated at source)

The system is still working on all the same lists it had before the system was updated.
(that is to say all the files in /var/db/pfblockerng/original have time stamps from before the update applied)
those would have to update first and that doesn't appear to be happening, through cron, or manual.
i was able to get one update by adding an IP to the custom list at the bottom, but the feeds on the same set above it never pulled new files.
just doesn't seem to be any error logged in any file I can find.

[james2432]

@tbr281 said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:

@BBcan177 I upgrated to pfsense 23.01 last night and that crashed pfblocker... im not too worried about it since i wanted a clean install of pfblocker.. just wanted to give you a heads up. hope the new build corrects the issue.

I fixed mine by accessing pfBlockerNG's settings page via the dashboard tool(wrench).

Going to "keep settings" and unchecking. Save. Uninstall pfBlockerNG, reinstall.

[BBcan177]

@jrey Can you pm or email (bbcan177 at gmail.com) your pfblockerng.log for review?

[jrey]

@bbcan177

So not sure this will help.

I went into the previously mentioned /var/db/pfblockerng/origianal directory and deleted one of the smaller files created within a larger group of feeds.

Cron didn't even know it was missing -- (maybe there is an index that needs to be adjusted, didn't look)

logged
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 16:00:01 ]

No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED
end

from here I went to the force a manual "update" the file in question didn't even appear on the list of files that "exist" but it didn't download it either... interesting.

wait a few minutes, and update again this time "reload" all file except the one I had deleted indicated "reload [date] completed"
the file I had deleted guess what...
Downloading update [ 02/18/23 16:20:15 ] .. 200 OK. completed ..

based on historical updates I would expect this particular file will update (at source) again in the next few hours, so I am going to leave everything now - and see if it updates itself next time (when the source updates).

Deleting original file
Cron - did not download the file (or recognize it as missing)
Manual update - does not download the file (didn't list it here either)
Manual reload - yeah a new updated file !!!

no changes made to the feeds or other settings at all in between tests.

[jrey]

@bbcan177 said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:

Can you pm or email (bbcan177 at gmail.com) your pfblockerng.log for review?

There is honestly nothing much in it since the update
every cron job that has run since the update only has entries like those already provided
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 02:00:00 ]

No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED

the reload and update I tried earlier (without the file being delete) show
either nothing downloaded just a list of the files the "exist" and then a reload compete for each of the existing files and never downloading anything.
There are not other messages of concern in these, the update and/or reload just didn't achieve the results of getting the newer files.

I finally thought just delete one of the existing files and see what happens, that is described in my other post.

What I have noticed has not been logged since the update is entries like for each file. (in my lists) but then again the logs have been pretty boring.

Remote timestamp: Thu, 16 Feb 2023 11:19:37 GMT
Local timestamp: Wed, 15 Feb 2023 11:19:43 GMT Update found

there is nothing in the error.og since the update.. prior to the update it would occasionally show (for example)
[ pfB_PRI1_v4 - Talos_BL_v4 ] Download Fail [ 02/16/23 10:00:18 ]
DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.
[ 02/16/23 10:00:18 ]
Restoring previously downloaded file contents... [ 02/16/23 10:00:18 ]

that entry is in fact is the last entry in the error log.
(I wasn't concerned about it because as several others have noted that particular feed has been up and down) a bunch and it would always get it later)
The other feeds I use have never logged any errors, prior to update or now (only the Talos one goes missing once and a while) although it also hasn't reported MIA since the update either, I don't think that has anything to do with the feed, but rather that it appears not to check at all)
My other post details the steps taken and the results.

[BBcan177]

@jrey next time delete from the deny folder. That's is the monitored folder. Not the original folder.

[jrey]

@bbcan177

Ok there was a cron within 10 minutes of your ask, so I nuked one file.

with the file now removed from the deny directory only the cron job now logged the same.
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 18:00:01 ]

No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED

and the file I removed from deny has not returned. If fact all the files in there still show the time stamp of my previous run at ~16:20 when the reload with the one file missing from original "reloaded" all the files and "downloaded" the one I had nuked.

So my next step delete the file in original (a different file from the one previously) but now the same one I just deleted from deny before cron and run a manual reload
Again all the files "reload" except the one in original that I nuked.
It downloaded
Downloading update [ 02/18/23 18:09:54 ] .. 200 OK. completed ..

and is now back in both "original" and "deny"

next test, delete another different one in deny and go directly to manual "update" not reload,
Downloading update .. 200 OK. completed ..
(no time stamp in that message however) but an update did download it and it updated the one in original as well

So at least I know at this point that I can delete the files in deny, run either a manual reload or update and they will download. kind of rules out the original question is it related to the DNS item I quoted.

Thanks
JR

[johnpoz]

@bbcan177 silly question, maybe I just need more coffee this morning.

I just updated to 23.01, and pfblocker seems to be working just fine and currently at 3.2.0_1, my take is that _2 just not available yet..

But my question is more to devel vs non devel version. I notice version numbers and package dependence for _1 seem to be in sync between the versions.

Is there any reason to move away from devel version and change over to just the NG version?

[SteveITS]

@johnpoz it was in the 23.01 release notes I believe ;) but they migrated devel to non devel so they are equivalent at the moment.

[johnpoz]

@steveits said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:

it was in the 23.01 release notes I believe

Well F me then ;) hehehe you are correct good sir..

https://docs.netgate.com/pfsense/en/latest/releases/23-01.html

The pfBlockerNG package has been updated to match pfBlockerNG-devel. After upgrade it is safe to uninstall pfBlockerNG-devel (keeping settings) and install pfBlockerNG instead.

How did I miss that? Doh! Thanks!

edit: successfully moved to just NG version of package vs -devel, all looking good here..

[jrey]

@johnpoz
@BBcan177

So I'm curious. is the cron job updating the feeds as expected?
I am on 3.2.0_1 and well as my previous posts are showing, the feeds are not updating through the cron job.
I can force them to update by deleting one or all of the txt files in the deny directory and running a manual update.
But when deleting the txt files in the deny directory and letting cron run, nothing happens - just returns "No Updates required."
I might be wasting my time, but was just going to read through the pfblockerng.php and see if I can spot anything different between the "update" and "cron" processing paths that might cause it.

looks to me list the function pfblockerng_sync_cron has a couple of points where it could not do anything and end update up exiting with only the "No required message" being logged. I might have another coffee myself and then add a couple of logging statements to at least see if there is a failure point in

looks like any of those first three if statements could just cause a "silent" unlogged by-pass of the enclosed -
that would lead to what is being logged.

[johnpoz]

@jrey what is not updating.. Your lists? Yeah if there is nothing to update, why should they update?

Be happy to try and duplicate what you think is problem.. I ran a manual update after my change over to NG vs -devel

 [ Force Reload Task - All ]
 UPDATE PROCESS START [ v3.2.0_1 ] [ 02/19/23 09:31:58 ]

===[  DNSBL Process  ]================================================


===[  GeoIP Process  ]============================================


===[  IPv4 Process  ]=================================================

[ MA_v4 ]			 exists. [ 02/19/23 09:31:59 ]
[ MA_rep_v4 ]			 exists.
[ PlexRemoteCheck_v4 ]		 exists.
[ SCake_v4 ]			 exists.
[ UptimeRobot_v4 ]		 exists.
[ US_v4 ]			 exists.
[ SCake_v4 ]			 exists.
[ Uptime_v4 ]			 exists.
[ dohIPlist_v4 ]		 exists.
[ GreatWalldohIPlist_v4 ]	 exists.
[ BlockDOH_custom_v4 ]		 exists.
[ shodan_v4 ]			 exists.
[ stretchoid_v4 ]		 exists.
[ shadowserver_v4 ]		 exists.
[ ScanDeny_custom_v4 ]		 exists.
[ AS14061_v4 ]			 exists.
[ AS39690_v4 ]			 exists.
[ AS62567_v4 ]			 exists.
[ AS133165_v4 ]			 exists.
[ AS135340_v4 ]			 exists.
[ AS200130_v4 ]			 exists.
[ AS201229_v4 ]			 exists.
[ AS202018_v4 ]			 exists.
[ AS202109_v4 ]			 exists.
[ AS205301_v4 ]			 exists.
[ AS393406_v4 ]			 exists.
[ AS394362_v4 ]			 exists.

===[  Aliastables / Rules  ]==========================================

No changes to Firewall rules, skipping Filter Reload
No Changes to Aliases, Skipping pfctl Update

 UPDATE PROCESS ENDED [ 02/19/23 09:32:00 ]

I am by no means a pfblocker expert, but this seems normal to me..

[MaxK 0]

@johnpoz said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:

How did I miss that? Doh! Thanks!

To be honest, I’m just trying to keep up with all the release notes, redmine issues, patches/suggested fixes, configuration changes, and forum discussions. I feel like I will definitely miss something when I go to upgrade.

[MoonKnight]

@johnpoz
Thanks for the heads up, just did the same as you did. No problem with NG version here. :)

[jrey]

@johnpoz

I would agree, with your screen capture when there is something being checked, that is what mine looked like prior to the update as well
But the lists haven't updated on my system except the ones where I have manually removed the file(s) from deny directory.
So this is clearly not the case here. and again a cron process hasn't listed anything other than 5 lines shown in the log (even if I delete a file from deny)
We do know that if I delete a file in deny and run a manual update it both downloads and updates producing log records similar to above) but only if I manually run it.
and yes I can verify that source lists have changed and should update, (in fact on source files was updated hours ago Sun 19 Feb 2023 12:10:03 PM CET) but it has not been picked up by any cron update, or manual.
the log as posted below and in prior posts, is only showing (nothing actually) exists (or list name even checked.) days worth of logging just like this:

CRON PROCESS START [ v3.2.0_1 ] [ 02/19/23 12:00:00 ]

No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED
CRON PROCESS START [ v3.2.0_1 ] [ 02/19/23 14:00:01 ]

No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED

I've now severals days of logs list this with nothing ever listed, again, unless I manually delete a file in the deny directory and in that case only a manual update or reload will download the file I deleted. Everything else remains untouched.

-Clearly not even telling me the files exists, need updates, is download. where it would then say downloading.

Something clearly didn't update well during the update process. But I can't see what, is preventing it.
The lists haven't changed, their update frequency hasn't changed.

Also the update process as used, was
uninstall pfBlockerNG (the keep settings was checked, this was recommended)
reboot system
update system from 22.05 -> 23.01
reinstall pfBlockerNG 3.2.0_1
manual reload

hasn't updated any list through the cron process since,

[Roger-S]

@BBcan177

I have observed the situation that under pfSense 23.01 with the current version of pfBlockerNG 3.2.0_1 both in the regular version as well as the DEVEL variant the update and reload of pfBlockerNG does not work properly.

The feeds are downloaded and run through individually, but when processing at the "TLD finalize" level, it hangs indefinitely until the pfSense is restarted. I have observed this phenomenon with both pfBlockerNG variants and also with classic DNSBL unbound mode as well as Python mode.

 UPDATE PROCESS START [ v3.2.0_1 ] [ 02/20/23 10:52:50 ]

===[  DNSBL Process  ]================================================

 Loading DNSBL Statistics... completed
 Loading DNSBL SafeSearch...  enabled
 Loading DNSBL Whitelist... completed
 DNSBL - SafeSearch changes found - Rebuilding!


[ EasyList ]			 Reload . completed ..
  Whitelist: adsafeprotected.com|amazon-adsystem.com|mail-ads.google.com|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  20441    20439      0          3          0          20436                
  ----------------------------------------------------------------------
  IPv4 count=24

[ EasyPrivacy ]			 Reload [ 02/20/23 10:52:56 ] . completed ..
  Whitelist: adfox.yandex.ru|adsdk.yandex.ru|an.yandex.ru|awaps.yandex.ru|bat.bing.com|bs.yandex.ru|collector.github.com|commerce.bing.com|copilot-telemetry.githubusercontent.com|fcmatch.google.com|fcmatch.youtube.com|informer.yandex.ru|mc.yandex.com|mc.yandex.ru|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  16418    15271      16         14         0          15241                
  ----------------------------------------------------------------------
  IPv4 count=6

[ Adaway ]			 Reload [ 02/20/23 10:53:02 ] . completed ..
  Whitelist: 
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  7355     7355       382        58         0          6915                 
  ----------------------------------------------------------------------

[ D_Me_ADs ]			 Reload [ 02/20/23 10:53:06 ] . completed ..
  Whitelist: ads.bing.com|ads.youtube.com|adserver.bing.com|amazon-adsystem.com|bs.yandex.ru|pagead.l.google.com|partnerad.l.google.com|pixel.adsafeprotected.com|video-stats.video.google.com|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  2701     2701       348        9          0          2344                 
  ----------------------------------------------------------------------

[ D_Me_Tracking ]		 Reload [ 02/20/23 10:53:10 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  34       34         14         0          0          20                   
  ----------------------------------------------------------------------

[ Yoyo ]			 Reload [ 02/20/23 10:53:14 ] . completed ..
  Whitelist: ads.bing.com|ads.youtube.com|adsafeprotected.com|adsdk.yandex.ru|adserver.bing.com|adservice.google.com|adservice.google.com.mt|amazon-adsystem.com|analytics.google.com|bat.bing.com|bs.yandex.ru|mail-ads.google.com|pagead.l.google.com|partnerad.l.google.com|video-stats.video.google.com|www-google-analytics.l.google.com|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  3680     3680       2507       16         0          1157                 
  ----------------------------------------------------------------------

[ Abuse_ThreatFox ]		 Reload [ 02/20/23 10:53:17 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  26782    26642      2          0          0          26640                
  ----------------------------------------------------------------------

[ D_Me_Malv ]			 Reload [ 02/20/23 10:53:24 ] . completed ..
  Whitelist: 
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  2735     2735       2725       9          0          1                    
  ----------------------------------------------------------------------

[ D_Me_Malw ]			 Reload [ 02/20/23 10:53:27 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  1        1          1          0          0          0                    
  ----------------------------------------------------------------------

[ Krisk_C19 ]			 Reload [ 02/20/23 10:53:31 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  2000     2000       0          0          0          2000                 
  ----------------------------------------------------------------------

[ Maltrail_BD ]			 Reload [ 02/20/23 10:53:35 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  272221   272221     5191       0          0          267030               
  ----------------------------------------------------------------------

[ MVPS ]			 Reload [ 02/20/23 10:54:11 ] . completed ..
  Whitelist: 
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  8730     8730       1126       43         0          7561                 
  ----------------------------------------------------------------------

[ SFS_Toxic_BD ]		 Reload [ 02/20/23 10:54:16 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  43273    43272      8          0          0          43264                
  ----------------------------------------------------------------------
  IPv4 count=2

[ Spam404 ]			 Reload [ 02/20/23 10:54:26 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  7066     7064       61         0          0          7003                 
  ----------------------------------------------------------------------

[ SWC ]				 Reload [ 02/20/23 10:54:30 ] . completed ..
  Whitelist: 
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  11397    11397      2666       42         0          8689                 
  ----------------------------------------------------------------------

[ Abuse_urlhaus ]		 Reload [ 02/20/23 10:54:38 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  553      553        84         0          0          469                  
  ----------------------------------------------------------------------

[ PhishTank ]			 Reload [ 02/20/23 10:54:42 ] . completed ..
  Whitelist: 543.sites.google.com|accounts.google.com|docs.google.com|drive.google.com|forms.yandex.com|play.google.com|s3.amazonaws.com|script.google.com|sites.google.com|storage.cloud.google.com|www.bing.com|www.google.com|www.sites.google.com|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  26468    21536      66         13         0          21457                
  ----------------------------------------------------------------------
  IPv4 count=97

[ OISD ]			 Reload [ 02/20/23 10:54:50 ] . completed ..
  Whitelist: 
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  894756   894756     98405      436        0          795915               
  ----------------------------------------------------------------------

[ AZORult_BD ]			 Reload [ 02/20/23 10:56:37 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  982      982        749        0          0          233                  
  ----------------------------------------------------------------------

[ Ponmocup ]			 Reload [ 02/20/23 10:56:43 ] . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  50       50         0          0          0          50                   
  ----------------------------------------------------------------------

[ H3X_1w ]			 Reload [ 02/20/23 10:56:49 ] . completed .
 No Domains Found! Ensure only domain based Feeds are used for DNSBL!

[ Rescure_DNSBL ]		 Reload . completed ..
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  500      500        148        0          0          352                  
  ----------------------------------------------------------------------

------------------------------------------------------------------------
Assembling DNSBL database...... completed [ 02/20/23 10:57:07 ]
TLD:
TLD analysis............. completed [ 02/20/23 10:58:16 ]
TLD finalize.

I have tried it on existing pfSense environments but also on new installations. In both cases exactly the same.

Thanks and greetings

[emikaadeo]

@roger-s
This is a known issue. Read the first post from @BBcan177
Full Details here

There is still a Regression with TLD Wildcard Feature in pfSense+ (possibly in pfSense 2.6 also) due to some recent changes to FreeBSD Grep command. This is being reviewed and will provide more updates as soon as we have a solution.
The only solution for now is to disable Wildcard Blocking (TLD) in pfBlockerNG