pfBlockerNG/pfBlockerNG-devel v3.2.0_2
-
@jrey Can you pm or email (bbcan177 at gmail.com) your pfblockerng.log for review?
-
So not sure this will help.
I went into the previously mentioned /var/db/pfblockerng/origianal directory and deleted one of the smaller files created within a larger group of feeds.
Cron didn't even know it was missing -- (maybe there is an index that needs to be adjusted, didn't look)
logged
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 16:00:01 ]No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED
endfrom here I went to the force a manual "update" the file in question didn't even appear on the list of files that "exist" but it didn't download it either... interesting.
wait a few minutes, and update again this time "reload" all file except the one I had deleted indicated "reload [date] completed"
the file I had deleted guess what...
Downloading update [ 02/18/23 16:20:15 ] .. 200 OK. completed ..based on historical updates I would expect this particular file will update (at source) again in the next few hours, so I am going to leave everything now - and see if it updates itself next time (when the source updates).
Deleting original file
Cron - did not download the file (or recognize it as missing)
Manual update - does not download the file (didn't list it here either)
Manual reload - yeah a new updated file !!!no changes made to the feeds or other settings at all in between tests.
-
@bbcan177 said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:
Can you pm or email (bbcan177 at gmail.com) your pfblockerng.log for review?
There is honestly nothing much in it since the update
every cron job that has run since the update only has entries like those already provided
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 02:00:00 ]No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDEDthe reload and update I tried earlier (without the file being delete) show
either nothing downloaded just a list of the files the "exist" and then a reload compete for each of the existing files and never downloading anything.
There are not other messages of concern in these, the update and/or reload just didn't achieve the results of getting the newer files.I finally thought just delete one of the existing files and see what happens, that is described in my other post.
What I have noticed has not been logged since the update is entries like for each file. (in my lists) but then again the logs have been pretty boring.
Remote timestamp: Thu, 16 Feb 2023 11:19:37 GMT
Local timestamp: Wed, 15 Feb 2023 11:19:43 GMT Update foundthere is nothing in the error.og since the update.. prior to the update it would occasionally show (for example)
[ pfB_PRI1_v4 - Talos_BL_v4 ] Download Fail [ 02/16/23 10:00:18 ]
DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.
[ 02/16/23 10:00:18 ]
Restoring previously downloaded file contents... [ 02/16/23 10:00:18 ]that entry is in fact is the last entry in the error log.
(I wasn't concerned about it because as several others have noted that particular feed has been up and down) a bunch and it would always get it later)
The other feeds I use have never logged any errors, prior to update or now (only the Talos one goes missing once and a while) although it also hasn't reported MIA since the update either, I don't think that has anything to do with the feed, but rather that it appears not to check at all)
My other post details the steps taken and the results. -
@jrey next time delete from the deny folder. That's is the monitored folder. Not the original folder.
"Experience is something you don't get until just after you need it."
Website: http://pfBlockerNG.com
Twitter: @BBcan177Ā #pfBlockerNG
Reddit: https://www.reddit.com/r/pfBlockerNG/new/ -
Ok there was a cron within 10 minutes of your ask, so I nuked one file.
with the file now removed from the deny directory only the cron job now logged the same.
CRON PROCESS START [ v3.2.0_1 ] [ 02/18/23 18:00:01 ]No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDEDand the file I removed from deny has not returned. If fact all the files in there still show the time stamp of my previous run at ~16:20 when the reload with the one file missing from original "reloaded" all the files and "downloaded" the one I had nuked.
So my next step delete the file in original (a different file from the one previously) but now the same one I just deleted from deny before cron and run a manual reload
Again all the files "reload" except the one in original that I nuked.
It downloaded
Downloading update [ 02/18/23 18:09:54 ] .. 200 OK. completed ..and is now back in both "original" and "deny"
next test, delete another different one in deny and go directly to manual "update" not reload,
Downloading update .. 200 OK. completed ..
(no time stamp in that message however) but an update did download it and it updated the one in original as wellSo at least I know at this point that I can delete the files in deny, run either a manual reload or update and they will download. kind of rules out the original question is it related to the DNS item I quoted.
Thanks
JR -
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
@bbcan177 silly question, maybe I just need more coffee this morning.
I just updated to 23.01, and pfblocker seems to be working just fine and currently at 3.2.0_1, my take is that _2 just not available yet..
But my question is more to devel vs non devel version. I notice version numbers and package dependence for _1 seem to be in sync between the versions.
Is there any reason to move away from devel version and change over to just the NG version?
-
@johnpoz it was in the 23.01 release notes I believe ;) but they migrated devel to non devel so they are equivalent at the moment.
-
@steveits said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:
it was in the 23.01 release notes I believe
Well F me then ;) hehehe you are correct good sir..
https://docs.netgate.com/pfsense/en/latest/releases/23-01.html
The pfBlockerNG package has been updated to match pfBlockerNG-devel. After upgrade it is safe to uninstall pfBlockerNG-devel (keeping settings) and install pfBlockerNG instead.How did I miss that? Doh! Thanks!
edit: successfully moved to just NG version of package vs -devel, all looking good here..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
So I'm curious. is the cron job updating the feeds as expected?
I am on 3.2.0_1 and well as my previous posts are showing, the feeds are not updating through the cron job.
I can force them to update by deleting one or all of the txt files in the deny directory and running a manual update.
But when deleting the txt files in the deny directory and letting cron run, nothing happens - just returns "No Updates required."
I might be wasting my time, but was just going to read through the pfblockerng.php and see if I can spot anything different between the "update" and "cron" processing paths that might cause it.looks to me list the function pfblockerng_sync_cron has a couple of points where it could not do anything and end update up exiting with only the "No required message" being logged. I might have another coffee myself and then add a couple of logging statements to at least see if there is a failure point in
looks like any of those first three if statements could just cause a "silent" unlogged by-pass of the enclosed -
that would lead to what is being logged. -
@jrey what is not updating.. Your lists? Yeah if there is nothing to update, why should they update?
Be happy to try and duplicate what you think is problem.. I ran a manual update after my change over to NG vs -devel
[ Force Reload Task - All ] UPDATE PROCESS START [ v3.2.0_1 ] [ 02/19/23 09:31:58 ] ===[ DNSBL Process ]================================================ ===[ GeoIP Process ]============================================ ===[ IPv4 Process ]================================================= [ MA_v4 ] exists. [ 02/19/23 09:31:59 ] [ MA_rep_v4 ] exists. [ PlexRemoteCheck_v4 ] exists. [ SCake_v4 ] exists. [ UptimeRobot_v4 ] exists. [ US_v4 ] exists. [ SCake_v4 ] exists. [ Uptime_v4 ] exists. [ dohIPlist_v4 ] exists. [ GreatWalldohIPlist_v4 ] exists. [ BlockDOH_custom_v4 ] exists. [ shodan_v4 ] exists. [ stretchoid_v4 ] exists. [ shadowserver_v4 ] exists. [ ScanDeny_custom_v4 ] exists. [ AS14061_v4 ] exists. [ AS39690_v4 ] exists. [ AS62567_v4 ] exists. [ AS133165_v4 ] exists. [ AS135340_v4 ] exists. [ AS200130_v4 ] exists. [ AS201229_v4 ] exists. [ AS202018_v4 ] exists. [ AS202109_v4 ] exists. [ AS205301_v4 ] exists. [ AS393406_v4 ] exists. [ AS394362_v4 ] exists. ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload No Changes to Aliases, Skipping pfctl Update UPDATE PROCESS ENDED [ 02/19/23 09:32:00 ]I am by no means a pfblocker expert, but this seems normal to me..
-
@johnpoz said in pfBlockerNG/pfBlockerNG-devel v3.2.0_2:
How did I miss that? Doh! Thanks!
To be honest, Iām just trying to keep up with all the release notes, redmine issues, patches/suggested fixes, configuration changes, and forum discussions. I feel like I will definitely miss something when I go to upgrade.
-
@johnpoz
Thanks for the heads up, just did the same as you did. No problem with NG version here. :) -
I would agree, with your screen capture when there is something being checked, that is what mine looked like prior to the update as well
But the lists haven't updated on my system except the ones where I have manually removed the file(s) from deny directory.
So this is clearly not the case here. and again a cron process hasn't listed anything other than 5 lines shown in the log (even if I delete a file from deny)
We do know that if I delete a file in deny and run a manual update it both downloads and updates producing log records similar to above) but only if I manually run it.
and yes I can verify that source lists have changed and should update, (in fact on source files was updated hours ago Sun 19 Feb 2023 12:10:03 PM CET) but it has not been picked up by any cron update, or manual.
the log as posted below and in prior posts, is only showing (nothing actually) exists (or list name even checked.) days worth of logging just like this:CRON PROCESS START [ v3.2.0_1 ] [ 02/19/23 12:00:00 ]
No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDED
CRON PROCESS START [ v3.2.0_1 ] [ 02/19/23 14:00:01 ]No Updates required.
CRON PROCESS ENDED
UPDATE PROCESS ENDEDI've now severals days of logs list this with nothing ever listed, again, unless I manually delete a file in the deny directory and in that case only a manual update or reload will download the file I deleted. Everything else remains untouched.
-Clearly not even telling me the files exists, need updates, is download. where it would then say downloading.
Something clearly didn't update well during the update process. But I can't see what, is preventing it.
The lists haven't changed, their update frequency hasn't changed.Also the update process as used, was
uninstall pfBlockerNG (the keep settings was checked, this was recommended)
reboot system
update system from 22.05 -> 23.01
reinstall pfBlockerNG 3.2.0_1
manual reloadhasn't updated any list through the cron process since,
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
I have observed the situation that under pfSense 23.01 with the current version of pfBlockerNG 3.2.0_1 both in the regular version as well as the DEVEL variant the update and reload of pfBlockerNG does not work properly.
The feeds are downloaded and run through individually, but when processing at the "TLD finalize" level, it hangs indefinitely until the pfSense is restarted. I have observed this phenomenon with both pfBlockerNG variants and also with classic DNSBL unbound mode as well as Python mode.
UPDATE PROCESS START [ v3.2.0_1 ] [ 02/20/23 10:52:50 ] ===[ DNSBL Process ]================================================ Loading DNSBL Statistics... completed Loading DNSBL SafeSearch... enabled Loading DNSBL Whitelist... completed DNSBL - SafeSearch changes found - Rebuilding! [ EasyList ] Reload . completed .. Whitelist: adsafeprotected.com|amazon-adsystem.com|mail-ads.google.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 20441 20439 0 3 0 20436 ---------------------------------------------------------------------- IPv4 count=24 [ EasyPrivacy ] Reload [ 02/20/23 10:52:56 ] . completed .. Whitelist: adfox.yandex.ru|adsdk.yandex.ru|an.yandex.ru|awaps.yandex.ru|bat.bing.com|bs.yandex.ru|collector.github.com|commerce.bing.com|copilot-telemetry.githubusercontent.com|fcmatch.google.com|fcmatch.youtube.com|informer.yandex.ru|mc.yandex.com|mc.yandex.ru| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 16418 15271 16 14 0 15241 ---------------------------------------------------------------------- IPv4 count=6 [ Adaway ] Reload [ 02/20/23 10:53:02 ] . completed .. Whitelist: ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 7355 7355 382 58 0 6915 ---------------------------------------------------------------------- [ D_Me_ADs ] Reload [ 02/20/23 10:53:06 ] . completed .. Whitelist: ads.bing.com|ads.youtube.com|adserver.bing.com|amazon-adsystem.com|bs.yandex.ru|pagead.l.google.com|partnerad.l.google.com|pixel.adsafeprotected.com|video-stats.video.google.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2701 2701 348 9 0 2344 ---------------------------------------------------------------------- [ D_Me_Tracking ] Reload [ 02/20/23 10:53:10 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 34 34 14 0 0 20 ---------------------------------------------------------------------- [ Yoyo ] Reload [ 02/20/23 10:53:14 ] . completed .. Whitelist: ads.bing.com|ads.youtube.com|adsafeprotected.com|adsdk.yandex.ru|adserver.bing.com|adservice.google.com|adservice.google.com.mt|amazon-adsystem.com|analytics.google.com|bat.bing.com|bs.yandex.ru|mail-ads.google.com|pagead.l.google.com|partnerad.l.google.com|video-stats.video.google.com|www-google-analytics.l.google.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3680 3680 2507 16 0 1157 ---------------------------------------------------------------------- [ Abuse_ThreatFox ] Reload [ 02/20/23 10:53:17 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 26782 26642 2 0 0 26640 ---------------------------------------------------------------------- [ D_Me_Malv ] Reload [ 02/20/23 10:53:24 ] . completed .. Whitelist: ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2735 2735 2725 9 0 1 ---------------------------------------------------------------------- [ D_Me_Malw ] Reload [ 02/20/23 10:53:27 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1 1 1 0 0 0 ---------------------------------------------------------------------- [ Krisk_C19 ] Reload [ 02/20/23 10:53:31 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2000 2000 0 0 0 2000 ---------------------------------------------------------------------- [ Maltrail_BD ] Reload [ 02/20/23 10:53:35 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 272221 272221 5191 0 0 267030 ---------------------------------------------------------------------- [ MVPS ] Reload [ 02/20/23 10:54:11 ] . completed .. Whitelist: ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 8730 8730 1126 43 0 7561 ---------------------------------------------------------------------- [ SFS_Toxic_BD ] Reload [ 02/20/23 10:54:16 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 43273 43272 8 0 0 43264 ---------------------------------------------------------------------- IPv4 count=2 [ Spam404 ] Reload [ 02/20/23 10:54:26 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 7066 7064 61 0 0 7003 ---------------------------------------------------------------------- [ SWC ] Reload [ 02/20/23 10:54:30 ] . completed .. Whitelist: ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 11397 11397 2666 42 0 8689 ---------------------------------------------------------------------- [ Abuse_urlhaus ] Reload [ 02/20/23 10:54:38 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 553 553 84 0 0 469 ---------------------------------------------------------------------- [ PhishTank ] Reload [ 02/20/23 10:54:42 ] . completed .. Whitelist: 543.sites.google.com|accounts.google.com|docs.google.com|drive.google.com|forms.yandex.com|play.google.com|s3.amazonaws.com|script.google.com|sites.google.com|storage.cloud.google.com|www.bing.com|www.google.com|www.sites.google.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 26468 21536 66 13 0 21457 ---------------------------------------------------------------------- IPv4 count=97 [ OISD ] Reload [ 02/20/23 10:54:50 ] . completed .. Whitelist: ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 894756 894756 98405 436 0 795915 ---------------------------------------------------------------------- [ AZORult_BD ] Reload [ 02/20/23 10:56:37 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 982 982 749 0 0 233 ---------------------------------------------------------------------- [ Ponmocup ] Reload [ 02/20/23 10:56:43 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 50 50 0 0 0 50 ---------------------------------------------------------------------- [ H3X_1w ] Reload [ 02/20/23 10:56:49 ] . completed . No Domains Found! Ensure only domain based Feeds are used for DNSBL! [ Rescure_DNSBL ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 500 500 148 0 0 352 ---------------------------------------------------------------------- ------------------------------------------------------------------------ Assembling DNSBL database...... completed [ 02/20/23 10:57:07 ] TLD: TLD analysis............. completed [ 02/20/23 10:58:16 ] TLD finalize.I have tried it on existing pfSense environments but also on new installations. In both cases exactly the same.
Thanks and greetings
-
@roger-s
This is a known issue. Read the first post from @BBcan177
Full Details hereThere is still a Regression with TLD Wildcard Feature in pfSense+ (possibly in pfSense 2.6 also) due to some recent changes to FreeBSD Grep command. This is being reviewed and will provide more updates as soon as we have a solution.
The only solution for now is to disable Wildcard Blocking (TLD) in pfBlockerNG -
B Beerman referenced this topic on
-
B Beerman referenced this topic on
-
B Beerman referenced this topic on
-
So I tried to update to 3.2.0_2 but the update is failing.
>>> Upgrading pfSense-pkg-pfBlockerNG... Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-pkg-pfBlockerNG: 3.2.0_1 -> 3.2.0_2 [pfSense] Number of packages to be upgraded: 1 2 MiB to be downloaded. pkg-static: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/All/pfSense-pkg-pfBlockerNG-3.2.0_2.pkg: Not Found FailedEDIT: Update works now please disregard.
-
Installed this version on my SG-3100, using DNSBL python mode, with geoIP.
It's working perfectly fine for me, thanks a lot for all your efforts in fixing the problems, BBcan177 and Netgate team.
-
@jrey BBcan177 posted a fix for the download problem: https://www.reddit.com/r/pfBlockerNG/comments/116fuie/temp_workaround_to_get_latest_v320_2_files/
-
Thanks, but I wasn't looking for the 3.2.0_2 package.
The issue for me had nothing to do with the lack of that package. I spent some time working through the code and providing feedback to @BBcan177 over the weekend. A local code change worked out to resolve the issue. So for my case (which was a real problem documented in this thread by the way) I'm not sure that changes made to test the change would be in the 3.2.0_2 package at this point. So I'm unlikely to even try until all my current testing is complete.
PS @BBcan177 I just waiting for a 12:00 time cycle to pass "12hours" and I'll send you the notes. Thanks JR
-
Looks like the update worked now, idk why it was not before.
Does this also fix the PHP errors and freezing of the Pfblockerng widget on the dashboard?
EDIT: Looks like this PHP error is still present, it freezes PfSense+ 23.01 so hopefully its fixed soon. I had to remove the pfblockerng widget and all is well again.
