PfBlockerNG not updating lists?
-
Good afternoon,
I seem to have a problem with pfBlockerNG not updating IP blocklists. I can force an update but it skips the list, merely logging "exists".
I created a new list with identical sources, this shows in the widget to have around 400000 IPs, the same number as the original list. As the log read "exists" I tried moving the list .txt file in /var/db/pfblockerng/deny and ran an update, this I think forced the list to recreate and the original list now has around 500000 IPs. The copy list still shows 400000 odd IPs.
Does anybody else see this behavior or is it just me? I am not in front of the pfSense right now but I did check for pfSense and package updates so versions are latest. Any thoughts appreciated.
-
Hi jonesr,
Did you set the "Update Frequency" setting for each pfBNG Alias?
-
Hi!
I have the same problem. The DNSBL seems to update fine but the coutrylists do not update. The only way I can update the countrylists is to do a "force reload".Jonna
-
I have the same problem. The DNSBL seems to update fine but the coutrylists do not update. The only way I can update the countrylists is to do a "force reload".
MaxMind is updated the first Tuesday of each month. So there are no changes to make to a Country Alias until either the MaxMind feed is updated, or you select/de-select Countries. So what you are experiencing is normal…
-
Thanks
Jonna -
Hi jonesr,
Did you set the "Update Frequency" setting for each pfBNG Alias?
Hi BBcan177,
Thanks for the response. Yes, both are set to once a day.
This might be completely subjective but if I force an update it completes near instantly, it takes less than a second to reload the page and the "live log viewer" report is already finished, just saying "exists" for each rule. When I renamed the list file as described before and forced an update the report would run through as it progressed, it looked like it was actually "doing something".
-
Thanks for the response. Yes, both are set to once a day.
This might be completely subjective but if I force an update it completes near instantly, it takes less than a second to reload the page and the "live log viewer" report is already finished, just saying "exists" for each rule. When I renamed the list file as described before and forced an update the report would run through as it progressed, it looked like it was actually "doing something".
If you have the alias set to "Once per day", then cron will update that list as per the Cron settings that are defined in the pfBNG General Tab.
You can check the last updated timestamp in the widget, and also at the bottom of the pfblockerng.log file in the "Last Updated List Summary" section.
-
Thanks BBcan177,
Cron is set for the default I think, I don't recall changing it - Every Hour, 0, 0, 0.
The widget said the lists were last updated 5 days ago. I wasn't sure if this meant "hasn't been able to update in 5 days" or "didn't need to update for 5 days, source has not changed" - but it has, if you read on.
The logs don't seem to show any attempt to update since my last forced one 5 days ago. I have forced an update and the widget count remained the same.
I have tried a "force reload" now, the widget count has gone up from ~500000 to ~600000 for both the original and duplicate test lists, so it has taken a manual process to resolve this but I didn't move the list file this time.
Have I just been reading this wrong?
Force Update will download any new Alias/Lists. –Does this just download something if I add an entirely new source list, for example? It makes sense now that it would seem to "finish too fast, not do anything" as I described before.
Force Cron will download any Alias/Lists that are within the Frequency Setting (due for Update).
Force Reload will reload all Lists using the existing Downloaded files. This is useful when Lists are out of 'sync' or Reputation changes were made. --I read this to mean it would not increase or decrease any list count if it was pulling from an existing local file?Thank you for your help, apologies if this is all just my misunderstanding. But I'm still not seeing why my config isn't "reloading" to reflect changes in the source lists automatically. Am I still missing something? I don't see anything that isn't set to update once per day, or any way to set "Cron Reload List Automatically".
-
The widget "Updated" column will report the last timestamp that the List was updated.
If you review your pfblockerng.log, you will see a section that looks like this:
CRON PROCESS START [ 03/19/16 0:15:00 ] [ Alienvault ] Remote timestamp: Sat, 19 Mar 2016 04:00:03 GMT Local timestamp: Sat, 19 Mar 2016 03:30:02 GMT Update found [ Atlas_Attacks ] Remote timestamp: Sat, 19 Mar 2016 00:05:54 GMT Local timestamp: Sat, 19 Mar 2016 00:05:54 GMT Update not requiredSo when Cron runs, it will check the remote timestamp and only update Lists that are newer than the last update.
Definition of the "Force" Commands: (My comments in Red)
Force Update will download any new Alias/Lists. –Does this just download something if I add an entirely new source list, for example? It makes sense now that it would seem to "finish too fast, not do anything" as I described before.
When you run "Force Update", it will download any new Lists that have not been added. After that, the only way to get the lists to update, is during a CRON event when the Lists "Frequency" setting is within the timeframe of the CRON event.
Force Cron will download any Alias/Lists that are within the Frequency Setting (due for Update).
When you run "Force Cron", it will download any Lists that are within the CRON/Frequency settings only.
Force Reload will reload all Lists using the existing Downloaded files. This is useful when Lists are out of 'sync' or Reputation changes were made. –I read this to mean it would not increase or decrease any list count if it was pulling from an existing local file?
Force Reload will also download any new Lists that haven't been previously downloaded, but it will also reset the database if Deduplication/Reputation is used. So when dedup is enabled, the Country Blocklists are configured first, then as each List is reloaded (Using the existing downloaded original file), it will skip any IPs that have already been added to the database for Deny Aliases.
Hope that helps!
-
BBcan177,
Thank you for taking the time to explain in such detail. I noted your post in another thread so I am going to wait for pfSense 2.3 and try this with a fresh install. I'm sure everything is ok and as you describe, but to satisfy my curiosity I will play around some more until it clicks. Thanks again.
-
In regards to what your explaining, there should be no difference in how the package is working in pfSense 2.2.6 or 2.3.
If you wanted to start fresh with the package… goto the pfBlockerNG: General Tab, and unclick "Enable pfBlockerNG" and "Keep Settings"… then hit "Save"… This will remove the database and files but leave the configuration intact... Re-click both checkboxes and "Save"…. Follow that with a "Force Update". You can then review the pfblockerng.log in the Update Tab window.
Depending on how you defined the pfBlockerNG Cron task, its typically defined to run "Every hour". You can goto the "Update Tab" tab, and click the "View" button before the Cron task is scheduled to run, and you will see in Realtime what is occurring…
If there are specifics, copy/paste those into this thread, or send me a PM and I can help guide you further...
