pfBlockerNG sync not working
-
@IT_Luke Just to clarify, I do not think it was always this way. I just do not know when the behavior changed. We've had pfB in HA for like 9 years now.
-
@Bob-Dig I confirm the failure to sync the updates from the Master at cron sync time (it only syncs with a forced-reload). Something is definitely broken with the change config detection. Will try to snoop around the .inc file and see if there's anything that stands out.
-
@SteveITS I've been using it more or less 6 years I guess and ever since the sync was always cron based (not pushed at the change moment like the native HA sync). I have always been using the devel version as I started using this for the extra features and stuck to it. I guess that possibly some time earlier the XML config sync may have been done differently - I have no idea though it appears awkward to me that the dev would revert to a cron based sync. I recall initially this was even buggy (like it is now!) in that for some reason it would ignore the configured slave node and adding it twice would make it work (apparently this was fixed). In general, while pfblocker is a great and valueable add, it has never excelled in this feat unfortunately.
-
@Bob-Dig @SteveITS I have taken some time to look into the pfblockerng.php and .inc files more in depth and (take this with a grain of salt) there is no calling the XML RPC with the normal update nor cron update (manual or scheduled). The only time the XMLRPC sync function is called is when there is a manual force-reload so it seems this is behavior "by design". However it is possible to add a variable in the array such as "sync_needed" every time the config changes and the user saves the changes (list order, update frequency etc). If the user then does not force a reload, the normal cron job (which calls the function pfblockerng_sync_cron() ) should check if this flag has been set and if so call the XMLRPC Sync function and reset the flag to 0. This way, if one does apply changes and forgets to execute a manual force reload the changes do get synched over to the slave where it will pick them up at next cron run. It is not a major modification and I'm sure @BBcan177 can implement it if requested. I will run some tests on a HA pair when I have more time and see if I can get this functionality working the way I'd like and keep you updated.
-
@IT_Luke Yay for open source. :) I'd try to help but I have a large programming project that moved up from summer to 3 weeks from now. ٩(͡๏̯͡๏)۶
-
@SteveITS @Bob-Dig ok so after doing some tests I I think I have found a very simple yet effective way of getting the Slave node synched without major modifications to the code for the moment and it seems to work fine for me. In short there is just one line to add so that the config is synched over to the slave every time the cron function is run or every time you select to run cron manually. This means that any changes you make to your lists whether ipv4 or DNSBL will be synced over every hour if you forget to sync it manually. I have placed the XMLRPC sync call at the very beginning of the cron function so that when they both run simultaneously on the nodes the slave node should already have received the updated config (but that really depends on their clock sync so it's not) and use that as base of the list updates and avoid missing firewall rule matches. In any case it would fix them at the next run, though you would get the notification which is normal. In order to avoid this with certainty a cron offset of a few minutes should e introduced on all slave HA nodes (similarly to the CARP Skew concept) so that the updated XML config is definitely received before the slave cron update process begins. This should be the orginal behavior of the list update method though I cannot vouch for it as I did not see any way the cron update would run the XMLRPC sync function apart from running a "force update all". In the case of a slave node the mod/patch won't do anything as the function already implements checking if there is a sync target configured or not, and if not just exits.
All you need to do is to add the following at line 544 (which should be blank) to the pfblockerng.php file under /usr/local/www/pfblockerng/ :
pfblockerng_sync_on_changes(); // Sync config to HA slave @cron time to ensure config alignmentFeel free to try and see how it works for you.
-
S SteveITS referenced this topic on
-
Looks like it's line 541 in the new version:
// Function to update Lists/Feeds as per Cron
function pfblockerng_sync_cron() {
global $config, $pfb, $pfbarr;
pfblockerng_sync_on_changes(); // Sync config to HA slave @cron time to ensure config alignment
$hour = date('G');
$dow = date('N'); -
@SteveITS Yep - or line 547 - as long as it's at the beginning of the function!
A few lines have been added/modified around line 492 for the extra Maxmind ID requirement. -
S SteveITS referenced this topic on
-
@Bob-Dig @SteveITS FYI I noticed (alas) that the latest pfBlocker 3.2.0_17 update after the 3.2.0_9 shenanigans still doesn't have the HA sync fix so if you change any settings on the HA Master node in pfB they won't be synched to the slave node @cron time (only if you do a full Reload manually which is not ideal). The line number where to insert the fix is still the same (541 - should be empty) as before so that hasn't changed:
pfblockerng_sync_on_changes(); // Sync config to HA slave @cron time to ensure config alignment
I thought BBCan would have implemented this by now, any idea what goes?
-
@IT_Luke Glad you revived this thread, I am seeing the same thing as well, noticed it last week when I was doing some big changes and nothing synced over.
Force reload always fixes it though. Still annoying to have to go in and do it.
-
@planedrop You can always (re)apply the one liner fix to the pfblockerng.php file under /usr/local/www/pfblockerng/ on the Master node. Haven't had any issues since.
-
@SteveITS @Bob-Dig The latest 3.2.0_18 update still doesn't have the cron update slave sync fix. To apply this again you just have to add the same oneliner to line 590 in the same pfblocker.php file (under /usr/local/www/pfblockerng/):
// Function to update Lists/Feeds as per Cron function pfblockerng_sync_cron() { global $config, $pfb, $pfbarr; **pfblockerng_sync_on_changes(); // Sync config to HA slave @cron time to ensure config alignment** $hour = date('G'); $dow = date('N'); -
@planedrop @SteveITS @Bob-Dig The latest 3.2.0_19 update has fixed the missing XML_RPC sync issue with the slave - it syncs whether you manually run Update or Cron (and of course a full reload) so the workaround is no longer necessary.
-
@IT_Luke OK this is great news, thanks for sharing!
-
And it seems that certain special characters in the password don't work either.
I have a special user for HA Sync. And when syncing, I always got “User or Password not correct” (or something like that) on the console and the sync failed. To rule out that it is the user, I tested it with the “admin” user. This user also has special characters in the password and the result was the same.
I then generated a new password for the HA user, using only letters (upper and lower case) and numbers. The sync now works without any problems.
-
@Beerman That's curious: have you tried resetting the password using your special characters again to see if that still fails? Maybe it's one particular character which breaks it (if I recall correctly there was an issue sometime back which is why I didn't use any).
-
...and 3.2.0_20 released yesterday to fix GEOIP / ASN entry issues.
-
@IT_Luke This may be really dumb, but do we have any timeline ideas for the non-Devel edition?
I moved all my installs back to non-Devel a while back, I can't recall the exact article/post why but my understanding was the main branch was basically becoming the Devel branch. Clearly I misunderstood that but I'm probably not going to back to Devel on any production installs anyway.
-
@planedrop You didn't misunderstand, that's exactly what was said some months ago: the intention of merging and maintaining solely the non devel version and the suggestion to switch over. I stuck with the devel version on all my installs as as a tester myself with my team we always have a devel branch and a production branch, especially when you roll out changes that are hard to debug in the immediate. I suspect that the devel branch will remain, but hey don't take my word for it, it's just a gut feeling. They may very well drop it in the future and keep a "closed" branch meaning that they will only do this internally or with official beta testers, but then they won't get a large feedback test pool like they have now if you see what I mean (that's the beauty of Open Source and having an active community with developers). Maybe something has changed but personally I'm sticking with the devel branch until further notice. I understand also that things are changing for pfSense itself (as you see the releases are going to be available only with the installer and the ISOs will be removed from download from the still accessible repository) so all this must blend in somehow even if apparently pfB is a seperate service (but we all know that it's become one of the most valueable add-ons for the product). I'm sure the updates will be eventually rolled out on the non devel branch - however I don't know anything about the timeline or intentions and why these initial hints regarding the devel version (BBCan can only shed light on that), I'm just speculating based on my intuitions!
-
@IT_Luke OK glad I am not crazy, couldn't really find that much info on it Googling for some reason (though Google has become terrible recently so I'm not totally surprised) but knew I had seen it somewhere.
I will probably move back to Devel on my main testing installation so I can check out some of the new stuff until further notice.
