HA XMLRPC sync appears to “merge” but does not actually write changes on the Backup

w0w

pfSense Plus in HA (Primary ⇄ Backup) with XMLRPC config sync enabled. pfBlockerNG present. Sync logs show normal activity on the Backup node.

XMLRPC sync runs and claims sections were merged, but the Backup’s config.xml content does not change (only the <revision> timestamp updates), I have checked this with config history. New rules/sections do not appear on the Backup at all, same for aliases.

After editing on the Primary (e.g., adding a simple LAN rule with a unique description), the same object should be present in the Backup GUI and in /cf/conf/config.xml. This isn’t working now, and I’m not sure when it broke—maybe on a 25.11 snapshots, or even earlier.

marcosm

Can you reproduce that starting with a minimal config? It's hard to say what could be happening without reproducible steps. It works in my testing.

SteveITS

@w0w in pfB the sync happens on a force reload only. Theres a one line fix to have it happen at cron intervals…see this thread:
https://forum.netgate.com/topic/179060/pfblockerng-sync-not-working/50

(The line number has changed over time)

w0w

I will not be able to try a minimal configuration in the near future, unfortunately. But I might have some time to dig a bit deeper. For now, at least I am sure that the receiving side receives everything, and I can even see my test rule in the dump, but it is not clear why the block is not being written… Maybe it is failing a validation check.

marcosm

If you're referring to the changes from pfBlockerNG then it's likely the cron thing already mentioned. Otherwise something else to try is temporarily removing packages from both nodes and testing.

w0w

@marcosm
I have removed only pfBlocker, and the configuration has synced successfully.

w0w

Even with synchronization completely disabled, simply having pfBlocker installed prevents synchronization between the firewalls.

SteveITS

@w0w oh do you mean any change, not just pfB? Then disregard my post above. That’s only pfB.

w0w

@SteveITS said in HA XMLRPC sync appears to “merge” but does not actually write changes on the Backup:

do you mean any change, not just pfB?

Exactly. Anyway it looks like this bug is related to pfB somehow.

w0w

It looks like config sync stops working when pfBlocker is installed on the secondary node. Even if I completely remove all pfBlocker settings, on a new install sync still stops, even when pfBlocker is not configured at all.

w0w

<package>
			<name>pfBlockerNG</name>
			<descr><![CDATA[Manage IPv4/v6 List Sources into &#039;Deny, Permit or Match&#039; formats.&lt;br /&gt;
			GeoIP database by MaxMind Inc. (GeoLite2 Free version).&lt;br /&gt;
			De-Duplication, Suppression, and Reputation enhancements.&lt;br /&gt;
			Provision to download from diverse List formats.&lt;br /&gt;
			Advanced Integration for Proofpoint ET IQRisk IP Reputation Threat Sources.&lt;br /&gt;
			Domain Name (DNSBL) blocking via Unbound DNS Resolver.]]></descr>
			<pkginfolink>https://docs.netgate.com/pfsense/en/latest/packages/pfblocker.html</pkginfolink>
			<version>3.2.9_1</version>
			<configurationfile>pfblockerng.xml</configurationfile>
			<include_file>/usr/local/pkg/pfblockerng/pfblockerng.inc</include_file>
			<plugins>
				<item>
					<type>plugin_xmlrpc_send</type>
				</item>
				<item>
					<type>plugin_xmlrpc_recv</type>
				</item>
			</plugins>
		</package>

If I remove the section shown below on the secondary firewall, sync starts working again immediately.

			<plugins>
				<item>
					<type>plugin_xmlrpc_send</type>
				</item>
				<item>
					<type>plugin_xmlrpc_recv</type>
				</item>
			</plugins>

w0w

This post is deleted!

w0w

I have ended up modifying function pkg_call_plugins from

/etc/inc/pfsense-utils.inc

During XMLRPC config merges the core calls pkg_call_plugins("plugin_xmlrpc_recv", $pkg_sections). In stock code this invokes every installed package that registers a plugin_xmlrpc_recv* handler, regardless of whether that package’s own config is present in the incoming payload. That means unrelated plugins (e.g., pfBlockerNG on the secondary) can run on every merge and interfere with sync.

I modified pkg_call_plugins() in /etc/inc/pfsense-utils.inc to add one small guard:

For plugin_xmlrpc_recv* calls only, call a package’s handler iff the XMLRPC payload contains installedpackages/<pkgname>* keys for that package.

All other plugin types (send, normal hooks, etc.) are untouched.

In other words: only the packages whose config is actually being synced get their recv-hook called. This is just a workaround. Please don’t judge me too harshly.

function pkg_call_plugins($plugin_type, $plugin_params) {
    $results = array();
    $is_recv = (strncmp($plugin_type, 'plugin_xmlrpc_recv', 19) === 0);

    foreach ((array)config_get_path('installedpackages/package', []) as $package) {
        $items = (array)array_get_path($package, 'plugins/item', []);
        if (empty($items)) {
            continue;
        }

        // Derive package code name from configuration file (e.g. "pfblockerng.xml" -> "pfblockerng")
        $cfg = $package['configurationfile'] ?? '';
        $pkgname = $cfg ? substr(reverse_strrchr($cfg, '.'), 0, -1) : ($package['name'] ?? '');

        foreach ($items as $plugin) {
            if (!is_array($plugin) || (($plugin['type'] ?? '') !== $plugin_type)) {
                continue;
            }

            // Minimal guard: on XMLRPC receive, only call the plugin if payload has installedpackages/<pkgname>*
            if ($is_recv) {
                $ip = $plugin_params['installedpackages'] ?? null;
                $has_match = false;
                if (is_array($ip)) {
                    foreach ($ip as $k => $_) {
                        if (strncmp($k, $pkgname, strlen($pkgname)) === 0) {
                            $has_match = true;
                            break;
                        }
                    }
                }
                if (!$has_match) {
                    continue;
                }
            }

            $inc = $package['include_file'] ?? '';
            if (!$inc || !file_exists($inc)) {
                continue;
            }
            require_once($inc);

            $fn = $pkgname . '_' . $plugin_type;
            if (function_exists($fn)) {
                $results[$pkgname] = call_user_func($fn, $plugin_params);
            }
        }
    }

    return $results;
}

marcosm

I installed pfBlockerNG on an HA system and tested with and without the package sync enabled. Config changes made on the primary correctly synced to the secondary in both cases.

First check that the pkg version is correct. See:
https://forum.netgate.com/post/1231033

Then, if you haven't yet, try reinstalling pfBlockerNG.

You can also try testing pfBlockerNG-devel - make sure to make a config backup since that makes config changes to the DNSBL VIP that are incompatible with pfBlockerNG.

w0w

@marcosm

Yes, I’ve tried removing all packages and installing the devel version, and even just installing it without configuring it, and ONLY on the backup node; there was no pfBlocker at all on the primary. All of these variants lead to the same result: if pfBlocker is present on the backup node, there is no synchronization.

The only thing I haven’t tried yet is wiping everything and installing from scratch. With the patch, synchronization of the main settings works in any scenario, as well as package sync, at least for Filer. I haven’t checked pfBlocker itself.

It’s possible that the actual bug is somewhere else, maybe in the rules… but then there’s still the question: why is pfBlocker, for which synchronization is disabled and which is not configured at all, interfering with the main config sync at all? Anyway, these are just my rhetorical questions from a non-expert…

By the way, here are my sync settings:

And on the backup everything is unchecked.

w0w

I installed another secondary pfSense node and configured only the necessary interfaces, VIPs, and HA. I restored only the firewall rules and tried to sync — it worked fine. Then I deleted all rules except those needed on the SYNC interface. After that, I installed pfBlockerNG (without configuring anything). Sync stopped working immediately.
As always reports synced sections... but test firewall rule is not synced. configuration diff shows only date/time changed.

Nov 26 17:41:15 	pkg-static 	26347 	pfSense-pkg-pfBlockerNG-3.2.9_1 installed
Nov 26 17:41:17 	php-fpm 	582 	/rc.filter_configure_sync: Gateway Recovery: killed policy routing states for tier 2 in failOVERWANS
Nov 26 17:41:17 	php-fpm 	582 	/rc.filter_configure_sync: Gateway Recovery: killed policy routing states for tier 2 in IPV6_group
Nov 26 17:42:06 	php-fpm 	84114 	/xmlrpc.php: Configuration Change: (system)@10.0.88.1: Merged in config (dhcpdv6, staticroutes, gateways, virtualip, system, hasync, aliases, ca, cert, crl, dhcpd, dnshaper, dnsmasq, filter, ipsec, kea, kea6, nat, schedules, shaper, unbound, wol sections) from XMLRPC client.
Nov 26 17:42:06 	check_reload_status 	653 	Syncing firewall
Nov 26 17:42:06 	check_reload_status 	653 	Reloading filter

This leads me to think there is some underlying problem on the primary node, or that it is somehow related to the interface configuration. I’m not sure.

marcosm

I've pushed a fix for the HA sync issue. Please update to the latest pfBlockerNG version and test again there.

w0w

@marcosm
Thank you. Updated and now it is working just fine.