2.5.1-RELEASE -> 2.5.2rc : XMLRPC method restore_config_section

Luca De Andreis

Hi all,
I've two pfsense virtual nodes on 2.5.1-RELEASE.
Cluster running on management 8443 (not standard).
Work fine, no error, no alert.

I've tried to update to 2.5.2.r.20210615.1851 and... ONLY when I switch:

Enter Persistent CARP Maintenance Mode <> Leave Persistent CARP Maintenance Mode

I can see an alert:
A communications error occurred while attempting to call XMLRPC method restore_config_section:

Any idea ?
I repeat, on 2.5.1-RELEASE works correctly

Very thanks

Luca De Andreis

This post is deleted!

jimp

At first I couldn't reproduce it, but now I can every time. Not sure what changed. It didn't show up in the logs or notices initially, but after I restarted php (16/11 from the console) it was logged.

I noticed that the error message was incomplete. I managed to fix that up but it isn't terribly helpful:

Jun 16 10:49:20 	php-fpm 	66863 	/rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section: Error writing request

I checked out the request body and it is the whole config sync, though I checked and the configuration is syncing.

On the target system, the XMLRPC lock file in /tmp/xmlrpc.lock is left, so something on the secondary node must be bailing before it finishes the sync process.

There is nothing in the logs on the secondary, however, to indicate a problem.

juliokele

same Alert after "Leave Persistent CARP Maintenance Mode":

A communications error occurred while attempting to call XMLRPC method restore_config_section: @ 2021-06-16 18:57:20
A communications error occurred while attempting to call XMLRPC method restore_config_section: @ 2021-06-16 18:57:21

2.5.2-RC (amd64)
built on Tue Jun 15 18:52:47 EDT 2021
FreeBSD 12.2-STABLE

jimp

How many users do you have on that cluster in the System > User Manager?

Which configuration sections are you having it synchronize?

We're still looking into the cause of the errors but so far I'm the only one able to replicate it, and even then it only happens some of the time. It's very inconsistent.

The XMLRPC sync has a built-in retry mechanism and it almost always succeeds by the second try for me. Though the logs and notices are annoying, it is working.

juliokele

@jimp
two users in User Manager, all options are selected under Configuration Synchronization Settings (XMLRPC Sync). pfBlocker, suricata, haproxy settings are synchronized.

Luca De Andreis

Two users in User Manager, all options are selected, no additional packages installed.
Yes is very inconsistent, the problem does not always arise.

jimp

I tried some older snapshots and the problem started between 20210609-0300 and 20210610-1655.

I have been able to narrow this down further to a single package update:

php74-pear-HTTP_Request2: 2.3.0,1 -> 2.4.2,1

If I go back to a working snapshot and update only that one package, the XMLRPC request starts failing again.

So we are going to roll that version back in our repository to get things working again, and investigate the problem in more depth after 2.5.2.

jimp

The newest RC build is up now and it includes the workaround for this. It's been solid for me, I'm unable to make the sync fail now.

When you get a chance, update and try it out.

Luca De Andreis

@jimp
Yes, release 2.5.2.r.20210617.1709 works great !!!!

juliokele

@jimp
Yes, it works.