pfSense Plus 23.01 XMLRPC Sync problems when addiing CARP VIPs

santanna · Feb 24, 2023, 9:36 PM

Dear all,

I was configuring a HA cluster between 2 pfSense Plus 23.01 firewalls and the HA sync part is ok. When I start creating de CARP VIPs on the primary firewall, this configurations are not sent to the secondary firewall and I get some errors.

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[24-Feb-2023 18:17:00 America/Sao_Paulo] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399
[24-Feb-2023 18:18:10 America/Sao_Paulo] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399
[24-Feb-2023 18:19:21 America/Sao_Paulo] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399
[24-Feb-2023 18:20:31 America/Sao_Paulo] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399
[24-Feb-2023 18:21:42 America/Sao_Paulo] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399

No FreeBSD crash data found.

Any help on this issue? Both firewalls were installed and upgraded to version 23.01. The same configuration works on pfSense 2.6.0-CE.

empbilly · Feb 25, 2023, 9:28 PM

@santanna

How is the HA sync configuration on master and backup? Post a screenshot

santanna · Feb 26, 2023, 9:19 PM

@empbilly

Firewall #1

login-to-view

Firewall #2

login-to-view

empbilly · Feb 26, 2023, 9:19 PM

@santanna

Under Diagnostics > Backup & Restore > Configuration History you have the version of the XMLRPC configuration.

Which versions does it show in master and backup?

santanna · Feb 26, 2023, 11:33 PM

@empbilly

Firewall #1 (Master)

login-to-view

Firewall #2 (Secondary)

login-to-view

roesh · Mar 7, 2023, 6:14 AM

Hello @santanna,

I am running 23.01 too and have the same issue. It seems to be a an error on the XMLRPC receiving side, when it processes Virtual IPs config.

My stack trace:

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[07-Mar-2023 00:09:46 Europe/Berlin] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
  thrown in /usr/local/www/xmlrpc.php on line 399
[07-Mar-2023 00:10:58 Europe/Berlin] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
  thrown in /usr/local/www/xmlrpc.php on line 399
[07-Mar-2023 00:12:10 Europe/Berlin] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
  thrown in /usr/local/www/xmlrpc.php on line 399



No FreeBSD crash data found.

Line 399 is around this code part in xmlrpc.php:

                $vipbackup = array();
                $oldvips = array();
                if (isset($sections['virtualip']) &&
                    is_array($config['virtualip']['vip'])) {
                        foreach ($config['virtualip']['vip'] as $vip) {

Therefore, I deselected "Virtual IP" under "Select options to sync" and at least it does not generate these crash reports anymore. Of course, it wont sync the VIP settings.

PS: I opened a ticket at Netgate's TAC Support.

roesh · Mar 7, 2023, 6:14 AM

It seems to be a known issue: https://redmine.pfsense.org/issues/14034

williwinkie · Mar 8, 2023, 12:15 PM

Same problem here, got two new 1541's[08-Mar-2023 12:23:52 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php:399
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->restore_config_section(Array, 900)
#1 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.restore...', Array)
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#3 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#4 {main}
thrown in /usr/local/www/xmlrpc.php on line 399

Not the first time I've setup HA Systems, usually work straight away but this one is making me go bald

jimp · Mar 8, 2023, 2:35 PM

As mentioned above it's a bug in 23.01 (#14034)

The configuration on the secondary has an empty <virtualips> tag which is tripping up that PHP code. If you go on the secondary and add a dummy VIP, then the sync will succeed and you can delete the dummy entry if the sync doesn't do that on its own.

roesh · Mar 11, 2023, 9:22 PM

@jimp Yes, once the secondary pfsense has some CARP virtual IP addresses, the error vanishes.

Deselecting the "Virtual IP" under "Select options to sync" does not work in the end, if one likes to also use DHCP failover.

nick.loenders · Mar 27, 2023, 12:25 PM

@roesh @jimp

I added another new CARP VIP, but the interface stays DOWN ?