CARP Causing Fatal Error and causing PHP-FPM to crash

dramsell · May 3, 2016, 11:36 AM

Hi all,

I am experiencing some strange behaviour when CARP is running. It's a simple setup, nothing complicated. I have a CARP cluster (2 pfSense 2.3). When CARP is configured, there appears to be issues copying the XML file across to the backup. I receive the following crash report on the primary:

Primary
_Crash report begins. Anonymous machine information:

amd64
10.3-RELEASE
FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[29-Apr-2016 09:45:14 Etc/UTC] PHP Fatal error: Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205
[29-Apr-2016 09:45:14 Etc/UTC] PHP Fatal error: Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205

Filename: /var/crash/minfree
2048_

The primary will then hang and I receive a:
504 Gateway Time-out - nginx

In the system logs of the primary, I get:
nginx: 2016/04/29 10:06:59 [error] 26627#0: *239 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.20.1.12, server: , request: "GET /firewall_virtual_ip.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.20.1.15", referrer: "https://10.20.1.15/"

On the secondary, I receive the below
Secondary
_Crash report begins. Anonymous machine information:

amd64
10.3-RELEASE
FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[29-Apr-2016 09:42:20 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:42:20 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:42:20 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:42:20 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:42:20 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431
[29-Apr-2016 09:42:20 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:42:20 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:42:20 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:42:20 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:42:20 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431
[29-Apr-2016 09:42:45 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:42:45 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:42:45 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:42:45 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:42:45 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431
[29-Apr-2016 09:42:45 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:42:45 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:42:45 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:42:45 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:42:45 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431
[29-Apr-2016 09:43:33 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:43:33 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:43:33 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:43:33 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:43:33 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431
[29-Apr-2016 09:43:33 Etc/UTC] PHP Stack trace:
[29-Apr-2016 09:43:33 Etc/UTC] PHP 1. {main}() /usr/local/www/xmlrpc.php:0
[29-Apr-2016 09:43:33 Etc/UTC] PHP 2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
[29-Apr-2016 09:43:33 Etc/UTC] PHP 3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
[29-Apr-2016 09:43:33 Etc/UTC] PHP 4. header() /etc/inc/xmlrpc_server.inc:431_

The secondary GUI will also then timeout with the 504 error.

In the system logs of the secondary, I get:
nginx: 2016/04/29 10:00:07 [error] 22195#0: *234 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.20.1.12, server: , request: "POST /pkg_mgr_installed.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.20.1.13", referrer: "https://10.20.1.13/pkg_mgr_installed.php"

Restarting PHP-FPM from the command line gets the GUI back up and running again, at least until I make additional changes that are needed to be sync'd.

Any help here would be much appreciated.

dramsell · May 3, 2016, 11:30 AM

Does anyone have any ideas here? This occurs on 2 separate firewall clusters

jimp · May 9, 2016, 7:07 PM

Can you try a 2.3.1 snapshot on one of those HA pairs? There was a fix or two for XMLRPC last week or so, would be worth trying out.