CARP Causing Fatal Error and causing PHP-FPM to crash



  • Hi all,

    I am experiencing some strange behaviour when CARP is running. It's a simple setup, nothing complicated. I have a CARP cluster (2 pfSense 2.3). When CARP is configured, there appears to be issues copying the XML file across to the backup. I receive the following crash report on the primary:

    Primary
    _Crash report begins.  Anonymous machine information:

    amd64
    10.3-RELEASE
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    PHP Errors:
    [29-Apr-2016 09:45:14 Etc/UTC] PHP Fatal error:  Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205
    [29-Apr-2016 09:45:14 Etc/UTC] PHP Fatal error:  Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205

    Filename: /var/crash/minfree
    2048_

    The primary will then hang and I receive a:
    504 Gateway Time-out - nginx

    In the system logs of the primary, I get:
    nginx: 2016/04/29 10:06:59 [error] 26627#0: *239 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.20.1.12, server: , request: "GET /firewall_virtual_ip.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.20.1.15", referrer: "https://10.20.1.15/"

    On the secondary, I receive the below
    Secondary
    _Crash report begins.  Anonymous machine information:

    amd64
    10.3-RELEASE
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    PHP Errors:
    [29-Apr-2016 09:42:20 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431
    [29-Apr-2016 09:42:20 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:42:20 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431
    [29-Apr-2016 09:42:45 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431
    [29-Apr-2016 09:42:45 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:42:45 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431
    [29-Apr-2016 09:43:33 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431
    [29-Apr-2016 09:43:33 Etc/UTC] PHP Stack trace:
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  1. {main}() /usr/local/www/xmlrpc.php:0
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  2. XML_RPC_Server->XML_RPC_Server() /usr/local/www/xmlrpc.php:585
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  3. XML_RPC_Server->service() /etc/inc/xmlrpc_server.inc:347
    [29-Apr-2016 09:43:33 Etc/UTC] PHP  4. header() /etc/inc/xmlrpc_server.inc:431_

    The secondary GUI will also then timeout with the 504 error.

    In the system logs of the secondary, I get:
    nginx: 2016/04/29 10:00:07 [error] 22195#0: *234 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.20.1.12, server: , request: "POST /pkg_mgr_installed.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.20.1.13", referrer: "https://10.20.1.13/pkg_mgr_installed.php"

    Restarting PHP-FPM from the command line gets the GUI back up and running again, at least until I make additional changes that are needed to be sync'd.

    Any help here would be much appreciated.



  • Does anyone have any ideas here? This occurs on 2 separate firewall clusters


  • Rebel Alliance Developer Netgate

    Can you try a 2.3.1 snapshot on one of those HA pairs? There was a fix or two for XMLRPC last week or so, would be worth trying out.


Log in to reply