XMLRPC issues - php-fm - Webconfigurator processes?

[xonacs]

Hi everyone!

We have a pfsense setup with carp failover installed.

Actually, we have some 20 setups installed this way.  They are all running version 2.3.2 and running on VMWare ESXi 5.5 or VMWare ESXi 6.

Unfortunately, we are getting a LOT of Failures of this type:

/rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin https://x.x.x.x:443.

I've noticed that option 16, php-fm restart fixes this issues once it happens.  I've also noticed that the issue seems to happen LESS frequently if I increase the number of webconfigurator processes.

This definately seems to be some kind of bug.

The php-fm option 16 restart indicates an xmlrpc lock exists and needs to be removed when I do it.  It seems some connection is not closing properly on the secondary unit in the carp setup.

Can someone help?

[thebembino]

I am also experiencing this issue after I upgraded 4 boxes running various levels of 2.3.1 up to 2.3.2.

Same environment: pfsense on ESXi 5.5/6.0 hosts.

On my end though, It might be an issue with how Suricata is using the XMLRPC sync? Appears to be breaking once a rules refresh is complete. Suricata is setup to NOT ask the target slave to refresh their own rules.

Haven't had any XMLRPC errors from the other pair of 2.3.2 VMs that DO NOT have Suricata

Sep 7 00:31:49 	php-fpm 	7073 	/rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443.
Sep 7 00:31:49 	php-fpm 	7073 	/rc.filter_synchronize: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443.
Sep 7 00:31:49 	php-fpm 	7073 	/rc.filter_synchronize: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103
Sep 7 00:30:33 	check_reload_status 		Syncing firewall
Sep 7 00:30:33 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] The Rules update has finished.
Sep 7 00:30:29 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Snort GPLv2 Community Rules are up to date...
Sep 7 00:30:28 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules file update downloaded successfully.
Sep 7 00:30:09 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2983.tar.gz...
Sep 7 00:30:08 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Sep 7 00:30:06 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Sep 6 17:59:01 	lonrogfw-bluesteel.voyageurtransportation.ca 		nginx: 2016/09/06 17:59:01 [error] 57076#100061: *2654 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/"
Sep 6 17:27:55 	php-cgi 		rc.restart_webgui: Creating rrd update script
Sep 6 17:27:53 	rc.php-fpm_restart 	54531 	>>> Restarting php-fpm
Sep 6 17:27:50 	lonrogfw-bluesteel.voyageurtransportation.ca 		nginx: 2016/09/06 17:27:50 [alert] 28036#100081: *38216 kevent() reported about an closed connection (53: Software caused connection abort) while reading response header from upstream, client: 10.1.100.7, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.1.100.30"
Sep 6 17:27:47 	login 		login on ttyv0 as root
Sep 6 17:26:50 	php-fpm 	44517 	/pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443.
Sep 6 17:26:50 	php-fpm 	44517 	/pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: enablesid-sample.conf
Sep 6 17:26:50 	php-fpm 	44517 	/pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: enablesid-sample.conf
Sep 6 17:26:50 	php-fpm 	44517 	/pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103
Sep 6 17:26:05 	lonrogfw-bluesteel.voyageurtransportation.ca 		nginx: 2016/09/06 17:26:05 [error] 28036#100081: *38216 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /pkg_edit.php?xml=suricata/suricata_sync.xml HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/pkg_edit.php?xml=suricata/suricata_sync.xml"
Sep 6 17:25:35 	php-fpm 	44517 	/pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443.
Sep 6 17:25:35 	php-fpm 	44517 	/pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: dropsid-sample.conf
Sep 6 17:25:35 	php-fpm 	44517 	/pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: dropsid-sample.conf
Sep 6 17:25:35 	php-fpm 	44517 	/pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103
Sep 6 17:24:20 	php-fpm 	44517 	/pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443.
Sep 6 17:24:20 	php-fpm 	44517 	/pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: disablesid-sample.conf
Sep 6 17:24:20 	php-fpm 	44517 	/pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: disablesid-sample.conf
Sep 6 17:24:20 	php-fpm 	44517 	/pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103
Sep 6 17:23:05 	php-fpm 	44517 	/pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443.
Sep 6 17:23:05 	php-fpm 	44517 	/pkg_edit.php: [suricata] XMLRPC sync is starting.
Sep 6 17:23:05 	check_reload_status 		Syncing firewall
Sep 6 17:23:05 	check_reload_status 		Syncing firewall
Sep 6 12:31:22 	php-fpm 	49586 	/rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443.
Sep 6 12:31:22 	php-fpm 	49586 	/rc.filter_synchronize: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443.
Sep 6 12:31:22 	php-fpm 	49586 	/rc.filter_synchronize: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103
Sep 6 12:30:06 	check_reload_status 		Syncing firewall
Sep 6 12:30:06 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] The Rules update has finished.
Sep 6 12:30:06 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Snort GPLv2 Community Rules are up to date...
Sep 6 12:30:06 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules are up to date...
Sep 6 12:30:04 	php-cgi 		suricata_check_for_rule_updates.php: [Suricata] Emerging Threats Open rules are up to date...
Sep 6 12:16:07 	lonrogfw-bluesteel.voyageurtransportation.ca 		nginx: 2016/09/06 12:16:07 [error] 28036#100081: *12636 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/"