504 Gateway Time-out (nginx) - occurs whilst pfSense gateway isn't available
-
Hi,
We are currently bench testing pfSense with High Availability Sync and CARP after using in our office for general internet access for 6+ months.
In our testing environment pfSense is configured with a gateway but it's currently not available, thus it has no internet. We have a primary with sync enabled to the secondary (via dedicated Ethernet port) and we are using CARP.
This problem typically occurs on our secondary pfSense server, whilst applying changes to the primary over the course of an hour you will eventually get this problem. The sync from primary to secondary no longer works (A communications error occurred while attempting XMLRPC sync), and the secondary is no longer accessible via the Web GUI with error 504 Gateway Time-out (nginx).
We have also triggered this problem on both pfSense servers by going to System -> Update -> Update Settings -> 'Disable the automatic dashboard auto-update check' then click Save. Yet to confirm if changing this value fixes the issue in the long run, will confirm once we know.
In the logs we see these errors around the time the problem occurs (c.c.c.c is my LAN IP, h.h.h.h is pfSense LAN IP):
nginx: 2016/10/13 15:32:02 [error] 24041#100197: *1043 upstream timed out (60: Operation timed out) while reading response header from upstream, client: c.c.c.c, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "h.h.h.h"Temp Workaround: SSH to console then select 16 followed by 11 to fix temporarily
Other threads with this problem:
https://forum.pfsense.org/index.php?topic=110121.0
https://forum.pfsense.org/index.php?topic=110070.0
https://forum.pfsense.org/index.php?topic=110116.0Can confirm this is not resolved in 2.3.1_1
Would be nice if there was a permanent fix before we bring pfSense into production.
Anyone else experiencing this issue, ideas?
Thanks,
Shaun -
Just logged into the Web GUI this morning and found the following crash report on the secondary, no problems on the primary.
Crash report begins. Anonymous machine information:
amd64
10.3-RELEASE-p9
FreeBSD 10.3-RELEASE-p9 #1 5fc1b19(RELENG_2_3_2): Tue Sep 27 12:26:06 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[13-Oct-2016 16:39:14 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[13-Oct-2016 16:39:14 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80 -
The crash report on the secondary was from yesterday and appears to be the result of making changes to the primary which replicate to the secondary via XMLRPC. We're able to re-produce this problem by updating firewall rules on the primary, each time we update a rule or add a new rule we see two more lines of PHP errors on the secondary but the rules are being correctly added\updated.
Crash report begins. Anonymous machine information:
amd64
10.3-RELEASE-p9
FreeBSD 10.3-RELEASE-p9 #1 5fc1b19(RELENG_2_3_2): Tue Sep 27 12:26:06 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[14-Oct-2016 08:15:02 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:15:02 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:15:32 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:15:32 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:15:59 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:16:00 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:16:26 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:16:26 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:16:40 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:16:40 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:17:35 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:17:35 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:17:50 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:17:50 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:18:00 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:18:00 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:18:09 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:18:09 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:19:07 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
[14-Oct-2016 08:19:07 Europe/London] PHP Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80 -
PHP PHP Fatal error Crash reports - Resolved
Rebooting resolved the issue with the re-occurring PHP Fatal error Crash reports, suspect this was a consequence of restarting the PHP service.
–---------
XMLRPC frequently failing to sync and 504 Gateway Time-out (nginx) - Still a problem; only when gateway is unavailable
We were still getting problems with XMLRPC frequently failing to sync and 504 Gateway Time-out (nginx) until we made the default gateway available with an active internet connection, this problem only seems to occur when the gateway is unavailable. I've also observed that pages in the pfSense Web GUI are much quicker to load when the default gateway is available, not sure why this should have any impact.
This isn't ideal for production environments and could be very misleading during an outage, if you're active gateway is down you need your firewall to be responsive so you can restore connectivity as soon as possible, you don't want to be side tracked by problems on the firewall.
Any ideas what could be causing this?
Thanks,
Shaun -
It would appear that 'Disable the automatic dashboard auto-update check' hadn't saved earlier when we attempted to disable auto update checks, enabling this option seems to fix the problem and would seem like the logical workaround.
-
It seems this is still a problem despite disabling automatic dashboard updates although would appear to be less frequent.
As before the problem only occurs when the default gateway isn't available.
The temporary fix is to either restart pfSense or restart the PHP service.
-
I'm suffering the same behaviour but I'm unable to 'Disable the automatic dashboard auto-update check'. When I click on the check box web gui freezes and then times out the same way you explained.
Only happening on the second box when no gateway is available.
On the first box I can apply changes but everything works really slow. On this one I do have 'Disable the automatic dashboard auto-update check'. -
Hell I have been getting the same problem, and I have to restart the pfsense computer dew to 504 gateway fail.
I do not use Sync and CARP like you, I have pfsense PC quad core 3.2 gig 6gig ram 120 HDD it has 1 WAN in and 2 Lan out with load balancing to a dual wan router that also has load balancing as well for my internal network, but primary Lan1 on pfsense PC drops out and then I have to reset pfsense computer to get it back.If I was a business running servers I would find this a big problem.