504_Gateway_Time-Out/Nginx problem

Not sure what happened but recently if I try to connect to the GUI or while I leave my dashboard on my monitor. I notice that theres a "Gateway timedout" with something to do with nginx below the message. I use a host override that points to my LAN IP of my home network of my pfSense box but haven't really touched the settings of this in awhile.

System Logs show this but with various widgets multiple times.

nginx: 2016/04/16 01:38:05 [crit] 19682#0: *53697 connect() to unix:/var/run/php-fpm.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.20, server: , request: "GET /widgets/widgets/thermal_sensors.widget.php?getThermalSensorsData=11460784783922 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:",
```followed by the hostname and the referrer.

Only way to fix or get access again is to go to the console hit 16 +11 and back in business. Oddly it only occurs after midnight and network runs fine regardless. I will also post a picture once I'm able to catch this problem again.

Edit: just happened again and only takes option 16 in console menu to get the GUI back up.
![2016-04-16.png](/public/_imported_attachments_/1/2016-04-16.png)
![2016-04-16.png_thumb](/public/_imported_attachments_/1/2016-04-16.png_thumb)

MMapplebeck

I'm having this exact same problem on 3 SG appliances, and one VM.

It happens almost every day on mu home VM, and the appliances has happened once each since upgrading to 2.3

Does anybody know what command is issued by using option 16? I'd like to just add a cron until this issue is verified and hopeully makes it into 2.3.1

vbentley

I have had this happen once in the last eight hours.

@Marc:

I'm having this exact same problem on 3 SG appliances, and one VM.

It happens almost every day on mu home VM, and the appliances has happened once each since upgrading to 2.3

Does anybody know what command is issued by using option 16? I'd like to just add a cron until this issue is verified and hopeully makes it into 2.3.1

I might know of the command I'll post it here later tonight if I can find it on my desktop. Thanks for the reply though ;D

vbentley

I just had a 502 Bad Gateway from nginx when trying to access the webGUI.
SSH to console and selecting option 16 to restart PHP-FPM regains webGUI access.

jahonix

https://forum.pfsense.org/index.php?topic=110116.msg613227#msg613227
similar but ssh access only via OPT interface, not from Lan…

MMapplebeck

Close, but not quite, I know in my case on all 4 hosts, traffic is still being routed, everything works fine, I can even access lightsquid on them, just no access to the nginx web interface.

@Marc:

Close, but not quite, I know in my case on all 4 hosts, traffic is still being routed, everything works fine, I can even access lightsquid on them, just no access to the nginx web interface.

Same here; I can ssh into the console so I can enter the command number 16 to restart php-fpm. Can't find the command for this so I can use cron to issue it so often as someone gave the idea here. Anyone know what the issue is or a way around this?

jahonix

@cmb:

Only thing I'm aware of there is if the system doesn't have Internet access, update checks can pile up and hang the GUI, resulting in a 504.
https://redmine.pfsense.org/issues/6177

That should be your problem

Has anyone seen if rebooting or gitsyncing your pfsense box works? I'll try it out tomorrow if no one hasn't; but it could be just a first boot up bug of 2.3 release.

MMapplebeck

Rebooting works as well, however, SSH in and restart php-fpm is easier, and does not cause any connection downtime. This is not just from first boot after an upgrade, in fact, my home VM is built from the 2.3 release iso and still shows the problem.

@Marc:

Rebooting works as well, however, SSH in and restart php-fpm is easier, and does not cause any connection downtime. This is not just from first boot after an upgrade, in fact, my home VM is built from the 2.3 release iso and still shows the problem.

Seems you're right as rebooting still brings up the same error. Almost lost my IPSec too >:( good thing I wasn't busy and ran into this to figure out how to fix that VPN. I also installed on a brand new machine fresh and later uploaded my config file.

Just curious does anyone have a host override in anyone of your systems?
Edit:
Nvm it would seem that still didn't solve the issue.

zanon

Hi,
same here .
I am trying to install clean 2.3 to replace my office box. Server board is Avoton A1SAi-2750F. installation went OK, but during setup starts to receive a lot of "504 Gateway Time-out " nginx erros .
its totally random and its not related to any specific page or form.

tail -f /var/log/nginx-error.log


2016/04/23 19:07:22 [error] 16067#0: *4538 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.104.132, server: , request: "GET /widgets/widgets/system_information.widget.php?getupdatestatus=1 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.104.11:11443", referrer: "https://192.168.104.11:11443/"
2016/04/23 19:07:29 [error] 16067#0: *4547 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.104.132, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.104.11:11443", referrer: "https://192.168.104.11:11443/"
2016/04/23 19:07:42 [error] 16067#0: *4550 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.104.132, server: , request: "POST / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.104.11:11443", referrer: "https://192.168.104.11:11443/"

2016/04/23 20:15:58 [error] 16067#0: *4581 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.104.132, server: , request: "GET /widgets/widgets/system_information.widget.php?getupdatestatus=1 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.104.11:11443", referrer: "https://192.168.104.11:11443/"

2016/04/24 12:35:52 [error] 19734#0: *293 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.104.132, server: , request: "POST /interfaces.php?if=opt3 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.104.11:11443", referrer: "https://192.168.104.11:11443/interfaces.php?if=opt3"

During installation most noticeable changes i made are :
1. Moved tmpfs to ram
2. activated aes ni and thermal for intel
3. disable ipv6 but preffer ipv4
4. enable all HW offload on network ifaces.

I could send config file or allow vpn connection for investigation.

I tried going to the 2.3.1 build and can confirm this issue transfers over as of the latest version of 2.3.1. I have how ever; removed all of my widgets (except system info since one widget is needed to still be up), rebooted my box, and then re added my widgets back. I didn't see any problems in the system logs about nginx before rebooting and re adding my widgets back. Will continue to test this method further and return if resolved or not.

MMapplebeck

If this is related to the below linked bug, it has been logged and assigned as high priority for 2.3.1, however, no work has been done on it yet(hence the problem being in the snaps). Has anybody had any luck figuring out what command can be used to restart php-fpm? It's starting to get really annoying as OpenVPN also breaks(now I've had to open up SSH externally so I can restart php-fpm remotely, I'm not a fan of that), this is an issue in our production environment.

https://redmine.pfsense.org/issues/6177

@Marc:

If this is related to the below linked bug, it has been logged and assigned as high priority for 2.3.1, however, no work has been done on it yet(hence the problem being in the snaps). Has anybody had any luck figuring out what command can be used to restart php-fpm? It's starting to get really annoying as OpenVPN also breaks(now I've had to open up SSH externally so I can restart php-fpm remotely, I'm not a fan of that), this is an issue in our production environment.

https://redmine.pfsense.org/issues/6177

Not sure if its exactly the same but I haven't ran into this nginx issue (so far). It usually throws so many log errors by midnight and thru the day till it breaks. I noticed after the procedure I did that my ipsec widget was still throwing up the nginx error. I removed the widget and haven't ran into an issue. If you can reproduce my workaround (2.3.1 might not be necessary) but try to remove all the widgets if possible (reboot if possible), and if any widgets are throwing up the error remove them for awhile. Periodically re add them to see if waiting it out will get them to work well with each other. I have OpenVPN and IPSec and have just recently been using IPSec more often. I can't say if I can't access either VPN while my issue persist as I can even ssh locally from my main computer to my pfsense box to issue the php restart. Lastly Im not sure if there is a command (at least currently known) that can automatically issue a php restart.

MMapplebeck

I just found another topic discussing this very same issue, apparently it is caused by the IPSec widget.

https://forum.pfsense.org/index.php?topic=110116.0

AEITS_Inc

@Marc:

I'm having this exact same problem on 3 SG appliances, and one VM.

It happens almost every day on mu home VM, and the appliances has happened once each since upgrading to 2.3

Does anybody know what command is issued by using option 16? I'd like to just add a cron until this issue is verified and hopeully makes it into 2.3.1

Option 16 runs /etc/rc.php-fpm_restart

Option 11 runs /etc/rc.restart_webgui

I added both of them to an hourly cron job a minute apart, and I have not been locked out by a 50? error since. Not really a fix, but it does seem to be a workaround of the problem.

For completeness, if anyone doesn't want to change the crontab using shell access, there is a cron package that will do the same thing from the web_gui.

wm408

Hi All,

There are some notes back in this thread also regarding widgets and also auto update: https://forum.pfsense.org/index.php?topic=109812.msg616466#msg616466

Just Fyi.