504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
-
I think I have the same situation here.
After a few days pfSense has basically stopped responding to clients (from any interface). I can't access it via HTTPS (nginx error 504). It doesn't respond to OpenVPN clients. It doesn't reply on ping.
But it's still passing traffic through it. Clients in the LAN can still access Internet. Clients from the Internet can still access every e-mail server and website located in the LAN.
It is a single-LAN multi-WAN system with PPPoE on all WAN OPT1 OPT2 etc. pfSense v.2.3-RELEASE (i386), upgraded from 2.2.6. The system is a virtual machine on IBM x3650 host server running VMware ESXi v. 4.1.
EDIT: PPPoE on OPT1 OPT2 etc, but DHCP on WAN.
EDIT 2: it still replies to ping and responds to IPsec remote peers (which are also pfSense v.2.3).
EDIT 3: sometimes nginx reports error 502 (Bad Gateway).
EDIT 4: it still responds to SSH connection. Tested on remote pfSense 2.3 devices that stopped responding.
-
It can be fixed by restarting PHP-FPM (console command #16), but not permanently. After a few hours it stops responding to HTTPS (nginx error 504 or 502) again.
EDIT: I think I've found the root cause here: https://forum.pfsense.org/index.php?topic=110070.0. It's the IPsec widget.
-
Could be.
In vanilla mode it runs flawlessly, my restored config has the IPsec widget in dashboard. I'll try to remove that and test again (when I have the time to). -
I had this problem (not exactly but same effect). I tried updating to the 2.3.1 snapshot and problem persisted. I removed every widget (as I saw there were multiple widgets along with IPSec that was being logged), rebooted, waited it out for a couple minutes, and slowly re added widgets. IPSec did pop back up with an nginx error but removed it again and just waited a couple hours longer. As far as I can tell my system running fine for the past +12 hours (and it usually breaks even during the day despite it waiting till midnight to break.
My post is here if you want to see what me and others were getting on this issue.
https://forum.pfsense.org/index.php?topic=110121.0
Edit: Seems it came back to haunt me once again, if you haven't done the above yet try to also clear your browser cache. It would seem the IPSec widget isn't playing well but I have reduced the amount of times this problem occurs (as far as Im experiencing so far). But it seems like its varying in occurrences, widget, and when it brings the webGUI down.
-
Look also here discussing the same problem! https://forum.pfsense.org/index.php?topic=110121.0
-
This was likely fixed in either 2.3.1 or 2.3.1_1 depending on which instance of the issue is responsible.
-
@cmb:
This was likely fixed in either 2.3.1 or 2.3.1_1 depending on which instance of the issue is responsible.
Yes, I can confirm that this issue is resolved (at least I haven't noticed the problem arising lately). Latest updates have been doing wonders :)
-
Hello
This issue is NOT resolved in version 2.3.1_1 . I contantly gett 502 BAD gateway after some time of usage.
I have removed IPSEC gadgets etc and the problem still persists.
I have to reboot the pfsense everyday because of this.Best regards
Toby -
There's no need to reboot, simply restart PHP-FPM and the webconfigurator from the shell menu. Plus you are two releases behind.
-
Hello
Running 2.3.2 P1 , sorry i did not check. But anyway the error is still there. I have it on several units
/Toby
-
Hello
Running 2.3.2 P1 , sorry i did not check. But anyway the error is still there. I have it on several units
/Toby
If you can please run this command, and try to capture the output; when the issue happens ps uxawww either by ssh, or the local terminal. Install pstree for an even better way to find the issue. One of the devs instructed me to do this in order to see what was the cause.
pkg install pstree
rehash
pstree -
I have the same issue here with version:
2.3.2-RELEASE-p1 (amd64)
built on Tue Sep 27 12:13:07 CDT 2016
FreeBSD 10.3-RELEASE-p9After some hours I get the "504 Gateway Time-out" or Bad Gateway error.
-
You are on a totally different version. Nano i386 vs amd64.
Did you restart PHP-FPM from console/ssh? What was the result? -
About ~100% of cases this is fixed by Restart PHP-FPM + Restart webConfigurator from console. (Not that it'd make me love the nginx thing, or the pkg's stupidity of being absolutely unable to work offline.)
And disable the updates checking on dashboard, plus definitely do NOT add the installed packages widget.
-
I know the version thing, I only want to say, that the issue is still alive :D
-Restart PHP-FPM is working every time, but its not really usable to do this so often
But I try to disable the updates checking thing. Installed packages widget is not installed.
Thanks :)
-
@crisdavid:
If you can please run this command, and try to capture the output; when the issue happens ps uxawww either by ssh, or the local terminal. Install pstree for an even better way to find the issue. One of the devs instructed me to do this in order to see what was the cause.
pkg install pstree
rehash
pstreeDon't bother installing that. ps forest/tree format under bsd:
ps auxdww
Gives more info on all processes.
("w" 's are for long line wrapping - 'd' is for forest view (tree) )https://www.freebsd.org/cgi/man.cgi?ps(1)
-
I tried that thing with uncheck updates - didn't work. After some days the http request timed out because ngin didn't respond
-
There was a problem with the IPsec widget a while back. If you have that on the dashboard then try to remove it.
-
bump
Setup :
- 2.3.2-RELEASE-p1 (i386)
- 2013 ALIX Engine
- 4G NanoBSD
- bare install, no-addons
I have been trying to taccle down similar issues as those mentionned in this topic :
-
nginx timeouts ( 504, 502 … ) or slowness
-
nginx timeouts in syslog
-
traffic still going trough, only the UI is impacted
My appliance is coupled w/ a 100Mbps DSL line ( which is well below the maximum troughput it can handle ), and I started noticying that the nginx issues were happening at high troughput ( >= 90Mbps ).
I disabled traffic shaping at first ( as it can be quite CPU intensive ), but it didn't change the behaviour.
The moment the troughput drops to values like 20Mbps, the UI becomes responsive again ( no slowness, no timeouts ).
What can I do to solve these problems ? :-\
-
bump
..Alix..
What can I do to solve these problems ?Bump in place ?
First off: Install a fast/excellent CF like a Transcend Industrial grade.
Then start with a 2.3 from scratch i.s.o. upgrading from a 2.2
Set the CF to read/write all the time.
2.3 works fine with an Alix.
