502 Bad Gateway (nginx) after Update to 2.3

Jailer

Thank you for this info.

Is it possible to force the option 16 ofer ssh?

When i log in my pfsense over SSH i just see the consol input window without any options to choose (1,2,3,…)

service nginx restart

Den

I had the same problem on clean install of 2.3. Restarting PHP-FPM and webConfigurator didn't help, had to reboot.

jahonix

@billyjp:

Is it possible to force the option 16 ofer ssh?

/etc/rc.php-fpm_restart

@Jailer:

service nginx restart

might give you: "Cannot 'restart' nginx. Set nginx_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'."

And "rc.restart_webgui" gives you option 11 "Restart webConfigurator" from console.

cmb

service commands don't do anything, don't run them. If you don't get the console menu, you're not logging in as root. You'll need the sudo package, then run sudo /etc/rc.initial

jahonix

You learn something new every day.
Thanks!

Guest

Disabling the automatic dashboard auto-update check at /system_update_settings.php seems to mitigate the issue for now.

edit: i'm getting more and more the feeling that the issue is somehow ipv6 related.. still investigating.

billyjp

Thanks for this tipp.

Can anyone approve this? I dont want to change to much important on my Main Firewall. But the nginx error comes no every 3 days :-[

luckman212

Just want to chime in since I feel the 2.3.1 release is imminent

I got this 502 Bad Gateway nginx error for the first time today. Had never seen it before. I am actually running a 2.3.2 snapshot according to the dash, based on 2.3.1.a.20160516.0651. I have been tracking the dev branch - Not sure how I landed on 2.3.2 but I assume that whatever fix was in for this should have been in there. So I don't believe that this is fixed. yet.

I do have the auto-update check enabled on my dashboard.

I did leave the dashboard page "up" in a browser window overnight so I guess it was sitting there for a long time refreshing every so often.

I do have the IPSEC dashboard widget turned on, with 2 tunnels that are both "UP"

This is on an SG2440

My logs were filled to the brim with the error below:

nginx: 2016/05/18 07:54:28 [error] 50536#0: *6944 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 2604:2000:xxxx:xx::116e, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "r1.xxx.xxx:8888"

I run my webconfigurator on HTTP port 8888

Not sure if that gives any clues

Screenshot below

Ambivalent

So after having this issue multiple times a day I dug and dug and dug and found the thing that fixed it for me. This will apply to you if, like me you had been leaving the main firewal page open with the widgets. This error appears to be caused by a widget. In my case I had to close my OpenVPN status widget and it stopped killing PHP and apparently that kills the whole thing until you reboot. This may or may not apply to you but it's easy to test and not a terrible workaround for now.

luckman212

I think the IPSec widget also causes this. Not sure why that is. Interesting info- not really a fix but it's a workaround. Sadly, at least for me those 2 widgets are among the most useful ones to see at a glance.

phil.davis

Make sure to retry all the widgets after upgrading to 2.3.1-RELEASE.
Then this can move forward if there are more issues reported.
The problems are related to the widgets doing updates every 10 seconds or so, asking the nginx server to do stuff. nginx gives the PHP to back-end PHP processes to do the work. If those things get delayed (or hung) too much then all the PHP processes get busy and nginx will have to give up.

Christos

Got the 502 Bad Gateway error with 2.3.1 :( :( :(

gordc

Has anyone been able to figure out what is happening with this. I get this every couple of days and the only resolution is to take the 20 minute walk to physically restart the firewall.
I can log in using ssh but the majority of the commands either give an error on execution or don't work. When I log in I don't get a menu. I start the menu with /etc/rc.initial
After that if I use options 11 or 16 I get errors
Restarting webConfigurator…Error: cannot open /var/etc/nginx-webConfigurator.conf in system_generate_nginx_config().

Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 80
PHP ERROR: Type: 1, File: /etc/inc/interfaces.inc, Line: 80, Message: Call to undefined function pfSense_interface_listget()
Fatal error: Call to undefined function gettext() in /etc/inc/rrd.inc on line 60
PHP ERROR: Type: 1, File: /etc/inc/rrd.inc, Line: 60, Message: Call to undefined function gettext()

Killing php-fpm
pkill: signalling pid 737: Operation not permitted
/etc/rc.php-fpm_restart: cannot create /tmp/php_errors.txt: Permission denied
Found XMLRPC lock. Removing.
rm: /tmp/xmlrpc.lock: Operation not permitted
Starting php-fpm
[ERROR] unable to bind listening socket for address '/var/run/php-fpm.socket': Address already in use (48)
[ERROR] FPM initialization failed

If I use option 5 to try to restart the system I get the prompt asking me to continue but the system does not reboot.

I have also tried /etc/rc.initial.reboot
Again I get the prompt asking to proceed but the system does not reboot.

This is costing me too much time.

jahonix

Just upgrade to 2.3.1_1 which was released today.

Honest Bob

I had the same issue even with 2.3.1-1. I dont have the openvpn widget open. I shut down the PFblockerNG and Snort widgets to see if they are the culprit. I am left with system information and interfaces.

samham

since upgrading from 2.2 to 2.3.1 I've been getting "502 Bad Gateway" error at least once a week which I correct by starting PHP-FPM, but this is getting very annoying especially because it causes some of my sessions to drop, does any one know of any permanent fix?

2.3.1-RELEASE-p1
IPsec
OPenVPN
PfBlockerNG

Capture.PNG_thumb

Perforado

Same here. Will try tomorrow without the ipsec widget for reference.

tweek

I'm also seeing the 502 Bad Gateway error. I'm running 2.3.1-RELEASE-p1 (i386 nanobsd) as a direct update from v 2.2.6, hardware is a Soekris net6501. Installed packages are Network UPS Tools v2.3.0 and openvpn-client-export v1.3.8.

When it hangs up I log in via SSH and choose the 16) Restart PHP-FPM item from the text interface. As per others on this and other threads I have removed the IPsec widget from the dashboard to see if that helps.

hekmel

I have received the 502 Bad Gateway error after upgrading to 2.3.1 Release.
I have the IPSec widget open. I will have to restart the firewall after working hours today, and disable the widget and see if that solves anything on our end.

gordc

I am still getting this error even after upgrading to 2.3.1_1
It has happened a couple of times now. I have also noticed that when it happens and I SSH in to the CLI I have to run sudo rc.initial to get the menu up. (If I don't run sudo none of the commands will work). I can then reset PHP and get access.