New 502 Bad Gateway
-
i dont use pfblocker and have 502 bad gateway errors on all most all my upgraded pfsense's
what should i do. -
I updated the pfBlockerNG package this AM and rebooted pfSense but I just got the 504 Gateway Time-out again. Not sure if there is something else going on?
i dont use pfblocker and have 502 bad gateway errors on all most all my upgraded pfsense's
what should i do.Gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and post it as a text file attachment.
-
I've been working with BBcan177 to help test some possible fixes in the pfBlockerNG package and just got the 502 error on one of the two pfSense machines that I maintain. I've attached a text file containing the output requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994. The only modification I made to the output from the commands was to replace instances of my WAN IP with <wan_ip>.
I don't know whether this is consistent with others' experiences with this error, but from all external appearances the machine still seems to be operating. By that I mean that Internet access is still operational. I just can't get to the web GUI. I can SSH in normally using a non-root user, but when I try to execute any command requiring root privileges (e.g. sudo pftop), the console hangs with no output, and does not respond to CTRL-C or CTRL-Z. I performed an upgrade from 2.3.4 so both of my machines are still UFS. I'm glad to provide any other information that may be useful.
pfSense_info_dump.txt</wan_ip>
-
It seems like this 502 issue has been around for a along time. Here's a thread on it starting from April 2016: https://forum.pfsense.org/index.php?topic=110515.0. And here is a non-pfSense-specific article about it from May 2014: http://jvdc.me/fix-502-bad-gateway-error-on-nginx-server-after-upgrading-php/. Another from 2012 https://wildlyinaccurate.com/solving-502-bad-gateway-with-nginx-php-fpm/. Presently I'm unable to attempt any of the suggestions from these pages because my machine exhibiting the problem is remote, and I cannot execute any commands requiring root privileges via SSH without the console freezing.
I have never seen the 502 error myself before 2.4.0 and I've run pfSense for many years. I only started running pfBlockerNG about 6 months ago or so, but the supposed link to pfBlockerNG specifically seems like it may be a red herring. The only common threads seem to be that it involves bad blood between PHP and nginx, and within the context of pfSense specifically users seem to often report it being linked to widgets on the dashboard.
-
I've been working with BBcan177 to help test some possible fixes in the pfBlockerNG package and just got the 502 error on one of the two pfSense machines that I maintain. I've attached a text file containing the output requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994. The only modification I made to the output from the commands was to replace instances of my WAN IP with <wan_ip>.
I don't know whether this is consistent with others' experiences with this error, but from all external appearances the machine still seems to be operating. By that I mean that Internet access is still operational. I just can't get to the web GUI. I can SSH in normally using a non-root user, but when I try to execute any command requiring root privileges (e.g. sudo pftop), the console hangs with no output, and does not respond to CTRL-C or CTRL-Z. I performed an upgrade from 2.3.4 so both of my machines are still UFS. I'm glad to provide any other information that may be useful.</wan_ip>
Yours is definitely still DNSBL. Make sure you are on the latest version of pfBlocker there (2.1.2) and that you have restarted the box after updating pfBlocker.
-
It seems like this 502 issue has been around for a along time. Here's a thread on it starting from April 2016: https://forum.pfsense.org/index.php?topic=110515.0. And here is a non-pfSense-specific article about it from May 2014: http://jvdc.me/fix-502-bad-gateway-error-on-nginx-server-after-upgrading-php/. Another from 2012 https://wildlyinaccurate.com/solving-502-bad-gateway-with-nginx-php-fpm/. Presently I'm unable to attempt any of the suggestions from these pages because my machine exhibiting the problem is remote, and I cannot execute any commands requiring root privileges via SSH without the console freezing.
I have never seen the 502 error myself before 2.4.0 and I've run pfSense for many years. I only started running pfBlockerNG about 6 months ago or so, but the supposed link to pfBlockerNG specifically seems like it may be a red herring. The only common threads seem to be that it involves bad blood between PHP and nginx, and within the context of pfSense specifically users seem to often report it being linked to widgets on the dashboard.
There are a large number of things that could cause a 502 error. This specific case is from DNSBL in pfBlocker. There are likely other issues out there that have not yet been fully identified, including one that can happen from the IPsec status/widget in some cases that are hard to reproduce. pfBlocker is not a red herring for the case discussed in this thread, but others who may have an unrelated 502 are assuming their issue is related and chime in. Having a 502 doesn't mean this thread is relevant, having a 502 with supporting detail matching the other symptoms exactly does. It's currently the most common cause, but not the only cause.
Without some supporting detail we can't track down what happened. Gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and put it in a text file attached to a response. If you can't login as root directly, install and configure the sudo pkg so you can login as another user and switch to root that way. For nearly everyone else here, hitting CTRL-Z and then running /bin/tcsh allowed them to use the console or ssh as root.
-
Thanks for the additional info Jim. I didn't intend to muddy the waters; this just seems like a tough one to nail down. I'm not sure why CTRL-Z isn't working in my case, but I've tried numerous times. I also already have sudo installed, but attempting to execute anything with sudo makes the console unresponsive as well. I think I'm dead in the water until I can get physical access to the machine later today. When I can though, I'll reboot it and go from there. I had updated to the latest pfBlockerNG, but had only restarted the service instead of the machine. So perhaps my issue is that a full reboot is required. Also I did gather the information requested from https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and attached it as a TXT file to my earlier post. But I wasn't sure if that request was directed at me specifically or just anyone still experiencing the issue.
-
I'm not sure why CTRL-Z isn't working in my case, but I've tried numerous times. I also already have sudo installed, but attempting to execute anything with sudo makes the console unresponsive as well.
Try leaving an ssh session and the console sitting at a root shell so you don't need to get past that step later (just be sure to secure your computer if you leave those open). And rather than running programs through sudo, try launching a shell with sudo: "sudo -s"
I had updated to the latest pfBlockerNG, but had only restarted the service instead of the machine. So perhaps my issue is that a full reboot is required.
A reboot is best because there will still be stale locks out there otherwise.
Also I did gather the information requested from https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and attached it as a TXT file to my earlier post. But I wasn't sure if that request was directed at me specifically or just anyone still experiencing the issue.
It's directed at everyone. Even if you took it before, when you are on the latest pfBlocker it might be different. We can't assume the outputs are identical when other factors have changed.
-
Unfortunately (in this case, at least) I have root login disabled for SSH. So my only recourse is to SSH in non-root and then try sudo. But I don't even receive a password prompt when I try to run things with sudo (I did just try "sudo -s" too). It just hangs as soon as I press Enter. However, not a big deal, I'll be able to reset the machine later this evening and I'll see how things go from there. If I get a 502 again following that, I will re-gather and re-post the requested information. If it might be helpful to have that information following a reboot but before the 502 error occurs, let me know and I'd be happy to grab that too. Thanks again.
-
Updated pfBlocker and am getting a 504 Gateway Time-out, after letting the page load for 3-4 minutes. Uptime was roughly 4 hours. Rules are also set up auto update every 4 hours I believe.
Output attached.
-
Updated pfBlocker and am getting a 504 Gateway Time-out, after letting the page load for 3-4 minutes. Uptime was roughly 4 hours. Rules are also set up auto update every 4 hours I believe.
Output attached.
The output still shows it's hung up on the DNSBL daemon. You might manually check the index.php page for dnsbl (linked earlier in this thread) and see if it's the updated code or not.
-
Updated pfBlocker and am getting a 504 Gateway Time-out, after letting the page load for 3-4 minutes. Uptime was roughly 4 hours. Rules are also set up auto update every 4 hours I believe.
Output attached.
The output still shows it's hung up on the DNSBL daemon. You might manually check the index.php page for dnsbl (linked earlier in this thread) and see if it's the updated code or not.
The code had not been removed from the index.php file after installing the update. I manually edited it and will report back with any issues.
-
Reading through the thread and I get a similar issue but with 504 Gateway Timeout not a 502. Just wanted to confirm this is the same problem.
PfSense locks up, 504 timeout on web admin. When I go to the machine and use the console via a keyboard, i try and initiate a shell via option 8 and get
Password for admin@pfSense.rice-bucket.com:
*** Welcome to pfSense 2.4.0-RELEASE (amd64) on pfSense ***And nothing further. The same if I ssh in. Via the shell I can use CTRL-Z and type reboot to reboot the server.
I am using pfBlocker, but did turn it off. However did not restart and its now locked again. I will reinstall pfsense fresh (this is an upgrade from 2.3 to latest 2.4) and restore from backup for now.
-
@ak:
Reading through the thread and I get a similar issue but with 504 Gateway Timeout not a 502. Just wanted to confirm this is the same problem.
PfSense locks up, 504 timeout on web admin. When I go to the machine and use the console via a keyboard, i try and initiate a shell via option 8 and get
Password for admin@pfSense.rice-bucket.com:
*** Welcome to pfSense 2.4.0-RELEASE (amd64) on pfSense ***And nothing further. The same if I ssh in. Via the shell I can use CTRL-Z and type reboot to reboot the server.
I am using pfBlocker, but did turn it off. However did not restart and its now locked again. I will reinstall pfsense fresh (this is an upgrade from 2.3 to latest 2.4) and restore from backup for now.
If it happens again, hit CTRL-Z, type /bin/tcsh, and then gather the info mentioned in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994
-
@ak:
Reading through the thread and I get a similar issue but with 504 Gateway Timeout not a 502. Just wanted to confirm this is the same problem.
PfSense locks up, 504 timeout on web admin. When I go to the machine and use the console via a keyboard, i try and initiate a shell via option 8 and get
Password for admin@pfSense.rice-bucket.com:
*** Welcome to pfSense 2.4.0-RELEASE (amd64) on pfSense ***And nothing further. The same if I ssh in. Via the shell I can use CTRL-Z and type reboot to reboot the server.
I am using pfBlocker, but did turn it off. However did not restart and its now locked again. I will reinstall pfsense fresh (this is an upgrade from 2.3 to latest 2.4) and restore from backup for now.
If it happens again, hit CTRL-Z, type /bin/tcsh, and then gather the info mentioned in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994
I just did:
https://pastebin.com/ryQSY9y8
-
@ak:
Reading through the thread and I get a similar issue but with 504 Gateway Timeout not a 502. Just wanted to confirm this is the same problem.
PfSense locks up, 504 timeout on web admin. When I go to the machine and use the console via a keyboard, i try and initiate a shell via option 8 and get
Password for admin@pfSense.rice-bucket.com:
*** Welcome to pfSense 2.4.0-RELEASE (amd64) on pfSense ***And nothing further. The same if I ssh in. Via the shell I can use CTRL-Z and type reboot to reboot the server.
I am using pfBlocker, but did turn it off. However did not restart and its now locked again. I will reinstall pfsense fresh (this is an upgrade from 2.3 to latest 2.4) and restore from backup for now.
If it happens again, hit CTRL-Z, type /bin/tcsh, and then gather the info mentioned in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994
I just did:
https://pastebin.com/ryQSY9y8
That still appears to be pfBlocker DNSBL getting backed up
-
Can someone with the DNSBL active and as the 'probable cause' of 502 issues perhaps try and see if adding code below to the top of the index.php would help?
ignore_user_abort(true); set_time_limit(0);I could reproduce xmlrpc locks sometimes hanging php due to 'lost' locks and made a PR for that:
(https://github.com/pfsense/pfsense/commit/ee0e2507b819c142b7ad6c331d401002188f1ab8), i dont have/use dnsbl myself so cant properly check.. Did try and hit index.php with AB and putting sleeps inside the index.php the code, but couldnt make it 'fail' in a way that it would just completely hang as described in the thread..Not sure if it will help for pfB.. but it is 'somewhat similar' and might be an easy fix.. or not..
-
That still appears to be pfBlocker DNSBL getting backed up
It is, I had a 502 today but unfortunately was not able to get the requested information. If and when it does again I will post it here.
-
Have upgraded to Coreboot 0.11 and Pfblocker 2.1.2 through the console connection. Have no access to the GUI via the normal LAN interface even immediately after a reboot, but can access the GUI login on the DMZ interface but it gives a 504 error timeout when I try to login. Problem only showed up after upgrading from 2.3.4 to 2.4.0
Logs at https://pastebin.com/xn7WMF0W
-
I successfully applied the 2.1.2 pfBlockerNG update. I did a reboot when the update completed. I am able to access the web GUI and SSH with no issues. I'll report back later today to make sure I can still access the web gui and SSH after it has been running for awhile.
It has been several days since I applied the pfBlockerNG 2.1.2 update. Everything is running normal. The upgrade fixed my 502 Bad Gateway and SSH logon issues.
