New 502 Bad Gateway

Hans_P

If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:
/usr/bin/netstat -Ln
/usr/bin/netstat -xn
/usr/sbin/swapinfo -h
/usr/bin/top | /usr/bin/head -n7
/bin/ps uxawwd
/usr/bin/sockstat
Attach the output in a text file as it will be too large to put inline on a forum post.

@jimp From my topic yesterday https://forum.pfsense.org/index.php?topic=138283.0

You referred me to get these command outputs when it happened again and it finally has. I have attached the text file. Let me know if you need anything else and thanks again.

[pfSense Command Outputs.txt](/public/imported_attachments/1/pfSense Command Outputs.txt)

gsmornot

@BBcan177:

I am going to make a PR shortly to hopefully fix this issue….

You could download these two files from my Github gist to get the updated code:
fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"

fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
You will need to restart the DNSBL Service for this to take effect.

A reboot is not required, but would be recommended.

Done and works fine so far as I can tell. Good work!
Side note-have been watching for V3. :-)

jimp

@Hans_P:

You referred me to get these command outputs when it happened again and it finally has. I have attached the text file. Let me know if you need anything else and thanks again.

Looks like the same pfblocker dnsbl issue as others in the thread. There is a pfBlocker update that will be out momentarily that should help, if not fix it entirely.

depster

@gsmornot:

I have not had any issues but based on this post I have disabled DNSBL and pfblocker until the next update. Bummer since this is one of my favorite parts of using pfsense.

I have been up and running for ~10 hours since disabling DNBL. Previsouly, I had just pfBlocker disabled - and it locked up overnight.

klank

So I have disabled pfblocker, that fixed the crashing or hang that was occurring every 8 hours or so. However now the gui and ssh become non responsive after 8 hours or so.

I'm going to try fresh install with ZFS to see if the problems continue.

m0nji

Very interesting. I did not have this 502 Bad Gateway Issue after upgrading to 2.4.0-RELEASE. (Uptime since Upgrade 1 Day 18 Hours).
I just installed now the new Update (Upgrading pfSense-pkg-pfBlockerNG from 2.1.1_11 to 2.1.2)

My Setup:
Kettop Mi19N (32GB SSD UFS Filesystem + 8GB RAM)
PFBlockerNG + DNSBL enabled
MultiWAN (Failover)
IPSEC + OpenVPN

jimp

To anyone experiencing the DNSBL issue in this thread: Update to the latest pfBlocker (2.1.2 or later) as it contains a fix for the locking issue leading to these conditions. After updating, ideally you will want to reboot the firewall to make sure it's starting with a clean slate. At the very least, stop and restart the DNSBL daemon.

I'll edit this into the first post of the thread for visibility.

Hans_P

@jimp:

To anyone experiencing the DNSBL issue in this thread: Update to the latest pfBlocker (2.1.2 or later) as it contains a fix for the locking issue leading to these conditions. After updating, ideally you will want to reboot the firewall to make sure it's starting with a clean slate. At the very least, stop and restart the DNSBL daemon.

Ok, I did install the latest update to pfBlocker (2.1.2) and seemed to install just fine. When I went to restart the box, it is now in a constant restart state. I am able to SSH into it to see what is going on. I can see the commands as the system tries to load, but it will stop at…

"cpu_reset_proxy: Stopped CPU 1" (see attached)

Then it just restarts. It will keep doing this. I am not sure if the update caused this or not so am posting it here. Anyways, I am just trying to factory reset now and load my current configuration, but can't get it to work. I've tried the button method and is still the same. It doesn't make it to the "main" command menu to run option #4, but I can pause it at the boot menu (see attached). Any suggestions on what to look for? Is a Factory Reset suggested? Any help would be appreciated. Thanks in advance.

![pfSense error.JPG](/public/imported_attachments/1/pfSense error.JPG)
![pfSense error.JPG_thumb](/public/imported_attachments/1/pfSense error.JPG_thumb)
![pfSense menu.JPG](/public/imported_attachments/1/pfSense menu.JPG)
![pfSense menu.JPG_thumb](/public/imported_attachments/1/pfSense menu.JPG_thumb)

Hans_P

Ok, I did install the latest update to pfBlocker (2.1.2) and seemed to install just fine. When I went to restart the box, it is now in a constant restart state. I am able to SSH into it to see what is going on. I can see the commands as the system tries to load, but it will stop at…

I was able to fix this. I found searching in the forums a command to use…

https://forum.pfsense.org/index.php?topic=135466.0

I performed the steps that @Derelict (Thank You) mentioned and it worked. I was able to reboot pfSense and also able to log into the Web GUI. It immediately came up with a crash reporter and wanted me to submit it the developers, which I did. I hope they can find what happened in my case.

As of right now, I am back up and running and didn't have to reset back to defaults or reload my config. I hope this post will help someone else.

klank

I've had to go back to 2.3 as 2.4 has been way to unstable for me. Even a fresh install with ZFS is having issues.

Omitted

I updated the pfBlockerNG package this AM and rebooted pfSense but I just got the 504 Gateway Time-out again. Not sure if there is something else going on?

Xentrk

I successfully applied the 2.1.2 pfBlockerNG update. I did a reboot when the update completed. I am able to access the web GUI and SSH with no issues. I'll report back later today to make sure I can still access the web gui and SSH after it has been running for awhile.

ha11oga11o

I successfully update 2.1.2 pfBlockerNG . I did NOT reboot my unit jet. I am able to access the web GUI and SSH with no issues for 2 days so far. Before it was hang after 2-3 hours.

It seems it works fine for me.

Thanks

hdejongh

i dont use pfblocker and have 502 bad gateway errors on all most all my upgraded pfsense's
what should i do.

jimp

@Omitted:

I updated the pfBlockerNG package this AM and rebooted pfSense but I just got the 504 Gateway Time-out again. Not sure if there is something else going on?

@hdejongh:

i dont use pfblocker and have 502 bad gateway errors on all most all my upgraded pfsense's
what should i do.

Gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and post it as a text file attachment.

TheNarc

I've been working with BBcan177 to help test some possible fixes in the pfBlockerNG package and just got the 502 error on one of the two pfSense machines that I maintain. I've attached a text file containing the output requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994. The only modification I made to the output from the commands was to replace instances of my WAN IP with <wan_ip>.

I don't know whether this is consistent with others' experiences with this error, but from all external appearances the machine still seems to be operating. By that I mean that Internet access is still operational. I just can't get to the web GUI. I can SSH in normally using a non-root user, but when I try to execute any command requiring root privileges (e.g. sudo pftop), the console hangs with no output, and does not respond to CTRL-C or CTRL-Z. I performed an upgrade from 2.3.4 so both of my machines are still UFS. I'm glad to provide any other information that may be useful.

pfSense_info_dump.txt</wan_ip>

TheNarc

It seems like this 502 issue has been around for a along time. Here's a thread on it starting from April 2016: https://forum.pfsense.org/index.php?topic=110515.0. And here is a non-pfSense-specific article about it from May 2014: http://jvdc.me/fix-502-bad-gateway-error-on-nginx-server-after-upgrading-php/. Another from 2012 https://wildlyinaccurate.com/solving-502-bad-gateway-with-nginx-php-fpm/. Presently I'm unable to attempt any of the suggestions from these pages because my machine exhibiting the problem is remote, and I cannot execute any commands requiring root privileges via SSH without the console freezing.

I have never seen the 502 error myself before 2.4.0 and I've run pfSense for many years. I only started running pfBlockerNG about 6 months ago or so, but the supposed link to pfBlockerNG specifically seems like it may be a red herring. The only common threads seem to be that it involves bad blood between PHP and nginx, and within the context of pfSense specifically users seem to often report it being linked to widgets on the dashboard.

jimp

@TheNarc:

I've been working with BBcan177 to help test some possible fixes in the pfBlockerNG package and just got the 502 error on one of the two pfSense machines that I maintain. I've attached a text file containing the output requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994. The only modification I made to the output from the commands was to replace instances of my WAN IP with <wan_ip>.

I don't know whether this is consistent with others' experiences with this error, but from all external appearances the machine still seems to be operating. By that I mean that Internet access is still operational. I just can't get to the web GUI. I can SSH in normally using a non-root user, but when I try to execute any command requiring root privileges (e.g. sudo pftop), the console hangs with no output, and does not respond to CTRL-C or CTRL-Z. I performed an upgrade from 2.3.4 so both of my machines are still UFS. I'm glad to provide any other information that may be useful.</wan_ip>

Yours is definitely still DNSBL. Make sure you are on the latest version of pfBlocker there (2.1.2) and that you have restarted the box after updating pfBlocker.

jimp

@TheNarc:

It seems like this 502 issue has been around for a along time. Here's a thread on it starting from April 2016: https://forum.pfsense.org/index.php?topic=110515.0. And here is a non-pfSense-specific article about it from May 2014: http://jvdc.me/fix-502-bad-gateway-error-on-nginx-server-after-upgrading-php/. Another from 2012 https://wildlyinaccurate.com/solving-502-bad-gateway-with-nginx-php-fpm/. Presently I'm unable to attempt any of the suggestions from these pages because my machine exhibiting the problem is remote, and I cannot execute any commands requiring root privileges via SSH without the console freezing.

I have never seen the 502 error myself before 2.4.0 and I've run pfSense for many years. I only started running pfBlockerNG about 6 months ago or so, but the supposed link to pfBlockerNG specifically seems like it may be a red herring. The only common threads seem to be that it involves bad blood between PHP and nginx, and within the context of pfSense specifically users seem to often report it being linked to widgets on the dashboard.

There are a large number of things that could cause a 502 error. This specific case is from DNSBL in pfBlocker. There are likely other issues out there that have not yet been fully identified, including one that can happen from the IPsec status/widget in some cases that are hard to reproduce. pfBlocker is not a red herring for the case discussed in this thread, but others who may have an unrelated 502 are assuming their issue is related and chime in. Having a 502 doesn't mean this thread is relevant, having a 502 with supporting detail matching the other symptoms exactly does. It's currently the most common cause, but not the only cause.

Without some supporting detail we can't track down what happened. Gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and put it in a text file attached to a response. If you can't login as root directly, install and configure the sudo pkg so you can login as another user and switch to root that way. For nearly everyone else here, hitting CTRL-Z and then running /bin/tcsh allowed them to use the console or ssh as root.

TheNarc

Thanks for the additional info Jim. I didn't intend to muddy the waters; this just seems like a tough one to nail down. I'm not sure why CTRL-Z isn't working in my case, but I've tried numerous times. I also already have sudo installed, but attempting to execute anything with sudo makes the console unresponsive as well. I think I'm dead in the water until I can get physical access to the machine later today. When I can though, I'll reboot it and go from there. I had updated to the latest pfBlockerNG, but had only restarted the service instead of the machine. So perhaps my issue is that a full reboot is required. Also I did gather the information requested from https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 and attached it as a TXT file to my earlier post. But I wasn't sure if that request was directed at me specifically or just anyone still experiencing the issue.