New 502 Bad Gateway

Steve_B

I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed. Those changes will be in the next 2.4 snapshot (probably early Oct 18th)

I would appreciate any feedback Better/worse/no change . .

Thanks

Xentrk

I also get the 502 Bad Gateway error when trying to access the pfSense Web GUI. I can logon a SSH session. But it hangs after entering my password. I noticed it about 24 hours after upgrade to 2.4. I use pfBlockerNG. Thanks for the information in this thread. I will continue to monitor for updates to the issue.

AhnHEL

@Steve_B:

I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed.

I, myself never had the IPSEC dashboard widget displayed when I was getting the 502 Bad Gateway issue. Was this problem somehow thought to be related? I don't use IPSEC at all.

depster

@jimp:

If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:
/usr/bin/netstat -Ln
/usr/bin/netstat -xn
/usr/sbin/swapinfo -h
/usr/bin/top | /usr/bin/head -n7
/bin/ps uxawwd
/usr/bin/sockstat
Attach the output in a text file as it will be too large to put inline on a forum post.

I believe that I am experiencing this problem; I have attached the requested files to confirm. I had disabled pfBlockerNG and found the system locked up again this morning; however, I had not disabled DNSBL.

Php-FpmSocket_503Lockup.txt

jimp

@depster:

I believe that I am experiencing this problem; I have attached the requested files to confirm. I had disabled pfBlockerNG and found the system locked up again this morning; however, I had not disabled DNSBL.

The output you posted confirms you are hitting this issue, it's definitely the pfBlocker DNSBL process getting stuck like others here in the thread.

jimp

@AhnHEL:

@Steve_B:

I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed.

I, myself never had the IPSEC dashboard widget displayed when I was getting the 502 Bad Gateway issue. Was this problem somehow thought to be related? I don't use IPSEC at all.

In the past, the IPsec widget has been a different source of 502 errors. So it's possible that someone looking at this thread might be hitting that issue and not the DNSBL issue.

gsmornot

I have not had any issues but based on this post I have disabled DNSBL and pfblocker until the next update. Bummer since this is one of my favorite parts of using pfsense.

mloiterman

Can anyone confirm that the issue is not present provided that DNSBL is disabled? Is that all that needs to be done until upgrade?

vomcliff

@BreeOge:

The file is at this location

/usr/local/www/pfblockerng/www/index.php

cd /usr/local/www/pfblockerng/www/

cp index.php index.old = do this so you have a copy of the original before you remove the section.

Edit index.php with your favorite editor, and remove this section at the bottom.

if (!empty($pfb_query)) {
	// Increment DNSBL Alias Counter
	$dnsbl_info = '/var/db/pfblockerng/dnsbl_info';
	if (($handle = @fopen("{$dnsbl_info}", 'r')) !== FALSE) {
		flock($handle, LOCK_EX);
		$pfb_output = @fopen("{$dnsbl_info}.bk", 'w');
		flock($pfb_output, LOCK_EX);
		// Find line with corresponding DNSBL Aliasname
		while (($line = @fgetcsv($handle)) !== FALSE) {
			if ($line[0] == $pfb_query) {
				$line[3] += 1;
			}
			@fputcsv($pfb_output, $line);
		}
		@fclose($pfb_output);
		@fclose($handle);
		@rename("{$dnsbl_info}.bk", "{$dnsbl_info}");
	}
}

Just to add to this thread, I can confirm that the above fix worked for me. I had this issue after pushing out the upgrade to 2.4 and followed the post above (I commented the lines out rather than deleting them). Since then it has been stable and all pfSense routers in my environment have stopped giving the bad gateway error.

BBcan177

I am going to make a PR shortly to hopefully fix this issue….

You could download these two files from my Github gist to get the updated code:

fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"

fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"

You will need to restart the DNSBL Service for this to take effect.

A reboot is not required, but would be recommended.

Hans_P

@jimp:

If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:
/usr/bin/netstat -Ln
/usr/bin/netstat -xn
/usr/sbin/swapinfo -h
/usr/bin/top | /usr/bin/head -n7
/bin/ps uxawwd
/usr/bin/sockstat
Attach the output in a text file as it will be too large to put inline on a forum post.

@jimp From my topic yesterday https://forum.pfsense.org/index.php?topic=138283.0

You referred me to get these command outputs when it happened again and it finally has. I have attached the text file. Let me know if you need anything else and thanks again.

[pfSense Command Outputs.txt](/public/imported_attachments/1/pfSense Command Outputs.txt)

gsmornot

@BBcan177:

I am going to make a PR shortly to hopefully fix this issue….

You could download these two files from my Github gist to get the updated code:
fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"

fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
You will need to restart the DNSBL Service for this to take effect.

A reboot is not required, but would be recommended.

Done and works fine so far as I can tell. Good work!
Side note-have been watching for V3. :-)

jimp

@Hans_P:

You referred me to get these command outputs when it happened again and it finally has. I have attached the text file. Let me know if you need anything else and thanks again.

Looks like the same pfblocker dnsbl issue as others in the thread. There is a pfBlocker update that will be out momentarily that should help, if not fix it entirely.

depster

@gsmornot:

I have not had any issues but based on this post I have disabled DNSBL and pfblocker until the next update. Bummer since this is one of my favorite parts of using pfsense.

I have been up and running for ~10 hours since disabling DNBL. Previsouly, I had just pfBlocker disabled - and it locked up overnight.

klank

So I have disabled pfblocker, that fixed the crashing or hang that was occurring every 8 hours or so. However now the gui and ssh become non responsive after 8 hours or so.

I'm going to try fresh install with ZFS to see if the problems continue.

m0nji

Very interesting. I did not have this 502 Bad Gateway Issue after upgrading to 2.4.0-RELEASE. (Uptime since Upgrade 1 Day 18 Hours).
I just installed now the new Update (Upgrading pfSense-pkg-pfBlockerNG from 2.1.1_11 to 2.1.2)

My Setup:
Kettop Mi19N (32GB SSD UFS Filesystem + 8GB RAM)
PFBlockerNG + DNSBL enabled
MultiWAN (Failover)
IPSEC + OpenVPN

jimp

To anyone experiencing the DNSBL issue in this thread: Update to the latest pfBlocker (2.1.2 or later) as it contains a fix for the locking issue leading to these conditions. After updating, ideally you will want to reboot the firewall to make sure it's starting with a clean slate. At the very least, stop and restart the DNSBL daemon.

I'll edit this into the first post of the thread for visibility.

Hans_P

@jimp:

To anyone experiencing the DNSBL issue in this thread: Update to the latest pfBlocker (2.1.2 or later) as it contains a fix for the locking issue leading to these conditions. After updating, ideally you will want to reboot the firewall to make sure it's starting with a clean slate. At the very least, stop and restart the DNSBL daemon.

Ok, I did install the latest update to pfBlocker (2.1.2) and seemed to install just fine. When I went to restart the box, it is now in a constant restart state. I am able to SSH into it to see what is going on. I can see the commands as the system tries to load, but it will stop at…

"cpu_reset_proxy: Stopped CPU 1" (see attached)

Then it just restarts. It will keep doing this. I am not sure if the update caused this or not so am posting it here. Anyways, I am just trying to factory reset now and load my current configuration, but can't get it to work. I've tried the button method and is still the same. It doesn't make it to the "main" command menu to run option #4, but I can pause it at the boot menu (see attached). Any suggestions on what to look for? Is a Factory Reset suggested? Any help would be appreciated. Thanks in advance.

![pfSense error.JPG](/public/imported_attachments/1/pfSense error.JPG)
![pfSense error.JPG_thumb](/public/imported_attachments/1/pfSense error.JPG_thumb)
![pfSense menu.JPG](/public/imported_attachments/1/pfSense menu.JPG)
![pfSense menu.JPG_thumb](/public/imported_attachments/1/pfSense menu.JPG_thumb)

Hans_P

Ok, I did install the latest update to pfBlocker (2.1.2) and seemed to install just fine. When I went to restart the box, it is now in a constant restart state. I am able to SSH into it to see what is going on. I can see the commands as the system tries to load, but it will stop at…

I was able to fix this. I found searching in the forums a command to use…

https://forum.pfsense.org/index.php?topic=135466.0

I performed the steps that @Derelict (Thank You) mentioned and it worked. I was able to reboot pfSense and also able to log into the Web GUI. It immediately came up with a crash reporter and wanted me to submit it the developers, which I did. I hope they can find what happened in my case.

As of right now, I am back up and running and didn't have to reset back to defaults or reload my config. I hope this post will help someone else.

klank

I've had to go back to 2.3 as 2.4 has been way to unstable for me. Even a fresh install with ZFS is having issues.