New 502 Bad Gateway

Hans_P · Oct 23, 2017, 4:24 PM

@PiBa:

That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

If that were the case there should be some other sign of stress on the PHP-FPM socket, and there isn't. Most likely it's a completely unrelated problem that happens to present with one common symptom.

It has started becoming an issue for me again. I have attached my outputs as requested.

Netgate Sg-4860
Ver. 2.4.0
pfBlocker
Ver. 2.1.2

[pfSense Freeze.txt](/public/imported_attachments/1/pfSense Freeze.txt)

Jailer · Oct 23, 2017, 6:18 PM

@jimp:

@PiBa:

That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

If that were the case there should be some other sign of stress on the PHP-FPM socket, and there isn't. Most likely it's a completely unrelated problem that happens to present with one common symptom.

If thats the case I’m curious as to why a ZFS install mitigates it?

ak · Oct 23, 2017, 6:25 PM

@PiBa:

@jimp:

@ntct:

I also have 502 Bad Gateway on pfSense 2.4 and pfblockerng 2.1.2…... :-\

And judging by the output, it's still DNSBL getting backed up.

That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

I did a fresh install of 2.4 and had pfblockerng with the issue. I didn't come on the forum so didn't know of the instructions to get log details.

However, I disabled pfblockerng and have not had a lock up for over 24 hours. Let me run without pfblockerng for a bit more then switch it back on and see.

msvuze · Oct 23, 2017, 10:07 PM

mine crashes right away if enabled but with pfblockerng disabled it crashes a lot later like 24-28 hrs after.

Is this fixed if I select the developer update ? or is this still a work in progress>

PiBa · Oct 23, 2017, 10:22 PM

@msvuze:

mine crashes right away if enabled but with pfblockerng disabled it crashes a lot later like 24-28 hrs after.

Is this fixed if I select the developer update ? or is this still a work in progress>

If you have this same issue with pfBlocker and DNSbl disabled, then please provide the information requested to see if it is really the same behavior in the background, that might help tell what causes it. And once the root cause is known, a fix can be made, not before..

BBcan177 · Oct 23, 2017, 10:37 PM

As per jimp's suggestion, please try these two patched files which use a pfSense function called try_lock() as opposed to flock().

Run the following commands to download the patched version of the two files from my Github Gist:

fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"
fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"

Recommend a reboot after downloading the patches.

You can review the Gist revisions here:

index.php
https://gist.github.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/revisions

pfblockerng.inc
https://gist.github.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/revisions

Note: The try_lock() function calls might require increasing the timeout setting from the default setting of "5" (seconds)

BBcan177 · Oct 23, 2017, 11:29 PM

Made a couple changes to the code… So if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes.... Thanks!

ntct · Oct 24, 2017, 1:12 AM

@BBcan177:

Made a couple changes to the code… So if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes.... Thanks!

Hi BBcan177

Can you make a PR?

BBcan177 · Oct 24, 2017, 4:28 AM

Added some temp file removal to the patches. So please re-download this new code if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes…. Thanks!

A reboot is recommended following these patches.

Once the code has been tested, I will submit this as a PR.

Please report back your feedback.

Thanks!

hdejongh · Oct 24, 2017, 9:07 AM

@jimp:

@hdejongh:

hee Jim,

https://pastebin.com/JVMQTWbY

bad gateway 502 issue

That looks like it was taken just after a reboot, not when the problem was happening. No sign of anything getting backed up in there.

Hee Jim,

im 99,99999% certain the issue was happening at that time.
Ill do it again.

this one is from an other firewall.
it hangs on "sockstat" 15 minutes now so i think those wont come anymore.

https://pastebin.com/Ek9R0qkh

ha11oga11o · Oct 24, 2017, 2:45 PM

Here is more data with 502 err:

https://pastebin.com/TqSBTKEK

OpenVPN clients cannot connect also, which is mine major problem for now.

hope will help.

hdejongh · Oct 24, 2017, 3:36 PM

@ha11oga11o:

Here is more data with 502 err:

https://pastebin.com/TqSBTKEK

OpenVPN clients cannot connect also, which is mine major problem for now.

hope will help.

it starts to become a major issue as well for us. Im about to restore backups..
All virtual firewall's that we upgraded have the same problem…
we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

jimp · Oct 24, 2017, 3:38 PM

@hdejongh:

https://pastebin.com/Ek9R0qkh

it starts to become a major issue as well for us. Im about to restore backups..
All virtual firewall's that we upgraded have the same problem…
we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.

jimp · Oct 24, 2017, 3:40 PM

@ha11oga11o:

Here is more data with 502 err:

https://pastebin.com/TqSBTKEK

OpenVPN clients cannot connect also, which is mine major problem for now.

That's the same as others here, pfBlocker DNSBL getting stuck waiting. Try to apply the fixed files from bbcan a few posts above yours.

beernutmark · Oct 24, 2017, 3:54 PM

Just another "me too" post. I have just applied the fixes above and will report back tomorrow morning if it doesn't lock up. It has locked up within 24 hours ever since the 2.4 upgrade so hopefully a clean system in the morning will show success.

hdejongh · Oct 24, 2017, 4:03 PM

@jimp:

@hdejongh:

https://pastebin.com/Ek9R0qkh

it starts to become a major issue as well for us. Im about to restore backups..
All virtual firewall's that we upgraded have the same problem…
we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.

ok, i will first upgrade to 2.4.1 and report back then!

ha11oga11o · Oct 24, 2017, 4:04 PM

@jimp:

@ha11oga11o:

Here is more data with 502 err:

https://pastebin.com/TqSBTKEK

OpenVPN clients cannot connect also, which is mine major problem for now.

That's the same as others here, pfBlocker DNSBL getting stuck waiting. Try to apply the fixed files from bbcan a few posts above yours.

I just did. Will revert with output if hangs.

Cheers!

gsmornot · Oct 24, 2017, 5:34 PM

I'm still learning. Can I just enter the two files from reply 165 into the command box via Diagnostics, Command? I am a GUI user.
I have not had any issues yet but following along to prevent any issues. I do use DNSBL and I am on 2.4.1. Was on 2.4 and the related RC's.

ha11oga11o · Oct 24, 2017, 7:17 PM

@gsmornot:

I'm still learning. Can I just enter the two files from reply 165 into the command box via Diagnostics, Command? I am a GUI user.
I have not had any issues yet but following along to prevent any issues. I do use DNSBL and I am on 2.4.1. Was on 2.4 and the related RC's.

Use putty as stated here, and when you log in choose shell and copy paste one row after another. Reboot unit and thats it. And use login "root" not "admin" as stated in video.

https://www.youtube.com/watch?v=krNuKDGEjvQ

Cheers!

hdejongh · Oct 25, 2017, 7:03 AM

@jimp:

@hdejongh:

https://pastebin.com/Ek9R0qkh

it starts to become a major issue as well for us. Im about to restore backups..
All virtual firewall's that we upgraded have the same problem…
we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.

since upgrade tot 2.4.1 no problems yet!