New 502 Bad Gateway



  • @jimp:

    @PiBa:

    That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

    If that were the case there should be some other sign of stress on the PHP-FPM socket, and there isn't. Most likely it's a completely unrelated problem that happens to present with one common symptom.

    It has started becoming an issue for me again.  I have attached my outputs as requested.

    Netgate Sg-4860
    Ver. 2.4.0
    pfBlocker
    Ver. 2.1.2

    [pfSense Freeze.txt](/public/imported_attachments/1/pfSense Freeze.txt)



  • @jimp:

    @PiBa:

    That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

    If that were the case there should be some other sign of stress on the PHP-FPM socket, and there isn't. Most likely it's a completely unrelated problem that happens to present with one common symptom.

    If thats the case I’m curious as to why a ZFS install mitigates it?



  • @PiBa:

    @jimp:

    @ntct:

    I also have 502 Bad Gateway on pfSense 2.4 and pfblockerng 2.1.2…... :-\

    And judging by the output, it's still DNSBL getting backed up.

    That why the report of @rightnow would be interesting. As he has this error without any packages installed. Makes me wonder if some other component causes (part of) the problem, and maybe pfBlocker is just the most visible to suffer from that 'defect'.

    I did a fresh install of 2.4 and had pfblockerng with the issue. I didn't come on the forum so didn't know of the instructions to get log details.

    However, I disabled pfblockerng and have not had a lock up for over 24 hours. Let me run without pfblockerng for a bit more then switch it back on and see.



  • mine crashes right away if enabled but with pfblockerng disabled it crashes a lot later like 24-28 hrs after.

    Is this fixed if I select the developer update ? or is this still a work in progress>



  • @msvuze:

    mine crashes right away if enabled but with pfblockerng disabled it crashes a lot later like 24-28 hrs after.

    Is this fixed if I select the developer update ? or is this still a work in progress>

    If you have this same issue with pfBlocker and DNSbl disabled, then please provide the information requested to see if it is really the same behavior in the background, that might help tell what causes it. And once the root cause is known, a fix can be made, not before..


  • Moderator

    As per jimp's suggestion, please try these two patched files which use a pfSense function called try_lock() as opposed to flock().

    Run the following commands to download the patched version of the two files from my Github Gist:

    fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"
    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
    

    Recommend a reboot after downloading the patches.

    You can review the Gist revisions here:

    index.php
        https://gist.github.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/revisions

    pfblockerng.inc
        https://gist.github.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/revisions

    Note: The try_lock() function calls might require increasing the timeout setting from the default setting of "5" (seconds)


  • Moderator

    Made a couple changes to the code… So if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes.... Thanks!



  • @BBcan177:

    Made a couple changes to the code… So if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes.... Thanks!

    Hi BBcan177

    Can you make a PR?


  • Moderator

    Added some temp file removal to the patches. So please re-download this new code if you have downloaded these files prior to this post, please re-download the same URLs above to get the new changes…. Thanks!

    A reboot is recommended following these patches.

    Once the code has been tested, I will submit this as a PR.

    Please report back your feedback.

    Thanks!



  • @jimp:

    @hdejongh:

    hee Jim,

    https://pastebin.com/JVMQTWbY

    bad gateway 502 issue

    That looks like it was taken just after a reboot, not when the problem was happening. No sign of anything getting backed up in there.

    Hee Jim,

    im 99,99999% certain the issue was happening at that time.
    Ill do it again.

    this one is from an other firewall.
    it hangs on "sockstat" 15 minutes now so i think those wont come anymore.

    https://pastebin.com/Ek9R0qkh



  • Here is more data with 502 err:

    https://pastebin.com/TqSBTKEK

    OpenVPN clients cannot connect also, which is mine major problem for now.

    hope will help.



  • @ha11oga11o:

    Here is more data with 502 err:

    https://pastebin.com/TqSBTKEK

    OpenVPN clients cannot connect also, which is mine major problem for now.

    hope will help.

    it starts to become a major issue as well for us. Im about to restore backups..
    All virtual firewall's that we upgraded have the same problem…
    we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.


  • Rebel Alliance Developer Netgate

    @hdejongh:

    https://pastebin.com/Ek9R0qkh

    it starts to become a major issue as well for us. Im about to restore backups..
    All virtual firewall's that we upgraded have the same problem…
    we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

    If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.


  • Rebel Alliance Developer Netgate

    @ha11oga11o:

    Here is more data with 502 err:

    https://pastebin.com/TqSBTKEK

    OpenVPN clients cannot connect also, which is mine major problem for now.

    That's the same as others here, pfBlocker DNSBL getting stuck waiting. Try to apply the fixed files from bbcan a few posts above yours.



  • Just another "me too" post.  I have just applied the fixes above and will report back tomorrow morning if it doesn't lock up.  It has locked up within 24 hours ever since the 2.4 upgrade so hopefully a clean system in the morning will show success.



  • @jimp:

    @hdejongh:

    https://pastebin.com/Ek9R0qkh

    it starts to become a major issue as well for us. Im about to restore backups..
    All virtual firewall's that we upgraded have the same problem…
    we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

    If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.

    ok, i will first upgrade to 2.4.1 and report back then!



  • @jimp:

    @ha11oga11o:

    Here is more data with 502 err:

    https://pastebin.com/TqSBTKEK

    OpenVPN clients cannot connect also, which is mine major problem for now.

    That's the same as others here, pfBlocker DNSBL getting stuck waiting. Try to apply the fixed files from bbcan a few posts above yours.

    I just did. Will revert with output if hangs.

    Cheers!



  • I'm still learning. Can I just enter the two files from reply 165 into the command box via Diagnostics, Command? I am a GUI user.
    I have not had any issues yet but following along to prevent any issues. I do use DNSBL and I am on 2.4.1. Was on 2.4 and the related RC's.



  • @gsmornot:

    I'm still learning. Can I just enter the two files from reply 165 into the command box via Diagnostics, Command? I am a GUI user.
    I have not had any issues yet but following along to prevent any issues. I do use DNSBL and I am on 2.4.1. Was on 2.4 and the related RC's.

    Use putty as stated here, and when you log in choose shell and copy paste one row after another. Reboot unit and thats it. And use login "root" not "admin" as stated in video.

    https://www.youtube.com/watch?v=krNuKDGEjvQ

    Cheers!



  • @jimp:

    @hdejongh:

    https://pastebin.com/Ek9R0qkh

    it starts to become a major issue as well for us. Im about to restore backups..
    All virtual firewall's that we upgraded have the same problem…
    we have to reboot them multiple times per day to get it working, otherwise ipsec's and openvpn stop working.

    If that is the output when you have the problem, then it's NOT this problem. No sign of pfBlocker or anything blocking PHP. Start a new thread, it's probably something already solved on 2.4.1 if it's a VM issue. Check the release notes.

    since upgrade tot 2.4.1 no problems yet!



  • So far so good with the updated files.



  • @ha11oga11o:

    @gsmornot:

    I'm still learning. Can I just enter the two files from reply 165 into the command box via Diagnostics, Command? I am a GUI user.
    I have not had any issues yet but following along to prevent any issues. I do use DNSBL and I am on 2.4.1. Was on 2.4 and the related RC's.

    Use putty as stated here, and when you log in choose shell and copy paste one row after another. Reboot unit and thats it. And use login "root" not "admin" as stated in video.

    https://www.youtube.com/watch?v=krNuKDGEjvQ

    Cheers!

    MAC user so I used terminal. Thank you for the point in the right direction, patched this morning after waking up to 502 Bad Gateway.



  • @BBcan177:

    As per jimp's suggestion, please try these two patched files which use a pfSense function called try_lock() as opposed to flock().

    Run the following commands to download the patched version of the two files from my Github Gist:

    fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"
    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
    

    Recommend a reboot after downloading the patches.

    You can review the Gist revisions here:

    index.php
        https://gist.github.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/revisions

    pfblockerng.inc
        https://gist.github.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/revisions

    Note: The try_lock() function calls might require increasing the timeout setting from the default setting of "5" (seconds)

    So Far running for more than 24 hrs without problems on pfsense 2.4.0 - I would say it looks pretty good.

    I will upgrade to pfsense 2.4.1 tonight.



  • I haven't posted here but have been following the thread as I've had similar issues. After so many hours(less than half a day) pfsense gui and shell would become completely unresponsive even though clients still had internet access.

    Replacing the files "index.php" and "pfblockerng.inc" with the ones BBcan177 posted seems to have fixed the issue for me. I've been up for 41hrs now without a problem.



  • Guess I spoke too soon. Just got the 502.  Also this seems to stop the firewall schedules from working as a schedule that was allowing access should have been stopped but access was still available.  Hopefully it's not affecting other aspects of the firewall security.



  • Is 2.4.1 supposed to fix this without the updated files here?

    I saw the redmine ticket was closed due to a new version of pfbng fixing this. Is that automatically installed in 2.4.1? I'm not seeing a package update on my 2.4.1 RELEASE box.


  • Rebel Alliance Developer Netgate

    If you ran the 2.4.1 update after the new package was uploaded (2.1.2_1) then it would pick up the new files automatically.



  • I still have 502 issue when use pfSense 2.4.1 and pfblockerng 2.1.2_1.  :(

    output.txt



  • Still having the issue after 2.4.1, had to disable DNSBL again.



  • Hey all,

    Try disabling the Dashboard auto-update check. It seems unrelated, but I made this change last weekend and my system has been stable ever since.



  • I am also experiencing this issue. Error 502 Bad Gateway roughly every 15 hours. Latest version of pfSense and pfBlocker as of the date of this post. I'd post the actual versions but I need to reboot the router to get into the GUI and I can't afford to take down the internet here at this very moment.

    As a side note, I've used pfSense for years now and this '502 Bad Gateway' seems to be a reoccurring theme..



  • Running 2.4.1 and 2.1.2_1 and still getting 502 but also getting a few crash reports to send in each day as well.

    PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 2496, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes)	@ 2017-10-28 07:02:39
    


  • Hi,

    same here - updates onto last version of pfsene & pfblocker - after about a day it becomes unresponsive.



  • Disabling Snort and it's updates has kept me up and running for 2 days now.  I'm going to wait another day and then re-enable snort and see what happens.  Perhaps Snort needs the same changes that pfblocker has gotten.


  • Moderator

    I made some additional mods to the code.  Run the following command to download the patched version from my Github Gist:

    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
    

    Recommend a reboot after downloading the patch.

    Please let me know your feedback!



  • For me after  2.4.1-RELEASE (amd64)  and pfBlockerNG 2.1.2_1 Finally no more err 502 or 504. Open VPN keep connections.

    Sistem running for  2 Days 09 Hours 02 Minutes 23 Seconds. Before i had issues after 6-9 hrs.

    Many thnx fo all.



  • @mindframe:

    Hey all,

    Try disabling the Dashboard auto-update check. It seems unrelated, but I made this change last weekend and my system has been stable ever since.

    Well it finally failed after a week of being stable…



  • @BBcan177:

    I made some additional mods to the code.  Run the following command to download the patched version from my Github Gist:

    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
    

    Recommend a reboot after downloading the patch.

    Please let me know your feedback!

    I installed this today and after 6 hours of running my pFsense VM increased disk usage of over 20gb and crashed the VM and needed to be rebuilt.


  • Moderator

    @morph0:

    I installed this today and after 6 hours of running my pFsense VM increased disk usage of over 20gb and crashed the VM and needed to be rebuilt.

    I don't think the patch would have done that… Confirmed with a few other users. Check your PM for details and we can go from there... Thanks!



  • @morph0:

    @BBcan177:

    I made some additional mods to the code.  Run the following command to download the patched version from my Github Gist:

    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
    

    Recommend a reboot after downloading the patch.

    Please let me know your feedback!

    I installed this today and after 6 hours of running my pFsense VM increased disk usage of over 20gb and crashed the VM and needed to be rebuilt.

    Works on my machines since 4 days without a hassle and without filling up the disks.
    What was filled up, did you have had a look on the files?