Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New 502 Bad Gateway

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    281 Posts 67 Posters 209.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      @Music:

      I have the same problem.

      I run Pfblocker and Snort.

      i did remove the code and it seemed to work, but after like 20 min or so i got the same problem again. So im looking to do a complete reinstall and put back a backup i made before i went to 2.4.0.

      when i use the old kernel on 2.4.0 all is fine.

      Did you gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 when it was stopped? You may have been hitting a different issue entirely.

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • M
        musicwizard
        last edited by

        @jimp:

        @Music:

        I have the same problem.

        I run Pfblocker and Snort.

        i did remove the code and it seemed to work, but after like 20 min or so i got the same problem again. So im looking to do a complete reinstall and put back a backup i made before i went to 2.4.0.

        when i use the old kernel on 2.4.0 all is fine.

        Did you gather the information requested in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 when it was stopped? You may have been hitting a different issue entirely.

        Funny thing is i did have the information  in a txt file was still copying some information out of putty. And my cat stept on the power connector of my computer which is lying on the floor. so i lost that file.  :(

        So i just did a clean install and used the recover config and installed it on ZFS now. Only have pfblocker running atm and no problems yet.

        1 Reply Last reply Reply Quote 0
        • B
          belt9
          last edited by

          @BreeOge:

          Currently you have 3 choices

          1. Remove PFblocker till BBcan177 can get the effected coded fixed.  He is working on it, so I wouldn't expect it will be that long.
          2. Reinstall with the ZFS file system.
          3. Keep rebooting till update comes out.

          Just checking in as my system was unreachable and it looks like this is the culprit, I just updated to 2.4.0-RELEASE form a month old snapshot last night.

          I'm not sure about #2 helping you, as I've got a ZFS raidz2 install and this still happened to me. I've disabled DNSBL until a fix comes out, BBCan177 is top notch. I'm sure the fix will be available shortly if it's at all feasible.

          1 Reply Last reply Reply Quote 0
          • M
            musicwizard
            last edited by

            well since i have a clean reinstall if 2.4.0 with zfs and im running both Pfblocker and snort i haven't had a " lockup"  of the webgui etc.

            Uptime 15 Hours 14 Minutes 49 Seconds

            Al tho i did get a few Updates from BBcan :) because im using the test/beta version.

            It goes seem that 2.4.0 using more Cpu tho.

            1 Reply Last reply Reply Quote 0
            • T
              TheNarc
              last edited by

              With respect to the workaround that involves editing /usr/local/www/pfblockerng/www/index.php, I am not a PHP programmer, so I may be speaking out of turn, but I noticed that in the relevant code section two locks are acquired and never released:

              
                      if (!empty($pfb_query)) {
              	// Increment DNSBL Alias Counter
              	$dnsbl_info = '/var/db/pfblockerng/dnsbl_info';
              	if (($handle = @fopen("{$dnsbl_info}", 'r')) !== FALSE) {
              		flock($handle, LOCK_EX);
              		$pfb_output = @fopen("{$dnsbl_info}.bk", 'w');
              		flock($pfb_output, LOCK_EX);
              
              		// Find line with corresponding DNSBL Aliasname
              		while (($line = @fgetcsv($handle)) !== FALSE) {
              			if ($line[0] == $pfb_query) {
              				$line[3] += 1;
              			}
              			@fputcsv($pfb_output, $line);
              		}
              
              		@fclose($pfb_output);
              		@fclose($handle);
              		@rename("{$dnsbl_info}.bk", "{$dnsbl_info}");
              	}
              }
              

              Referring to the PHP documentation for the flock function, apparently this used to be okay because the locks were implicitly released when the file handles were closed, but that is no longer so starting from version 5.3.2:  https://secure.php.net/manual/en/function.flock.php

              That said, I don't know if the update from 2.3.4 to 2.4.0 happened to jump from a pre-5.3.2 version of PHP to a post-5.3.2 version.  But regardless, I assume that explicitly releasing locks couldn't hurt anything.  Also, the return value of one of the two fopen calls is checked but the other is not.  I'm going to try the following modification on my system, which is still UFS, and will report back:

              
              if (!empty($pfb_query)) {
              	// Increment DNSBL Alias Counter
              	$dnsbl_info = '/var/db/pfblockerng/dnsbl_info';
              	if (($handle = @fopen("{$dnsbl_info}", 'r')) !== FALSE) {
              		if(flock($handle, LOCK_EX)) {
                       if (($pfb_output = @fopen("{$dnsbl_info}.bk", 'w')) !== FALSE) {
                          if(flock($pfb_output, LOCK_EX)) {
                             // Find line with corresponding DNSBL Aliasname
                             while (($line = @fgetcsv($handle)) !== FALSE) {
                                if ($line[0] == $pfb_query) {
                                   $line[3] += 1;
                                }
                                @fputcsv($pfb_output, $line);
                             }
                             flock($pfb_output, LOCK_UN);
                          }
                          @fclose($pfb_output);
                       }
                       flock($handle, LOCK_UN);
                      }
              	@fclose($handle);
              	@rename("{$dnsbl_info}.bk", "{$dnsbl_info}");
              	}
              }
              
              

              And sorry for the lousy formatting; I tried pasting several times and couldn't manage to get the indentation non-wonky.

              1 Reply Last reply Reply Quote 0
              • T
                TheNarc
                last edited by

                It looks like pfSense has been using PHP 5.3.2 since around version 2.1 (the change log for 2.1 https://doc.pfsense.org/index.php/2.1_New_Features_and_Changes just says "PHP to 5.3.x").  So technically as far back as then all locks acquired with PHP's flock should have been manually released.  Nevertheless, maybe some other change in 2.4.0 resulted in the lack of these explicit releases causing trouble where they had not before.  I have two pfSense machines running 2.4.0 with pfBlockerNG and my modification now so I'll just see how they fare.

                1 Reply Last reply Reply Quote 0
                • S
                  strigona
                  last edited by

                  @TheNarc:

                  It looks like pfSense has been using PHP 5.3.2 since around version 2.1 (the change log for 2.1 https://doc.pfsense.org/index.php/2.1_New_Features_and_Changes just says "PHP to 5.3.x").  So technically as far back as then all locks acquired with PHP's flock should have been manually released.  Nevertheless, maybe some other change in 2.4.0 resulted in the lack of these explicit releases causing trouble where they had not before.  I have two pfSense machines running 2.4.0 with pfBlockerNG and my modification now so I'll just see how they fare.

                  You are correct. I just ran 'php -v' on my 2.3.4 system:

                  PHP 5.6.31 (cli) (built: Jul 14 2017 19:45:37)
                  Copyright (c) 1997-2016 The PHP Group
                  Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
                      with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
                      with Xdebug v2.5.0, Copyright (c) 2002-2016, by Derick Rethans
                      with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH
                  
                  1 Reply Last reply Reply Quote 0
                  • B
                    BreeOge
                    last edited by

                    BBcan177 is currently working on an update to move this to a sqlite database, and get it off the text file.  So this should never be an issue in the future.

                    All the effected code does is keep the updates for the desktop widget.  Omitting the code does not break pfBlockerNG it only affects the widget reporting.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Steve_B Netgate
                      last edited by

                      I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed.  Those changes will be in the next 2.4 snapshot (probably early Oct 18th)

                      I would appreciate any feedback Better/worse/no change . .

                      Thanks

                      Als ik kan

                      1 Reply Last reply Reply Quote 0
                      • XentrkX
                        Xentrk
                        last edited by

                        I also get the 502 Bad Gateway error when trying to access the pfSense Web GUI. I can logon a SSH session. But it hangs after entering my password. I noticed it about 24 hours after upgrade to 2.4.  I use pfBlockerNG. Thanks for the information in this thread. I will continue to monitor for updates to the issue.

                        pfSense 2.4.4_2 | Intel i5-3450 @ 3.10GHz  | AES-NI enabled |  pfBlockerNG | Snort
                        Blog Site: https://x3mtek.com || GitHub: https://github.com/Xentrk

                        1 Reply Last reply Reply Quote 0
                        • AhnHELA
                          AhnHEL
                          last edited by

                          @Steve_B:

                          I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed.

                          I, myself never had the IPSEC dashboard widget displayed when I was getting the 502 Bad Gateway issue.  Was this problem somehow thought to be related?  I don't use IPSEC at all.

                          AhnHEL (Angel)

                          1 Reply Last reply Reply Quote 0
                          • D
                            depster
                            last edited by

                            @jimp:

                            If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:

                            /usr/bin/netstat -Ln
                            /usr/bin/netstat -xn
                            /usr/sbin/swapinfo -h
                            /usr/bin/top | /usr/bin/head -n7
                            /bin/ps uxawwd
                            /usr/bin/sockstat
                            
                            

                            Attach the output in a text file as it will be too large to put inline on a forum post.

                            I believe that I am experiencing this problem; I have attached the requested files to confirm. I had disabled pfBlockerNG and found the system locked up again this morning; however, I had not disabled DNSBL.

                            Php-FpmSocket_503Lockup.txt

                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by

                              @depster:

                              I believe that I am experiencing this problem; I have attached the requested files to confirm. I had disabled pfBlockerNG and found the system locked up again this morning; however, I had not disabled DNSBL.

                              The output you posted confirms you are hitting this issue, it's definitely the pfBlocker DNSBL process getting stuck like others here in the thread.

                              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                @AhnHEL:

                                @Steve_B:

                                I pushed some experimental changes today that are designed to help track down the issue of gateway timeouts occurring when the IPSec dashboard widget is displayed.

                                I, myself never had the IPSEC dashboard widget displayed when I was getting the 502 Bad Gateway issue.  Was this problem somehow thought to be related?  I don't use IPSEC at all.

                                In the past, the IPsec widget has been a different source of 502 errors. So it's possible that someone looking at this thread might be hitting that issue and not the DNSBL issue.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • G
                                  gsmornot
                                  last edited by

                                  I have not had any issues but based on this post I have disabled DNSBL and pfblocker until the next update. Bummer since this is one of my favorite parts of using pfsense.

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    mloiterman
                                    last edited by

                                    Can anyone confirm that the issue is not present provided that DNSBL is disabled?  Is that all that needs to be done until upgrade?

                                    1 Reply Last reply Reply Quote 0
                                    • V
                                      vomcliff
                                      last edited by

                                      @BreeOge:

                                      The file is at this location

                                      /usr/local/www/pfblockerng/www/index.php

                                      cd /usr/local/www/pfblockerng/www/
                                      
                                      cp index.php index.old = do this so you have a copy of the original before you remove the section.
                                      

                                      Edit index.php with your favorite editor, and remove this section at the bottom.

                                      if (!empty($pfb_query)) {
                                      	// Increment DNSBL Alias Counter
                                      	$dnsbl_info = '/var/db/pfblockerng/dnsbl_info';
                                      	if (($handle = @fopen("{$dnsbl_info}", 'r')) !== FALSE) {
                                      		flock($handle, LOCK_EX);
                                      		$pfb_output = @fopen("{$dnsbl_info}.bk", 'w');
                                      		flock($pfb_output, LOCK_EX);
                                      		// Find line with corresponding DNSBL Aliasname
                                      		while (($line = @fgetcsv($handle)) !== FALSE) {
                                      			if ($line[0] == $pfb_query) {
                                      				$line[3] += 1;
                                      			}
                                      			@fputcsv($pfb_output, $line);
                                      		}
                                      		@fclose($pfb_output);
                                      		@fclose($handle);
                                      		@rename("{$dnsbl_info}.bk", "{$dnsbl_info}");
                                      	}
                                      }
                                      
                                      

                                      Just to add to this thread, I can confirm that the above fix worked for me. I had this issue after pushing out the upgrade to 2.4 and followed the post above (I commented the lines out rather than deleting them). Since then it has been stable and all pfSense routers in my environment have stopped giving the bad gateway error.

                                      1 Reply Last reply Reply Quote 0
                                      • BBcan177B
                                        BBcan177 Moderator
                                        last edited by

                                        I am going to make a PR shortly to hopefully fix this issue….

                                        You could download these two files from my Github gist to get the updated code:

                                        fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"
                                        
                                        fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
                                        

                                        You will need to restart the DNSBL Service for this to take effect.

                                        A reboot is not required, but would be recommended.

                                        "Experience is something you don't get until just after you need it."

                                        Website: http://pfBlockerNG.com
                                        Twitter: @BBcan177  #pfBlockerNG
                                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                        1 Reply Last reply Reply Quote 0
                                        • H
                                          Hans_P
                                          last edited by

                                          @jimp:

                                          If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:

                                          /usr/bin/netstat -Ln
                                          /usr/bin/netstat -xn
                                          /usr/sbin/swapinfo -h
                                          /usr/bin/top | /usr/bin/head -n7
                                          /bin/ps uxawwd
                                          /usr/bin/sockstat
                                          
                                          

                                          Attach the output in a text file as it will be too large to put inline on a forum post.

                                          @jimp From my topic yesterday https://forum.pfsense.org/index.php?topic=138283.0

                                          You referred me to get these command outputs when it happened again and it finally has.  I have attached the text file.  Let me know if you need anything else and thanks again.

                                          [pfSense Command Outputs.txt](/public/imported_attachments/1/pfSense Command Outputs.txt)

                                          1 Reply Last reply Reply Quote 0
                                          • G
                                            gsmornot
                                            last edited by

                                            @BBcan177:

                                            I am going to make a PR shortly to hopefully fix this issue….

                                            You could download these two files from my Github gist to get the updated code:

                                            fetch -o /usr/local/www/pfblockerng/www/index.php "https://gist.githubusercontent.com/BBcan177/9f9c8e62b166cee07ad16cd4ff59103c/raw"
                                            
                                            fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://gist.githubusercontent.com/BBcan177/7ff15715be0f02afdbe0a00c676aedce/raw"
                                            

                                            You will need to restart the DNSBL Service for this to take effect.

                                            A reboot is not required, but would be recommended.

                                            Done and works fine so far as I can tell. Good work!
                                            Side note-have been watching for V3. :-)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.