Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New 502 Bad Gateway

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    281 Posts 67 Posters 205.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      steky9
      last edited by

      If anything removing TrafficTotals has made things worse rather than better. I only rebooted on Sunday night to get management back, and now 48 hours later I'm getting the 502's again. Maybe its just a freak occurence, I don't know, but as is somethings badly broken.

      1 Reply Last reply Reply Quote 0
      • P
        PiBa
        last edited by

        From what i 'think' happens TrafficTotals does not cause the problem. But might experience the same symptom as pfBlocker..

        For those using TrafficTotals and experiencing the problem it would be nice to get some information in 'why' the vnstat process hangs during a set of actions mostly unrelated to what pfBlocker does as they might have the same root cause for 'hanging'.. So truss, lsof and possibly gdb output would be nice to see especially as its a easy separated process unlike pfBlocker/lighttpd/php chain thats harder to run separately. Also it would be good to see if vnstat still hangs when gathering stats for less and or different interfaces.. Both while there are already hanging vnstat processes, gather details, and after killing all those processes try if it still hangs. (without rebooting..)

        Yes 'something' is broken. but only a +1 wont help fix it.. i think its already high on the attention list of BBcan177, and i'm interested in this topic as well and probably a few others that passively 'monitor' this thread, but i doubt it can be properly fixed without detailed information and a good understanding of why the problem happens.

        Please gather information some about running vnstat processes (when more than 1 starts running.) for those that have TrafficTotals installed. Preferably on pfSense 2.4.2 though not sure if that would help for anything..

        1 Reply Last reply Reply Quote 0
        • S
          steky9
          last edited by

          I'm happy to help if any more command output is required. I haven't rebooted it yet since the 502's started again on Tuesday. Given the day thats in it, I'll leave it be unless it starts dropping traffic.

          1 Reply Last reply Reply Quote 0
          • C
            Corbinm3
            last edited by

            We were also having issues with 502 Bad Gateway, 2.4.0 release, PFBlockerNG, Snort, OpenVPN Exporter…I noticed our issues after enabling DNS Resolver. Maybe it is just a coincidence but not even 30 minutes after turning that on and we got 502 errors. Rebooted, disabled PFBlockerNG, same problem the following day. Rebooted again, switched back to DNS forwarder and uninstalled PFBlockerNG, been stable since. Without the DNS resolver enabled we were stable with PFBlockerNG installed for over a week. After enabling the DNS Resolver, we went down quick. I'll keep this forum up to date if we do go down again and I have to rule out the DNS Resolver as the culprit, but for now, that's what it looks like from our end.

            Edit: Forgot to post this is on Netgate hardware, can't remember which one though and I'm not in the office.

            1 Reply Last reply Reply Quote 0
            • P
              PiBa
              last edited by

              For those willing to give some new code a try i have made a few changes to the 'file locking' code of pfBlockerNG.. :)
              Could some of you try if the changes made improve things?

              https://github.com/PiBa-NL/FreeBSD-ports/commit/1766713b26c8f388ad6e7909b2e971f7d74cdfea

              Changes are as following:

              • include globals.inc so the /tmp/ folder is know to be used for placing lock files instead of the root /
              • dont try and lock a resource handle with try_lock as a 'resource-(descriptive)-name' is expected
              • use 1 lock around the stats file re-writing code, having 2 locks for the same piece of code is not needed.
              • remove the force_unlock called on a 'Resource #10' which wasn't used to create a lock anyhow..

              It should be possible to apply the patch with systempatches package.
              To add a new patch that way press add then fill in:
              Description: pfBlocker_dnsbl_statsfile_locking
              File: https://github.com/PiBa-NL/FreeBSD-ports/commit/1766713b26c8f388ad6e7909b2e971f7d74cdfea.patch
              PathStrip: 4
              Base: /
              IgnoreWhitespace: Checked
              AutoApply: Unchecked

              Save, Fetch, Apply

              A message should show "Patch applied successfully".

              To revert it should be possible to just press 'Revert' which appears after the patch is applied.. If all fails, reinstall pfBlocker package "pkg install -f pfSense-pkg-pfBlockerNG"

              Edit :
              FYI: pfBlocker 2.1.2_2 includes this patch.

              1 Reply Last reply Reply Quote 0
              • D
                dstroot
                last edited by

                Hmm…  test output is not inspiring confidence.  Patch test output:

                /usr/bin/patch --directory=/ -f -p4 -i /var/patches/5a17809ef2593.patch --check --reverse --ignore-whitespace
                
                Hmm...  Looks like a unified diff to me...
                The text leading up to this was:
                --------------------------
                |From 1766713b26c8f388ad6e7909b2e971f7d74cdfea Mon Sep 17 00:00:00 2001
                |From: PiBa-NL
                |Date: Fri, 24 Nov 2017 01:37:34 +0100
                |Subject: [PATCH] pfBlockerNG, implement proper locking of dnsbl_info file to
                | avoid possible corruption
                |
                |–-
                | .../usr/local/pkg/pfblockerng/pfblockerng.inc      | 30 ++++++++-----------
                | .../files/usr/local/www/pfblockerng/www/index.php  | 35 ++++++++++------------
                | 2 files changed, 27 insertions(+), 38 deletions(-)
                |
                |diff --git a/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc b/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc
                |index 0fddd745065b..c6379b8dab38 100644
                |--- a/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc
                |+++ b/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc
                --------------------------
                Patching file usr/local/pkg/pfblockerng/pfblockerng.inc using Plan A...
                Hunk #1 failed at 2500.
                1 out of 1 hunks failed while patching usr/local/pkg/pfblockerng/pfblockerng.inc
                Hmm...  The next patch looks like a unified diff to me...
                The text leading up to this was:
                --------------------------
                |diff --git a/net/pfSense-pkg-pfBlockerNG/files/usr/local/www/pfblockerng/www/index.php b/net/pfSense-pkg-pfBlockerNG/files/usr/local/www/pfblockerng/www/index.php
                |index 0b864797146e..8992f4a0342f 100644
                |--- a/net/pfSense-pkg-pfBlockerNG/files/usr/local/www/pfblockerng/www/index.php
                |+++ b/net/pfSense-pkg-pfBlockerNG/files/usr/local/www/pfblockerng/www/index.php
                --------------------------
                Patching file usr/local/www/pfblockerng/www/index.php using Plan A...
                Hunk #1 failed at 28.
                Hunk #2 failed at 71.
                2 out of 2 hunks failed while patching usr/local/www/pfblockerng/www/index.php
                done
                
                1 Reply Last reply Reply Quote 0
                • P
                  PiBa
                  last edited by

                  dstroot, are you running latest/unmodified pfBlockerNG 2.1.2_1 version? On that version, the patch above should apply cleanly.

                  1 Reply Last reply Reply Quote 0
                  • D
                    dstroot
                    last edited by

                    Yes, see attached.  Cheers!

                    firewall_home_lan_-_System__Package_Manager__Installed_Packages.png
                    firewall_home_lan_-_System__Package_Manager__Installed_Packages.png_thumb

                    1 Reply Last reply Reply Quote 0
                    • P
                      PiBa
                      last edited by

                      Wierd, can you reinstall it? 'pkg install -f pfSense-pkg-pfBlockerNG'
                      Same patch applies without issue on my 2.1.2_1, you havn't replaced anything with one of the previous files from bbcan's links or manually changed any part?

                      Or did you check the 'revert' test results? Below would be the expected test result:

                      Patch can be applied cleanly (detail)
                      Patch can NOT be reverted cleanly (detail)
                      

                      Revert only works after the patch is applied already.

                      1 Reply Last reply Reply Quote 0
                      • D
                        dstroot
                        last edited by

                        I think the "Can NOT be reverted cleanly" through me off.  I just checked and mine and got the results you show so I applied the patch and it applied successfully.  I will report if the 502s go away.  Cheers!

                        1 Reply Last reply Reply Quote 0
                        • F
                          fortrash
                          last edited by

                          Same error here. It is quite annoying as my pfsense installation ran smoothly. Since 2.4  I have to schedule reboots.
                          Any workaround except commenting out the code?
                          Are the developer working on it? Otherwise I do test the competitor product.

                          Thanks,

                          1 Reply Last reply Reply Quote 0
                          • P
                            PiBa
                            last edited by

                            Fortrash, please try the patch: https://forum.pfsense.org/index.php?topic=137103.msg767259#msg767259

                            1 Reply Last reply Reply Quote 0
                            • F
                              fortrash
                              last edited by

                              Thanks any chance to help the developer?

                              1 Reply Last reply Reply Quote 0
                              • P
                                PiBa
                                last edited by

                                @fortrash:

                                Thanks any chance to help the developer?

                                How do you mean?

                                Patches are made by developers.. Testing if it indeed fixes the issue would help, but thats something 'you' (users that experience the actual issue) have to do..

                                1 Reply Last reply Reply Quote 0
                                • F
                                  fortrash
                                  last edited by

                                  Thanks, I am testing the patch.

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    chrcoluk
                                    last edited by

                                    guys check my post here

                                    https://forum.pfsense.org/index.php?topic=110515.msg766964#msg766964

                                    pfSense CE 2.7.2

                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      PatPend
                                      last edited by

                                      Running pfsense 2.4.1-RELEASE with pfBlockerNG on 2.1.2_1 installed and also running OpenVPN server.  When the Bad Gateway error happened, OpenVPN clients couldn't connect as well.  Rebooting fixed it for now.  Any news on when the patch might be rolled into an update?  Thanks, Russ

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        PiBa
                                        last edited by

                                        PatPend, any news on if the patch helps? ::) may i presume your running with pfBlocker dns blocklists enabled as well?

                                        If there is no positive feedback on its results then there is no need to commit it right? I hope there are no new problems reported by the people that applied the patch, and that they can confirm the problem did not return or at the very least took longer to re-appear.. Would be nice that in a week time they could say it has been running stable..

                                        In the mean time please feel free to apply the patch to your own installation and test it out as well.

                                        1 Reply Last reply Reply Quote 0
                                        • P
                                          PatPend
                                          last edited by

                                          @PiBa:

                                          PatPend, any news on if the patch helps? ::) may i presume your running with pfBlocker dns blocklists enabled as well?

                                          If there is no positive feedback on its results then there is no need to commit it right? I hope there are no new problems reported by the people that applied the patch, and that they can confirm the problem did not return or at the very least took longer to re-appear.. Would be nice that in a week time they could say it has been running stable..

                                          In the mean time please feel free to apply the patch to your own installation and test it out as well.

                                          I applied the patch today, so far so good.  I'm running with easylist and easylist privacy plus 28 custom entries to block smart TV traffic.  Last time it took about 10 days before hitting the 502 (of course when it did I was out of town when it also took out OpenVPN ::) ).

                                          1 Reply Last reply Reply Quote 0
                                          • BBcan177B
                                            BBcan177 Moderator
                                            last edited by

                                            @PiBa:

                                            For those willing to give some new code a try i have made a few changes to the 'file locking' code of pfBlockerNG.. :)

                                            Thanks PiBa, hopefully we get some feedback to see if this resolves this issue… and we can get it merged!

                                            "Experience is something you don't get until just after you need it."

                                            Website: http://pfBlockerNG.com
                                            Twitter: @BBcan177  #pfBlockerNG
                                            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.