Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New 502 Bad Gateway

    Scheduled Pinned Locked Moved
    2.4 Development Snapshots
    67
    281
    167.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      doktornotor Banned
      last edited by

      @BreeOge:

      That was not it, did it again, and I received 0 errors /var/run/php-fpm.socket failed. Actually the logs didn't show anything at all.. or nothing out of the normal should I say.

      So, you actually mean that it worked?

      1 Reply Last reply Reply Quote 0
      • B
        BreeOge
        last edited by

        No, it cleared the errors that you said was the cause, however I am still getting the 502 bad gateway error, not as often, but still getting them.

        1 Reply Last reply Reply Quote 0
        • D
          doktornotor Banned
          last edited by

          Well, then you'll need this as well, probably…

          1 Reply Last reply Reply Quote 0
          • B
            BreeOge
            last edited by

            Thank you sir.  I will try that as well. I thank you for your help.

            1 Reply Last reply Reply Quote 0
            • B
              BreeOge
              last edited by

              I have a question Doktornotor, will rc.php_ini_setup get overwritten on updates?

              1 Reply Last reply Reply Quote 0
              • B
                BreeOge
                last edited by

                Also its still happening.. even with the changes you suggested :(

                1 Reply Last reply Reply Quote 0
                • B
                  BreeOge
                  last edited by

                  Update* Reverted back to 2.4.0 and its no longer happening, only happens with the 2.4.1 version!

                  1 Reply Last reply Reply Quote 0
                  • B
                    BreeOge
                    last edited by

                    Worked great for the 2 days, updated to the OCT 3 update, and now all my sites are getting 502 bad gateway.  Trying the new update today.. But something is seriously messed up.

                    1 Reply Last reply Reply Quote 0
                    • B
                      BreeOge
                      last edited by

                      All I get now is upstream timed out, however nothing before this starts, it acts as if everything is ok, then bam 502 error. No Listen queue overflow nothing.  something is killing Nginx.

                      Oct 4 14:18:53	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:53 [error] 34017#100128: *892 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144432540 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:43	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:43 [error] 34017#100128: *890 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144422541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:42	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:42 [error] 49658#100106: *381 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.139, server: , request: "GET /decide?version=1&lib=android&token=0350ff19c3e127ef2c92f5cc82284391&distinct_id=d15cf0351e8da34a2092d588fd3e33ed7fecd29e HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "decide.mixpanel.com"
Oct 4 14:18:33	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:33 [error] 34017#100128: *888 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144412542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:32	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:32 [error] 49658#100106: *379 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.152, server: , request: "GET /MFEwTzBNMEswSTAJBgUrDgMCGgUABBTBL0V27RVZ7LBduom%2FnYB45SPUEwQU5Z1ZMIJHWMys%2BghUNoZ7OrUETfACEAi4elAbvpzaLRZNPjlRv1U%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "ocsp.digicert.com"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *765 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144402542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *879 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144382542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *651 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144392541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:14	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:14 [error] 49658#100106: *375 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:18:09	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:09 [error] 49658#100106: *373 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:18:01	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:01 [error] 49658#100106: *371 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:59	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:59 [error] 49658#100106: *369 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:59	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:59 [error] 49658#100106: *367 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:56	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:56 [error] 49658#100106: *363 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:54	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:54 [error] 49658#100106: *360 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:54	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:54 [error] 49411#100110: *359 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:15:53	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:53 [error] 34017#100128: *892 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144372542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:43	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:43 [error] 34017#100128: *890 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144362541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:33	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:33 [error] 34017#100128: *888 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144352542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:32	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:32 [error] 49658#100106: *335 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.152, server: , request: "GET /GIAG2.crl HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "pki.google.com"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *765 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/snort_alerts.widget.php?getNewAlerts=1507144342542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *879 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "POST /widgets/widgets/gateways.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *651 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144342542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
                      1 Reply Last reply Reply Quote 0
                      • D
                        doktornotor Banned
                        last edited by

                        As noted, you are simply overloading nginx with your CP. All the hints are already above, go on with some better tuning. Noone will do it for you.

                        1 Reply Last reply Reply Quote 0
                        • B
                          BreeOge
                          last edited by

                          I have upped the tuning, all the things you suggested. Also they does not explain the fact that it worked fine for 2 full days, after I reverted to 2.4 then on OCT 3 after the update less than an hour it started happening.  Its not overloading the CP, it looks like it is, but its not.  It just dies.. you see the CP because when it dies, it tosses the 502 and the CP stops.  I have the settings all set way up, to stop this, and it has had no effect at all!  Once the Nginx crashes nothing can work so it starts tossing out errors.  Also I have turned off the CP just to see if that was the issue, it still happened.

                          I had PFblocker installed, just removed it to see if that is causing the issue.  30 min, no crash as of yet.

                          1 Reply Last reply Reply Quote 0
                          • D
                            doktornotor Banned
                            last edited by

                            Eh well, it not just overloads nginx, it overloads the entire box apparently as mentioned earlier.

                            
Sep 25 11:25:10	kernel		sonewconn: pcb 0xfffff8004c40f690: Listen queue overflow: 193 already in queue awaiting acceptance (9 occurrences)
[/code)[/code]
                            1 Reply Last reply Reply Quote 0
                            • B
                              BreeOge
                              last edited by

                              Still good to go since I removed PFblocker, I am thinking it may have something to do with PFblocker / Snort/ or Squidguard working together on 2.4  What was updated on 2.4.0 on the 3rd?  I looked for patch notes, but could not find it.  Any file that shows what was updated on past updates?

                              When it does lock up, the SSH becomes unresponsive as well, it only shows the basic login, I have to CTRL-Z to close the process.

                              *** Welcome to pfSense 2.4.0-RC (amd64) on fort-smith ***
                              ^ZYou have stopped jobs.
                              [1] + Suspended              /etc/rc.initial

                              reboot

                              That is the only way to get it to free up.  If you try to restart PHP it will just lock up as well.

                              I also updated the listen Queue overflow, and the error for overflow does not happen now till WAY later, if its been locked up for 15 min or more.  That is a symptom.  But some file that changed on the 3rd is causing issues between pfblocker, snort, or squidguard working with each other.

                              Still working good since I removed Pfblocker, no errors what so ever in the logs, running like a champ.

                              1 Reply Last reply Reply Quote 0
                              • D
                                doktornotor Banned
                                last edited by

                                If you read the above message, your box is completely overloaded with queued ingress connections. So yeah, in that state NOTHING works properly. Even linked you to the kernel tunable.

                                Also:  http://nginx.org/en/docs/freebsd_tuning.html

                                1 Reply Last reply Reply Quote 0
                                • B
                                  BreeOge
                                  last edited by

                                  I really appreciate the help doktornotor, and I understand where your coming from and why your giving me the solutions you are suggesting.  I guess I have not explained it well enough though.  I understand that its being queued with ingress connections, but the problem I have is why does it only happen when I updated on the 3rd to the newest 2.4.0 version and not before, system load has not changed, actually if anything system load is currently low.  Problem only showed up on 2.4.1 and on 2.4.0 after Tuesdays update.  Before that I never had an issue with Overloaded ingress connections even with pfblocker installed.  I have had the system with a heaver load than it currently has and never received these errors.  I have tried the settings you suggested mine are currently

                                  kern.ipc.somaxconn=4096

                                  and

                                  backlog= 4096

                                  and I still have the issue.

                                  However when I removed pfblocker the issue stopped, but its never been an issue before, and like I said its actually low load at the current time.  It only raised its head after the 2.4.0 update on the 3rd or on the 2.4.1 updates.  Before then even on the 2.4.0 I had no issues even during high loads.  That is why I posted in the 2.4 Development Snapshots, because it only showed up after an update, something changed to cause the problem.  All I did was update to newest version, no load change or anything else. Update then issue showed up.  Again I do appreciate your help, but I guess I will have to wait till more are having the issue.  I know they will have issues, I have 2 boxes set up, one only has one to two users at any given time so super low load and even it started after the update.

                                  Thank you for your assistance. Sorry for taking your time.

                                  BreeOge

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    seanr22a
                                    last edited by

                                    I'm seeing the same thing - 502 errors. I'm running three sites all of them worked perfect until 2.4.0-RC changed from bsd 11.0 to bsd 11.1 (a few days ago)
                                    Strange thing it's only one site that have problems. The problem site is the site with most users/traffic.

                                    This morning I removed pfBlocker on the problem site and so far everything good. It's to much of a coincidence that the problem started when the move was made from bsd 11 to 11.1 a couple of days ago.

                                    I had to remove squid on all three sites (XMLRPC Sync stopped working giving a constant flow of errors) and now Pfblocker is removed on one site after the introduktion of bsd 11.1.

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      BreeOge
                                      last edited by

                                      Yea, it happens more on the one that has higher traffic, but if you leave it even on the low traffic ones they will eventually get it as well, just takes longer. Its like its not releasing something, and keeps building till it locks up..

                                      I removed all my PFblockers too.. Interesting side note as well

                                      I have 3 systems, 2 of them have Pfblocker, squid, and squidguard installed. Those are the ones getting the 502.  I have another one set up, with only Squid, and Pfblocker, and Squidguard disabled.  It has yet to get a 502 Bad gateway.  It seems to be something with the 11.1 and either Pfblocker or Squidgard or all working together, but if you  have Squid and Squidguard and no pfblocker it does not happen, same goes with Squid and Pfblocker with no Squidguard.  Its only when all 3 are present.

                                      Seanr22a do you have those 3 installed on all your systems or a mix?

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        seanr22a
                                        last edited by

                                        @BreeOge:

                                        Seanr22a do you have those 3 installed on all your systems or a mix?

                                        I have two SG-2440 and one SG-8860. They all have Snort, Squid and Pfblocker. The configuration is identical at these sites so I use XMLRPC Sync in all three packages.
                                        At the two sites still running Pfblocker I have moved the IP blocklists  to Snort and only using DNSbl in the Pfblocker package. I will wait a couple of days if its still fine I will install Pfblocker using only DNSbl at the main site as well and give it a try. My problem with squid is the XMLRPC Sync but all this started to happen with bsd 11.1

                                        To me it looks like it's running out of resources of some kind. I have a lot memory and diskspace available when it happens so its something else.

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          BreeOge
                                          last edited by

                                          I have a lot memory and diskspace available when it happens so its something else.

                                          Same here..

                                          The release notes for 11.1 are here https://www.freebsd.org/releases/11.1R/relnotes.html#kernel

                                          Lots of changed for the kernel.  I suspect one of these are the cause, due to it crashed the kernel but I could be wrong on that.  Also I have no idea what would cause it.  just something to look at.

                                          I know it looks like the system is being overloaded like doktornotor said, but hard to believe a system that never overloaded before the update now overloads constantly with the same load or less.  Does not make sense.

                                          1 Reply Last reply Reply Quote 0
                                          • luckman212L
                                            luckman212 LAYER 8
                                            last edited by

                                            Hmm I'm not running into any issues on my test 2.4.1 box, but to be fair, I'm not running Snort of pfBNG. Hope this doesn't turn out to be a nasty bug in the kernel that could delay 2.4.x getting out the door. Seems like every time 2.4 is about to drop something else comes up. Been 542 days since 2.3-RELEASE :)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post