New 502 Bad Gateway

BreeOge

EDIT: tl;dr version of the thread: To anyone running DNSBL and experiencing periodic 502 errors or other PHP failures, it is due to a file locking issue in DNSBL.

Update to the latest pfBlocker (2.1.2 or later) as it contains a fix for the locking issue leading to these conditions. After updating, ideally you will want to reboot the firewall to make sure it's starting with a clean slate. At the very least, stop and restart the DNSBL daemon.

-jimp

Per Jim Pingle this is not the same 502 error other have been having. So I am posting here to see if I can get some help..

I am still getting 502 Bad Gateway every day, so bad its about once an hour.. Here is what I can get when its locked up.. I have been fighting with this for days. I have 2 systems exactly the same, this is the ONLY one having this issue, however this system does have a much higher utilization as well.

In the logs all I ever receive is

Sep 25 11:25:10	kernel		sonewconn: pcb 0xfffff8004c40f690: Listen queue overflow: 193 already in queue awaiting acceptance (9 occurrences)
Sep 25 11:25:10	fort-smith.rentlynnwood.com		nginx: 2017/09/25 11:25:10 [error] 49549#100109: *18700 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.11, server: , request: "GET /generate_204 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "connectivitycheck.gstatic.com"
Sep 25 11:24:54	fort-smith.rentlynnwood.com		nginx: 2017/09/25 11:24:54 [error] 49549#100109: *18697 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.98, server: , request: "POST /c/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "helptaptapdash1.ksmobile.com"
Sep 25 11:24:54	fort-smith.rentlynnwood.com		nginx: 2017/09/25 11:24:54 [error] 49549#100109: *18695 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.98, server: , request: "POST /c/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "helptaptapdash1.ksmobile.com"
Sep 25 11:24:52	fort-smith.rentlynnwood.com		nginx: 2017/09/25 11:24:52 [error] 49549#100109: *18693 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.98, server: , request: "POST /c/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "helptaptapdash1.ksmobile.com"
Sep 25 11:24:52	fort-smith.rentlynnwood.com		nginx: 2017/09/25 11:24:52 [error] 49549#100109: *18691 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.98, server: , request: "POST /c/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "helptaptapdash1.ksmobile.com"

This is some logs I can get before I have to reboot.

*** Welcome to pfSense 2.4.1-DEVELOPMENT (amd64) on fort-smith ***
^ZYou have stopped jobs.
[1] + Suspended               /etc/rc.initial
# uname -a
FreeBSD fort-smith.rentlynnwood.com 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #300   r313908+13ee0afae40(RELENG_2_4): Sun Sep 24 21:54:58 CDT 2017     root@buildbot  2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense  amd64
# pkg info -x pfsense -x php
^X^Z[2] + Suspended               pkg info -x pfsense -x php
# cat /var/log/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p1 #300 r313908+13ee0afae40(RELENG_2_4): Sun Sep 24 21:54:58 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(efifb): resolution 800x600
CPU: Intel(R) Pentium(R) CPU G3220 @ 3.00GHz (3000.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x4ddaebbf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,xsave,osxsave,rdrand>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x21 <lahf,abm>Structured Extended Features=0x2603 <fsgsbase,tscadj,erms,invpcid,nfpusg>XSAVE Features=0x1 <xsaveopt>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8159846400 (7781 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <superm smci--mb="">
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1500032174 Hz quality 1000
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80683490, 0) error 1
random: entropy device external interface
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c570, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c620, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c6d0, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80683330, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806833e0, 0) error 1
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG" 
nexus0
cryptosoft0: <software crypto=""> on motherboard
padlock0: No ACE support.
acpi0: <superm smci--mb=""> on motherboard
acpi0: Power Button (fixed)
cpu0: <acpi cpu=""> on acpi0
cpu1: <acpi cpu=""> on acpi0
hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
Event timer "HPET5" frequency 14318180 Hz quality 440
Event timer "HPET6" frequency 14318180 Hz quality 440
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <acpi pci="" bus=""> on pcib0
vgapci0: <vga-compatible display=""> port 0xf000-0xf03f mem 0xf7400000-0xf77fffff,0xe8000000-0xefffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
hdac0: <intel haswell="" hda="" controller=""> mem 0xf7c34000-0xf7c37fff irq 16 at device 3.0 on pci0
xhci0: <intel lynx="" point="" usb="" 3.0="" controller=""> mem 0xf7c20000-0xf7c2ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms=""> at device 22.0 (no driver attached)
em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k=""> port 0xf060-0xf07f mem 0xf7c00000-0xf7c1ffff,0xf7c3d000-0xf7c3dfff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 0c:c4:7a:6b:41:00
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0: <intel lynx="" point="" usb="" 2.0="" controller="" usb-b=""> mem 0xf7c3c000-0xf7c3c3ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
usbus1: 480Mbps High Speed USB v2.0
hdac1: <intel lynx="" point="" hda="" controller=""> mem 0xf7c30000-0xf7c33fff irq 22 at device 27.0 on pci0
pcib1: <acpi pci-pci="" bridge=""> irq 16 at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pcib2: <acpi pci-pci="" bridge=""> irq 19 at device 28.3 on pci0
pci1: <acpi pci="" bus=""> on pcib2
igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0xe000-0xe01f mem 0xf7b00000-0xf7b7ffff,0xf7b80000-0xf7b83fff irq 19 at device 0.0 on pci1
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 0c:c4:7a:6b:41:01
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <acpi pci-pci="" bridge=""> irq 17 at device 28.5 on pci0
pci2: <acpi pci="" bus=""> on pcib3
pcib4: <acpi pci-pci="" bridge=""> at device 0.0 on pci2
pci3: <acpi pci="" bus=""> on pcib4
fwohci0: <texas instruments="" tsb43ab22="" a=""> mem 0xf7a04000-0xf7a047ff,0xf7a00000-0xf7a03fff irq 16 at device 3.0 on pci3
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:30:48:00:00:12:1a:63
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <ieee1394(firewire) bus=""> on fwohci0
sbp0: <sbp-2 scsi="" over="" firewire=""> on firewire0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: PhysicalUpperBound register is not implemented.  Physical memory access is limited to the first 4GB
fwohci0: PhysicalUpperBound = 0x00000000
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
pcib5: <acpi pci-pci="" bridge=""> irq 18 at device 28.6 on pci0
pci4: <acpi pci="" bus=""> on pcib5
xhci1: <nec upd720202="" usb="" 3.0="" controller=""> mem 0xf7900000-0xf7901fff irq 18 at device 0.0 on pci4
xhci1: 64 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table
usbus2 on xhci1
usbus2: 5.0Gbps Super Speed USB v3.0
pcib6: <acpi pci-pci="" bridge=""> irq 19 at device 28.7 on pci0
pci5: <acpi pci="" bus=""> on pcib6
ahci0: <asmedia asm1062="" ahci="" sata="" controller=""> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xf7800000-0xf78001ff irq 19 at device 0.0 on pci5
ahci0: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahci0: quirks=0xc00000 <noccs,noaux>ahcich0: <ahci channel=""> at channel 0 on ahci0
ahcich1: <ahci channel=""> at channel 1 on ahci0
ehci1: <intel lynx="" point="" usb="" 2.0="" controller="" usb-a=""> mem 0xf7c3b000-0xf7c3b3ff irq 23 at device 29.0 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
usbus3: 480Mbps High Speed USB v2.0
isab0: <pci-isa bridge=""> at device 31.0 on pci0
isa0: <isa bus=""> on isab0
ahci1: <intel lynx="" point="" ahci="" sata="" controller=""> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf040-0xf05f mem 0xf7c3a000-0xf7c3a7ff irq 19 at device 31.2 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich2: <ahci channel=""> at channel 0 on ahci1
ahcich3: <ahci channel=""> at channel 1 on ahci1
ahcich4: <ahci channel=""> at channel 2 on ahci1
ahcich5: <ahci channel=""> at channel 3 on ahci1
ahcich6: <ahci channel=""> at channel 4 on ahci1
ahcich7: <ahci channel=""> at channel 5 on ahci1
ahciem0: <ahci enclosure="" management="" bridge=""> on ahci1
acpi_button0: <power button=""> on acpi0
acpi_button1: <sleep button=""> on acpi0
acpi_tz0: <thermal zone=""> on acpi0
acpi_tz1: <thermal zone=""> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: <enhanced speedstep="" frequency="" control=""> on cpu0
est1: <enhanced speedstep="" frequency="" control=""> on cpu1
Timecounters tick every 1.000 msec
nvme cam probe device init
hdacc0: <intel haswell="" hda="" codec=""> at cad 0 on hdac0
hdaa0: <intel haswell="" audio="" function="" group=""> at nid 1 on hdacc0
pcm0: <intel haswell="" (hdmi="" dp="" 8ch)=""> at nid 3 on hdaa0
hdacc1: <realtek alc1150="" hda="" codec=""> at cad 0 on hdac1
hdaa1: <realtek alc1150="" audio="" function="" group=""> at nid 1 on hdacc1
pcm1: <realtek alc1150="" (analog="" 5.1+hp="" 2.0)=""> at nid 20,22,21,27 and 24,25,26 on hdaa1
pcm2: <realtek alc1150="" (internal="" digital)=""> at nid 17 on hdaa1
pcm3: <realtek alc1150="" (rear="" digital)=""> at nid 30 on hdaa1
ugen3.1: <intel ehci="" root="" hub=""> at usbus3
ugen1.1: <intel ehci="" root="" hub=""> at usbus1
uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus3
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus1
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen2.1: <0x1912 XHCI root HUB> at usbus2
uhub3: <0x1912 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2
uhub3: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub2: 21 ports with 21 removable, self powered
ugen1.2: <vendor 0x8087="" product="" 0x8008=""> at usbus1
uhub4 on uhub1
uhub4: <vendor 2="" 9="" 0x8087="" product="" 0x8008,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr=""> on usbus1
ugen3.2: <vendor 0x8087="" product="" 0x8000=""> at usbus3
uhub5 on uhub0
uhub5: <vendor 2="" 9="" 0x8087="" product="" 0x8000,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr=""> on usbus3
uhub4: 6 ports with 6 removable, self powered
uhub5: 8 ports with 8 removable, self powered
ada0 at ahcich2 bus 0 scbus3 target 0 lun 0
ada0: <wdc wd10eurx-63fh1y0="" 01.01a01=""> ATA8-ACS SATA 3.x device
ada0: Serial Number WD-WCC1U4873857
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors)
ada0: quirks=0x1<4K>
ses0 at ahciem0 bus 0 scbus9 target 0 lun 0
ses0: <ahci sgpio="" enclosure="" 1.00="" 0001=""> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
Trying to mount root from ufs:/dev/gptid/d28d5113-7e41-11e7-b94c-0cc47a6b4100 [rw]...
#</ahci></wdc></vendor></vendor></vendor></vendor></intel></intel></intel></intel></realtek></realtek></realtek></realtek></realtek></intel></intel></intel></enhanced></enhanced></at></keyboard></thermal></thermal></sleep></power></ahci></ahci></ahci></ahci></ahci></ahci></ahci></intel></isa></pci-isa></intel></ahci></ahci></noccs,noaux></asmedia></acpi></acpi></nec></acpi></acpi></sbp-2></ieee1394(firewire)></texas></acpi></acpi></acpi></acpi></intel(r)></acpi></acpi></acpi></intel></intel></intel(r)></simple></intel></intel></vga-compatible></acpi></acpi></at></at></high></acpi></acpi></superm></software></version></superm></xsaveopt></fsgsbase,tscadj,erms,invpcid,nfpusg></lahf,abm></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,xsave,osxsave,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>

Here is what I am getting AFTER a reboot, you can see I am on the most recent snapshot.

[2.4.1-DEVELOPMENT][root@fort-smith.rentlynnwood.com]/root: uname -a
FreeBSD fort-smith.rentlynnwood.com 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #300 r313908+13ee0afae40(RELENG_2_4): Sun Sep 24 21:54:58 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense  amd64
[2.4.1-DEVELOPMENT][root@fort-smith.rentlynnwood.com]/root: pkg info -x pfsense -x php
pfSense-2.4.1.a.20170924.1424
pfSense-Status_Monitoring-1.7.2
pfSense-base-2.4.1.a.20170924.2137
pfSense-default-config-2.4.1.a.20170924.2137
pfSense-kernel-pfSense-2.4.1.a.20170924.2137
pfSense-pkg-Cron-0.3.7_2
pfSense-pkg-Notes-0.2.9_2
pfSense-pkg-Service_Watchdog-1.8.4
pfSense-pkg-Shellcmd-1.0.3
pfSense-pkg-acme-0.1.19_1
pfSense-pkg-freeradius3-0.15.1
pfSense-pkg-ntopng-0.8.10
pfSense-pkg-pfBlockerNG-2.1.1_10
pfSense-pkg-snort-3.2.9.5_1
pfSense-pkg-squid-0.4.39
pfSense-pkg-squidGuard-1.16.3
pfSense-rc-2.4.1.a.20170924.2137
pfSense-repo-2.4.1.a.20170924.1424
pfSense-upgrade-0.27_3
php56-pfSense-module-0.51
pear-XML_RPC2-1.1.3
php-xdebug-2.5.0
php56-xml-5.6.31
php56-xmlreader-5.6.31
php56-xmlwriter-5.6.31
php-suhosin-0.9.38_3
php-xdebug-2.5.0
php56-5.6.31
php56-bcmath-5.6.31
php56-bz2-5.6.31
php56-ctype-5.6.31
php56-curl-5.6.31
php56-dom-5.6.31
php56-filter-5.6.31
php56-ftp-5.6.31
php56-gettext-5.6.31
php56-hash-5.6.31
php56-json-5.6.31
php56-ldap-5.6.31
php56-mbstring-5.6.31_1
php56-mcrypt-5.6.31
php56-mysql-5.6.31
php56-opcache-5.6.31
php56-openssl-5.6.31
php56-pcntl-5.6.31
php56-pdo-5.6.31
php56-pdo_sqlite-5.6.31
php56-pfSense-module-0.51
php56-posix-5.6.31
php56-readline-5.6.31
php56-session-5.6.31
php56-shmop-5.6.31
php56-simplexml-5.6.31
php56-sockets-5.6.31
php56-sqlite3-5.6.31
php56-sysvmsg-5.6.31
php56-sysvsem-5.6.31
php56-sysvshm-5.6.31
php56-tokenizer-5.6.31
php56-xml-5.6.31
php56-xmlreader-5.6.31
php56-xmlwriter-5.6.31
php56-zlib-5.6.31
[2.4.1-DEVELOPMENT][root@fort-smith.rentlynnwood.com]/root: cat /var/log/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p1 #300 r313908+13ee0afae40(RELENG_2_4): Sun Sep 24 21:54:58 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(efifb): resolution 800x600
CPU: Intel(R) Pentium(R) CPU G3220 @ 3.00GHz (3000.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x4ddaebbf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,xsave,osxsave,rdrand>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x21 <lahf,abm>Structured Extended Features=0x2603 <fsgsbase,tscadj,erms,invpcid,nfpusg>XSAVE Features=0x1 <xsaveopt>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8159846400 (7781 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <superm smci--mb="">
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1500030774 Hz quality 1000
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80683490, 0) error 1
random: entropy device external interface
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c570, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c620, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c6d0, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80683330, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806833e0, 0) error 1
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG" 
nexus0
cryptosoft0: <software crypto=""> on motherboard
padlock0: No ACE support.
acpi0: <superm smci--mb=""> on motherboard
acpi0: Power Button (fixed)
cpu0: <acpi cpu=""> on acpi0
cpu1: <acpi cpu=""> on acpi0
hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
Event timer "HPET5" frequency 14318180 Hz quality 440
Event timer "HPET6" frequency 14318180 Hz quality 440
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <acpi pci="" bus=""> on pcib0
vgapci0: <vga-compatible display=""> port 0xf000-0xf03f mem 0xf7400000-0xf77fffff,0xe8000000-0xefffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
hdac0: <intel haswell="" hda="" controller=""> mem 0xf7c34000-0xf7c37fff irq 16 at device 3.0 on pci0
xhci0: <intel lynx="" point="" usb="" 3.0="" controller=""> mem 0xf7c20000-0xf7c2ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms=""> at device 22.0 (no driver attached)
em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k=""> port 0xf060-0xf07f mem 0xf7c00000-0xf7c1ffff,0xf7c3d000-0xf7c3dfff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 0c:c4:7a:6b:41:00
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0: <intel lynx="" point="" usb="" 2.0="" controller="" usb-b=""> mem 0xf7c3c000-0xf7c3c3ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
usbus1: 480Mbps High Speed USB v2.0
hdac1: <intel lynx="" point="" hda="" controller=""> mem 0xf7c30000-0xf7c33fff irq 22 at device 27.0 on pci0
pcib1: <acpi pci-pci="" bridge=""> irq 16 at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pcib2: <acpi pci-pci="" bridge=""> irq 19 at device 28.3 on pci0
pci1: <acpi pci="" bus=""> on pcib2
igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0xe000-0xe01f mem 0xf7b00000-0xf7b7ffff,0xf7b80000-0xf7b83fff irq 19 at device 0.0 on pci1
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 0c:c4:7a:6b:41:01
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <acpi pci-pci="" bridge=""> irq 17 at device 28.5 on pci0
pci2: <acpi pci="" bus=""> on pcib3
pcib4: <acpi pci-pci="" bridge=""> at device 0.0 on pci2
pci3: <acpi pci="" bus=""> on pcib4
fwohci0: <texas instruments="" tsb43ab22="" a=""> mem 0xf7a04000-0xf7a047ff,0xf7a00000-0xf7a03fff irq 16 at device 3.0 on pci3
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:30:48:00:00:12:1a:63
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <ieee1394(firewire) bus=""> on fwohci0
sbp0: <sbp-2 scsi="" over="" firewire=""> on firewire0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: PhysicalUpperBound register is not implemented.  Physical memory access is limited to the first 4GB
fwohci0: PhysicalUpperBound = 0x00000000
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
pcib5: <acpi pci-pci="" bridge=""> irq 18 at device 28.6 on pci0
pci4: <acpi pci="" bus=""> on pcib5
xhci1: <nec upd720202="" usb="" 3.0="" controller=""> mem 0xf7900000-0xf7901fff irq 18 at device 0.0 on pci4
xhci1: 64 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table
usbus2 on xhci1
usbus2: 5.0Gbps Super Speed USB v3.0
pcib6: <acpi pci-pci="" bridge=""> irq 19 at device 28.7 on pci0
pci5: <acpi pci="" bus=""> on pcib6
ahci0: <asmedia asm1062="" ahci="" sata="" controller=""> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xf7800000-0xf78001ff irq 19 at device 0.0 on pci5
ahci0: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahci0: quirks=0xc00000 <noccs,noaux>ahcich0: <ahci channel=""> at channel 0 on ahci0
ahcich1: <ahci channel=""> at channel 1 on ahci0
ehci1: <intel lynx="" point="" usb="" 2.0="" controller="" usb-a=""> mem 0xf7c3b000-0xf7c3b3ff irq 23 at device 29.0 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
usbus3: 480Mbps High Speed USB v2.0
isab0: <pci-isa bridge=""> at device 31.0 on pci0
isa0: <isa bus=""> on isab0
ahci1: <intel lynx="" point="" ahci="" sata="" controller=""> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf040-0xf05f mem 0xf7c3a000-0xf7c3a7ff irq 19 at device 31.2 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich2: <ahci channel=""> at channel 0 on ahci1
ahcich3: <ahci channel=""> at channel 1 on ahci1
ahcich4: <ahci channel=""> at channel 2 on ahci1
ahcich5: <ahci channel=""> at channel 3 on ahci1
ahcich6: <ahci channel=""> at channel 4 on ahci1
ahcich7: <ahci channel=""> at channel 5 on ahci1
ahciem0: <ahci enclosure="" management="" bridge=""> on ahci1
acpi_button0: <power button=""> on acpi0
acpi_button1: <sleep button=""> on acpi0
acpi_tz0: <thermal zone=""> on acpi0
acpi_tz1: <thermal zone=""> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: <enhanced speedstep="" frequency="" control=""> on cpu0
est1: <enhanced speedstep="" frequency="" control=""> on cpu1
Timecounters tick every 1.000 msec
nvme cam probe device init
hdacc0: <intel haswell="" hda="" codec=""> at cad 0 on hdac0
hdaa0: <intel haswell="" audio="" function="" group=""> at nid 1 on hdacc0
pcm0: <intel haswell="" (hdmi="" dp="" 8ch)=""> at nid 3 on hdaa0
hdacc1: <realtek alc1150="" hda="" codec=""> at cad 0 on hdac1
hdaa1: <realtek alc1150="" audio="" function="" group=""> at nid 1 on hdacc1
pcm1: <realtek alc1150="" (analog="" 5.1+hp="" 2.0)=""> at nid 20,22,21,27 and 24,25,26 on hdaa1
pcm2: <realtek alc1150="" (internal="" digital)=""> at nid 17 on hdaa1
pcm3: <realtek alc1150="" (rear="" digital)=""> at nid 30 on hdaa1
ugen1.1: <intel ehci="" root="" hub=""> at usbus1
ugen3.1: <intel ehci="" root="" hub=""> at usbus3
uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus1
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus3
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen2.1: <0x1912 XHCI root HUB> at usbus2
uhub3: <0x1912 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2
uhub3: 4 ports with 4 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 21 ports with 21 removable, self powered
ugen3.2: <vendor 0x8087="" product="" 0x8000=""> at usbus3
uhub4 on uhub1
uhub4: <vendor 2="" 9="" 0x8087="" product="" 0x8000,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr=""> on usbus3
ugen1.2: <vendor 0x8087="" product="" 0x8008=""> at usbus1
uhub5 on uhub0
uhub5: <vendor 2="" 9="" 0x8087="" product="" 0x8008,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr=""> on usbus1
uhub5: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ada0 at ahcich2 bus 0 scbus3 target 0 lun 0
ada0: <wdc wd10eurx-63fh1y0="" 01.01a01=""> ATA8-ACS SATA 3.x device
ada0: Serial Number WD-WCC1U4873857
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors)
ada0: quirks=0x1<4K>
ses0 at ahciem0 bus 0 scbus9 target 0 lun 0
ses0: <ahci sgpio="" enclosure="" 1.00="" 0001=""> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
Trying to mount root from ufs:/dev/gptid/d28d5113-7e41-11e7-b94c-0cc47a6b4100 [rw]...
WARNING: / was not properly dismounted</ahci></wdc></vendor></vendor></vendor></vendor></intel></intel></intel></intel></realtek></realtek></realtek></realtek></realtek></intel></intel></intel></enhanced></enhanced></at></keyboard></thermal></thermal></sleep></power></ahci></ahci></ahci></ahci></ahci></ahci></ahci></intel></isa></pci-isa></intel></ahci></ahci></noccs,noaux></asmedia></acpi></acpi></nec></acpi></acpi></sbp-2></ieee1394(firewire)></texas></acpi></acpi></acpi></acpi></intel(r)></acpi></acpi></acpi></intel></intel></intel(r)></simple></intel></intel></vga-compatible></acpi></acpi></at></at></high></acpi></acpi></superm></software></version></superm></xsaveopt></fsgsbase,tscadj,erms,invpcid,nfpusg></lahf,abm></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,movbe,popcnt,tscdlt,xsave,osxsave,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>

doktornotor

Read this: https://forum.pfsense.org/index.php?topic=136847.msg748761#msg748761

BreeOge

Thank you, ill try that to see if it works..

BreeOge

That was not it, did it again, and I received 0 errors /var/run/php-fpm.socket failed. Actually the logs didn't show anything at all.. or nothing out of the normal should I say. Ill keep watching it, and if it happens gain try to find out something that might help in figuring out what is causing it.

doktornotor

@BreeOge:

That was not it, did it again, and I received 0 errors /var/run/php-fpm.socket failed. Actually the logs didn't show anything at all.. or nothing out of the normal should I say.

So, you actually mean that it worked?

BreeOge

No, it cleared the errors that you said was the cause, however I am still getting the 502 bad gateway error, not as often, but still getting them.

doktornotor

Well, then you'll need this as well, probably…

BreeOge

Thank you sir. I will try that as well. I thank you for your help.

BreeOge

I have a question Doktornotor, will rc.php_ini_setup get overwritten on updates?

BreeOge

Also its still happening.. even with the changes you suggested :(

BreeOge

Update* Reverted back to 2.4.0 and its no longer happening, only happens with the 2.4.1 version!

BreeOge

Worked great for the 2 days, updated to the OCT 3 update, and now all my sites are getting 502 bad gateway. Trying the new update today.. But something is seriously messed up.

BreeOge

All I get now is upstream timed out, however nothing before this starts, it acts as if everything is ok, then bam 502 error. No Listen queue overflow nothing. something is killing Nginx.

Oct 4 14:18:53	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:53 [error] 34017#100128: *892 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144432540 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:43	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:43 [error] 34017#100128: *890 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144422541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:42	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:42 [error] 49658#100106: *381 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.139, server: , request: "GET /decide?version=1&lib=android&token=0350ff19c3e127ef2c92f5cc82284391&distinct_id=d15cf0351e8da34a2092d588fd3e33ed7fecd29e HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "decide.mixpanel.com"
Oct 4 14:18:33	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:33 [error] 34017#100128: *888 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144412542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:32	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:32 [error] 49658#100106: *379 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.152, server: , request: "GET /MFEwTzBNMEswSTAJBgUrDgMCGgUABBTBL0V27RVZ7LBduom%2FnYB45SPUEwQU5Z1ZMIJHWMys%2BghUNoZ7OrUETfACEAi4elAbvpzaLRZNPjlRv1U%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "ocsp.digicert.com"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *765 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144402542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *879 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144382542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:23 [error] 34017#100128: *651 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144392541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:18:14	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:14 [error] 49658#100106: *375 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:18:09	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:09 [error] 49658#100106: *373 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:18:01	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:18:01 [error] 49658#100106: *371 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:59	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:59 [error] 49658#100106: *369 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:59	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:59 [error] 49658#100106: *367 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:56	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:56 [error] 49658#100106: *363 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:54	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:54 [error] 49658#100106: *360 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:17:54	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:17:54 [error] 49411#100110: *359 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.124, server: , request: "POST /news/report HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "n.m.ksmobile.net"
Oct 4 14:15:53	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:53 [error] 34017#100128: *892 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144372542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:43	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:43 [error] 34017#100128: *890 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144362541 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:33	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:33 [error] 34017#100128: *888 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144352542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:32	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:32 [error] 49658#100106: *335 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.152, server: , request: "GET /GIAG2.crl HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "pki.google.com"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *765 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/snort_alerts.widget.php?getNewAlerts=1507144342542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *879 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "POST /widgets/widgets/gateways.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"
Oct 4 14:15:23	fort-smith.rentlynnwood.com		nginx: 2017/10/04 14:15:23 [error] 34017#100128: *651 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 70.178.196.154, server: , request: "GET /widgets/widgets/pfblockerng.widget.php?getNewCounts=1507144342542 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fort-smith.rentlynnwood.com", referrer: "https://fort-smith.rentlynnwood.com/index.php"

doktornotor

As noted, you are simply overloading nginx with your CP. All the hints are already above, go on with some better tuning. Noone will do it for you.

BreeOge

I have upped the tuning, all the things you suggested. Also they does not explain the fact that it worked fine for 2 full days, after I reverted to 2.4 then on OCT 3 after the update less than an hour it started happening. Its not overloading the CP, it looks like it is, but its not. It just dies.. you see the CP because when it dies, it tosses the 502 and the CP stops. I have the settings all set way up, to stop this, and it has had no effect at all! Once the Nginx crashes nothing can work so it starts tossing out errors. Also I have turned off the CP just to see if that was the issue, it still happened.

I had PFblocker installed, just removed it to see if that is causing the issue. 30 min, no crash as of yet.

doktornotor

Eh well, it not just overloads nginx, it overloads the entire box apparently as mentioned earlier.


Sep 25 11:25:10	kernel		sonewconn: pcb 0xfffff8004c40f690: Listen queue overflow: 193 already in queue awaiting acceptance (9 occurrences)
[/code)[/code]

BreeOge

Still good to go since I removed PFblocker, I am thinking it may have something to do with PFblocker / Snort/ or Squidguard working together on 2.4 What was updated on 2.4.0 on the 3rd? I looked for patch notes, but could not find it. Any file that shows what was updated on past updates?

When it does lock up, the SSH becomes unresponsive as well, it only shows the basic login, I have to CTRL-Z to close the process.

*** Welcome to pfSense 2.4.0-RC (amd64) on fort-smith ***
^ZYou have stopped jobs.
[1] + Suspended /etc/rc.initial

reboot

That is the only way to get it to free up. If you try to restart PHP it will just lock up as well.

I also updated the listen Queue overflow, and the error for overflow does not happen now till WAY later, if its been locked up for 15 min or more. That is a symptom. But some file that changed on the 3rd is causing issues between pfblocker, snort, or squidguard working with each other.

Still working good since I removed Pfblocker, no errors what so ever in the logs, running like a champ.

doktornotor

If you read the above message, your box is completely overloaded with queued ingress connections. So yeah, in that state NOTHING works properly. Even linked you to the kernel tunable.

Also: http://nginx.org/en/docs/freebsd_tuning.html

BreeOge

I really appreciate the help doktornotor, and I understand where your coming from and why your giving me the solutions you are suggesting. I guess I have not explained it well enough though. I understand that its being queued with ingress connections, but the problem I have is why does it only happen when I updated on the 3rd to the newest 2.4.0 version and not before, system load has not changed, actually if anything system load is currently low. Problem only showed up on 2.4.1 and on 2.4.0 after Tuesdays update. Before that I never had an issue with Overloaded ingress connections even with pfblocker installed. I have had the system with a heaver load than it currently has and never received these errors. I have tried the settings you suggested mine are currently

kern.ipc.somaxconn=4096

and

backlog= 4096

and I still have the issue.

However when I removed pfblocker the issue stopped, but its never been an issue before, and like I said its actually low load at the current time. It only raised its head after the 2.4.0 update on the 3rd or on the 2.4.1 updates. Before then even on the 2.4.0 I had no issues even during high loads. That is why I posted in the 2.4 Development Snapshots, because it only showed up after an update, something changed to cause the problem. All I did was update to newest version, no load change or anything else. Update then issue showed up. Again I do appreciate your help, but I guess I will have to wait till more are having the issue. I know they will have issues, I have 2 boxes set up, one only has one to two users at any given time so super low load and even it started after the update.

Thank you for your assistance. Sorry for taking your time.

BreeOge

seanr22a

I'm seeing the same thing - 502 errors. I'm running three sites all of them worked perfect until 2.4.0-RC changed from bsd 11.0 to bsd 11.1 (a few days ago)
Strange thing it's only one site that have problems. The problem site is the site with most users/traffic.

This morning I removed pfBlocker on the problem site and so far everything good. It's to much of a coincidence that the problem started when the move was made from bsd 11 to 11.1 a couple of days ago.

I had to remove squid on all three sites (XMLRPC Sync stopped working giving a constant flow of errors) and now Pfblocker is removed on one site after the introduktion of bsd 11.1.