Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New 502 Bad Gateway

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    281 Posts 67 Posters 218.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      BreeOge
      last edited by

      @AhnHEL:

      I have one box using the ZFS file structure, the other is using UFS, both using pfBlockerNG.  The ZFS is rock solid, and the UFS one gets the Bad Gateway after some time.  Wondering if that is a possible reason why two similar boxes with similar settings exhibit different behavior using the same snapshot and same packages.

      Both running 20171009 Snapshots for 2.4.0

      Just a thought

      Very interesting mine are UFS as well..

      1 Reply Last reply Reply Quote 0
      • john_galtJ
        john_galt
        last edited by

        BreeOge

        I have a time frame on my pfSense. I had to reboot when I got home at 4pm. It's now 7:50pm and I am getting the 502 Bad Gateway error. I'm going to reboot and scrub the system log between those times and if someone want's to look at it fine. Regardless I'm going ahead with my plans to reinstall this weekend.

        Doug

        Doug

        1 Reply Last reply Reply Quote 0
        • B
          BreeOge
          last edited by

          It locked, here is the logs from the info you wanted..  This is from my heavier load box.  My low load box also crashed, I will make a new post for it.

          # /usr/sbin/swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/gptid/d2a5a9dd-7e41-11e7-b   3684016       0B     3.5G     0%


          # /usr/bin/top | /usr/bin/head -n7
last pid: 50984;  load averages:  0.07,  0.14,  0.15  up 0+23:33:41    21:18:05
134 processes: 1 running, 131 sleeping, 2 stopped

Mem: 1036M Active, 2017M Inact, 1009M Wired, 693M Buf, 3775M Free
Swap: 3598M Total, 3598M Free


          /usr/bin/netstat -Ln
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
Netgraph sockets
Type  Recv-Q Send-Q Node Address   #Hooks
ctrl       0      0 [1b31]:           0
ctrl       0      0 [1b22]:           0
ctrl       0      0 [1ad8]:           0
ctrl       0      0 [1ad2]:           0
ctrl       0      0 [14fb]:           0
ctrl       0      0 [5]:              0
unix  100/0/200                        /var/run/clamav/clamd.sock
unix  0/0/80                           /tmp/mysql.sock
unix  0/0/5                            /var/run/dpinger_WAN_DHCP~70.178.22.158~70.178.22.1.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/30                           /var/run/check_reload_status
unix  193/0/128                        /var/run/php-fpm.socket
unix  0/0/4                            /var/run/devd.seqpacket.pipe

          Had to pastbin /usr/bin/netstat -xn

          https://pastebin.com/RKzat3ue

          1 Reply Last reply Reply Quote 0
          • B
            BreeOge
            last edited by

            This is on my lower load box

            # /usr/sbin/swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/label/swap0  33554428       0B      32G     0%

            # /usr/bin/top | /usr/bin/head -n7
last pid: 31956;  load averages:  0.02,  0.07,  0.07  up 0+07:41:27    21:21:32
96 processes:  1 running, 93 sleeping, 2 stopped

Mem: 601M Active, 1095M Inact, 1002M Wired, 636M Buf, 13G Free
Swap: 32G Total, 32G Free

            Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
Netgraph sockets
Type  Recv-Q Send-Q Node Address   #Hooks
ctrl       0      0 [23c]:            0
ctrl       0      0 [22b]:            0
ctrl       0      0 [1e3]:            0
ctrl       0      0 [199]:            0
ctrl       0      0 [f]:              0
ctrl       0      0 [5]:              0
unix  100/0/128                        /var/run/clamav/clamd.sock
unix  0/0/5                            /var/run/dpinger_WAN_DHCP~70.178.196.154~70.178.196.1.sock
unix  0/0/5                            /var/run/dpinger_Steve_Telephone~192.168.16.2~10.10.10.2.sock
unix  0/0/5                            /var/run/dpinger_Raymond_Telephone~192.168.16.2~10.10.12.2.sock
unix  0/0/5                            /var/run/dpinger_Kevin_Telephone~192.168.16.2~10.10.11.2.sock
unix  0/0/5                            /var/run/dpinger_Cisco_Router~192.168.16.2~192.168.16.201.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/30                           /var/run/check_reload_status
unix  13/0/128                         /var/run/php-fpm.socket
unix  0/0/4                            /var/run/devd.seqpacket.pipe

            Pastbin for /usr/bin/netstat -xn

            https://pastebin.com/6tEs3Fs0

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              Lots of port 1344, do you have squid+clamav active as well? Can you try shutting that off?

              Also that netstat -x output is too big to put inline, you should attach that in a .txt file

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • B
                BreeOge
                last edited by

                @jimp:

                Lots of port 1344, do you have squid+clamav active as well? Can you try shutting that off?

                Also that netstat -x output is too big to put inline, you should attach that in a .txt file

                I disabled squid+Clamav, when it crashes ill get ya the logs again for that as well.

                1 Reply Last reply Reply Quote 0
                • john_galtJ
                  john_galt
                  last edited by

                  Hi

                  This morning I could not connect to the GUI nor was the serial console working. I rebooted then downloaded the system.log
                  It was nothing but what you see below. I only posted the last few lines before I restarted the firewall. I hope this helps.

                  Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 12 05:44:37 pfSense syslogd: kernel boot file is /boot/kernel/kernel
Oct 12 05:44:37 pfSense kernel: Copyright (c) 1992-2017 The FreeBSD Project.
Oct 12 05:44:37 pfSense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Oct 12 05:44:37 pfSense kernel: The Regents of the University of California. All rights reserved.
Oct 12 05:44:37 pfSense kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Oct 12 05:44:37 pfSense kernel: FreeBSD 11.1-RELEASE-p1 #82 r313908+d77c47fe50c(RELENG_2_4_0): Mon Oct  9 18:04:16 CDT 2017
Oct 12 05:44:37 pfSense kernel: root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense amd64
Oct 12 05:44:37 pfSense kernel: FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Oct 12 05:44:37 pfSense kernel: VT(efifb): resolution 800x600
Oct 12 05:44:37 pfSense kernel: CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz (1596.34-MHz K8-class CPU)
Oct 12 05:44:37 pfSense kernel: Origin="GenuineIntel"  Id=0x306d4  Family=0x6  Model=0x3d  Stepping=4
Oct 12 05:44:37 pfSense kernel: Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Oct 12 05:44:37 pfSense kernel: Features2=0x7ffafbbf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand>Oct 12 05:44:37 pfSense kernel: AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>Oct 12 05:44:37 pfSense kernel: AMD Features2=0x121 <lahf,abm,prefetch>Oct 12 05:44:37 pfSense kernel: Structured Extended Features=0x21c27ab <fsgsbase,tscadj,bmi1,avx2,smep,bmi2,erms,invpcid,nfpusg,rdseed,adx,smap,proctrace>Oct 12 05:44:37 pfSense kernel: XSAVE Features=0x1 <xsaveopt>Oct 12 05:44:37 pfSense kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Oct 12 05:44:37 pfSense kernel: TSC: P-state invariant, performance statistics
Oct 12 05:44:37 pfSense kernel: real memory  = 8589934592 (8192 MB)
Oct 12 05:44:37 pfSense kernel: avail memory = 8168407040 (7790 MB)
Oct 12 05:44:37 pfSense kernel: Event timer "LAPIC" quality 600
Oct 12 05:44:37 pfSense kernel: ACPI APIC Table: <alaska a="" m="" i="">Oct 12 05:44:37 pfSense kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Oct 12 05:44:37 pfSense kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
Oct 12 05:44:37 pfSense kernel: random: unblocking device.
Oct 12 05:44:37 pfSense kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS
Oct 12 05:44:37 pfSense kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS
Oct 12 05:44:37 pfSense kernel: ioapic0 <version 2.0="">irqs 0-39 on motherboard
Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #1 Launched!
Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #3 Launched!
Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #2 Launched!
Oct 12 05:44:37 pfSense kernel: Timecounter "TSC" frequency 1596341116 Hz quality 1000
Oct 12 05:44:37 pfSense kernel: random: entropy device external interface
Oct 12 05:44:37 pfSense kernel: wlan: mac acl policy registered
Oct 12 05:44:37 pfSense kernel: kbd1 at kbdmux0
Oct 12 05:44:37 pfSense kernel: netmap: loaded module
Oct 12 05:44:37 pfSense kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
Oct 12 05:44:37 pfSense kernel: random: registering fast source Intel Secure Key RNG
Oct 12 05:44:37 pfSense kernel: random: fast provider: "Intel Secure Key RNG"
Oct 12 05:44:37 pfSense kernel: nexus0
Oct 12 05:44:37 pfSense kernel: cryptosoft0: <software crypto="">on motherboard
Oct 12 05:44:37 pfSense kernel: padlock0: No ACE support.
Oct 12 05:44:37 pfSense kernel: acpi0: <alaska a="" m="" i="">on motherboard
Oct 12 05:44:37 pfSense kernel: acpi0: Power Button (fixed)
Oct 12 05:44:37 pfSense kernel: cpu0: <acpi cpu="">on acpi0
Oct 12 05:44:37 pfSense kernel: cpu1: <acpi cpu="">on acpi0
Oct 12 05:44:37 pfSense kernel: cpu2: <acpi cpu="">on acpi0
Oct 12 05:44:37 pfSense kernel: cpu3: <acpi cpu="">on acpi0</acpi></acpi></acpi></acpi></alaska></software></version></alaska></xsaveopt></fsgsbase,tscadj,bmi1,avx2,smep,bmi2,erms,invpcid,nfpusg,rdseed,adx,smap,proctrace></lahf,abm,prefetch></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>

                  After scrolling through all 6,000 lines of the system.log file I see several of these lines they appear to be at 1 minute intervals

                  Oct 12 05:43:10 pfSense kernel: sonewconn: pcb 0xfffff80008c430f0: Listen queue overflow: 193 already in queue awaiting acceptance (708 occurrences)

                  Some Googling returned this https://redmine.pfsense.org/issues/6406

                  Doug

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    @john_galt:

                    After scrolling through all 6,000 lines of the system.log file I see several of these lines they appear to be at 1 minute intervals

                    Oct 12 05:43:10 pfSense kernel: sonewconn: pcb 0xfffff80008c430f0: Listen queue overflow: 193 already in queue awaiting acceptance (708 occurrences)

                    For that, check the current value of kern.ipc.soacceptqueue  (run "sysctl kern.ipc.soacceptqueue"). It's probably at the default of 128. Set that to 1024 or higher (e.g. 4096), make a tunable for that under System > Advanced, Tunables tab.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • john_galtJ
                      john_galt
                      last edited by

                      jimp

                      Indeed it was set to 128. I added the tunable and set it to 4096 and applied. I'm doing this via openVPN connection. Do I need to reboot it?

                      Doug

                      Doug

                      1 Reply Last reply Reply Quote 0
                      • jimpJ
                        jimp Rebel Alliance Developer Netgate
                        last edited by

                        No need to reboot, that is one that can be changed at run-time.

                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                        Need help fast? Netgate Global Support!

                        Do not Chat/PM for help!

                        1 Reply Last reply Reply Quote 0
                        • john_galtJ
                          john_galt
                          last edited by

                          Okay. Will keep an eye on it and thanks.

                          Doug

                          1 Reply Last reply Reply Quote 0
                          • S
                            seanr22a
                            last edited by

                            Now the SG-2440 boxes stopped working as well  :( it lasted 6 days before I got 502 error.  Now all of them are back running 2.3.4p1 and  XMLRPC Sync is working as well. I will wait for the final release before I give 2.4 a new try.

                            1 Reply Last reply Reply Quote 0
                            • M
                              MaxPF
                              last edited by

                              Same thing here this morning. I had to force a reboot. I also increased sysctl kern.ipc.soacceptqueue to 4096 as suggested. While I was at it, I updated to latest build and it now says 2.4 RELEASE.

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                For anyone still seeing the problem after updating to 2.4.0-RELEASE, please gather the info I asked for in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 before rebooting the firewall and also supply a full list of installed packages that are running.

                                pfBlocker is mentioned a lot, but at least in the output shown so far, squid+clamav appeared to be more likely at fault.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • B
                                  BreeOge
                                  last edited by

                                  @jimp:

                                  Lots of port 1344, do you have squid+clamav active as well? Can you try shutting that off?

                                  Also that netstat -x output is too big to put inline, you should attach that in a .txt file

                                  Locked up again, without squid+clamav installed..

                                  
# /usr/sbin/swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/label/swap0  33554428       0B      32G     0%
#


                                  
# /usr/bin/top | /usr/bin/head -n7
last pid: 44796;  load averages:  0.10,  0.13,  0.12  up 0+14:58:53    12:32:05
88 processes:  1 running, 85 sleeping, 2 stopped

Mem: 475M Active, 827M Inact, 1187M Wired, 832M Buf, 13G Free
Swap: 32G Total, 32G Free


                                  
# /usr/bin/netstat -Ln
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
Netgraph sockets
Type  Recv-Q Send-Q Node Address   #Hooks
ctrl       0      0 [46b]:            0
ctrl       0      0 [468]:            0
ctrl       0      0 [44b]:            0
ctrl       0      0 [443]:            0
ctrl       0      0 [3f9]:            0
ctrl       0      0 [3b0]:            0
ctrl       0      0 [367]:            0
ctrl       0      0 [31e]:            0
ctrl       0      0 [2d5]:            0
ctrl       0      0 [28c]:            0
ctrl       0      0 [243]:            0
ctrl       0      0 [1fb]:            0
ctrl       0      0 [1b0]:            0
ctrl       0      0 [11]:             0
ctrl       0      0 [5]:              0
unix  0/0/5                            /var/run/dpinger_WAN_DHCP~70.178.196.154~70.178.196.1.sock
unix  0/0/5                            /var/run/dpinger_Steve_Telephone~192.168.16.2~10.10.10.2.sock
unix  0/0/5                            /var/run/dpinger_Raymond_Telephone~192.168.16.2~10.10.12.2.sock
unix  0/0/5                            /var/run/dpinger_Kevin_Telephone~192.168.16.2~10.10.11.2.sock
unix  0/0/5                            /var/run/dpinger_Cisco_Router~192.168.16.2~192.168.16.201.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/30                           /var/run/check_reload_status
unix  193/0/128                        /var/run/php-fpm.socket
unix  0/0/4                            /var/run/devd.seqpacket.pipe

                                  pastbin /usr/bin/netstat -xn

                                  https://pastebin.com/ZFujW9Kp

                                  1 Reply Last reply Reply Quote 0
                                  • P
                                    pyrodex
                                    last edited by

                                    @jimp:

                                    For anyone still seeing the problem after updating to 2.4.0-RELEASE, please gather the info I asked for in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 before rebooting the firewall and also supply a full list of installed packages that are running.

                                    pfBlocker is mentioned a lot, but at least in the output shown so far, squid+clamav appeared to be more likely at fault.

                                    I have never run squid+clamav and experienced it with only pfBlocker. I removed it the other day to restore usability and haven't had a lockup since.

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      BreeOge
                                      last edited by

                                      Here is the logs from another system, i removed the Squid+Clamav as well on this system. it locked up as well

                                      
# /usr/sbin/swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/gptid/d2a5a9dd-7e41-11e7-b   3684016       0B     3.5G     0%


                                      
# /usr/bin/top | /usr/bin/head -n7
last pid: 33447;  load averages:  0.08,  0.10,  0.08  up 0+12:43:35    12:46:18
113 processes: 1 running, 110 sleeping, 2 stopped

Mem: 689M Active, 1655M Inact, 945M Wired, 650M Buf, 4548M Free
Swap: 3598M Total, 3598M Free


                                      
# /usr/bin/netstat -Ln
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
Netgraph sockets
Type  Recv-Q Send-Q Node Address   #Hooks
ctrl       0      0 [ea1]:            0
ctrl       0      0 [e92]:            0
ctrl       0      0 [e73]:            0
ctrl       0      0 [e6b]:            0
ctrl       0      0 [e66]:            0
ctrl       0      0 [e]:              0
ctrl       0      0 [5]:              0
unix  0/0/80                           /tmp/mysql.sock
unix  0/0/5                            /var/run/dpinger_WAN_DHCP~70.178.22.158~70.178.22.1.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/30                           /var/run/check_reload_status
unix  193/0/128                        /var/run/php-fpm.socket
unix  0/0/4                            /var/run/devd.seqpacket.pipe

                                      /usr/bin/netstat -xn = See file attached.

                                      netstat-xn.txt

                                      1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by

                                        @BreeOge:

                                        
# /usr/bin/netstat -Ln
unix  193/0/128                        /var/run/php-fpm.socket


                                        
[...]
tcp4       0      0 127.0.0.1.8081         192.168.16.73.43834         0      0      0      0  65700  65700      1   2048      0      0 525600 525600    0.00    0.00    0.00    0.00    0.00 3797.36
[...]
fffff8000d6e7960 stream   1116      0                0 fffff8000d617a50                0                0 /var/run/php-fpm.socket
fffff8000d617a50 stream      0      0                0 fffff8000d6e7960                0                0
[...]

                                        So there are ~190+ things stuck doing a PHP operation, and the same number of stuck connections hitting the dnsbl daemon. The only thing pfBlocker does with lighty is run /usr/local/www/pfblockerng/www/index.php

                                        So something in that file is getting stuck and making those pile up.  Probably its file lock operation, maybe something isn't giving up a lock and everything else is stuck waiting.

                                        Try editing /usr/local/www/pfblockerng/www/index.php and commenting out or removing the whole "Increment DNSBL Alias Counter" block and see if it makes a difference. Keep a backup so you can put it back later if there is no change.

                                        Someone should probably bring this to bbcan's attention in the meantime.

                                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          BreeOge
                                          last edited by

                                          Done, let ya know if it crashes again.

                                          Thank you for helping resolve this issue.  Me and many of the people here thank you for your time on this.

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            MaxPF
                                            last edited by

                                            @jimp:

                                            For anyone still seeing the problem after updating to 2.4.0-RELEASE, please gather the info I asked for in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 before rebooting the firewall and also supply a full list of installed packages that are running.

                                            pfBlocker is mentioned a lot, but at least in the output shown so far, squid+clamav appeared to be more likely at fault.

                                            The problem with that, at least in my case, is that the console is unusable either locally or remotely. There is no menu, just a black screen and no matter what command I try nothing happens. Even CTRL+C just show ^C.

                                            Running 2.4 Release now. See how it goes.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.