New 502 Bad Gateway
-
DSNBL Default settings
I have 2 DSNBL Feeds, mostly for ads, and MalwareApproximately how many clients do you have on your local network that hit DNSBL? Any idea how busy it usually is?
The only Error messages we get are 502 Bad Gateway, but we can not access SSH without closing out process with CTRL-Z, and commands such as find, ect do not work. they just hang.
Most likely you just need to start a proper shell, try running /bin/tcsh
I have one site that has currently has
LAN 192.168.1.175 192.168.1.199 = 5 users
LAN 192.168.1.5 192.168.1.174 = 19 usersThis site will lock up in less than 12 hours.
Another site
LAN 192.168.1.41 192.168.1.50 = 3 users
LAN 192.168.1.5 192.168.1.40 = 4 usersIt will take up to 24 hours.
and another
Interface Pool Start Pool End # of leases in use
LAN 192.168.16.50 192.168.16.75 = 5 usersThis one will take up to 24 to 48 hours. However since the last update 2.4.0.r.20171009.1758 it has been happening more frequent around every 12 hours itself.
I currently have the pfBlockerNG uninstalled on the top 2 because they are used for apartments, and didn't want to bug the tenants when it locks up the CP becomes unresponsive. However for testing purpose I do re-install, and let it run with any changes for testing. I can usually get a crash within 6 to 12 hours. or less sometimes.
Just a note the amount of users, has not changed from before to after this issue started. Load has remained the same.
Thank you, I will test the /bin/tcsh and see if that gives me a good shell again. I will test that now on the top 2, will have a result back after it crashes.
Thank you, any logs you would like to see, please let me know I will post as much as I can.
-
[
The devs can perhaps assist with tracing, meanwhile I'm simply once again annoyed by the course taken here. We've had this some ~3 years ago when the repos have been taken offline altogether, accompanied by some giant noise about trademark violations. It's getting old. Waste of time. Annoying. Disrespectful to people who've been contributing to pfSense, even after that CLA/copyright assignment/re-licensing nonsense etc. People wanting to build this thing for development purposes get absolutely zero assistance whatsoever and are being [url=https://forum.pfsense.org/index.php?topic=109089.0]deliberately sabotaged by the so-called build scripts.Open-source touted all over the website, and all you get is this. Getting on a tipping point again here.
I never knew about that, I do not have any bad intentions for this. Sorry if I came off that way, I was not aware.
-
If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:
/usr/bin/netstat -Ln /usr/bin/netstat -xn /usr/sbin/swapinfo -h /usr/bin/top | /usr/bin/head -n7 /bin/ps uxawwd /usr/bin/sockstat
Attach the output in a text file as it will be too large to put inline on a forum post.
-
I got home and my pfSense GUI and Serial Console was unresponsive. As stated earlier from a client point of view everything worked, wired and wireless connections in and outbound.
My setup is is almost the same as BreeOge's, with even less users. Only me with a lot of gadgets. When I left for work at 6am I had just restarted and everything worked. When I got home at 4pm it was in the reported condition.
2.4.0-RC (amd64)
built on Mon Oct 09 17:58:12 CDT 2017
FreeBSD 11.1-RELEASE-p1What else can I provide?
Editβ¦. I also have openVPN with one user.
Doug
-
I have one box using the ZFS file structure, the other is using UFS, both using pfBlockerNG. The ZFS is rock solid, and the UFS one gets the Bad Gateway after some time. Wondering if that is a possible reason why two similar boxes with similar settings exhibit different behavior using the same snapshot and same packages.
Both running 20171009 Snapshots for 2.4.0
Just a thought
-
AhnHEL
I don't know but I had planned to do a fresh install of my UFS pfSense to ZFS this weekend and restore the same config.
Doug
-
I have one box using the ZFS file structure, the other is using UFS, both using pfBlockerNG. The ZFS is rock solid, and the UFS one gets the Bad Gateway after some time. Wondering if that is a possible reason why two similar boxes with similar settings exhibit different behavior using the same snapshot and same packages.
Both running 20171009 Snapshots for 2.4.0
Just a thought
Very interesting mine are UFS as well..
-
BreeOge
I have a time frame on my pfSense. I had to reboot when I got home at 4pm. It's now 7:50pm and I am getting the 502 Bad Gateway error. I'm going to reboot and scrub the system log between those times and if someone want's to look at it fine. Regardless I'm going ahead with my plans to reinstall this weekend.
Doug
-
It locked, here is the logs from the info you wanted.. This is from my heavier load box. My low load box also crashed, I will make a new post for it.
# /usr/sbin/swapinfo -h Device 1K-blocks Used Avail Capacity /dev/gptid/d2a5a9dd-7e41-11e7-b 3684016 0B 3.5G 0%
# /usr/bin/top | /usr/bin/head -n7 last pid: 50984; load averages: 0.07, 0.14, 0.15 up 0+23:33:41 21:18:05 134 processes: 1 running, 131 sleeping, 2 stopped Mem: 1036M Active, 2017M Inact, 1009M Wired, 693M Buf, 3775M Free Swap: 3598M Total, 3598M Free
/usr/bin/netstat -Ln Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address Netgraph sockets Type Recv-Q Send-Q Node Address #Hooks ctrl 0 0 [1b31]: 0 ctrl 0 0 [1b22]: 0 ctrl 0 0 [1ad8]: 0 ctrl 0 0 [1ad2]: 0 ctrl 0 0 [14fb]: 0 ctrl 0 0 [5]: 0 unix 100/0/200 /var/run/clamav/clamd.sock unix 0/0/80 /tmp/mysql.sock unix 0/0/5 /var/run/dpinger_WAN_DHCP~70.178.22.158~70.178.22.1.sock unix 0/0/4 /var/run/devd.pipe unix 0/0/30 /var/run/check_reload_status unix 193/0/128 /var/run/php-fpm.socket unix 0/0/4 /var/run/devd.seqpacket.pipe
Had to pastbin /usr/bin/netstat -xn
https://pastebin.com/RKzat3ue
-
This is on my lower load box
# /usr/sbin/swapinfo -h Device 1K-blocks Used Avail Capacity /dev/label/swap0 33554428 0B 32G 0%
# /usr/bin/top | /usr/bin/head -n7 last pid: 31956; load averages: 0.02, 0.07, 0.07 up 0+07:41:27 21:21:32 96 processes: 1 running, 93 sleeping, 2 stopped Mem: 601M Active, 1095M Inact, 1002M Wired, 636M Buf, 13G Free Swap: 32G Total, 32G Free
Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address Netgraph sockets Type Recv-Q Send-Q Node Address #Hooks ctrl 0 0 [23c]: 0 ctrl 0 0 [22b]: 0 ctrl 0 0 [1e3]: 0 ctrl 0 0 [199]: 0 ctrl 0 0 [f]: 0 ctrl 0 0 [5]: 0 unix 100/0/128 /var/run/clamav/clamd.sock unix 0/0/5 /var/run/dpinger_WAN_DHCP~70.178.196.154~70.178.196.1.sock unix 0/0/5 /var/run/dpinger_Steve_Telephone~192.168.16.2~10.10.10.2.sock unix 0/0/5 /var/run/dpinger_Raymond_Telephone~192.168.16.2~10.10.12.2.sock unix 0/0/5 /var/run/dpinger_Kevin_Telephone~192.168.16.2~10.10.11.2.sock unix 0/0/5 /var/run/dpinger_Cisco_Router~192.168.16.2~192.168.16.201.sock unix 0/0/4 /var/run/devd.pipe unix 0/0/30 /var/run/check_reload_status unix 13/0/128 /var/run/php-fpm.socket unix 0/0/4 /var/run/devd.seqpacket.pipe
Pastbin for /usr/bin/netstat -xn
https://pastebin.com/6tEs3Fs0
-
Lots of port 1344, do you have squid+clamav active as well? Can you try shutting that off?
Also that netstat -x output is too big to put inline, you should attach that in a .txt file
-
Lots of port 1344, do you have squid+clamav active as well? Can you try shutting that off?
Also that netstat -x output is too big to put inline, you should attach that in a .txt file
I disabled squid+Clamav, when it crashes ill get ya the logs again for that as well.
-
Hi
This morning I could not connect to the GUI nor was the serial console working. I rebooted then downloaded the system.log
It was nothing but what you see below. I only posted the last few lines before I restarted the firewall. I hope this helps.Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:40 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:41 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:42 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:43 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:43:45 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket Oct 12 05:44:37 pfSense syslogd: kernel boot file is /boot/kernel/kernel Oct 12 05:44:37 pfSense kernel: Copyright (c) 1992-2017 The FreeBSD Project. Oct 12 05:44:37 pfSense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Oct 12 05:44:37 pfSense kernel: The Regents of the University of California. All rights reserved. Oct 12 05:44:37 pfSense kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Oct 12 05:44:37 pfSense kernel: FreeBSD 11.1-RELEASE-p1 #82 r313908+d77c47fe50c(RELENG_2_4_0): Mon Oct 9 18:04:16 CDT 2017 Oct 12 05:44:37 pfSense kernel: root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense amd64 Oct 12 05:44:37 pfSense kernel: FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Oct 12 05:44:37 pfSense kernel: VT(efifb): resolution 800x600 Oct 12 05:44:37 pfSense kernel: CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz (1596.34-MHz K8-class CPU) Oct 12 05:44:37 pfSense kernel: Origin="GenuineIntel" Id=0x306d4 Family=0x6 Model=0x3d Stepping=4 Oct 12 05:44:37 pfSense kernel: Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Oct 12 05:44:37 pfSense kernel: Features2=0x7ffafbbf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand>Oct 12 05:44:37 pfSense kernel: AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>Oct 12 05:44:37 pfSense kernel: AMD Features2=0x121 <lahf,abm,prefetch>Oct 12 05:44:37 pfSense kernel: Structured Extended Features=0x21c27ab <fsgsbase,tscadj,bmi1,avx2,smep,bmi2,erms,invpcid,nfpusg,rdseed,adx,smap,proctrace>Oct 12 05:44:37 pfSense kernel: XSAVE Features=0x1 <xsaveopt>Oct 12 05:44:37 pfSense kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID Oct 12 05:44:37 pfSense kernel: TSC: P-state invariant, performance statistics Oct 12 05:44:37 pfSense kernel: real memory = 8589934592 (8192 MB) Oct 12 05:44:37 pfSense kernel: avail memory = 8168407040 (7790 MB) Oct 12 05:44:37 pfSense kernel: Event timer "LAPIC" quality 600 Oct 12 05:44:37 pfSense kernel: ACPI APIC Table: <alaska a="" m="" i="">Oct 12 05:44:37 pfSense kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Oct 12 05:44:37 pfSense kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads Oct 12 05:44:37 pfSense kernel: random: unblocking device. Oct 12 05:44:37 pfSense kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS Oct 12 05:44:37 pfSense kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS Oct 12 05:44:37 pfSense kernel: ioapic0 <version 2.0="">irqs 0-39 on motherboard Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #1 Launched! Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #3 Launched! Oct 12 05:44:37 pfSense kernel: SMP: AP CPU #2 Launched! Oct 12 05:44:37 pfSense kernel: Timecounter "TSC" frequency 1596341116 Hz quality 1000 Oct 12 05:44:37 pfSense kernel: random: entropy device external interface Oct 12 05:44:37 pfSense kernel: wlan: mac acl policy registered Oct 12 05:44:37 pfSense kernel: kbd1 at kbdmux0 Oct 12 05:44:37 pfSense kernel: netmap: loaded module Oct 12 05:44:37 pfSense kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19 Oct 12 05:44:37 pfSense kernel: random: registering fast source Intel Secure Key RNG Oct 12 05:44:37 pfSense kernel: random: fast provider: "Intel Secure Key RNG" Oct 12 05:44:37 pfSense kernel: nexus0 Oct 12 05:44:37 pfSense kernel: cryptosoft0: <software crypto="">on motherboard Oct 12 05:44:37 pfSense kernel: padlock0: No ACE support. Oct 12 05:44:37 pfSense kernel: acpi0: <alaska a="" m="" i="">on motherboard Oct 12 05:44:37 pfSense kernel: acpi0: Power Button (fixed) Oct 12 05:44:37 pfSense kernel: cpu0: <acpi cpu="">on acpi0 Oct 12 05:44:37 pfSense kernel: cpu1: <acpi cpu="">on acpi0 Oct 12 05:44:37 pfSense kernel: cpu2: <acpi cpu="">on acpi0 Oct 12 05:44:37 pfSense kernel: cpu3: <acpi cpu="">on acpi0</acpi></acpi></acpi></acpi></alaska></software></version></alaska></xsaveopt></fsgsbase,tscadj,bmi1,avx2,smep,bmi2,erms,invpcid,nfpusg,rdseed,adx,smap,proctrace></lahf,abm,prefetch></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>
After scrolling through all 6,000 lines of the system.log file I see several of these lines they appear to be at 1 minute intervals
Oct 12 05:43:10 pfSense kernel: sonewconn: pcb 0xfffff80008c430f0: Listen queue overflow: 193 already in queue awaiting acceptance (708 occurrences)
Some Googling returned this https://redmine.pfsense.org/issues/6406
-
After scrolling through all 6,000 lines of the system.log file I see several of these lines they appear to be at 1 minute intervals
Oct 12 05:43:10 pfSense kernel: sonewconn: pcb 0xfffff80008c430f0: Listen queue overflow: 193 already in queue awaiting acceptance (708 occurrences)
For that, check the current value of kern.ipc.soacceptqueue (run "sysctl kern.ipc.soacceptqueue"). It's probably at the default of 128. Set that to 1024 or higher (e.g. 4096), make a tunable for that under System > Advanced, Tunables tab.
-
jimp
Indeed it was set to 128. I added the tunable and set it to 4096 and applied. I'm doing this via openVPN connection. Do I need to reboot it?
Doug
-
No need to reboot, that is one that can be changed at run-time.
-
Okay. Will keep an eye on it and thanks.
-
Now the SG-2440 boxes stopped working as well :( it lasted 6 days before I got 502 error. Now all of them are back running 2.3.4p1 and XMLRPC Sync is working as well. I will wait for the final release before I give 2.4 a new try.
-
Same thing here this morning. I had to force a reboot. I also increased sysctl kern.ipc.soacceptqueue to 4096 as suggested. While I was at it, I updated to latest build and it now says 2.4 RELEASE.
-
For anyone still seeing the problem after updating to 2.4.0-RELEASE, please gather the info I asked for in https://forum.pfsense.org/index.php?topic=137103.msg753994#msg753994 before rebooting the firewall and also supply a full list of installed packages that are running.
pfBlocker is mentioned a lot, but at least in the output shown so far, squid+clamav appeared to be more likely at fault.