WebGUI 504 Timeout Issue
-
Hello Everyone,
I have been having 504 timeout issue since last once week now. No updates and no additional package installed lately. I am currently using pfSense version 2.3.2 with standard settings, pfBlockerNG & Snort is configured and the services are executing as expected.
I am able to access the shell and the firewall is running smooth without any issues. Can someone please shed some light, thanks in advance.
In the meantime, i have changed below values;
max_execution_time = 900 -> /etc/php.ini
request_terminate_timeout = 300 -> /usr/local/etc/php-fpm.conf2016/08/16 20:42:58 [error] 20217#0: *1 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/16 21:26:02 [error] 18966#0: *1 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/16 21:33:05 [error] 18966#0: *5 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/17 09:51:20 [error] 20268#0: *1 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/17 16:41:59 [error] 18589#0: *9 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/17 16:49:59 [error] 18589#0: *20 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1", referrer: "http://10.10.10.1/" 2016/08/17 17:13:56 [error] 18589#0: *23 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.10.10.101, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.10.10.1" -
Something has blocked PHP execution. Look at the output of "ps uxawwd" to see if anything is a child of a php process.
From ssh or the console, use option 16, then option 11. That should get you back in.
-
Hi Jimp,
Thank you for the reply, below is the output of "ps uxawwd"
You're correct, i managed to get to the pfSense login screen after executing option 11 & 16, upon clicking the login/submit button the process freezes for a long time before once again showing the Nginx 504 page. I have even tried to reboot the box several times, but each time i end up with no luck.
The only thing that looks suspicious to me is "pfB_Europe_v6", is there a way to stop pfBlockerNG service from the shell level to confirm if this culprit is the cause? Your guidance is much appreciated, thank you.
Wan Interface : AAA.AAA.AAA.AAA
WAN Gateway : BBB.BBB.BBB.BBBUSER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 0 0.0 0.0 0 96 - DLs 9:09PM 0:02.92 [kernel] root 11 100.0 0.0 0 8 - RL 9:09PM 142:05.60 - [idle] root 1 0.0 0.0 9060 620 - ILs 9:09PM 0:00.00 - /sbin/init -- root 270 0.0 0.8 95788 21688 - Ss 9:10PM 0:00.06 |-- php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) root 48622 0.0 1.2 103980 34780 - I 9:16PM 0:00.34 | `-- php-fpm: pool nginx (php-fpm) root 89868 0.0 0.1 10460 2084 - I 9:16PM 0:00.00 | `-- sh -c /sbin/pfctl -t pfB_Europe_v6 -T show | wc -l root 90092 0.0 1.1 43228 30788 - D 9:16PM 0:00.03 | |-- /sbin/pfctl -t pfB_Europe_v6 -T show root 90095 0.0 0.1 5856 1544 - I 9:16PM 0:00.00 | `-- wc -l root 300 0.0 0.1 10388 2040 - INs 9:10PM 0:00.00 |-- /usr/local/sbin/check_reload_status root 302 0.0 0.1 10388 1904 - IN 9:10PM 0:00.00 | `-- check_reload_status: Monitoring daemon of check_reload_status root 313 0.0 0.1 9404 3616 - Ss 9:10PM 0:00.00 |-- /sbin/devd -q root 5140 0.0 0.1 10152 1884 - Ss 9:10PM 0:00.20 |-- /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf root 12264 0.0 0.1 10232 1768 - Is 9:10PM 0:00.00 | `-- /usr/local/sbin/sshlockout_pf 15 root 8875 0.0 0.1 10236 1868 - Ss 9:10PM 0:00.45 |-- /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid root 9201 0.0 0.1 12632 4136 - Ss 9:10PM 0:00.08 |-- /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 10481 0.0 0.1 10424 2028 - Is 9:10PM 0:00.00 |-- /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid root 11114 0.0 0.2 12856 4584 - S 10:00PM 0:00.03 |-- /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf root 11845 0.0 0.9 38376 25584 - S 10:00PM 0:01.05 |-- /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl root 12088 0.0 0.2 15032 5428 - Ss 9:10PM 0:00.00 |-- /usr/sbin/sshd root 53860 0.3 0.2 17644 5944 - Ss 11:33PM 0:00.08 | `-- sshd: root@pts/0 (sshd) root 54240 0.3 0.1 10460 2188 0 Ss 11:33PM 0:00.00 | `-- -sh (sh) root 54469 0.3 0.1 10460 2088 0 S 11:33PM 0:00.00 | `-- /bin/sh /etc/rc.initial root 55183 0.4 0.1 10820 2784 0 S 11:33PM 0:00.00 | `-- /bin/tcsh root 56110 0.0 0.1 10204 1872 0 R+ 11:33PM 0:00.00 | `-- ps uxawwd root 16242 0.0 0.1 10632 1840 - Is 9:10PM 0:00.28 |-- /usr/local/bin/dpinger -S -r 0 -i UniFi_GW -B AAA.AAA.AAA.AAA -p /var/run/dpinger_UniFi_GW~AAA.AAA.AAA.AAA~BBB.BBB.BBB.BBB.pid -u /var/run/dpinger_UniFi_GW~AAA.AAA.AAA.AAA~BBB.BBB.BBB.BBB.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 BBB.BBB.BBB.BBB root 20890 0.0 0.2 13680 4948 - Is 9:10PM 0:00.00 |-- nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 21159 0.0 0.2 13680 5288 - I 9:10PM 0:00.00 | `-- nginx: worker process (nginx) root 21504 0.0 0.1 10108 1844 - Is 9:10PM 0:00.02 |-- /usr/sbin/cron -s root 22597 0.0 0.5 13012 13044 - Ss 9:10PM 0:00.19 |-- /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid nobody 26261 0.0 0.1 11340 3776 - I 9:10PM 0:00.01 |-- /usr/local/sbin/dnsmasq --all-servers --rebind-localhost-ok --stop-dns-rebind --dns-forward-max=5000 --cache-size=10000 --local-ttl=1 root 21 0.0 0.1 10464 1756 v0 Is+ 9:09PM 0:00.01 `-- sh /etc/rc autoboot root 304 0.0 1.1 99796 31984 v0 I+ 9:10PM 0:06.97 `-- /usr/local/bin/php-cgi -f /etc/rc.bootup root 28785 0.0 1.6 51420 45812 v0 D+ 9:10PM 0:06.24 `-- /sbin/pfctl -o basic -f /tmp/rules.debug.old root 2 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [crypto] root 3 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [crypto returns] root 4 0.0 0.0 0 16 - DL 9:09PM 0:00.05 - [cam] root 5 0.0 0.0 0 8 - DL 9:09PM 0:00.83 - [pf purge] root 6 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [sctp_iterator] root 7 0.0 0.0 0 16 - DL 9:09PM 0:00.39 - [pagedaemon] root 8 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [vmdaemon] root 9 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [idlepoll] root 10 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [audit] root 12 0.0 0.0 0 136 - WL 9:09PM 0:21.82 - [intr] root 13 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [ng_queue] root 14 0.0 0.0 0 24 - DL 9:09PM 0:00.01 - [geom] root 15 0.0 0.0 0 8 - DL 9:09PM 0:05.20 - [rand_harvestq] root 16 0.0 0.0 0 80 - DL 9:09PM 0:00.06 - [usb] root 17 0.0 0.0 0 8 - DL 9:09PM 0:00.00 - [pagezero] root 18 0.0 0.0 0 8 - DL 9:09PM 0:00.01 - [bufdaemon] root 19 0.0 0.0 0 8 - DL 9:09PM 0:00.12 - [syncer] root 20 0.0 0.0 0 8 - DL 9:09PM 0:00.01 - [vnlru] root 55 0.0 0.0 0 8 - DL 9:09PM 0:00.07 - [md0] -
When this happens, try to access the WebGUI with the firewall IP and not the FQDN.
You can also use another browser or computer to see if the WebGUI is responding.One thing that will freeze all tab in a browser is when you view the pfblockerng.log in Firewall / pfBlockerNG / Update. You have to click on the End View button to get the WebGUI back in that browser.
-
Hi Handy,
A very good day to you. Yes, i have tried to access the pfSense box with it's LAN IP, different computers and other browsers, but still no luck. Could you kindly explain/guide to me if there is a way to stop pfBlockerNG service from the shell level of perhaps remove pfBlockerNG temporarily to isolate the root cause? Many thanks.
-
Does the WebGUI return when you do 11) Restart webConfigurator from the shell ?
or 16) Restart PHP-FPM ? -
Not all the time, some times it does. Especially option 16 to restart PHP-FPM, no luck with option 11. Mostly i had to reboot the box to be able to at least access the pfSense login screen.
-
So do you have the time to go to pfblockerng and disable it ?
-
If you only have shell access, maybe you could modify the config.xml to disable pfBlockerNG and see if the issue goes away.
https://forum.pfsense.org/index.php?topic=102470.msg648231#msg648231
But I normally use the WebGUI to restore a config file, not sure how to do it from the shell.
You could modify one of the backup file in /cf/conf/backup (maybe make a copy to another folder in case you need it later), then use the 15) Restore recent configuration -
Thank you, i shall try as you suggested. Will update the outcome soon.
-
I managed to login to pfSense box, attached is the error notification
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-08-19 10:03:28