Nginx with IPsec widget causes 504?
-
When having IPSec widget up on the dashboard I get a 504 Gateway Timeout with Nginx and been having this since 2.3 release. Seems if I remove the widget everything is fine.
Although regardless I see this being output when entering the command "ps auwwx"
root 35838 0.3 1.0 268244 38740 - S 3:26AM 0:00.31 php-fpm: pool nginx (php-fpm) root 41479 0.0 0.2 38844 6324 - Is 10:43PM 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 41650 0.0 0.2 38844 7380 - S 10:43PM 0:04.95 nginx: worker process (nginx) root 42060 0.0 0.2 38844 7380 - S 10:43PM 0:05.05 nginx: worker process (nginx) root 42294 0.0 0.2 38844 7364 - S 10:43PM 0:06.86 nginx: worker process (nginx)Not sure if theres supposed to be three worker process or if the problem lies elsewhere this is what i've have found while on the 2.3.1 snapshot. Does anyone have a similar result?
-
What do you see for php-fpm when that happens? The timeout you see is not nginx timing out, but nginx saying that what it was trying to load timed out. Usually there is something stuck in PHP when that happens. If you can show the full "ps uxawww" output it may be possible to spot.
Or, even better:
pkg install pstree rehash pstreeSimilar output but it makes it easier to spot which processes are children of others.
-
Thank you for the reply
Give me some time to get this to reproduce and i'll provide the necessary information.Edit:
pstree output displays the following-+= 00001 root /sbin/init -- |-+= 00318 root php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fp | |--- 37870 root php-fpm: pool nginx (php-fpm) | |--- 39664 root php-fpm: pool nginx (php-fpm) | |--- 43229 root php-fpm: pool nginx (php-fpm) | \--- 83249 root php-fpm: pool nginx (php-fpm) |-+= 00356 root /usr/local/sbin/check_reload_status | \--- 00358 root check_reload_status: Monitoring daemon of check_reload_stat |--= 00371 root /sbin/devd -q |-+= 12981 root /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/ | \--= 40178 root /usr/local/sbin/sshlockout_pf 15 |-+= 14052 root /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/loca | \--- 14294 root minicron: helper /usr/local/bin/ping_hosts.sh (minicron) |-+= 14504 root /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /us | \-+- 14756 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expirea | \--- 76576 root /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts |-+= 14678 root /usr/sbin/sshd | |-+= 12555 root sshd: admin@notty (sshd) | | \-+= 12838 root /bin/sh /etc/rc.initial -c /usr/libexec/sftp-server | | \--- 13111 root /usr/libexec/sftp-server | |-+= 57044 root sshd: root@pts/0 (sshd) | | \-+= 57647 root -sh (sh) | | \-+= 57715 root /bin/sh /etc/rc.initial | | \--= 59472 root /bin/tcsh | |-+= 57334 root sshd: root@notty (sshd) | | \--= 57548 root /usr/libexec/sftp-server | \-+= 84179 root sshd: admin@pts/1 (sshd) | \-+= 13338 root /bin/sh /etc/rc.initial | \-+= 14582 root /bin/tcsh | \-+= 15630 root pstree | \--- 15653 root ps -axwwo user,pid,ppid,pgid,command |--= 14939 root /usr/local/sbin/sshlockout_pf 15 |-+= 14991 root /usr/local/bin/minicron 86400 /var/run/update_alias_url_data. | \--- 14999 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_ |--= 15732 root dhclient: em0 [priv] (dhclient) |--= 21103 _dhcp dhclient: em0 (dhclient) |--= 24559 root /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid |--= 25401 root /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.con |--= 27077 root /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf |--= 40849 root /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B xxx.xxx.xxx.xxx - |-+= 41267 root nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx | |--- 41426 root nginx: worker process (nginx) | |--- 41427 root nginx: worker process (nginx) | \--- 41717 root nginx: worker process (nginx) |--= 42091 root /usr/sbin/cron -s |--= 43657 unbound /usr/local/sbin/unbound -c /var/unbound/unbound.conf |--= 44423 root /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases |--= 44782 root /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp |--= 58083 dhcpd /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/ |-+= 62095 root /usr/local/libexec/ipsec/starter --daemon charon | \--= 62399 root /usr/local/libexec/ipsec/charon --use-syslog |--= 70428 root /usr/sbin/powerd -b min -a adp -n adp |--= 75906 root /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/ |--= 39550 root /usr/libexec/getty Pc ttyv0 \-+- 66263 root /bin/sh /var/db/rrd/updaterrd.sh \--- 12539 root sleep 60if it helps here is the ps auwwx output
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 200.0 0.0 0 32 - RL 4:59PM 573:26.08 [idle] root 0 0.0 0.0 0 240 - DLs 4:59PM 0:53.63 [kernel] root 1 0.0 0.0 9136 812 - ILs 4:59PM 0:00.00 /sbin/init -- root 2 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [crypto] root 3 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [crypto returns] root 4 0.0 0.0 0 32 - DL 4:59PM 0:00.00 [cam] root 5 0.0 0.0 0 16 - DL 4:59PM 0:04.43 [pf purge] root 6 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [sctp_iterator] root 7 0.0 0.0 0 16 - DL 4:59PM 0:00.01 [enc_daemon0] root 8 0.0 0.0 0 32 - DL 4:59PM 0:00.23 [pagedaemon] root 9 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [vmdaemon] root 10 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [audit] root 12 0.0 0.0 0 528 - WL 4:59PM 0:36.50 [intr] root 13 0.0 0.0 0 32 - DL 4:59PM 0:00.00 [ng_queue] root 14 0.0 0.0 0 48 - DL 4:59PM 0:00.64 [geom] root 15 0.0 0.0 0 16 - DL 4:59PM 0:14.76 [rand_harvestq] root 16 0.0 0.0 0 560 - DL 4:59PM 0:00.37 [usb] root 17 0.0 0.0 0 16 - DL 4:59PM 0:00.00 [pagezero] root 18 0.0 0.0 0 16 - DL 4:59PM 0:00.01 [idlepoll] root 19 0.0 0.0 0 32 - DL 4:59PM 0:00.24 [bufdaemon] root 20 0.0 0.0 0 16 - DL 4:59PM 0:02.70 [syncer] root 21 0.0 0.0 0 16 - DL 4:59PM 0:00.04 [vnlru] root 56 0.0 0.0 0 16 - DL 4:59PM 0:00.02 [md0] root 318 0.0 0.7 268244 26916 - Ss 4:59PM 0:00.75 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) root 356 0.0 0.1 18888 2504 - INs 4:59PM 0:00.00 /usr/local/sbin/check_reload_status root 358 0.0 0.1 18888 2392 - IN 4:59PM 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 371 0.0 0.1 13624 5200 - Is 4:59PM 0:00.01 /sbin/devd -q root 12555 0.0 0.2 82264 8412 - Ss 10:07PM 0:00.14 sshd: admin@notty (sshd) root 12838 0.0 0.1 17000 2512 - Is 10:07PM 0:00.00 /bin/sh /etc/rc.initial -c /usr/libexec/sftp-server root 12981 0.0 0.1 14516 2316 - Ss 5:00PM 0:01.73 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf root 13111 0.0 0.2 50292 7412 - I 10:07PM 0:00.01 /usr/libexec/sftp-server root 14052 0.0 0.0 12268 1872 - Is 5:00PM 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 14294 0.0 0.0 12268 1884 - I 5:00PM 0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) root 14504 0.0 0.0 12268 1872 - Is 5:00PM 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts root 14678 0.0 0.2 59064 8072 - Is 4:59PM 0:00.00 /usr/sbin/sshd root 14756 0.0 0.0 12268 1884 - I 5:00PM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) root 14939 0.0 0.1 14612 2180 - Is 4:59PM 0:00.00 /usr/local/sbin/sshlockout_pf 15 root 14991 0.0 0.0 12268 1872 - Is 5:00PM 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data root 14999 0.0 0.0 12268 1884 - I 5:00PM 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron) root 15732 0.0 0.1 14564 2264 - Is 4:59PM 0:00.00 dhclient: em0 [priv] (dhclient) _dhcp 21103 0.0 0.1 14564 2344 - Is 4:59PM 0:00.02 dhclient: em0 (dhclient) root 24559 0.0 0.1 16676 2428 - Ss 4:59PM 0:00.48 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid root 25401 0.0 0.1 21624 5548 - Ss 4:59PM 0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 27077 0.0 0.1 18896 2488 - Is 4:59PM 0:00.00 /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid root 37870 0.0 0.9 281056 36780 - I 5:20PM 0:00.22 php-fpm: pool nginx (php-fpm) root 39664 0.0 0.9 281056 36932 - I 7:49PM 0:00.16 php-fpm: pool nginx (php-fpm) root 40178 0.0 0.1 14612 2180 - Is 5:00PM 0:00.00 /usr/local/sbin/sshlockout_pf 15 root 40849 0.0 0.1 15012 2292 - Is 4:59PM 0:01.34 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B xxx.xxx.xxx.xxx -p /var/run/dpinger_WAN_DHCP_xxx.xxx.xxx.xxx_xxx.xxx.xxx.xxx.pid -u /var/run/dpinger_WAN_DHCP_xxx.xxx.xxx.xxx_xxx.xxx.xxx.xxx.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 xxx.xxx.xxx.xxx root 41267 0.0 0.2 38844 6324 - Is 5:00PM 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 41426 0.0 0.2 38844 7312 - S 5:00PM 0:03.70 nginx: worker process (nginx) root 41427 0.0 0.2 38844 7336 - S 5:00PM 0:04.04 nginx: worker process (nginx) root 41717 0.0 0.2 38844 7308 - I 5:00PM 0:04.51 nginx: worker process (nginx) root 42091 0.0 0.1 16532 2288 - Ss 5:00PM 0:00.07 /usr/sbin/cron -s root 43229 0.0 0.9 281056 37000 - I 5:35PM 0:00.25 php-fpm: pool nginx (php-fpm) unbound 43657 0.0 0.5 43220 21020 - Ss 5:00PM 0:01.97 /usr/local/sbin/unbound -c /var/unbound/unbound.conf root 44423 0.0 0.1 12272 2032 - Is 5:00PM 0:00.00 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d mauro.manor -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /var/etc/hosts root 44782 0.0 0.4 30136 17964 - Ss 5:00PM 0:00.72 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 57044 0.0 0.2 82264 8668 - Ss 10:02PM 0:00.03 sshd: root@pts/0 (sshd) root 57334 0.0 0.2 82264 8412 - Ss 10:02PM 0:00.14 sshd: root@notty (sshd) root 57548 0.0 0.2 50292 7412 - Is 10:02PM 0:00.01 /usr/libexec/sftp-server dhcpd 58083 0.0 0.3 24804 13564 - Ss 5:00PM 0:00.80 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1 root 62095 0.0 0.1 30380 3460 - Is 5:00PM 0:00.00 /usr/local/libexec/ipsec/starter --daemon charon root 62399 0.0 0.4 222312 14172 - Is 5:00PM 0:00.54 /usr/local/libexec/ipsec/charon --use-syslog root 65977 0.0 0.0 8168 1824 - IN 10:12PM 0:00.00 sleep 60 root 70428 0.0 0.0 14408 1956 - Ss 5:00PM 0:01.03 /usr/sbin/powerd -b min -a adp -n adp root 75906 0.0 0.1 21032 4904 - Is 5:00PM 0:00.32 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid root 76576 0.0 0.0 14328 1964 - I 8:00PM 0:00.00 /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts root 83249 0.0 0.9 281052 38048 - I 5:14PM 0:00.24 php-fpm: pool nginx (php-fpm) root 84179 0.0 0.2 82264 8660 - Ss 10:07PM 0:00.03 sshd: admin@pts/1 (sshd) root 39550 0.0 0.0 14428 1988 v0 Is+ 5:00PM 0:00.00 /usr/libexec/getty Pc ttyv0 root 66263 0.0 0.1 17000 2580 v0- IN 5:00PM 0:02.44 /bin/sh /var/db/rrd/updaterrd.sh root 57647 0.0 0.1 17000 2632 0 Is 10:02PM 0:00.00 -sh (sh) root 57715 0.0 0.1 17000 2540 0 I 10:02PM 0:00.00 /bin/sh /etc/rc.initial root 59472 0.0 0.1 17340 3660 0 I+ 10:02PM 0:00.01 /bin/tcsh root 13338 0.0 0.1 17000 2536 1 Is 10:07PM 0:00.00 /bin/sh /etc/rc.initial root 14582 0.0 0.1 17340 3572 1 S 10:07PM 0:00.01 /bin/tcsh root 67673 0.0 0.1 18676 2264 1 R+ 10:13PM 0:00.00 ps auwwxI've replaced my public ip address with xxx.xxx.xxx.xxx
-
For future reference, when you edit a post, it doesn't notify that you've edited, so I didn't see that until I randomly stumbled back on this thread – make a new reply rather than editing and it will show up easier for those who have already replied and visited the thread.
The only thing that stands out is this:
|-+= 14504 root /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /us | \-+- 14756 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expirea | \--- 76576 root /usr/local/sbin/fcgicli -f /etc/rc.expireaccountsroot 76576 0.0 0.0 14328 1964 - I 8:00PM 0:00.00 /usr/local/sbin/fcgicli -f /etc/rc.expireaccountsIt appears to have gotten stuck in that script expiring user accounts.
Do you have any other accounts defined besides "admin"? Any groups? Can you share the account usernames and group names?
If that happens again and you see that specific process in the ps output, try to kill it and see if the GUI works again. The fcgicli process itself, that is, not the minicron or helper thread.
-
My apologies about that. I created an admin group for myself and another member so I could keep the actual admin account better secured; other than that I created three other users in another group with only access to the WOL page and the dashboard. I remember accidentally disabling or deleting one of the users in my admin group that I use to login most of the time but was able to restore it later. Would the reason be that or does that process refer to something else? Oddly once I remove the IPSec widget everything returns to normal but once I add it to the dashboard it'll return with the 504 later on until I restart php.
-
Extra users and groups are fine, I was mostly curious since before 2.3 released we had observed an issue with spaces in group names causing a problem with accounts being synchronized, I thought it might have been related.
