Can't login to WebGUI after a couple of days

megapearl

Problem returned after 1 day, restarting php-fpm from ssh does work.

I'm on:

2.3.1-DEVELOPMENT (amd64)
built on Mon May 16 06:51:39 CDT 2016
FreeBSD 10.3-RELEASE-p2

The system is on the latest version.

cmb

You have the IPsec widget on your dashboard? If so, if you remove that, does it stop happening? Seems a couple have seen that, haven't run into it myself, but would be good to narrow it down.

megapearl

Yes, having that one als widget, turning it off now.

glreed735

I started having the exact same problem with the update to 2.3. I updated to 2.3.1 hopeful that this problem was resolved, but it still happens with the IPSEC widget active

vronp

Same problem here on 2.3.1. I do not have the IPSec widget.

Menu item 16 fixes issue for a time. Also, it seem the GUI slows down significantly over a short time after doing the reset.

vronp

@cmb:

You have the IPsec widget on your dashboard? If so, if you remove that, does it stop happening? Seems a couple have seen that, haven't run into it myself, but would be good to narrow it down.

No IPSec on my dashboard. While the OP mentioned the 504 returning after "days", I see it return within an hour or two.

thanks,
Dave

cmb

Most if not all remaining instances of 504 gateway timeouts were fixed in 2.3.1_1.

vronp

@cmb:

Most if not all remaining instances of 504 gateway timeouts were fixed in 2.3.1_1.

I'm still experiencing the problem with 2.3.1_1

Dave

2chemlud

+1

2.3.1_1 64bit full, especially when re-starting a snort interface.

Intel(R) Atom(TM) CPU D525 @ 1.80GHz 4 CPUs, with 4 GB RAM and 120 GB SSD.

afreaken

I just had this happen at our remote site, not using the ipsec widget.

Netgate SG-2440

2.3.1-RELEASE-p1 (amd64)
built on Wed May 25 14:56:42 CDT 2016

afreaken

@afreaken:

I just had this happen at our remote site, not using the ipsec widget.

Netgate SG-2440

2.3.1-RELEASE-p1 (amd64)
built on Wed May 25 14:56:42 CDT 2016

Just happened to our local pfsense box, same as above (2.3.1_1). Will try to restart the web interface from console.

luckman212

https://redmine.pfsense.org/issues/6396 is marked closed. Is that a different bug, or should it be re-opened? I know there are 502's and 504's going on.

In any case, what is the correct redmine ticket for the remaining issue being discussed here?

edit: I found https://redmine.pfsense.org/issues/6406 which seems to be "this"

afreaken

@luckman212:

https://redmine.pfsense.org/issues/6396 is marked closed. Is that a different bug, or should it be re-opened? I know there are 502's and 504's going on.

In any case, what is the correct redmine ticket for the remaining issue being discussed here?

edit: I found https://redmine.pfsense.org/issues/6406 which seems to be "this"

I'm getting
"504 Gateway Time-out

nginx"

It just seems to happen randomly. Doing the reset from the console fixes it for some time.

My one box was up for ~12 days before it happened, the other was ~9 days. So…

EDIT: The widgets I'm using on both are: System Information, Interfaces, Gateways, Traffic Graphs

Not sure if that helps. Also running ipsec tunnel between locations.

The Brave Sir Robin

Just happened to me. 2.3.1_1. No IPSec or OpenVPN widgets. 20 days uptime. 504 error. I have an IPSec site-to-site running.

:(

OpenVPN server also stops working

Doing an option 16) Restart PHP-FPM did allow me to access the web gui but did not fix OpenVPN. Had to restart OpenVPN server to fix OpenVPN access.

afreaken

Happened again a couple days ago… I want to say this problem has persisted through the "fixes" in 2.3.1_1

Derelict

Do people seeing this with any regularity have anything interesting in the system log? Try filtering on frag.

afreaken

@Derelict:

Do people seeing this with any regularity have anything interesting in the system log? Try filtering on frag.

well… could I make a suggestion, could the "Reset Log Files" button not be the default accept button on the logs settings page, rather than the save button... I was trying to increase the number of logs per page, hit enter and instinctively clicked OK as I read the message, and instantly regretted my decision.

Not sure if this is the same error, but I saw a lot of these in the logs before the above happened.

nginx: 2016/06/20 08:37:47 [error] 18895#0: *1464191 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.27, server: , request: "POST /status_ipsec.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.10.2:445", referrer: "https://192.168.10.2:445/status_ipsec.php"

I see there is an update 5, running update 5 and will report back if it happens again, though it took 9 days of uptime to happen last time.

afreaken

@afreaken:

@Derelict:

Do people seeing this with any regularity have anything interesting in the system log? Try filtering on frag.

well… could I make a suggestion, could the "Reset Log Files" button not be the default accept button on the logs settings page, rather than the save button... I was trying to increase the number of logs per page, hit enter and instinctively clicked OK as I read the message, and instantly regretted my decision.

Not sure if this is the same error, but I saw a lot of these in the logs before the above happened.

nginx: 2016/06/20 08:37:47 [error] 18895#0: *1464191 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.27, server: , request: "POST /status_ipsec.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.10.2:445", referrer: "https://192.168.10.2:445/status_ipsec.php"

I see there is an update 5, running update 5 and will report back if it happens again, though it took 9 days of uptime to happen last time.

UPDATE: 10days 8Hours uptime, hasn't happened yet, so maybe the issue has been resolved between release 1 and release 5?

UPDATE: 22 Days uptime without issue. I'd say whatever caused my specific issue has been resolved.

cmb

@afreaken:

UPDATE: 10days 8Hours uptime, hasn't happened yet, so maybe the issue has been resolved between release 1 and release 5?

Several possibilities for that to happen were fixed between update 1 and 5.

matsan

@cmb:

Several possibilities for that to happen were fixed between update 1 and 5.

Just happened to me in 2.3.1_p5. Same error logged about the thermal sensors and OpenVPN required a restart to allow connections again. I left the WebGUI open during the night on the status screen (my mistake)

nginx: 2016/07/01 06:07:42 [error] 49003#0: *57704 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.17.18, server: , request: "GET /widgets/widgets/thermal_sensors.widget.php?getThermalSensorsData=11467344975628 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.17.4", referrer: "https://192.168.17.4/"

The first message logged is about the IPSec widget:

2016/07/01 05:11:08 [error] 49003#0: *57391 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.17.18, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.17.4", referrer: "https://192.168.17.4/"

Luckily I am able to login using SSH to an internal host and can then go back to the pfsense box and restart php-fpm (option 16) to get the WebGUI restarted and then restart OpenVPN. With vacations coming up I see a lot of calls from colleagues not being able to connect… :'(