Several PHP hangs after upgrade from 2.2.6 to 2.3



  • Hi, After yesterdays's upgrade to 2.3, the WebGUI has left responding at least twice and I needed to restart PHP-FPM via ssh (option 16)

    Any Thoughts?

    Thanks,

    Xavi.



  • What are you running on the system? What services, packages, etc.



  • pfsense is acting as office firewall and it has a ipsec vpn to our datacenter.

    packages installed: openvpn-client-export

    nginx log:

    nginx: 2016/04/15 13:27:41 [error] 98163#0: *2782 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 172.17.0.10, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "172.17.0.1", referrer: "https://172.17.0.1/"

    Thanks,

    Xavi.



  • @xavibj

    Are you running a NanoBSD instance ?

    I have a NanoBSD pfsense appliance and I noticed the same behaviour ( GUI responding very slowly or running in timeout, with the same syslog messages ) after the upgrade.

    I turned on the permanent Read-Write ( NanoBSD options ) and the problems disappeared.



  • @craymore, No, I have pfSense-memstick-2.2.6-RELEASE-amd64.img upgraded to 2.3.

    I have removed ipsec widget from dashboard and there was no hang since last friday.



  • @xavibj:

    @craymore, No, I have pfSense-memstick-2.2.6-RELEASE-amd64.img upgraded to 2.3.

    I have removed ipsec widget from dashboard and there was no hang since last friday.

    Good idea.  I have two systems that need php-fpm restarting regularly since upgrading.  A third box does not use IPSEC and the php-fpm has not needed restarting.

    One of them is a build from scratch with the config from the old one and the other is an upgrade.  Both are VMware VMs - x64, full install, both use IPSEC and both have the IPSEC widget on the dashboard.

    I'll know within a couple of days if the IPSEC widget is a potential candidate for the cause of the problem.

    Cheers
    Jon



  • I don't have IPsec widget on my dashboard and it still crashes. And even sometimes i need to restart PHP-FPM twice to get it working.

    I do have system info + term widget + interface and gateway status.
    I had traffic and openvpn widgets, but removing those also didn't helped …



  • This used to be a problem on earlier versions too.



  • I didn't had such issue on 2.2.6 (with all widgets working)



  • Just wanted to drop in and report that I am seeing the same behavior with the IPsec widget enabled on the 2.3 dashboard after upgrading a memstick to SSD install from 2.2.6 to 2.3.

    If I leave my browser open on the dashboard then php-fpm locks up and things start failing - such as OpenVPN certificate validation which effectively blocks all OpenVPN connections. At this point I have to ssh in and restart php-fpm (option 16).

    When I remove the IPsec widget from the dashboard then all is fine. I have done that and been running stable without a single error in nginx-error.log since.

    Currently running these widgets on 2.3 dashboard:
    System Information
    Interfaces
    Interface Statistics
    CARP Status
    Gateways
    Traffic Graphs
    S.M.A.R.T. Status
    Snort Alerts

    This is the output in /var/log/nginx-error.log below when it starts locking up:

    
    2016/04/18 09:18:10 [error] 47708#0: *3565 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 10:11:50 [error] 47747#0: *8108 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 10:21:40 [error] 47747#0: *9413 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:00:37 [error] 47708#0: *16167 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:00:38 [error] 47708#0: *16049 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /widgets/widgets/snort_alerts.widget.php?getNewAlerts=1461005858598 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:00:38 [error] 47473#0: *16108 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /ifstats.php?if=em2 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/graph.php?ifnum=lan&ifname=LAN&timeint=1&initdelay=4"
    2016/04/18 12:00:38 [error] 47708#0: *16373 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /ifstats.php?if=em0 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/graph.php?ifnum=wan&ifname=WAN&timeint=1&initdelay=2"
    2016/04/18 12:00:40 [error] 47747#0: *16197 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/interface_statistics.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:00:40 [error] 47473#0: *16343 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "POST /widgets/widgets/gateways.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:03:37 [error] 47708#0: *16167 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:03:58 [error] 47473#0: *16343 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:03:58 [error] 47708#0: *16373 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:03:58 [error] 47747#0: *16197 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:04:00 [error] 47473#0: *16108 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:04:00 [error] 47708#0: *16049 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:06:38 [error] 47708#0: *16167 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:07:18 [error] 47708#0: *16049 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:07:18 [error] 47473#0: *16108 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:07:18 [error] 47747#0: *16197 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    2016/04/18 12:07:20 [error] 47708#0: *16373 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.215, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw-1", referrer: "https://fw-1/"
    
    


  • I'm seeing the same behavior whether it's an upgrade of a 2.2.X on physical hardware, an upgrade of 2.2.X in a VM or an fresh install of 2.3 in a VM. Other issues that have been reported on other thread. For instance, no 2.3 install can check for updates much less load packages.

    This is true with Soekris 6501 i386 and VMs that are AMD64 and i386 in VirtualBox.



  • I'm having this issue as well on an ALIX 2D3 running the 2GB NanoBSD image.  I have no IPsec widget on the dashboard, and the NanoBSD setting to make permanent read/write had no effect either.

    Also, on the dashboard, the "Obtaining update status" indicator spins forever.

    Has pfSense left the ALIX behind with version 2.3?  Should those of us using ALIX with a CF card stay on 2.2.6?



  • Has pfSense left the ALIX behind with version 2.3?  Should those of us using ALIX with a CF card stay on 2.2.6?

    No, 2.3 runs OK on Alix. Much better if mounted permanent RW.

    Life might be even better if you turn off the dashboard update checks (System->Updates, Settings). And for that to have real effect behind the scenes, you need this change to the System widget, which really stops the checks being done in the background:
    https://github.com/pfsense/pfsense/commit/048f29dce62d8398bf0187a9ca9ff1d0541031f5
    (That will be part of 2.3.1 when it comes)



  • Running ALIX 2.3 with the same symptoms as Gloomrider. I cannot even get to the System>Update page. After about 3 minutes the page times out with a 504 Gateway Time-out.



  • @jim0266:

    Running ALIX 2.3 with the same symptoms as Gloomrider. I cannot even get to the System>Update page. After about 3 minutes the page times out with a 504 Gateway Time-out.

    Diag>NanoBSD, switch to permanent rw.


Log in to reply