Pfsense 2.2 + hyper-v - webconfigurator periodically dies



  • I just wanted to add this one as I have seen it since later Alpha phases.

    The web configurator works usually for at minimum a week then suddenly starts giving 503/ 500 errors.

    What I have been doing is just using a revert to checkpoint feature in Hyper-V so it is not really THAT much of an issue, but it is still happening.

    I am still trying to figure out a why but this is about the 8th time I have seen it starting somewhere in the July or later Alpha builds and it is happening on an October Beta build.

    Underlying architecture:
    Supermicro A1SAi-2550F
    Hyper-V Server 2012 R2
    32GB RAM w/ 8GB assigned to pfsense
    Intel 320 series 80GB SSD (15% full)

    Other notes - pfsense still routes traffic so that is not an issue. I tried doing a shell restart of the web configurator and that did not work. I also verified using IE, FF and Chrome on 2x different PCs.



  • what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?



  • Will check this later. I took a snapshot the last time it was happening. I just do not want to swap to that one remotely in the event it goes wrong.

    Hyper-V makes this easy, but not so easy to explain via phone to the wife :-)



  • @cmb:

    what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?

    Upgraded to the the 10/12 build and it just happened again. Tons of lines it seems:

    check_reload_status: Could no connect to /var/run/php-fpm.socket

    This also persisted through reboots (both orderly and hard reboot of the VM.



  • So given that error I just tried doing option 16 after logging in via SSH and got:
    pkill: Cannot open pidfile `/var/run/php-fpm.pid': No such file or directory

    Also saw this now that I have SSH access:
    Oct 14 22:46:50 hvpfsense check_reload_status: Syncing firewall
    Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/var/run/php-fpm.socket
    Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3346) response not received, request sent: 1374 on socket: unix:/var/run/php-fpm.socket for /pkg_edit.php?, closing connection
    Oct 14 22:46:51 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:52 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:53 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:54 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket
    Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
    Oct 14 22:46:55 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:56 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:57 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:57 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket
    Oct 14 22:46:58 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
    Oct 14 22:46:59 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket

    Very interesting bit here is that I am even on today's 10/14 and saw it happen again.



  • Unlikely that has anything to do with hyper-v, as for PHP it doesn't really matter what you're running on. php-fpm is dying for some reason that isn't clear from that. What features are you using on that system?



  • The bandwidthd package seems to trigger killing php-fpm – for some users.  Are you using that package?  If so, try removing it



  • @charliem:

    The bandwidthd package seems to trigger killing php-fpm – for some users.  Are you using that package?  If so, try removing it

    I had this happen with bandwidthd on and off. The old -ALPHA snapshot I reverted to did not have bandwidthd.

    I will snapshot and try to upgrade again later this week.

    In terms of other stuff pfsense is running pfblocker is basically it.