Pfsense 2.2 + hyper-v - webconfigurator periodically dies

[pjkenned]

I just wanted to add this one as I have seen it since later Alpha phases.

The web configurator works usually for at minimum a week then suddenly starts giving 503/ 500 errors.

What I have been doing is just using a revert to checkpoint feature in Hyper-V so it is not really THAT much of an issue, but it is still happening.

I am still trying to figure out a why but this is about the 8th time I have seen it starting somewhere in the July or later Alpha builds and it is happening on an October Beta build.

Underlying architecture:
Supermicro A1SAi-2550F
Hyper-V Server 2012 R2
32GB RAM w/ 8GB assigned to pfsense
Intel 320 series 80GB SSD (15% full)

Other notes - pfsense still routes traffic so that is not an issue. I tried doing a shell restart of the web configurator and that did not work. I also verified using IE, FF and Chrome on 2x different PCs.

[cmb]

what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?

[pjkenned]

Will check this later. I took a snapshot the last time it was happening. I just do not want to swap to that one remotely in the event it goes wrong.

Hyper-V makes this easy, but not so easy to explain via phone to the wife :-)

[pjkenned]

@cmb:

what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?

Upgraded to the the 10/12 build and it just happened again. Tons of lines it seems:

check_reload_status: Could no connect to /var/run/php-fpm.socket

This also persisted through reboots (both orderly and hard reboot of the VM.

[pjkenned]

So given that error I just tried doing option 16 after logging in via SSH and got:
pkill: Cannot open pidfile `/var/run/php-fpm.pid': No such file or directory

Also saw this now that I have SSH access:
Oct 14 22:46:50 hvpfsense check_reload_status: Syncing firewall
Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/var/run/php-fpm.socket
Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3346) response not received, request sent: 1374 on socket: unix:/var/run/php-fpm.socket for /pkg_edit.php?, closing connection
Oct 14 22:46:51 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:52 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:53 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
Oct 14 22:46:55 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:56 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:57 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:57 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket
Oct 14 22:46:58 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:59 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket

Very interesting bit here is that I am even on today's 10/14 and saw it happen again.

[cmb]

Unlikely that has anything to do with hyper-v, as for PHP it doesn't really matter what you're running on. php-fpm is dying for some reason that isn't clear from that. What features are you using on that system?

[charliem]

The bandwidthd package seems to trigger killing php-fpm – for some users.  Are you using that package?  If so, try removing it

[pjkenned]

@charliem:

The bandwidthd package seems to trigger killing php-fpm – for some users.  Are you using that package?  If so, try removing it

I had this happen with bandwidthd on and off. The old -ALPHA snapshot I reverted to did not have bandwidthd.

I will snapshot and try to upgrade again later this week.

In terms of other stuff pfsense is running pfblocker is basically it.