Pfsense 2.2 + hyper-v - webconfigurator periodically dies
-
I just wanted to add this one as I have seen it since later Alpha phases.
The web configurator works usually for at minimum a week then suddenly starts giving 503/ 500 errors.
What I have been doing is just using a revert to checkpoint feature in Hyper-V so it is not really THAT much of an issue, but it is still happening.
I am still trying to figure out a why but this is about the 8th time I have seen it starting somewhere in the July or later Alpha builds and it is happening on an October Beta build.
Underlying architecture:
Supermicro A1SAi-2550F
Hyper-V Server 2012 R2
32GB RAM w/ 8GB assigned to pfsense
Intel 320 series 80GB SSD (15% full)Other notes - pfsense still routes traffic so that is not an issue. I tried doing a shell restart of the web configurator and that did not work. I also verified using IE, FF and Chrome on 2x different PCs.
-
what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?
-
Will check this later. I took a snapshot the last time it was happening. I just do not want to swap to that one remotely in the event it goes wrong.
Hyper-V makes this easy, but not so easy to explain via phone to the wife :-)
-
@cmb:
what's in the system log at the time? 'clog /var/log/system.log' at SSH if you can't get to web. What packages do you have installed?
Upgraded to the the 10/12 build and it just happened again. Tons of lines it seems:
check_reload_status: Could no connect to /var/run/php-fpm.socket
This also persisted through reboots (both orderly and hard reboot of the VM.
-
So given that error I just tried doing option 16 after logging in via SSH and got:
pkill: Cannot open pidfile `/var/run/php-fpm.pid': No such file or directoryAlso saw this now that I have SSH access:
Oct 14 22:46:50 hvpfsense check_reload_status: Syncing firewall
Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/var/run/php-fpm.socket
Oct 14 22:46:50 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3346) response not received, request sent: 1374 on socket: unix:/var/run/php-fpm.socket for /pkg_edit.php?, closing connection
Oct 14 22:46:51 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:52 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:53 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket
Oct 14 22:46:54 hvpfsense lighttpd[48013]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
Oct 14 22:46:55 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:56 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:57 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:57 hvpfsense lighttpd[48013]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket
Oct 14 22:46:58 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socket
Oct 14 22:46:59 hvpfsense check_reload_status: Could not connect to /var/run/php-fpm.socketVery interesting bit here is that I am even on today's 10/14 and saw it happen again.
-
Unlikely that has anything to do with hyper-v, as for PHP it doesn't really matter what you're running on. php-fpm is dying for some reason that isn't clear from that. What features are you using on that system?
-
The bandwidthd package seems to trigger killing php-fpm – for some users. Are you using that package? If so, try removing it
-
The bandwidthd package seems to trigger killing php-fpm – for some users. Are you using that package? If so, try removing it
I had this happen with bandwidthd on and off. The old -ALPHA snapshot I reverted to did not have bandwidthd.
I will snapshot and try to upgrade again later this week.
In terms of other stuff pfsense is running pfblocker is basically it.