No Web Interface on Thu May 29 08:48:37 CDT 2014
-
Great work! I know that commenting out the start_packages line from /etc/rc is not enough to get the webgui working, as you found out. There are a few minicron entries after that line in /etc/rc that use fcgicli as well, one hourly account expire and one daily alias url updater. If I understand the problem correctly, they should be commented out as well, right?
In the old days, I'd look through the recent commits to the pfSense-tools tree, but I haven't taken the steps to regain access to that yet.
Hopefully the devs can find a fix, now that you've narrowed down the problem even more.
-
I am still getting the 100% CPU by check_reload_status too. I killed that and restarted it and the CPU went back to normal again.
-
I'm in the process of cloning the pfsense-tools repo to have a look through the commits. Can someone give me a timeframe for when this issue started showing up?
-
If the previous snapshots are available I can start installing them backwards and see when the issue disappears.
The first version I noticed the problem was the 29th or the 30th build.
EDIT: I am not sure what version I was running previous to the 29th build. It might have been the 26th or 27th. I don't see anything in the logs to show that I rebooted on the 28th. I put in a request for logging the version on boot so that I can easily keep track of what version was installed by going through the logs on my remote syslog server. I am sure the devs are busy though to worry about such things :).
-
If the previous snapshots are available I can start installing them backwards and see when the issue disappears.
The first version I noticed the problem was the 29th or the 30th build.
EDIT: I am not sure what version I was running previous to the 29th build. It might have been the 26th or 27th. I don't see anything in the logs to show that I rebooted on the 28th. I put in a request for logging the version on boot so that I can easily keep track of what version was installed by going through the logs on my remote syslog server. I am sure the devs are busy though to worry about such things :) .
http://snapshots.pfsense.org/FreeBSD_stable/10/amd64/pfSense_HEAD/updates/?C=M;O=D
There were a few versions that showed up on the 29th. Look at the 4G non VGA 21:41hrs and 23:36hrs as an example. still too small up to the last snaps out.
-
I'm having dramas trying to clone the repo, not too sure what's going on but it doesn't look like I'll be able to pull the commit logs any time soon.
-
Also, be aware that there are some issues with the iso and update image names taking an earlier date (in the filename) than they should have. Just FYI, but it can add to the confusion when trying to back out what image was built when, and identify when a problem showed up.
https://forum.pfsense.org/index.php?topic=76744.0 -
Updated to the version below and same issue just as an FYI…
2.2-ALPHA (i386)
built on Mon Jun 02 06:28:31 CDT 2014
FreeBSD 10.0-STABLEManually restart php-fpm and the gui works again through ssh.
killall php-fpm; sleep 2; /usr/local/sbin/php-fpm -c /usr/local/lib/php.ini -y /usr/local/lib/php-fpm.conf -RD 2>&1 >/dev/null
I notice that check_reload_status sometimes goes to 100% too(mainly after reboot). I manually kill and restart that and it goes back to normal low cpu usage. I have to force this one with -9 .
killall -9 check_reload_status; sleep 2; /usr/bin/nice -n20 /usr/local/sbin/check_reload_status
-
Any time I make a change to Suricata php-fpm has to be restarted again. I tried changing the log file size. The web page just sat there forever waiting for the post response I assume. I restart php-fpm and the change was done to to setting. I then tried to start Suricata and got the same waiting forever. I restarted php-fpm and the web gui started working again so I looked at the service did start. It seems like the commands are getting through before the gui stops working(at least enough to make it look like they did anyway).
I am thinking about uninstalling Suricata for testing 2.2 for now just so I don't have to deal with that.
EDIT: I just tried stopping Suricata and it stopped. php-fpm just seems to randomly (seemingly) stop working. Suricata might be a different issue as it goes to 100% CPU when I try to start it and doesn't seem to start anymore. Suricata does eventually go to normal CPU usage but the webui never returns when telling it to start. I still have to restart php-fpm.
Too many things to troubleshoot right now so I am removing Suricata.
-
I'm surprised that more things are not broken, given what you've found with php-fpm.
Do you have other packages installed that work OK, with just Suricata being a problem? The author of Suricata package did suggest that problems be posted in the packages sub-forum, but your problem is quite likely an issue with current state of 2.2 rather than the package:
https://forum.pfsense.org/index.php?topic=77311.msg421820#msg421820 -
I am sure more things will break php-fpm or are broken by php-fpm… whichever the case may be. I just have only been messing with Suricata so that is where I was seeing the issues.
I went ahead and added a rule and applied the changes and that worked I then went to change the client DHCP range in the openvpn config and that locked up php-fpm too.
So this is a more general failure of php-fpm it seems.
EDIT: I just checked the openvpn config after restarting php-fpm and it did make the change to the openvpn configuration even though php-fpm (and gui) stopped working.
-
Maybe it's time to create a bug in redmine, pointing back to these threads; so far, we don't even know if the devs are aware of the issue. I'll do that later tonight unless someone else can get to it first.
That would also be the right place to enter the feature request for version info to go into remote syslog files.
-
https://redmine.pfsense.org/issues/3690
-
I'm surprised that more things are not broken, given what you've found with php-fpm.
Do you have other packages installed that work OK, with just Suricata being a problem? The author of Suricata package did suggest that problems be posted in the packages sub-forum, but your problem is quite likely an issue with current state of 2.2 rather than the package:
https://forum.pfsense.org/index.php?topic=77311.msg421820#msg421820I had the current Suricata package working fine on an earlier 2.2 snapshot (before the php-fpm and web GUI hang-ups started). So I think Suricata is OK on 2.2, but for the moment 2.2 itself seems to have issues that often manifest themselves with any action using the GUI.
Bill
-
Looks like everything's working again in the latest build (amd64-20140602-1822).
-
3rd of June 64bit Nano updates are still far too small. :(
Steve
-
3rd of June 64bit Nano updates are still far too small. :(
Steve
Maybe to speed up full snaps. Better to anyone play a bell on devs to start compile nanobsd normally again.
-
3rd of June 64bit Nano updates are still far too small.
Tried applying one anyway? Maybe try it in a VM, it could still be a valid update.
-
The update files for NanoBSD are an image of the slice that is to be replaced. The usual size for the compressed image (for the 1GB Nano type) is ~68MB, the current update files are ~0.8MB. Now I'm willing to accept the compression algorithm may have been updated but it would be impressive if it can get 100X smaller! ;)
Steve
-
The GUI works on the latest snapshot. SSHD is down now though which was working in the snapshot from a day ago. Trying to start it from the GUI doesn't work. Staring it manually at the console with '/etc/sshd start' works though. I would much rather have to start sshd manually than the GUI stop working at times. This is definitely progress! :).
