Web Configurator failure

[madmage]

Noted a couple of older threads about this, but.. well, they're old.

Frequently, the web configurator stops working. completely.  nothing will bring it back, not even killing it and starting it over. not even reverting to http, not disabling the redirect, not putting it on a custom port, nothing.

this is becoming quite a nuisance.  I've read somewhere else that doing a firmware upgrade fixes it, ive done that a couple times now and it seems to work, but we're back to square one now.  I'm off to update it to the latest now and see what happens.

HELP!

[eri--]

I have not seen this in quite a while.
But killing php should solve the issue.

[madmage]

@ermal:

I have not seen this in quite a while.
But killing php should solve the issue.

wouldn't a reboot kill php?  rebooting doesn't fix it either.

[madmage]

doing a firmware upgrade doesn't fix it.

:(

[eri--]

Can you clarify what you mean by not working?

[madmage]

@ermal:

Can you clarify what you mean by not working?

not working means does not connect.  the browser throbber just sits and spins.

[seh2000]

@madmage:

@ermal:

Can you clarify what you mean by not working?

not working means does not connect.  the browser throbber just sits and spins.

Hi!
I had this a while back either the web browser would be slow slow or just sitting there. This happend when on 2.0 RC1 (i386) now I am on 2.0 RC3 (i386) and I have not seen this fenomen for weeks. THANKS pfSense team!!!!!
For me this happend in connection with my Cable Modem handing out private IP 192.168.100.nnn instead of the public address this due to "connection" issues between Cable Modem and ISP.
I tried to do traces with wireshark between Browser <> pfSense box and Cable Modem <> pfSense box, but never really found a reason, but a reboot of pfSense fixed the problem. I saw some processes keept asking for the old GW IP despite ben given the private IP.
I have not done any testings as for me 2.0 RC3 (i386) is very stable, so if it is not broken don't touch it :o) and as I work together with the IPS (he acknowledge I knew a bit about cable & internet & telecom) to monitor the network.
Not sure if this helps, but if you on a Cable Modem I would check what the modem do when you see the issue.
Steen

[madmage]

nothing like that. we actually were previously only using it for DHCP service. we haven't set it up to route out until recently. we still aren't using it as our primary route out the the 'net yet.

i think yesterday i was looking at the live traffic graph and it ceased working.  previously, i'd make changes to the dhcp server or the ntp server, just silly stuff and it just stops responding.

its real frustrating. :(

[cmb]

If a reboot doesn't bring it back you have something pretty seriously wrong, most likely some unstable package is hosing something. Check the system log with 'clog /var/log/system.log', check if lighty is running with 'ps ax|grep lightt' and if it's not, run /etc/rc.restart_webgui and see what that gives you.

[madmage]

@cmb:

If a reboot doesn't bring it back you have something pretty seriously wrong, most likely some unstable package is hosing something. Check the system log with 'clog /var/log/system.log', check if lighty is running with 'ps ax|grep lightt' and if it's not, run /etc/rc.restart_webgui and see what that gives you.

there is nothing in the logs. no packages are installed. lighty is running. restarting it using the console command ends up with it just sitting there with the ….............  forever.  killing the proc in command line and restarting doesn't work either.

this is a fresh install, nothing but the dhcp and dns settings configured.

[madmage]

Nothing on this? I can't reinstall right now, and am getting tired of having to manually edit the config file.

[jimp]

What is the snapshot date you are on?

Try this to restart:

killall -9 php; killall -9 lighttpd; /etc/rc.restart_webgui

[Bill48105]

Had this happen to be this morning with RC3 nano.  I had just replaced a m0n0 box that had been 100% reliable for over 3 years just yesterday & was anxious about it working reliably so 1st thing this morning I opened up web configurator & saw the dashboard but the 2 traffic graphs stuck on 'collecting initial data' so I clicked on Status Traffic Graph & that did the same thing, showing 'collecting initial data' but never showed a graph. I then clicked on RRD Graphs & that didn't load at all.  Entering just the router IP would just sit there trying to connect & go nowhere so not even dashboard would load.

I about went into panic mode & thinking I'd throw the m0n0 box back in place but traffic was still flowing OK so I left well enough alone & tried to diagnose the problem. I was able to SSH in & running top showed inetd using a lot of cpu (around 30% of my dual core atom although not sure if related or not) over & over. Otherwise mostly idle.  I exited back to the console menu & tried the Restart Webconfigurator option & it said it was restarting but showed ….. forever.  I left it for about 20 minutes but it was clear it wasn't helping as it just kept adding dots & web configurator would still not accept connections.  I opened 2nd SSH connection & left the ...'s going in the 1st.  I did killall php which returned to prompt then I did killall lighthttpd which said process not found & POOF immediately the ..'s stopped & it said stopping (or something along those lines) & went back to the menu screen & low & behold the web configurator worked again.

I found this thread after getting mine working by searching for pfsense killall php & figured I'd post up in case it this i nfo helpful.  I did not restart pfsense (since it was 9am which is a very busy time for the servers) to know if that would have helped but would have likely been my 1st move had it been off hours.

Yell if any info or logs from my pfsense box here is useful to figure this out.  Maybe killall php in the web configurator script might be useful?  (Hadn't looked for all I know there is already but maybe needs to be done in a loop or something..)  I was thinking of increasing the webconfigurator max processes above default of 2 to see if that helps as it was as if there were no php available to accept the web admin connections.
Thx
Bill

EDIT: I apologize if I should have started my own thread instead of saying 'me too' as after reading the posts again maybe not the same issue if rebooting doesn't fix the OP issue but as I said the killall php does the trick for my unresponsive webconfigurator here.

[madmage]

as i noted before, killing it doesn't fix it. Rebooting doesn't do it either.

I have had to move on. I've abandoned this project for the moment.

I think this needs to be addressed before the product comes out of RC, as I'm not the only person to have this issue.

[Bill48105]

madmage: Yeah perhaps might not be the same issue I had today since killall php solved it for me but the real question is what caused the hang up and that might be a key to your issue if in fact the causes were related. We both did say looking at live graph lead to it so maybe that is somehow a key. Only way to find out is to research & do some trial and error and compare to what others have experienced and that is why I replied.  The fact you have issue after reboot is definitely odd & likely points to something happening during bootup that is running/initializing something putting it back to the previous state so perhaps your config file would be helpful to devs. Did resetting to defaults help?  If resetting to defaults helps then restore your settings brings the problem back can be pretty sure it's in there.  Always a chance it is bad hardware or specific hardware loading certain drivers etc.  It'd also be interesting to know if setting your max processes value higher makes a difference for you, I'm trying that now cuz as I said in my previous post it was as if pfsense ran out of php's so requests couldn't be processed.  What browser are you using as who knows if that can come into play. I've had oddities with IE9 vs Firefox 5 in various places of the web configurator.

I feel your frustration as I gave up on pfsense a couple times now but RC3 has been working quite well for me so far & I REALLY need load balancing & failover to make use of my multiple lines in an orderly fashion. RC3 is definitely more solid that my experience with RC1 although I have still run into a few far less serious issues I can live with.  But without feedback & follow thru it's tough for issues to get resolved so I'm trying to do my part to help out. :)  I am working to find what logs might have clues that could be helpful but not familiar enough with pfsense/bsd atm but working on it. ;)
Bill

[madmage]

@Bill48105:

madmage: Yeah perhaps might not be the same issue I had today since killall php solved it for me but the real question is what caused the hang up and that might be a key to your issue if in fact the causes were related. We both did say looking at live graph lead to it so maybe that is somehow a key. Only way to find out is to research & do some trial and error and compare to what others have experienced and that is why I replied.  The fact you have issue after reboot is definitely odd & likely points to something happening during bootup that is running/initializing something putting it back to the previous state so perhaps your config file would be helpful to devs. Did resetting to defaults help?  If resetting to defaults helps then restore your settings brings the problem back can be pretty sure it's in there.  Always a chance it is bad hardware or specific hardware loading certain drivers etc.  It'd also be interesting to know if setting your max processes value higher makes a difference for you, I'm trying that now cuz as I said in my previous post it was as if pfsense ran out of php's so requests couldn't be processed.  What browser are you using as who knows if that can come into play. I've had oddities with IE9 vs Firefox 5 in various places of the web configurator.

I feel your frustration as I gave up on pfsense a couple times now but RC3 has been working quite well for me so far & I REALLY need load balancing & failover to make use of my multiple lines in an orderly fashion. RC3 is definitely more solid that my experience with RC1 although I have still run into a few far less serious issues I can live with.  But without feedback & follow thru it's tough for issues to get resolved so I'm trying to do my part to help out. :)  I am working to find what logs might have clues that could be helpful but not familiar enough with pfsense/bsd atm but working on it. ;)
Bill

I've used pfSense forever, and am a rabid fan…. so I'm not ready to give up yet!  We've ordered a new machine to replace the one I had pfSense on.  I am betting its a hardware problem.

[Bill48105]

Well fact it survives a reboot doesn't seem too promising so maybe new hardware will help even if it is just starting fresh that does the trick. ;)

Not sure if this is helpful or not but so far the only related logs I found are in /var/log/lighthttpd.error.log & it shows AFTER I did killall php around 9:38am not anything for when the thing actually froze around 9am

2011-08-15 09:38:28: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 43715 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 09:38:28: (mod_fastcgi.c.3309) child exited, pid: 43715 status: 0
2011-08-15 09:38:28: (mod_fastcgi.c.3356) response not received, request sent: 870 on socket: unix:/tmp/php-fastcgi.socket-1 for /ifstats.php?if=em1, closing connection
2011-08-15 09:38:28: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 19657 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 09:38:28: (mod_fastcgi.c.3356) response not received, request sent: 692 on socket: unix:/tmp/php-fastcgi.socket-1 for /index.php?, closing connection
2011-08-15 09:52:41: (mod_fastcgi.c.1734) connect failed: Connection refused on unix:/tmp/php-fastcgi.socket-1
2011-08-15 09:52:41: (mod_fastcgi.c.3027) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 4

By looks of the output I'd guess ifstats.php shown in the log is the traffic graph requesting data to be put on the graph which would make sense as I was viewing the traffic graph when it happened and that request was waiting until I killed php to finally die..

lighthttpd.log looks to be binary so need to figure out how to read/view those and not found a php log yet.

I did notice in php.ini:

max_execution_time = 99999999
max_input_time = 99999999
set_time_limit = 0

So guess php will hang around a LONG while like if ifstats.php got hung up on get_real_interface() or pfSense_get_interface_stats() for example. :D

I'm trying to duplicate the issue to be able to gather more clues but like many things it probably won't happen if I try. lol
Bill

[Bill48105]

OK I just had it happen again while viewing the traffic graph. This time I left it on the traffic graph & eventually in blue letters it says "Cannot get data about interface em1" mixed in with the white 'gathering' wording which made it tough to read.  I waited about 2 minutes then tried other menus & nothing would load so I did killall php & got similar lighthttpd.error.log entries:

2011-08-15 22:59:19: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 60352 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 22:59:19: (mod_fastcgi.c.3309) child exited, pid: 60352 status: 0
2011-08-15 22:59:19: (mod_fastcgi.c.3356) response not received, request sent: 852 on socket: unix:/tmp/php-fastcgi.socket-1 for /diag_logs.php?, closing connection
2011-08-15 22:59:20: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 9390 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 22:59:20: (mod_fastcgi.c.3356) response not received, request sent: 946 on socket: unix:/tmp/php-fastcgi.socket-1 for bandwidth_by_ip.php?if=wan, closing connection
2011-08-15 22:59:20: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 9390 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 22:59:20: (mod_fastcgi.c.3356) response not received, request sent: 870 on socket: unix:/tmp/php-fastcgi.socket-1 for /ifstats.php?if=em1, closing connection
2011-08-15 22:59:20: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 9390 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 22:59:20: (mod_fastcgi.c.3356) response not received, request sent: 852 on socket: unix:/tmp/php-fastcgi.socket-1 for /diag_logs.php?, closing connection
2011-08-15 22:59:20: (mod_fastcgi.c.2568) unexpected end-of-file (perhaps the fastcgi process died): pid: 9390 socket: unix:/tmp/php-fastcgi.socket-1
2011-08-15 22:59:20: (mod_fastcgi.c.3356) response not received, request sent: 870 on socket: unix:/tmp/php-fastcgi.socket-1 for /ifstats.php?if=em1, closing connection

You can see me trying to view the system logs from the menu there too..
I had increased max admin connections to 5 to see if that helped.

[mlabenda]

got the saem issue after installing 2.0RC3
will an update fix the issue ?