Web Configurator drops out consistantly
-
I have pfSense 2.0 beta 5 latest image loaded in test environment consisting of VMWare workstation on an Intel i7 with 12gb of DDR3.
I have installed the product successfully and have configured 2 WAN and 1 LAN interfaces. Thus far everything seems to work fine other than I consistently lose access to the webConfigurator after about 2 to 5 minutes of working with it and it always requires a reboot of the virtual device to get access back. There doesn't seem to be any rhyme or reason to it… i.e. the selection of anything in-particular other than it occurs after a short period of time and it does happen every time.
Any thoughts?
I am accessing it through the VM host which is Win7 ultimate and it occurs with both IE and Firefox. I should note that even restarting the webconfigurator through the console menu doesn't allow re-entry from a browser. I have to reboot it every time. Also.... the ports are still pingable when this occurs.
.... later
Have just confirmed that it also affects a ssh session as well. Thought I might have found a predictable cause by turning off webConfigurator redirect however it is still doing it and as indicated... it affects both http and telnet protocols when administering the system. Both drop at the same time and it does require a reboot to regain access. While it was dropped I attempted to configure a remote monitoring system to monitor the snmp of this device and while it could ping it... it could not otherwise communicate with it... reported it did not exist. Rebooted the pfsense device and the monitor popped it right up with all snmp info.
After configuring the remote monitor successfully it sees the pfsense device just fine. Once the ssh and http mgt. access drops out... the remote monitoring prgm continues to show the device up.
..... still later (passed my bedtime)
Ok.... reset device to factory defaults and recreated WAN1 and LAN.... so far so good... then created WAN2 (opt1).... problem back. Disabled WAN2 and problem disappeared. So it appears that even tho I am accessing the webconfigurator and ssh through the LAN port... if the opt1 port is enabled (but not configured for any traffic) the LAN port becomes problematic for traffic other than a simple ICMP. If the opt1 port is disabled then traffic works just fine indefinitely.
...... morning
Well.... seen that RC1 was available and did the shell based upgrade to it. Spoke to soon on the above. SSH and WebConfigurator traffic still goes out to lunch in spite of the fact that opt1 is disabled. PFsense is not locked up as I can still control through direct console port and can still ping wan and lan sites from the shell. Again... still requires a full reboot to gain access from SSH and WebConfigurator again.
Anyone else having this issue?
-
My configuration is as follows for the test invironment:
LAN 192.168.60.1/24
WAN1 192.168.70.70/24 GW 192.168.70.1
WAN2 192.168.50.2/24 GW 192.168.50.1 (disabled)No additional rules have been created. Working from stock install.
The only thing that seems to trigger this is an eventual change of the page through the webconfigurator however it does affect both the active ssh connection and the http or https connection when it occurs.
-
Anything reported in the system log around the time access to the web GUI failed? (Dump the system log by shell command # clog /var/log/system.log)
Was the web server still running? (what does the shell command # ps ax | grep ttp report?)
Was the ssh daemon still running? (What does the shell command # ps ax | grep sshd report?)
-
Hi there wallabybob,
Those were the first places I went to. Nothing out of the ordinary in the system.log file and ps reports that both ttp and sshd are still running.
I actually rebooted it this morning… connected via ssh, started a ping to a local IP and just walked away from it... pfsense stopped responding after 491 pings (8 minutes) without any other user activity going on through the ssh or https interface so I have confirmed at least that it isn't any user action through those interfaces causing the problem. Again... a direct console connection continues to operate just fine and everything looks to still be running as I can ping outbound on both interfaces and I can navigate at the command prompt at will. I tried a kill -HUP on both the ttp and sshd processes to no avail. Is a bit perplexing given I can still ping the interfaces both externally and from the inside out.
..... a bit later
Performed that sequence again... rebooted... just initiated an ssh connection this time... set a constant ping... only got to 294 (almost 5 minutes) before stopping. I would begin to suspect something in the vmware environment if it wasn't for the fact that I can still ping both sides of the pfsense interfaces.
-
When you stop getting a PING response do the packets still show up in a pfSense packet trace on the LAN interface?
-
Remember that I can still ping to the interfaces from the outside and through the interfaces from the inside.
Not sure how to check that at the command prompt. Can you provide the command?
Just performed latest update and symptom still exists.
-
I can confirm I am running into this same issue. I can successfully ping pfsense from another vm however pftop shows nothing which is really weird.
I am running virtualbox 4 using amd64 and the latest b5 snapshot -
Just an update. Restoring factory defaults and reconfiguring pfsense seems to have given me access to the webGui.
-
I should clarify that the 'constant ping' process I invoked was through an ssh connection at the ssh prompt. At the time pinging stopped is when ssh and http stopped responding… I could still ping the interfaces from the outside (no ssh) and also from the direct console prompt with the VM outbound on all interfaces.
I will emphasize that this was happening on both beta 5 and now RC1 and continues.
-
This is wierd so I want to check my understanding.
You ssh to pfSense and in the ssh session then start a ping to an external system. After a time the ping stops reporting anything at all and the web GUI is unresponsive but a ping to pfSense from an external system logs responses. Is that an accurate summary?
On my Ubuntu netbook terminal sessions (including ssh sessions) sometimes "freeze" but unfreeze if I open a new tab. That might explain what you report about ssh but not what you report about the GUI.
When the freeze happens I suggest you run a packet trace on the pfSense console, shell command # pfsense -c 100 -i where you replace <if>by the FreeBSD interface name (e.g. em0, sf1, re0 etc) of the interface used for the outgoing ping. If everything was working normally the outgoing ping and incoming responses will appear there.
Other readers of these forums use VMWARE products. It might be useful to compare version numbers.</if>
-
This is wierd so I want to check my understanding.
You ssh to pfSense and in the ssh session then start a ping to an external system. After a time the ping stops reporting anything at all and the web GUI is unresponsive but a ping to pfSense from an external system logs responses. Is that an accurate summary?
On my Ubuntu netbook terminal sessions (including ssh sessions) sometimes "freeze" but unfreeze if I open a new tab. That might explain what you report about ssh but not what you report about the GUI.
When the freeze happens I suggest you run a packet trace on the pfSense console, shell command # pfsense -c 100 -i where you replace <if>by the FreeBSD interface name (e.g. em0, sf1, re0 etc) of the interface used for the outgoing ping. If everything was working normally the outgoing ping and incoming responses will appear there.
Other readers of these forums use VMWARE products. It might be useful to compare version numbers.</if>
Well… almost. The symptom is that I loose the ability to communicate with ssh and HTTP (webConfigurator) arbitrarily. It occurs with no rhyme or reason and within minutes of bringing the pfsense vm up. When this communication is lost it is still possible to ping the interface ports of the pfsense vm from the outside in and it is possible to ping outside network devices from the pfsense direct console. pfsense is not locked up when this occurs as I can do most anything I like within the direct console. Just can't communicate (control) the pfsense box through the webConfigurator or through an ssh connection until I fully reboot the pfsense box again.
I suspect your example above for the command line should not be 'pfsense' but rather some other command?
The version of vmware I am using is vmware workstation 7.1
-
I suspect your example above for the command line should not be 'pfsense' but rather some other command?
Sorry, the command should have been # tcpdump -c 100 -i where you replace <if>by the FreeBSD interface name (e.g. em0, sf1, re0 etc) of the interface used for the outgoing ping.</if>
-
Ok…. this is becoming a real pain in the behind. I cleared my vm of pfsense and started from scratch with the latest beta5..... or so I thought. Downloaded pfSense-2.0-BETA5-i386-20110218-0524.iso.gz and installed from scratch. Once I completed the install I remoted into the webconfigurator and was prompted that there was an update. Started the update and got as far as shown below before the webconfigurator communications dropped out again. Same ol same ol. As I have not done anything from a configuration standpoint I have ask the question... has ANYONE else successfully created a vm in vmware workstation 7.1 of pfsense 2.0 beta 5? I have done it with the 1.x stable versions till the cows come home with no issues... but have yet to have 'total' success with 2.0 in any version.
Auto Update Download Status
Current Version : 2.0-BETA5
Latest Version : Fri Feb 18 06:31:46 EST 2011
File size : 90206413
Downloaded : 86184409
Percent : 96%Again... I iterate.... it appears to only be ssh and http that hang.... the install is still running... interfaces can still be pinged from the outside (i.e. from the LAN and WAN to the respective interfaces) and from the vm console I can operate anything internally to the install to include pinging anything on the outside through each interface respectively. A restart of the weconfigurator from the console menu makes no difference. What I can say is that the upgrade did stop dead in it's tracks at the 96% and did not complete. Rebooting the system and initiating it again repeated the process and hung again. Upon the 3rd attempt the file was completely downloaded and once downloaded it completed the upgrade in spite of the webconfiguration going out to lunch again.
After successfully loading the update.... I put the 'traffic graph' page up. It continued to to tick off for approximately an hour... the moment I clicked to go to another page.... communications ceased and the page did not come up. Again.... had to reboot from the vm console to pfsense.
I am absolutely happy to provide what ever information anyone might inquire of that may help to narrow this down. At this stage... I haven't been able to get this to work in two separate servers... both of which are running vm's of 1.2.3 without issue and have been running for a year now.
Anyone?
-
Ok…. digging around here. I have discovered something that is occurring every time. The filter.log file has appropriate listings up until the point it appears of the stoppage of the http and ssh communications... after which the file is loaded with constant @^ repeated indefinitely to the end of the file. It does appear that this is occurring at the point communications is lost. I will confirm by repeating this and checking time stamps. There are no proper lines at any point after the @^ start.
After a few tests I have been able to determine that it doesn't require any user input/activity for this to occur. It occurs regardless but appears to happen much sooner if user activity with the webconfigurator or ssh connection is happening.
-
The filter.log file has appropriate listings up until the point it appears of the stoppage of the http and ssh communications… after which the file is loaded with constant @^ repeated indefinitely to the end of the file.
What are you using to read filter.log? It is a "circular log" file which should be displayed by the clog command. The file is pre-allocated and once it is full old entries drop out of the file to make room for new entries. Here's the last few lines output by # more /var/log/filter.log on my system:
Feb 19 06:39:01 pfsense2 pf: 192.168.a.b.47573 > 192.168.x.y.143: Flags [P.], cksum 0xd742 (correct), ack 2578260386, win 267, options [nop,nop,TS val 107624139 ecr 79237729], length 6
Feb 19 06:39:02 pfsense2 pf:CLOG^A^@^@^@L<93>^D^@|<d0>^G^@^@^@^@^@</d0>Here's the last few lines output by # clog /var/log/flter.log on the same system:
Feb 21 08:32:25 pfsense2 pf: 00:00:00.499862 rule 69/8(ip-option): pass in on vr0: (tos 0x0, ttl 1, id 29561, offset 0, flags [none], proto IGMP (2), length 40, options (RA))
Feb 21 08:32:25 pfsense2 pf: 192.168.a.b > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.255.250 to_ex, 0 source(s)]Have you run out of kernel memory for networking when ssh and GUI hang? (Please post the output of pfSense command netstat -m run when ssh or gui hang.)
Does the filter.log have any entries for port 22 (ssh), port 80 (http) or port 443 (https)?
What you report is wierd. It is possible you have run into a number of separate problems.
ssh: On my Ubuntu 10.04 netbook terminal sessions sometimes hang. This includes local terminal sessions and ssh sessions to either of my pfSense boxes. It doesn't seem a communications hang in that if I open a new terminal tab and hit the enter key once or twice in the new tab and go back to the hung session and hit the enter key I almost always get a response. If your ssh problem is different it would be helpful to have a packet trace. I suggest you leave tcpdump running on the console:
tcpdump -i <if>port 22</if>
where <if>is your LAN interface. This will display all the ssh traffic (its not helpful to do this over a SSH session!). Start a SSH session into the pfSense box over the LAN interface and have it display something continuously (e.g. from a system status program like top or systat) then when your ssh session appears to hang paste the tcpdump output into a reply here.
http: As with ssh start tcpdump on your console but this time the port number will not be 22 but 80 (if you use http to the web GUI or 443 if you use https to the web GUI; assuming standard port assignments) and on a system on your LAN point the web browser to the pfSense box. Then when you detect the browser has hung paste the tcpdump output into a reply here.</if>
-
I have seen this behavior before with a physical machine with 6 nics.
2 nics where broadcom and a quad intel card. While I pulled out the Intel or broadcom all was working.
While it happens I was able to telnet on on port 443 or 22 to the pfsense the socket was open but the actual process didn't Reply. -
Will get that information together this late afternoon. Thanks for responding wollabybob!