WebGUI very slow - Rest of system doing just fine

[heper]

i've noticed delays in the webgui when there are dns issues.

so for anyone else having long delays …. check dns settings

[the.it.dude]

Is this only an issue in v2.0?  If so, why?

I usually configure my firewalls offline before plugging them into the network.  Never had any problems with v1.2.3.  I have all sorts of delays and timeouts trying to configure v2.0.

Funny thing is… I have 1 v2.0 box in production just for a VPN endpoint.  I don't remember having any issues configuring that.  But, I may have had that one plugged into the live network while I was configuring it.

[ripleydraven]

A DNS issue is an interesting point within this problem.
But I don´t believe it realy.
Which side should have the DNS issue WAN or LAN interface?
In my testings the WAN interface was connected to a router and got all needed IP´s by DHCP. There was no DNS problems.
The LAN interface was just configured with an IP address without DNS. For this interface it could be possible.
But I had no problems in a special szenario with the 32Bit Version of pfsense 2.0-RC1 for a long time.
The problem came up first when I added a third network card to the configuration.
With the 64Bit Version the problems started directly.
But maybe I tell you the whole story so it may help us all to figure out the problem.

I apologise that my first post is exorbitantly long.

I installed pfSense 2.0-RC1 on two new machines and run into many Problems with the WebGui.
The Servers are Dell PowerEdge 610 systems.
Configuration is:

• CPU Intel Xeon E5620 2,4 GHz,
• 4 GB RAM,
• 2 x 73 GB SAS,
• 4 x NIC Broadcom NetXtreme II 5709c Gigabit Ethernet Controller on Board
• 2 x Intel Gigabit ET Quad Port Server-Adapter, Cu, PCIe x4
    (Means two network cards with 4 network ports in each card.)
    So we have at least 12 Network Ports.

All of the hardware is found automatically by the pfSense Software.
So there was no need to install any hardware driver.

The Broadcom on Board NIC´s are identified as bce0, bce1, bce2 and bce3 
The Intel NIC´s on the Quad Cards are identified as igb0, igb1, igb2, igb3, igb4, igb5, igb6 and igb7
So this looks OK for me.

Because it is a new machine I first installed the 64Bit Version of pfSense 2.0-RC1.
I configured the bce0 card as WAN and the bce1 card as LAN.
WAN got an IP address by DHCP from a little router.
(It´s just testing scenario at this time.)
LAN Interface has a static IP address 192.168.101.254
This was configured by using the pfSense Console Setup.

After this I was able to login to the WebGui. In the WebGui I configured the other network cards.
The other cards have been configured as optional cards by using the WebGui.
I just added them to the configuration but they were not activated at this time.
After saving this I wanted to have a look on the new Web interface.
I was able to look around for about ten minutes. Then the WebGui didn´t respond anymore and the Web browser timed out after some time.
I had to restart the server and could login again but also just for about ten minutes. Then the same problem occurred.

First I thought it could be something wrong with the network cards because I read something about performance problems with some Intel Gigabit Cards.
So I changed the settings  /etc/sysctl.conf and disabled the LRO feature for all 12 network cards.
by adding the lines
dev.bce.0.enable_lro=0
dev.bce.1.enable_lro=0
…..
dev.igb.0.enable_lro=0
dev.igb.1.enable_lro=0
.....
and rebooted the server.

Sadly this didn´t change anything on the behaviour of the WebGui. It still stopped responding after some minutes.
So after some more testing, log files checking, looking for info’s in the forum and installing the update to RC2, I decided to install the 32Bit Version.
I thought that the 32Bit version maybe could have a better support for the network cards.
I thought that maybe the driver support in the 64Bit version is not well-engineered at this time. These things happen.

What shall I say? After installing the 32 Bit version and configuring two of the integrated Broadcom cards the same way as before I had no problems anymore.
At this time I did not configure any optional cards.

I tested the WebGui for a long time, nearly 2 hours. I clicked on every link, let the WebGui stay open, returned back again after some time,
tried it out again and it was still responding in a really fast way. I thought I had my solution but I wanted to be really sure.
So I decided to let the WebGui stay open on one machine for the whole night and on the other machine I closed it.
On the other day there was no problem. I could work with both WebGui really fast.
So remember. At this point the LAN Interface was just configured with a static IP and no DNS informations.
The WAN interface still with DHCP and workings DNS informations.

The next thing I tested was the update.
So I did an update to RC2 on one machine. After the installation of the update I had the same problem with the WebGui back again.
Damn. But OK, the update is not a stable one. So no update until there is a stable release ;-).

Now I thought I could start with the real configuration of the carp cluster.
So I went to Interfaces, added the third on Board card, bge2 (Broadcom NetXtreme II 5709c) as an optional card, renamed it to Sync and enabled the card.
After saving this setting the WebGui didn’t came back, the Web browser ran into a timeout and I had my problem back again :-((

So it seems to be that if I just use two cards everything is fine. If I just add one card more the WebGui fails.
The log files do not say anything about this.

So what do you think about it?

[Perry]

ripleydraven just to eliminate 2 things make sure you have the latest bios and try enable System -> Advanced -> Networking Disable hardware checksum offload.

[ripleydraven]

Hello Perry,
nice idea.
So I talked to the Dell support to check the BIOS version with them.
They said that the version 3.0.0 we have installed is the last one at this time.

I did the changes in System -> Advanced -> Networking and disabled the hardware checksum offload by
enabling ;-) the checkbox  "Disable hardware checksum offload" .
The configuration is now still with only two network cards, LAN and WAN.

To see which BIOS version is installed I had to start the server once again.
After this the WebGui usually is fast for some minutes.
After changing the settings as written above the behaviour now is different to the last one on Friday.
In the first ten minutes the WebGui is fast.
So I clicked every Link in every main category and also every tab on every page.
This worked fast for a while.
Then by clicking some tabs the WebGui did not load the next page again.
But if I use the Browser navigation buttons, stop loading the page and press the button to went one page backwards
I will get the last page. This did not work last week.
Sometimes the last page comes back fast sometimes it takes about 30 seconds.
When the last page is loaded and I press another button, this will bring up the new page fast.

But on some pages it is every time very slow.
This is on pages with a lot of informational calculation like on the RRD Graphs.
After trying to connect to one of these pages it takes a really long time.
I have to stop loading the page. Trying to go back to the last page does not work then.
I have to change the URL in the address line to just the IP address of the server.
Then the Dashboard comes up after some minutes and then I can navigate in a normal speed. 
But not to RRD Graphs. The page behind the tab "Pakets" does not realy load at all.

I do not think that there is also a problem with the support of the processor type.
I just used the default settings for the installation.
But in the meantime I also installed pfSense Version 1.2.3 on the other machine.
With this version there is no problem with the WebGui at any time.
Everything ist loading absolutely fast. Also the RRD Graphs.
This installation is also just done by choosing the default installation settings.
I also changed the WebGui of version 2.0-RC1 to a more simple one.
The one I am using now is the pfsense theme. But wit this I have the same problems.

So I believe it is not realy something with the WebGui of the 2.0-RC1 or with the network cards.
I think it is something inside the calculation of the websides or maybe something differnt in the version of the webserver.
Don´t get me wrong I´m not a prgrammer I just speculate.

If you have another idea let me know.
I will test it it here.

[ripleydraven]

Hello,

I just tested some more things, but still the problem exists.
With another WebGui "pfsense-dropdown" it works better.
But not realy good. The RRD Graphs Pages are still slow.
And from the time out in the browser still comes up after some time.
The lighthttp.error.log sais that there is a failed Socket connection, a SSL  failure and the fastcgi process died.

This is the whole lighthttp.error.log

2011-05-23 11:35:30: (log.c.166) server started
2011-05-23 11:42:52: (log.c.166) server started
2011-05-23 11:42:52: (network_writev.c.115) writev failed: Socket is not connected 19
2011-05-23 11:42:52: (mod_fastcgi.c.3094) connection was dropped after accept() (perhaps the fastcgi process died), write-offset: 0 socket: unix:/tmp/php-fastcgi.socket-1
2011-05-23 11:42:52: (connections.c.1710) SSL (error): 5 -1 32 Broken pipe
2011-05-23 11:49:47: (log.c.166) server started
2011-05-23 11:55:02: (connections.c.1710) SSL (error): 5 -1 32 Broken pipe
2011-05-23 11:55:30: (network_writev.c.115) writev failed: Socket is not connected 18
2011-05-23 11:55:30: (mod_fastcgi.c.3094) connection was dropped after accept() (perhaps the fastcgi process died), write-offset: 0 socket: unix:/tmp/php-fastcgi.socket-1

Has anybody any ideas?

[the.it.dude]

ripleydraven,

When my gui times out, I see a bunch of the following in the filter log:

rule 2/0(match): block out on em0:  192.168.13.1.443 > 192.168.13.4.xxxxx:  tcp 4 [bad hdr length xx - too short, < 20]

(The X's are random numbers)

Curious if you see the same?

Jeff

[ripleydraven]

the.it.dude,

sorry, I was in another customer project yesterday.

First to your question.
No I don´t see the same entry in the filter log.
what I see is the following:

This comes just one time by starting the filter log screen.
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use –v or –vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 96 bytes.

This comes up every 60 second in exactly this way. No random numbers. 
rule 1/0(match): block in on bce1: 192.168.208.1.1072 > 255.255.255.255.1947: UDP, length 40
The bce1 interface is my LAN interface. But it is in a complete different network.
The LAN IP is 172.16.173.109 (I changed it from the old one to put it in a network with a DNS Server on the LAN side for testing as told by Perry.)
Strange is that the IP address 192.168.208.1 also cannot be found in our whole network.
No pirate pc from a user in here ;-).

Second one question by me:
Maybe we are talking at cross purposes.
Just to be sure what you mean with time out of the WebGui.
In my case I do not get a time out message from the browser.
It just tries to load the Webpage endlessly. Also I´m able to connect to port 80 by telnet the hole time.
Just in some cases I see a HTTP Error 503 - Service unavailable

What I did in the meantime was to test around with another Firewall Distribution.
First I tried Endian Community Edition:

• No Problems with the WebGui. Never had a time out. WebGui is extremely fast.
• No Problems with the Network Cards. All Cards has been found automatically.
  This FW would work. Just does not fit my needs completely.

Second I tried ZeroShell. Seems to has nearly the same features like pfSense.
But does not start the installation because of a kernel panic after booting from the Live CD.

Regards,

Olaf.

[the.it.dude]

ripleydraven,

Yes, it sounds like our issues are a bit different.  My browser would actually timeout.  However, I posted in the 2.0 snapshot forum and found my solution.  It was to disable killing states when a Gateway was down under the Advanced, Miscellaneous tab.

Just googling UDP 1947 points to a Aladdin HASP license manager system.  We have this running for our Vectorworks lab.  Do you have any software that requires a license service? (Or maybe was installed at one time and forgot about?)

I had played with Endian and was about to give it a try in production.  But, I got scared off by the issues in the forums and the lack of support people were getting for the community edition.  Took a look at ZeroShell.  But, that looks like it is still in early Beta.  That scared me off too.

I was about to start looking at Sonicwall or Astaro commercial products before I got my issue resolved

Jeff

[ripleydraven]

the.it.dude
I tested around some things more.
What I found in the pftop log was that for nearly every click on a link in the WebGui, the LightHTTP Server starts a new session. That OK because the sessions have a small expiration time of 60 – to 90 seconds. But after some time there will be a session with an expiration time of over 85000 seconds and after this the WebGui is not reachable anymore.
Also after some time there will be a process called keglim and this has also the long running session time. Googling this process leads you to a thread in a FreeBSD forum.

http://forums.freebsd.org/archive/index.php/t-11880.html

It seems to be really a driver problem in FreeBSD with the Broadcom Network Cards. I changed the interface to one of the Intel NIC´s and do not have any problems at all with the WebGui for days.
So at this time it works for me in this way not to use the Broadcom NIC´s an just use 8 of the network cards in the servers.

In case of Endian and ZeroShell I agree with your opinion. These distributions are not really an alternative to pfSense at this time.

Olaf.