Web server pages rendering slowly going out WAN but fine internally - HELP PLS!

jobsoft

Very interesting indeed. What is your Internet setup configuration there? Since the two places that I tested it from (here and from Vermont) were also on cable internet and both behind NAT routers (each with a Linksys WRT54G running dd-wrt v23 SP2!). Both behaved the same way. I did ask the people at Barfield to test it out advise if performance or other problems and they said it look great to them too.

So, I wonder if the WRT54G's could be a factor in this anomaly??? I will have to rig my laptop direct to my cable this morning and see.

Also, as yet another "try this", I pulled up firefox here at my house from a fedora linux desktop and tried www.jobsoft.com. same thing! :-( But, I went ahead and captured some screen shots for the page render "progress" after the 1st, 2nd and 3rd minutes:

http://www.jobsoft.com/Screenshot_Jobsoft_Design_and_Development_1st_Minute.png
http://www.jobsoft.com/Screenshot_Jobsoft_Design_and_Development_2nd_Minute.png
http://www.jobsoft.com/Screenshot_Jobsoft_Design_and_Development_3rd_Minute.png

This is what I get no matter when I try it and from where and what here in the house behind the WRT54G NAT router. Notice in the 3rd minute the browser had given up and was "Done".

I can also remote VNC to a linux desktop at a customers site that has T1 and a cisco router this moming as well and see what it does from there.

Thanks for the feedback! It has helped to shift focus a bit.

jobsoft

One quick followup. Since I can packet capture on each end of this through the same event period, can anyone suggest what I might look for in wireshark that would be enlightening as to not necessarily what caused the problem to begin with, but what packet situation is resulting in the delays???

jobsoft

OK,

I have done the tcpdumps from 3 places:

http://www.jobsoft.com/packet-watch-dmz-filtered.cap
http://www.jobsoft.com/packet=watch-eth0-filtered.cap
http://www.jobsoft.com/packet-watch-wan-filtered.cap

All tcpdumps were 'tcpdump -s 1500 -i <iface>-w <capfile>.cap' and run simultaneously while I exercised the web pages from windows and linux here at my house. The anomalies did manifest themselves.

I then brought all 3 into Wireshark and filtered out only the packets to/from the web server and my external cable ip address and then saved those filtered sets back to the files above. I am making them available above as well in case anyone else wants to peek at them too, however, I certainly am already! :-)

DMZ was on the pfsense box xl1/DMZ interface. WAN was on the xl0/WAN interface. ETH0 was on the linux server at my house that I had firefox running from and it was listening on the inside wired lan.

What I did discover on the ETH0 stood out was several of the larger packets with HTTP payloads had checksum errors. While I have only just looked at these initially, something like that would trigger a retry. I also saw some "TCP DUP ACKs". What I will have to go back and do is trace one of these packets with the failed checksum back through WAN and DMZ and the see what followed. Ideally, if I could correlate the pauses in page rendering with the HTML contained in these retried packets, that would at least tie the browser behavior to the packet conditions. When I hook up my laptop direct to cable and then capture packets in the same way (just wireshark directly off the laptop on my house side).

The whole thing still puzzles me. ???</capfile></iface>

hoba

Do you see lots of errors or collisions at status>interfaces at one of the nics?

jobsoft

netstat -i

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
xl0 1500 <link#1>00:60:97:d0:14:fe 792325 0 813918 0 0
xl0 1500 fe80:1::260:9 fe80:1::260:97ff: 0 - 2 - -
xl0 1500 70-90-228-184 70-90-228-189-Nas 4167 - 5948 - -

xl1 1500 <link#2>00:10:4b:37:d3:5d 595782 26 554584 0 842
xl1 1500 fe80:2::210:4 fe80:2::210:4bff: 0 - 1 - -
xl1 1500 172.21/24 172.21.0.2 3187 - 7657 - -
xl2 1500 <link#3>00:50:04:76:95:f5 280118 133 294471 0 402
xl2 1500 fe80:3::250:4 fe80:3::250:4ff:f 0 - 1 - -
xl2 1500 192.168.1 gate 1885 - 2897 - -
pflog 33208 <link#4>0 0 0 0 0
lo0 16384 <link#5>9 0 9 0 0
lo0 16384 your-net localhost 445 - 0 - -
lo0 16384 localhost ::1 0 - 0 - -
lo0 16384 fe80:5::1 fe80:5::1 0 - 0 - -
pfsyn 2020 <link#6>0 0 0 0 0

Some Ierrs on xl1 (DMZ) and xl2 (LAN - not being considered at the moment) - none on xl0 (WAN)</link#6></link#5></link#4></link#3></link#2></link#1>

sullrich

Ahh yes. Checksum offloading errors.

From a shell:

ifconfig xl0 -rxsum
ifconfig xl1 -rxsum
ifconfig xl2 -rxsum

These seem like older cards, eh? I bet the checksum offloading is busted in FreeBSD.

jobsoft

Well, this gets even more interesting. I did what someone else suggested an hooked up my laptop directly to the cable modem. I had wireshark on it and started a capture and remoted with putty to my office and logged onto the pfsense box and start tcpdumps on DMZ and WAN. I then pulled up the browser on my laptop and the accessed the same pages. Here are the PCAP files:

http://www.jobsoft.com/packet-watch-dmz-lapdir-take2-filtered.pcap
http://www.jobsoft.com/packet-watch-eth0-lapdir-take2-filtered.pcap
http://www.jobsoft.com/packet-watch-wan-lapdir-take2-filtered.pcap

The amazing thing to me is the web pages rendered FLAWLESSLY! Yeah, a dropped packet here and there, but, NO delay in rendering!! 1-2 SECONDS tops! And none of the corrupt packets from when testing behind the WRT54G NAT.

OK, now, before PFSENSE, these sites also rendered in 1-2 seconds FROM BEHIND the WRT54G NAT!! So, when in combination of:

WEBSERVERS<–>DMZ<-->PFSENSE<-->WAN<-->WRT54G<-->LAPTOP

problems with packet turnaround/delays arise.

Again, before PFSENSE, it was Fedora6/Shorewall-IPTABLES. Under that scenario, some server were on LAN and some on WAN, but, all accessed with no problems whatsoever. With PFSENSE and take out WRT54G, things seem fine too. It is almost as if at the packet level some packets are getting corrupt and/or held up such that at the TCP/packet level no retransmission occurs until the browser (maybe?) tries the URL it is seeking. I haven't dug down that deep yet.

My next steps are 1) swap back in Fedora and capture/test, 2) different NICs and/or 3) "modern" PC hardware for PFSENSE.

Mark

jobsoft

@sullrich:

Ahh yes. Checksum offloading errors.

From a shell:

ifconfig xl0 -rxsum
ifconfig xl1 -rxsum
ifconfig xl2 -rxsum

These seem like older cards, eh? I bet the checksum offloading is busted in FreeBSD.

Slightly! :-)

xl0: <3Com 3c905-TX Fast Etherlink XL> port 0xd800-0xd83f irq 10 at device 18.0 on pci0
xl1: <3Com 3c905B-TX Fast Etherlink XL> port 0xdc00-0xdc7f mem 0xe8801000-0xe880107f irq 5 at device 19.0 on pci0
xlphy0: <3Com internal media interface> on miibus1
xl2: <3Com 3cSOHO100-TX OfficeConnect> port 0xe000-0xe07f mem 0xe8800000-0xe880007f irq 11 at device 20.0 on pci0
xlphy1: <3Com internal media interface> on miibus2

I have 3 newer AON-325 (Realtek 8139C) I was going to try. I also have a Compaq Netexpress (sp?) and can get my hands on another Intel Pro 100.

I had seen some reference to the issue with offloading in another thread. What still concerns me though is that I did try all 3 of these cards in different combinations of roles, so, I would suspect that one might work. and secondly, when I remove the WRT54G, and go laptop direct, no checksum errors at all (best I can tell). This could be an anomaly deriving from PFSENSE/FreeBSD and the WRT54G that only manifests itself when they come together (like chocolate and peanut better)! :-)

jobsoft

@sullrich:

Ahh yes. Checksum offloading errors.

From a shell:

ifconfig xl0 -rxsum
ifconfig xl1 -rxsum
ifconfig xl2 -rxsum

These seem like older cards, eh? I bet the checksum offloading is busted in FreeBSD.

forgot to add the results of the above commands you suggested:

ifconfig xl0 -rxsum

ifconfig: -rxsum: bad value

ifconfig xl1 -rxsum

ifconfig: -rxsum: bad value

ifconfig xl2 -rxsum

ifconfig: -rxsum: bad value

sullrich

Sorry, should be:

-rxcsum

jobsoft

Seems to have returned nothing on each:

ifconfig xl0 -rxcsum

ifconfig xl1 -rxcsum

ifconfig xl2 -rxcsum

Mark

sullrich

Then it worked. Do a ifconfig and compare the flags with what you posted originally.

jobsoft

WHOA!! ;D

That seems to have fixed it!!

I will have to go look up just what that option does.

Question is then, where can I work that into config.xml? I know there are options for media. I suppose I could look at the "master" config.xml that is over on the M0n0wall site as I suspect there are hidden options for something like this. I suppose I could tuck them into an rc file at bootup, but, I wonder if a "refresh" and/or change to an interface setting from within pfsense would reset the flags?!

But, at any length, this has advanced me considerably! I still intend to try and figure out why the WRT54G was mucking things up more that caused the checksum issue to rear up. But, at least it is functional and acceptable again.

I will post a follow to the WRT54G aspect at some point IFF something pertaining to pfsense is contributing.

Thanks all!!

Mark

sullrich

add a of shellcmd to config.inc from /cf/conf/config.xml

Download the configuration using the configuration backup and then add this under the <system>tag:

<shellcmd>ifconfig xl0 -rxcsum;ifconfig xl1 -rxcsum;ifconfig xl2 -rxcsum;</shellcmd>

Then re-upload the configuration back and note the firewall will reboot after restoring the changed configuration.. Afterwards the system will run the commands on reboot.

And FYI it appears that either your hardware's checksum offloading is busted or there is a driver issue. </system>

jobsoft

Thanks for this info. I am planning on getting 3 Intel chip cards ordered as I have read many others recommend them. Aside from Intel, any others good candidates?

sullrich

If you can swing it, get the intel gigabits. They have larger buffers and will decrease cpu overhead.

CryoGenID

Sorry for writing here into this discussion:
Are you sure the Intel NIC's won't make any problems?
We have a weird issue here and have also applied this "tweak" here and are
currently testing if our problems are resolved…
And we are using the Intel Dual Port GBit NIC's (and are also having the same
problem of lost/stuck packets with a brand new HP Bl20p G3-Blade)...

Has anybody running pfSense on a HP Blade successfully or on a DL380 with
two Intel Dual Port GBit NIC's?

Thanks for your reply :)

hoba

My homebox has 2x intel fxp onboard (ibm eserver). I don't see any issues with it. Not a zero in/out error. Same at the nexcom at our office or 2 other nexcoms that I have out with intel nics. However these drivers have support for several intel chipsets, so the problems might only arise with really new chipsets like in your hardware.