Dismal squid performance

jimp

Well it's hard to say what the exact cause might be in this case, I was just curious if the performance was the same or different on 32 vs 64. If they are both slow, it's probably more related to the network card or some other chipset issue.

clarknova

Right. I will give 32 bit a try.

jimp

Looks like amd64 packages are having a problem. Somehow some of them are being mixed up with their 32-bit counterparts, resulting in libs that won't load on amd64. I'm checking into it.

jimp

32-bit and 64-bit squid should be OK now. Both were rebuilt today and I've tried them out and they are starting again.

clarknova

I reinstalled squid 64-bit in a vm today and performance is worse than before. For example, downloading an iso from http://mirror.csclub.uwaterloo.ca, without squid it comes in ~6mbps. Using squid (non-transparent mode) it's a steady 1.75+/-.02 mbps. Using squid on a 32-bit pfsense 1.2.3 vm, the same download comes out of cache ~200 mbps. Both VMs are on the same ESXi server using 2 Xeon 5150 CPU cores and Intel GBE. PF 2.0 has 6GB RAM and 1.2.3 has 3.0GB, so hardware should not be an issue. Squid settings are identical between the two.

jimp

What is the output of "ifconfig -a" when running on 64-bit?

If it says anything about TSO and/or LRO, go into the advanced options and make sure the options to disable those are checked, and you may as well disable checksums.

clarknova

$ ifconfig -a
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:59:fa:4e
	inet6 fe80::20c:29ff:fe59:fa4e%em0 prefixlen 64 scopeid 0x1 
	inet 172.21.252.1 netmask 0xfffffe00 broadcast 172.21.253.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:59:fa:58
	inet 172.21.33.58 netmask 0xfffffe00 broadcast 172.21.33.255
	inet6 fe80::20c:29ff:fe59:fa58%em1 prefixlen 64 scopeid 0x2 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33152
enc0: flags=0<> metric 0 mtu 1536</promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

TSO and LRO were already disabled. I disabled checksum offloading and DL from cache is still 1.77mbps. This is on a VM.

jimp

Looks normal then in that regard.

Is it VMware?
If so, can you confirm if the normal VMware settings are still there?

/etc/sysctl.conf should have:

kern.timecounter.hardware=i8254

/boot/loader.conf should have:

kern.hz="100"

clarknova

I'll try those. By default my sysctl.conf has no uncommented lines, and loader.conf is empty.

clarknova

I copied /boot/loader.conf from another 2.0 machine, added those two lines from your post to their respective files, and rebooted. DL from cache is now 2.3 +/- .2 mbps. This is on ESXi, pfsense installed from pfsense.iso

clarknova

Are you getting good performance from yours now? If so I will try reinstalling it on my home pfsense. I just set up the vm because of the crashing that happened during the 32/64 fiasco. I want to minimize testing on my home box but if there's a chance that my issues are specific to this vm then I'll drop it like a rock.

jimp

I don't have a VM setup "behind" it to test right now. Mine is also in VirtualBox so it's a bit different setup.

clarknova

I installed squid on a fresh Beta4 i386 system this morning and….same problem. iso comes out of cache at 3.3 mbps. About halfway through the download it jumps to 10 mbps. No detectable signs of strain from any part of the system, just slow.

jimp

I tried it out on my amd64 box today, I moved an XP VM behind the amd64 pfSense VM with squid, and it came through at the full wire speed of my cable, 10Mbps.

clarknova

Uncached objects always download at the expected rate, it's the cached objects that are slow without apparent cause. Try deleting the download from your xp machine and then download it again. Assuming your squid cache was configured to cache objects of that size, you will see what I mean.

The expected behaviour is that cached objects should download at a speed that is bounded only by CPU, LAN network, or disk speed. Try this exercise in 1.2.3 and see the difference.

jimp

Ah, well I noticed in my haste of configuring squid I left its cache size as 100mb, not very effective if I want to store ISOs while testing…

So I bumped it up to 3GB, and tried again, and the download came to me at 17MByte/s (byte, not bit). I deleted the downloaded file and cleared firefox's cache between attempts. Squid's access log shows a TCP_HIT for the request so I know it was coming from the cache.

I'll try it a few more times with different download locations to see if I get any differences.

This is on an amd64 VM on a core i5-750, with the only tweak being the single loader.conf line tweaking nmbclusters to 32768.

jimp

Tried a few more iso downloads and the lowest one I got was 10MByte/s. (Though it went back and forth between 10-11) and the fastest was that first one, about 17MB/s.

I did notice that at least one place, Knoppix, looked like it was an http link on a mirror but actually redirected to an ftp link so it bypassed the cache entirely. (I only figured that out when I right clicked on the download in FF and copied the link to be sure I got the same thing twice).

clarknova

Well that's way better than I can get out of it. Can I copy your config? I must be doing something wrong, or there is a hardware problem that is common to my home machine and the vm at work.

jimp

Nothing special about the config, just a stock install (I haven't even changed the hostname or many other basic options really)

Squid and Lightsquid are the only packages on it.

Here's the squid sections:


		 <squid><config><active_interface>lan</active_interface>
				<allow_interface>on</allow_interface>
				<transparent_proxy>on</transparent_proxy>
				 <private_subnet_proxy_off><defined_ip_proxy_off><log_enabled>on</log_enabled>
				<log_dir>/var/squid/log</log_dir>
				 <log_rotate><proxy_port>3128</proxy_port>
				 <icp_port><visible_hostname>localhost</visible_hostname>
				<admin_email>admin@localhost</admin_email>
				<error_language>English</error_language>
				 <disable_xforward><disable_via><uri_whitespace>strip</uri_whitespace>
				 <dns_nameservers><disable_squidversion></disable_squidversion></dns_nameservers></disable_via></disable_xforward></icp_port></log_rotate></defined_ip_proxy_off></private_subnet_proxy_off></config></squid> 
		 <squidcache><config><harddisk_cache_size>3000</harddisk_cache_size>
				<harddisk_cache_system>aufs</harddisk_cache_system>
				<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
				<memory_cache_size>8</memory_cache_size>
				<minimum_object_size>0</minimum_object_size>
				<maximum_object_size>50000000</maximum_object_size>
				<level1_subdirs>16</level1_subdirs>
				<memory_replacement_policy>heap GDSF</memory_replacement_policy>
				<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
				<cache_swap_low>90</cache_swap_low>
				<cache_swap_high>95</cache_swap_high>
				 <donotcache><enable_offline></enable_offline></donotcache></config></squidcache>

rkelleyrtp

Just thought I would add my $0.02 here. I was running into the exact same problems with squid v2.7.9_1 on pfSense 2.0-BETA4 - Sept 1, 2010. After enabling squid, I was getting extremely poor performance from the local cache compared to just downloading from the internet and my ping times went thru the roof. Prior to squid, I ran a speed test (speedtest.net) and my ping times were 20ms-30ms. After enabling squid, my ping times went to +300ms and beyond. Also, once a file was in the local squid cache, the download rate of a file in squid cache would vary wildly from 1MB/sec to 25MB/sec (I was running running "wget <cached_url>" from my Mac terminal app).

After reading thru this thread, I made the changes both clarknova and jimp mentioned (/boot/loader.conf and /etc/sysctl.conf) as well as rebooting pfSense and stopping/(re)starting squid and mucking around with the /var/squid directory. Finally, something "clicked" on my firewall and squid started working properly. Here are some things I did which finally caused squid to behave properly:

Edited the /boot/loader.conf with this info:
–-----------------------------------------------
kern.ipc.nmbclusters="32768"
kern.maxfiles="65536"
kern.maxfilesperproc="32768"
net.inet.ip.portrange.last="65535"
Edit /etc/sysctl.conf with this info:

net.inet.tcp.inflight.enable=0
net.inet.tcp.hostcache.expire=1

Enabled all the TSL/LRO features in System-->Advanced-->Networking (including Disable hardware checksum offload, Disable hardware TCP segmentation offload, and Disable hardware large receive offload). I rebooted the system, saw no difference, then DISABLED these features again (I think this is what caused squid to start working properly).
Edited the squid configuration options:

Dropped the RAM cache to 8MB
Modified the disk cache to 50G
Changed the disk disk cache system to ufs (I think this is also what caused squid to work properly)
Recreated the /var/squid directory:

From the CLI, stopped squid (/usr/local/etc/rc.d/squid stop
Renamed /var/squid to /var/squid.old
Created a new /var/squid directory
Copied /var/squid.old/. /var/squid
Restarted squid (/usr/local/etc/rc.d/squid start)

After mucking around for about 2hrs, squid started working as it should, and I am now able to download from the local disk cache at +35MB/sec. I think it has something to do with my SSD and disk options set in the squid configuration.

All in all, I don't have the magic bullet to fix the problem - just a set of things I did that made my install magically work.

Hope this helps...</cached_url>