Squid slowness issues - also a bounty post
-
did you also checked the (RFC 1918) Checkbox?
-
did you also checked the (RFC 1918) Checkbox?
No. I didn't think that was needed for my network.
-
please check this box and test again
-
The problem is not the squid package itself.
We turned off write caching on hard drives which is surely causing your slowdowns:sysctl hw.ata.wc
hw.ata.wc: 0
Edit /etc/sysctl.conf and find this value and change to:
hw.ata.wc=1I've played with this setting and have not seen any change in throughput, however mine is a SCSI box so perhaps the command is different. Do you know off hand how to re-enable write caching for SCSI disks?
I've also noticed that proxy throughput is fine for a few seconds just after reboot. After reading through the system log, it appears that squid may be starting twice. Please find below an excerpt from my cache.log
2007/12/31 10:36:35| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1... 2007/12/31 10:36:35| Process ID 948 2007/12/31 10:36:35| With 7232 file descriptors available 2007/12/31 10:36:35| Using kqueue for the IO loop 2007/12/31 10:36:35| DNS Socket created at 0.0.0.0, port 9650, FD 5 2007/12/31 10:36:35| Adding nameserver 65.106.1.196 from /etc/resolv.conf 2007/12/31 10:36:35| Adding nameserver 65.106.7.196 from /etc/resolv.conf 2007/12/31 10:36:35| Unlinkd pipe opened on FD 10 2007/12/31 10:36:35| Swap maxSize 102400 KB, estimated 7876 objects 2007/12/31 10:36:35| Target number of buckets: 393 2007/12/31 10:36:35| Using 8192 Store buckets 2007/12/31 10:36:35| Max Mem size: 8192 KB 2007/12/31 10:36:35| Max Swap size: 102400 KB 2007/12/31 10:36:35| Store logging disabled 2007/12/31 10:36:35| Rebuilding storage in /var/squid/cache (CLEAN) 2007/12/31 10:36:35| Using Least Load store dir selection 2007/12/31 10:36:35| Current Directory is /tmp 2007/12/31 10:36:35| Loaded Icons. 2007/12/31 10:36:35| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12. 2007/12/31 10:36:35| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13. 2007/12/31 10:36:35| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14. 2007/12/31 10:36:35| WCCP Disabled. 2007/12/31 10:36:35| Ready to serve requests. 2007/12/31 10:36:35| Done reading /var/squid/cache swaplog (162 entries) 2007/12/31 10:36:35| Finished rebuilding storage from disk. 2007/12/31 10:36:35| 162 Entries scanned 2007/12/31 10:36:35| 0 Invalid entries. 2007/12/31 10:36:35| 0 With invalid flags. 2007/12/31 10:36:35| 162 Objects loaded. 2007/12/31 10:36:35| 0 Objects expired. 2007/12/31 10:36:35| 0 Objects cancelled. 2007/12/31 10:36:35| 0 Duplicate URLs purged. 2007/12/31 10:36:35| 0 Swapfile clashes avoided. 2007/12/31 10:36:35| Took 0.5 seconds ( 317.8 objects/sec). 2007/12/31 10:36:35| Beginning Validation Procedure 2007/12/31 10:36:35| Completed Validation Procedure 2007/12/31 10:36:35| Validated 162 Entries 2007/12/31 10:36:35| store_swap_size = 568k 2007/12/31 10:36:36| storeLateRelease: released 0 objects 2007/12/31 10:36:40| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1... 2007/12/31 10:36:40| Process ID 1000 2007/12/31 10:36:40| With 7232 file descriptors available 2007/12/31 10:36:40| Using kqueue for the IO loop 2007/12/31 10:36:40| DNS Socket created at 0.0.0.0, port 3353, FD 5 2007/12/31 10:36:40| Adding nameserver 65.106.1.196 from /etc/resolv.conf 2007/12/31 10:36:40| Adding nameserver 65.106.7.196 from /etc/resolv.conf 2007/12/31 10:36:40| Unlinkd pipe opened on FD 10 2007/12/31 10:36:40| Swap maxSize 102400 KB, estimated 7876 objects 2007/12/31 10:36:40| Target number of buckets: 393 2007/12/31 10:36:40| Using 8192 Store buckets 2007/12/31 10:36:40| Max Mem size: 8192 KB 2007/12/31 10:36:40| Max Swap size: 102400 KB 2007/12/31 10:36:40| Store logging disabled 2007/12/31 10:36:40| Rebuilding storage in /var/squid/cache (DIRTY) 2007/12/31 10:36:40| Using Least Load store dir selection 2007/12/31 10:36:40| Current Directory is /tmp 2007/12/31 10:36:40| Loaded Icons. 2007/12/31 10:36:40| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12. 2007/12/31 10:36:40| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13. 2007/12/31 10:36:40| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14. 2007/12/31 10:36:40| WCCP Disabled. 2007/12/31 10:36:40| Ready to serve requests. 2007/12/31 10:36:40| Done reading /var/squid/cache swaplog (162 entries) 2007/12/31 10:36:40| Finished rebuilding storage from disk. 2007/12/31 10:36:40| 162 Entries scanned 2007/12/31 10:36:40| 0 Invalid entries. 2007/12/31 10:36:40| 0 With invalid flags. 2007/12/31 10:36:40| 162 Objects loaded. 2007/12/31 10:36:40| 0 Objects expired. 2007/12/31 10:36:40| 0 Objects cancelled. 2007/12/31 10:36:40| 0 Duplicate URLs purged. 2007/12/31 10:36:40| 0 Swapfile clashes avoided. 2007/12/31 10:36:40| Took 0.3 seconds ( 554.1 objects/sec). 2007/12/31 10:36:40| Beginning Validation Procedure 2007/12/31 10:36:40| Completed Validation Procedure 2007/12/31 10:36:40| Validated 162 Entries 2007/12/31 10:36:40| store_swap_size = 568k 2007/12/31 10:36:41| storeLateRelease: released 0 objects
I appreciate all the help thus far…
-
I am able to get 50+ mb/s when copying files to and from the pfSense box via WinSCP. Perhaps this isn't a write caching issue after all. I'm still ready to pay a bounty for a solution to this problem as it is impacting our business.
-
I just downloaded and installed 1.2RC3 release, not the latest snapshot and speeds were fine after installing squid via the GUI. The issue must be in the pfsense code somewhere…
-
I am having this problem as well.
I have a total of 6 pFsense boxen running 1.2 rc4 with the latest available Squid version via Packages. All 6 boxen are experiencing this issue.
When going through the proxy pages load about 30% to 40% slower. I can test at 2800kbps on a speed test site without going through the proxy and only 2200kbps going through it. I can reproduce this every time.
I am using local authentication in normal mode, NOT transparent mode.
I didn't try using Squid until RC4 was out, so cannot confirm if its a problem on an older version.
-
Hello!
Are there any news with this issue?
-
Yes, please make a squid package update
-
-
Hi, same problem here, 4 Mb downstream syncronous.
pfsense 1.2-RC3, and really serious problems with downstream using proxy, traffic sharper disabled.
squid version 2.6.18
can anyone help? is there a solution? i'm working with pfsense since a year ago and really proud of it, but no clue how to solve this issue :/
kind regards
-
Please add to our bounty to fix the problem!
-
With 1.2 release and the latest Squid package (and traffic shaper enabled) I'm seeing download speeds much the same as before the upgrades - between 16 and 17 Mb/s on my 20 Mb/s line.
-
Hi!
I am having the same squid slow transfer problems on my institution.
I have a dual-wan system: one 25 MBps cable connection on WAN for internet access and one institutional adsl 1MBps on WAN OPT3 (also has internet access using another gateway).
The OPT3 is connected to a 1GBps switch and has a public IP (we have 32 public ips for mail, webpage, etc… and a cisco router).
If I download a huge file from our servers (connected on the WAN OPT3 interface) I can easily get 11000 MBytes/sec from our internal lan's, passing through squid.
Nevertheless, accessing the internet (through the cable connection on WAN) I can get maximum 50Kbytes/sec using squid and 2000KBytes/sec using only NAT and proxy off.
Why using the same squid traffic is slow in one interface and normal on another?Strangely if I discard the cable connection, deactivate WAN OPT3, and connect the 1MBps connection on WAN interface, I can get the full speed of it, downloading at about 200Kbytes/sec from the internet.
Can it be the different modems? On the cable I have a regular Motorola modem, and on the institucional I have a much higher quality adsl modem.
Can it be the NIC? When using cable we connect it to pfsense using a 100MBps 3Com card (xl driver) and the 1Mbps is on one VLAN that enters in pfsense throug a Intel PRO1000 (em driver) together with our internal networks...Can anyone shed some light on this problem? I am liking a lot pfsense, but I cannot have squid turned on because of such slow transfers!
Yours,
Antonio Paulino -
Your choice of NICs will pretty certainly be at the core - Intel NICs are much better than the 3Com ones. I suspect if you switch it for another Intel you'll find a big improvement.
-
Hi!
Thank you for your sugestion.
I exchanged the 3Com NIC for a Intel Pro100 (fxp driver) but the problem stays…Since it is the first time I use pfsense, does anyone knows when the problem first appeared? I don't need many fancy features of pfsense, so probably a old version will just suit me. I just need a simple multi wan routing/NAT/proxy box.
António Paulino
-
The problem appeared sometime after 1.2rc3.
There was a link in another post to older versions. I just checked it and it has been updated and now only includes 1.2 final. If anyone knows where some old 1.2 RCs are located, that would be great. I have a live cd iso for 1.2rc2, but I would hope that the community will respond to the bounty to fix the problem before we're stuck at rc3.
Have you tried using only two interfaces (only LAN/WAN, disable/remove all opt interfaces) and see if your problem still exists?
-
Hi!
I found the old version (including 1.0.1) in one of the mirrors, named loquefaltaba.
There is one directory "old" which old versions, full iso and upgrade packages.I haven't done it, but I will try, perhaps today, and then let you know.
António Paulino
-
Please can you guys provide the output of kldstat command or by any chance those that have issues are using spamd package?
-
From my working (fast) 1.2rc3 box:
Id Refs Address Size Name 1 3 0xc0400000 6df4ac kernel 2 1 0xc0ae0000 59e80 acpi.ko
From my working (very slow) 1.2-Release box:
Id Refs Address Size Name 1 4 0xc0400000 7fb834 kernel 2 1 0xc0bfc000 59e80 acpi.ko 3 1 0xc6f56000 d000 ipfw.ko
What do these outputs mean?
-
ipfw.ko is ipfw which is enabled for schedules and or captive portal.
-
Hi I have five computers with pfsense, all with squid and works very well, never get this problem on any machine, don´t use intel or 3com cards, all cards are realtek 8139, ps - only one point I have a link (1 Mb), all others using pppoe connections. ??? ???
-
Hi!
Finally today I tested an older version of pfsense to diagnose the problem with squid, namely 1.2-RC2.
I used a machine where 1.2-RELEASE and squid 2.6.18 is slow (tested it).
First with only a 25Mbps connection (in WAN) and LAN connection downloading through squid is blazing fast (the same speed without squid).
Next I downloaded the config.xml from the production machine (1.2-RELEASE), changed the interface IP's (to keep the two machines working at the same time) and uploaded to the 1.2-RC2 version. This includes two-WAN configuration and the three internal networks. The speed stayed fast.
This means the problem is with 1.2-RC4 and later distribution, because now the configuration is the same in both machines: the 1.2-Rc2 is downloading fast and 1.2-RELEASE is downloading slow!Any suggestions?
Paulino -
Thanks for the additional testing…it's appreciated.
@ Sullrich
Did the FreeBSD 6.3p?? change from rc3 to rc4 and beyond? I'm not familiar with what theses various releases mean or how much changes from version to version. Have there been any other symptoms that appeared with no changes in the pfsense code from rc3 to rc4? Still just trying to root out the cause of this one. Thanks for your continued input on this and amazing contributions with pfsense.
-
Thanks for the additional testing…it's appreciated.
@ Sullrich
Did the FreeBSD 6.3p?? change from rc3 to rc4 and beyond? I'm not familiar with what theses various releases mean or how much changes from version to version. Have there been any other symptoms that appeared with no changes in the pfsense code from rc3 to rc4? Still just trying to root out the cause of this one. Thanks for your continued input on this and amazing contributions with pfsense.
Not that I can recall.
-
I can confirm. Fresh install of 1.2. Nothing but squid running, latest package as of last night.
With transparent enabled, on our dual-bonded T1 here, I get 600-1100Kbit.
Disabled, I get ~ the full 3Mbit.Architecture is PentiumD with SMP kernel, 1GB RAM, 2GB Swap.
I monitored CPU usage with top. No difference. 99% idle either way.
-
Had the same issue:
Changed the following in /usr/local/pkg/squid.inc and it seems to have increased the speed significantly:
Original line: cache_dir disk_d $cachedir $disk_cache_size $level1 256
New line: cache_dir aufs $cachedir $disk_cache_size $level1 256basically changing from disk_d to aufs
save the file
restart squid. -
Good find, package updated!!
-
thanks everyone, am going to change as said and see if there is any improvement.
-
One warning, upgrading Squid kills it (or at least it did for me) - you have to blow away the old cache directories (in /usr/local/squid/cache) first.
-
@Cry:
One warning, upgrading Squid kills it (or at least it did for me) - you have to blow away the old cache directories (in /usr/local/squid/cache) first.
Hi, how to blow away the old cache /usr/local/squid/cache ?
I'm sorry i just installed pfsense+squid today. even the aufs changed, i still get same problem.
-
SSH in or go to the shell. Option 8 brings you to the command line. Then use the rm command to delete the files in that folder.
-
i have download the latest squid from pakages, its looks like cache_dir aufs $cachedir $disk_cache_size $level1 256 already changed by sullrich.
i have clear cache directories. but it is same like before. download still in a low rate.
i will try 1.2-RC2. thanks.
-
Same here, no improvements. I appreciate the update though as I had been reading that 'aufs' is pretty much the standard now and disk_d is becoming obsolete.
-
Confirmed again that the aufs change, with squid restart, does nothing. Last check, with squid running, downloads were running about 75% slower than with transparent proxy disabled.
-
I have install another pc with 1.2-RC2. And install squid(stable18) from latest packaged (updated from aufs to disk_d).
it's still not working. speed go down. :(
???
which squid can run on 1.2-RC2?
-
So far the only sure thing is pfsense 1.2rc3, though there have been so many bugfixes since then it hurts to go backwards. If you feel the need to, you can go here…We are still on 1.2rc3 and flying along at full speed.
http://pfsense.loquefaltaba.com/downloads/old/ -
Set up the new version on a 1.2 Release machine that's plugged into two bonded T1s. If downloads are any slower with the transparent proxy on, its a really a marginal difference.
-
just test with pfsense 1.2-RC3, and my squid working on it. ;D
maybe it have problem with 1.2release.
Thank you.
-
Set up the new version on a 1.2 Release machine that's plugged into two bonded T1s. If downloads are any slower with the transparent proxy on, its a really a marginal difference.
What sort of hardware do you run (NIC, CPU, scsi, sas, ide) also what T-1 modem/router are you using? If you have multiple processors, do you use SMP kernel or no?