Squid slowness issues - also a bounty post
-
The problem is not the squid package itself.
We turned off write caching on hard drives which is surely causing your slowdowns:sysctl hw.ata.wc
hw.ata.wc: 0
Edit /etc/sysctl.conf and find this value and change to:
hw.ata.wc=1I've played with this setting and have not seen any change in throughput, however mine is a SCSI box so perhaps the command is different. Do you know off hand how to re-enable write caching for SCSI disks?
I've also noticed that proxy throughput is fine for a few seconds just after reboot. After reading through the system log, it appears that squid may be starting twice. Please find below an excerpt from my cache.log
2007/12/31 10:36:35| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1... 2007/12/31 10:36:35| Process ID 948 2007/12/31 10:36:35| With 7232 file descriptors available 2007/12/31 10:36:35| Using kqueue for the IO loop 2007/12/31 10:36:35| DNS Socket created at 0.0.0.0, port 9650, FD 5 2007/12/31 10:36:35| Adding nameserver 65.106.1.196 from /etc/resolv.conf 2007/12/31 10:36:35| Adding nameserver 65.106.7.196 from /etc/resolv.conf 2007/12/31 10:36:35| Unlinkd pipe opened on FD 10 2007/12/31 10:36:35| Swap maxSize 102400 KB, estimated 7876 objects 2007/12/31 10:36:35| Target number of buckets: 393 2007/12/31 10:36:35| Using 8192 Store buckets 2007/12/31 10:36:35| Max Mem size: 8192 KB 2007/12/31 10:36:35| Max Swap size: 102400 KB 2007/12/31 10:36:35| Store logging disabled 2007/12/31 10:36:35| Rebuilding storage in /var/squid/cache (CLEAN) 2007/12/31 10:36:35| Using Least Load store dir selection 2007/12/31 10:36:35| Current Directory is /tmp 2007/12/31 10:36:35| Loaded Icons. 2007/12/31 10:36:35| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12. 2007/12/31 10:36:35| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13. 2007/12/31 10:36:35| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14. 2007/12/31 10:36:35| WCCP Disabled. 2007/12/31 10:36:35| Ready to serve requests. 2007/12/31 10:36:35| Done reading /var/squid/cache swaplog (162 entries) 2007/12/31 10:36:35| Finished rebuilding storage from disk. 2007/12/31 10:36:35| 162 Entries scanned 2007/12/31 10:36:35| 0 Invalid entries. 2007/12/31 10:36:35| 0 With invalid flags. 2007/12/31 10:36:35| 162 Objects loaded. 2007/12/31 10:36:35| 0 Objects expired. 2007/12/31 10:36:35| 0 Objects cancelled. 2007/12/31 10:36:35| 0 Duplicate URLs purged. 2007/12/31 10:36:35| 0 Swapfile clashes avoided. 2007/12/31 10:36:35| Took 0.5 seconds ( 317.8 objects/sec). 2007/12/31 10:36:35| Beginning Validation Procedure 2007/12/31 10:36:35| Completed Validation Procedure 2007/12/31 10:36:35| Validated 162 Entries 2007/12/31 10:36:35| store_swap_size = 568k 2007/12/31 10:36:36| storeLateRelease: released 0 objects 2007/12/31 10:36:40| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1... 2007/12/31 10:36:40| Process ID 1000 2007/12/31 10:36:40| With 7232 file descriptors available 2007/12/31 10:36:40| Using kqueue for the IO loop 2007/12/31 10:36:40| DNS Socket created at 0.0.0.0, port 3353, FD 5 2007/12/31 10:36:40| Adding nameserver 65.106.1.196 from /etc/resolv.conf 2007/12/31 10:36:40| Adding nameserver 65.106.7.196 from /etc/resolv.conf 2007/12/31 10:36:40| Unlinkd pipe opened on FD 10 2007/12/31 10:36:40| Swap maxSize 102400 KB, estimated 7876 objects 2007/12/31 10:36:40| Target number of buckets: 393 2007/12/31 10:36:40| Using 8192 Store buckets 2007/12/31 10:36:40| Max Mem size: 8192 KB 2007/12/31 10:36:40| Max Swap size: 102400 KB 2007/12/31 10:36:40| Store logging disabled 2007/12/31 10:36:40| Rebuilding storage in /var/squid/cache (DIRTY) 2007/12/31 10:36:40| Using Least Load store dir selection 2007/12/31 10:36:40| Current Directory is /tmp 2007/12/31 10:36:40| Loaded Icons. 2007/12/31 10:36:40| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12. 2007/12/31 10:36:40| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13. 2007/12/31 10:36:40| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14. 2007/12/31 10:36:40| WCCP Disabled. 2007/12/31 10:36:40| Ready to serve requests. 2007/12/31 10:36:40| Done reading /var/squid/cache swaplog (162 entries) 2007/12/31 10:36:40| Finished rebuilding storage from disk. 2007/12/31 10:36:40| 162 Entries scanned 2007/12/31 10:36:40| 0 Invalid entries. 2007/12/31 10:36:40| 0 With invalid flags. 2007/12/31 10:36:40| 162 Objects loaded. 2007/12/31 10:36:40| 0 Objects expired. 2007/12/31 10:36:40| 0 Objects cancelled. 2007/12/31 10:36:40| 0 Duplicate URLs purged. 2007/12/31 10:36:40| 0 Swapfile clashes avoided. 2007/12/31 10:36:40| Took 0.3 seconds ( 554.1 objects/sec). 2007/12/31 10:36:40| Beginning Validation Procedure 2007/12/31 10:36:40| Completed Validation Procedure 2007/12/31 10:36:40| Validated 162 Entries 2007/12/31 10:36:40| store_swap_size = 568k 2007/12/31 10:36:41| storeLateRelease: released 0 objects
I appreciate all the help thus far…
-
I am able to get 50+ mb/s when copying files to and from the pfSense box via WinSCP. Perhaps this isn't a write caching issue after all. I'm still ready to pay a bounty for a solution to this problem as it is impacting our business.
-
I just downloaded and installed 1.2RC3 release, not the latest snapshot and speeds were fine after installing squid via the GUI. The issue must be in the pfsense code somewhere…
-
I am having this problem as well.
I have a total of 6 pFsense boxen running 1.2 rc4 with the latest available Squid version via Packages. All 6 boxen are experiencing this issue.
When going through the proxy pages load about 30% to 40% slower. I can test at 2800kbps on a speed test site without going through the proxy and only 2200kbps going through it. I can reproduce this every time.
I am using local authentication in normal mode, NOT transparent mode.
I didn't try using Squid until RC4 was out, so cannot confirm if its a problem on an older version.
-
Hello!
Are there any news with this issue?
-
Yes, please make a squid package update
-
-
Hi, same problem here, 4 Mb downstream syncronous.
pfsense 1.2-RC3, and really serious problems with downstream using proxy, traffic sharper disabled.
squid version 2.6.18
can anyone help? is there a solution? i'm working with pfsense since a year ago and really proud of it, but no clue how to solve this issue :/
kind regards
-
Please add to our bounty to fix the problem!
-
With 1.2 release and the latest Squid package (and traffic shaper enabled) I'm seeing download speeds much the same as before the upgrades - between 16 and 17 Mb/s on my 20 Mb/s line.
-
Hi!
I am having the same squid slow transfer problems on my institution.
I have a dual-wan system: one 25 MBps cable connection on WAN for internet access and one institutional adsl 1MBps on WAN OPT3 (also has internet access using another gateway).
The OPT3 is connected to a 1GBps switch and has a public IP (we have 32 public ips for mail, webpage, etc… and a cisco router).
If I download a huge file from our servers (connected on the WAN OPT3 interface) I can easily get 11000 MBytes/sec from our internal lan's, passing through squid.
Nevertheless, accessing the internet (through the cable connection on WAN) I can get maximum 50Kbytes/sec using squid and 2000KBytes/sec using only NAT and proxy off.
Why using the same squid traffic is slow in one interface and normal on another?Strangely if I discard the cable connection, deactivate WAN OPT3, and connect the 1MBps connection on WAN interface, I can get the full speed of it, downloading at about 200Kbytes/sec from the internet.
Can it be the different modems? On the cable I have a regular Motorola modem, and on the institucional I have a much higher quality adsl modem.
Can it be the NIC? When using cable we connect it to pfsense using a 100MBps 3Com card (xl driver) and the 1Mbps is on one VLAN that enters in pfsense throug a Intel PRO1000 (em driver) together with our internal networks...Can anyone shed some light on this problem? I am liking a lot pfsense, but I cannot have squid turned on because of such slow transfers!
Yours,
Antonio Paulino -
Your choice of NICs will pretty certainly be at the core - Intel NICs are much better than the 3Com ones. I suspect if you switch it for another Intel you'll find a big improvement.
-
Hi!
Thank you for your sugestion.
I exchanged the 3Com NIC for a Intel Pro100 (fxp driver) but the problem stays…Since it is the first time I use pfsense, does anyone knows when the problem first appeared? I don't need many fancy features of pfsense, so probably a old version will just suit me. I just need a simple multi wan routing/NAT/proxy box.
António Paulino
-
The problem appeared sometime after 1.2rc3.
There was a link in another post to older versions. I just checked it and it has been updated and now only includes 1.2 final. If anyone knows where some old 1.2 RCs are located, that would be great. I have a live cd iso for 1.2rc2, but I would hope that the community will respond to the bounty to fix the problem before we're stuck at rc3.
Have you tried using only two interfaces (only LAN/WAN, disable/remove all opt interfaces) and see if your problem still exists?
-
Hi!
I found the old version (including 1.0.1) in one of the mirrors, named loquefaltaba.
There is one directory "old" which old versions, full iso and upgrade packages.I haven't done it, but I will try, perhaps today, and then let you know.
António Paulino
-
Please can you guys provide the output of kldstat command or by any chance those that have issues are using spamd package?
-
From my working (fast) 1.2rc3 box:
Id Refs Address Size Name 1 3 0xc0400000 6df4ac kernel 2 1 0xc0ae0000 59e80 acpi.ko
From my working (very slow) 1.2-Release box:
Id Refs Address Size Name 1 4 0xc0400000 7fb834 kernel 2 1 0xc0bfc000 59e80 acpi.ko 3 1 0xc6f56000 d000 ipfw.ko
What do these outputs mean?
-
ipfw.ko is ipfw which is enabled for schedules and or captive portal.
-
Hi I have five computers with pfsense, all with squid and works very well, never get this problem on any machine, don´t use intel or 3com cards, all cards are realtek 8139, ps - only one point I have a link (1 Mb), all others using pppoe connections. ??? ???
-
Hi!
Finally today I tested an older version of pfsense to diagnose the problem with squid, namely 1.2-RC2.
I used a machine where 1.2-RELEASE and squid 2.6.18 is slow (tested it).
First with only a 25Mbps connection (in WAN) and LAN connection downloading through squid is blazing fast (the same speed without squid).
Next I downloaded the config.xml from the production machine (1.2-RELEASE), changed the interface IP's (to keep the two machines working at the same time) and uploaded to the 1.2-RC2 version. This includes two-WAN configuration and the three internal networks. The speed stayed fast.
This means the problem is with 1.2-RC4 and later distribution, because now the configuration is the same in both machines: the 1.2-Rc2 is downloading fast and 1.2-RELEASE is downloading slow!Any suggestions?
Paulino