Fix the squid package {completed}



  • This discussion has been ongoing in the packages forum.  The squid package is now very slow for some reason.  It is also based on an outdated release.  To be awarded the bounty, update the squid package to the latest stable release (2.6.18), provide transparent proxy throughput at or very near non-proxy throughput, and make sure that sites like auctionsniper.com and evga.com (maybe others too?) display properly when opened through the proxy.  All of this must work on 1.2rc5 or 1.2_STABLE when it happens.  I am willing to contribute $250 to the bounty, and heiko $200 per one of the related posts below as long as he gets an invoice.  To get up to speed I've listed most (I think) of the other related posts below.

    http://forum.pfsense.org/index.php/topic,7887.0.html

    http://forum.pfsense.org/index.php/topic,7281.0.html

    http://forum.pfsense.org/index.php/topic,7538.0.html
    http://forum.pfsense.org/index.php/topic,6835.0.html

    http://forum.pfsense.org/index.php/topic,7652.0.html

    http://forum.pfsense.org/index.php/topic,6808.0.html
    http://cvstrac.pfsense.com/tktview?tn=1557



  • To be awarded the bounty, update the squid package to the latest stable release (2.6.18), provide transparent proxy throughput at or very near non-proxy throughput,

    Can you tell me what this is?!



  • I will contribute $50 if this can fix my issue here:

    http://forum.pfsense.org/index.php/topic,7919.0.html



  • As i saw, sullrich have updated the squid package to the latest 2.6.18 version!



  • Sullrich, thanks for being so amazingly prompt in getting the package updated, especially in light of 1.2 approaching release.

    The 'broken' sites appear to be working under the latest version.  The speeds look improved, but not fixed.  Under 2.6.18/1.2RC5 I can get 600kbps where I used to see about 250-300 kbps before the package bump.  This is on a bonded T-1 WAN 3mbps where I can see 2700kbps on squid 2.6.5/1.2RC2.

    Getting the package updated is a huge step, thanks…now where is all the bandwidth going?



  • Can you monitor your processor utilization? Does it max out when D/L-ing?

    Attached I have a screenshot when pulling this service pack (~370MB): http://www.microsoft.com/downloads/details.aspx?FamilyID=95ac1610-c232-4644-b828-c55eec605d55&DisplayLang=en

    I have squid (2.6.18) running in transparent mode. But notice the increase of CPU utilization (is that normal?). Anyway, I still get close to my theoretical max. of 16MB down

    ![DL 1700KB-s.png](/public/imported_attachments/1/DL 1700KB-s.png)
    ![DL 1700KB-s.png_thumb](/public/imported_attachments/1/DL 1700KB-s.png_thumb)



  • I'm not seeing CPU spikes like that, but my bandwidth is all over the place.




  • Can you easily switch off squid and test without it?
    Just to make sure the hardware, cables and stuff work as expected.



  • Same file download, without using the proxy.




  • This is a full install and you're using single or MP kernel?



  • MP kernel AFAIK, it's a dual processor machine.  I have had the same results on a single processor box as well.  Is there a shell command I can run to tell me which kernel I am running?



  • For sure it's somewhere here:  https://<your-pfsense-ip>/status.php</your-pfsense-ip>



  • Just a sampling…
    FreeBSD 6.2-RELEASE-p11 #0: Mon Feb 18 01:25:40 EST 2008
        sullrich@builder6.pfsense.com:/usr/obj.pfSense/usr/src/sys/pfSense_SMP.6

    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs



  • Are you by any case heeting swap?!



  • Heeting swap?  I don't understand what that means.



  • I am guessing 'needing swap'?

    Have you checked your throttling settings? More specifically is 'Throttle only specific extensions' [Proxy / traffic management] Checked? From my understanding, if that is not checked all file transfers get throttled by default, which may be contributing to your speed loss?



  • Just tried it again both ways, checked and unchecked on 'Throttle only specific extensions'…no change.



  • Are you by any case heeting swap?!

    I mean, is swap getting used?!



  • The top output on my 1.2RC3 box that is working properly shows 100k of swap in use.  The 1.2RC5 with the new squid package just shows 1024M available, 1024M free.  Perhaps you're on to something.



  • Can you please post the generated squid.conf.



  • Here are two squid.conf, one from our working box and one from the new slower box.  I have tested with only two interfaces on different boxes and see the same result.  I have the newer box in non-transparent mode so the few people who use it won't have to put up with the slowness.

    squid.conf.2.6.5.pf.1.2.rc3.txt
    squid.conf.2.6.18.pf.1.2.rc5.txt



  • So basically the differences are:

    diff squid.conf.2.6.5.pf.1.2.rc3.txt squid.conf.2.6.18.pf.1.2.rc5.txt

    1c1
    http_access deny allto be sureterface(s)nsacl/blacklist.acl"s.acl"025-65535
    \ No newline at end of file
    –-
    http_access deny allto be sureterface(s)nsquid/acl/throttle_exts.acl"-65535
    \ No newline at end of file



  • The diff related to throttle went away after I changed the box for 'Throttle only specific extensions'.  The do not cache entries are diff because there are no donotcache entries on the 2.6.18 box.  Other than that, they look the same to me, aside from a couple of cache size cfg and IP changes.



  • He means to ask if you are running into swap space. I asume you are not since you are on a Dual core machine and likely have atleast 1GB of ram.

    Can you try using the single processor kernel and see if the problem persists?



  • Is there an easier way to change kernel besides reinstall?



  • Mount the install CD (or copy the kernel to a USB key), backup the old kernel, copy the new one off the CD, reboot.



  • I bet it can be found in CVSweb as well, right?
    Anyone point me to the right link (still have my problems with this versioning system…)



  • Hi!

    I am having the same squid slow transfer problems on my institution.
    I have a dual-wan system: one 25 MBps cable connection on WAN for internet access and one institutional adsl 1MBps on WAN OPT3 (also has internet access using another gateway).
    The OPT3 is connected to a 1GBps switch and has a public IP (we have 32 public ips for mail, webpage, etc… and a cisco router).
    If I download a huge file from our servers (connected on the WAN OPT3 interface) I can easily get 11000 MBytes/sec from our internal lan's, passing through squid.
    Nevertheless, accessing the internet (through the cable connection on WAN) I can get maximum 50Kbytes/sec using squid and 2000KBytes/sec using only NAT and proxy off.
    Why using the same squid traffic is slow in one interface and normal on another?

    Strangely if I discard the cable connection, deactivate WAN OPT3, and connect the 1MBps connection on WAN interface, I can get the full speed of it, downloading at about 200Kbytes/sec from the internet.
    Can it be the different modems? On the cable I have a regular Motorola modem, and on the institucional I have a much higher quality adsl modem.
    Can it be the NIC? When using cable we connect it to pfsense using a 100MBps 3Com card (xl driver) and the 1Mbps is on one VLAN that enters in pfsense throug a Intel PRO1000 (em driver) together with our internal networks...

    Can anyone shed some light on this problem? I am liking a lot pfsense, but I cannot have squid turned on because of such slow transfers!

    Yours,
    Antonio Paulino



  • OK - finally got a chance to try 1.2 release on matching HW to our 1.2rc3 fully functional box.  Clean install, no changes except to add squid and setup the WAN interface.  Before squid I was getting 2.7 mbit, after squid barely 300kbit.  This is a single core kernel, pIII 1.7 w/ 1 gb ram, 80gb hd and one 3com, one intel NIC.



  • I can confirm. Fresh install of 1.2. Nothing but squid running, latest package as of last night.

    With transparent enabled, on our dual-bonded T1 here, I get 600-1100Kbit.
    Disabled, I get ~ the full 3Mbit.

    Running Pentium D architecture, with SMP kernel.

    I monitored CPU usage with top. No difference. 99% idle either way.

    Not good.



  • any update on this?

    Cheers.



  • I'm holding my breath that it may not be an issue with FreeBSD 7/pfSense 1.3b.  For now it looks like it's wait and see as nobody is actively working on a solution.



  • NEED TEST
    Looks like the throughput issue may have resolved itself in the FreeBSD 6.3 build of pfSense 1.2_RELEASE.  I have had success with the new build, please let us know if you see improvements as well.
    http://cvs.pfsense.org/~sullrich/testing_images/6/FreeBSD_RELENG_6_3/pfSense_RELENG_1_2/



  • @mhab12:

    NEED TEST
    Looks like the throughput issue may have resolved itself in the FreeBSD 6.3 build of pfSense 1.2_RELEASE.  I have had success with the new build, please let us know if you see improvements as well.
    http://cvs.pfsense.org/~sullrich/testing_images/6/FreeBSD_RELENG_6_3/pfSense_RELENG_1_2/

    Hi there,

    I just installed the version linked above with only squid package installed and can't confirm any changes related to this problem.
    The machine is a Celeron 700 MHz, 512MB RAM and 20GB HDD.
    Without Squid I can reach the full 8 Mbit/s of my connection. With active Squid the troughput drops down to approx. 5,5Mbit/s.
    I'll continue testing this version.



  • Interesting…we only have 3MBit of bandwidth and don't see ANY difference between squid or direct.



  • Well i do not think that your machine parameters can handle that speed with squid without some more tweaking to either squid or pfSense itself.

    One interesting tweak would be to mount the filesystem with the noatime option by setting it on /etc/fstab.
    Another somewhat risky would be to add the async to those option for the filesystem but this the later option is somewhat risky.

    Ermal



  • Okay I've tested another configuration.
    Machine was a dual Xeon 3 GHz with 2 Gigs of dual channel RAM and a 500 Gig HDD.
    Unfortunately the problem persists: w/o squid -> full speed, w/ squid -> only nearly 70% is reachable.

    The CPU usage was constantly 1% which was predictable. :)
    btw. CPU usage @ the Celeron was about 15-20% so this couldn't be the problem.



  • Care to contribute to the bounty?



  • I would but I'm in apprenticeship so I do not earn enough money to contribute. :/



  • My download is 13000KB/s with squid it goes down to around 22xxKB/s.

    Spec is:  Dual P3 600MHz, 512MB, Asus P2B-DS, 40GB HDD 7200rpm 16MB Cache.

    I've yet to test the "update" above never really have time to take the firewall down.


Log in to reply