How-to: using Squid for bandwidth throttling
-
I was looking for a way to throttle downloads from high speed servers such as Rapidshare, Megaupload etc, so that regular web browsing would still be fast enough while people are downloading large files from these servers. In fact later on I realized it would be even better to throttle downloads of large files, usually with certain extensions such as .rar, .zip, .avi, .iso etc.
While Pfsense features traffic shaping, this is based on ports and/or IPs, not URLs or file extensions. We can however use Squid as a transparent proxy: it offers throttling based on file extensions, in fact this can be configured conveniently through the Pfsense gui!
As this may be a common use case, I thought I'd do a little write up to describe what I did:
1. Install Squid through "System > Packages"
2. Manually install Squid version 2.6 from the console
Sadly, the version of Squid (2.7.7) that was installed through the gui package manager in step 1 didn't seem to have been compiled with support for "delay pools", hence being completely useless for bandwidth throttling. See http://forum.pfsense.org/index.php?topic=19989.
The workaround was to manually install version 2.6 from the console:
( do not uninstall Squid from the gui, as you will then lose the gui configuration screens for Squid )
pkg_delete squid-2.7.7 pkg_add -rv http://files.pfsense.org/packages/7/All/squid-2.6.21.tbz3. Workaround to make squid route to qOthersDownH
Unfortunately Squid will route all traffic to the qLANDef queue of the traffic shaper instead of qOthersDownH.
But I successfully used a workaround which involved mainly editing /usr/local/pkg/squid.inc : http://forum.pfsense.org/index.php?topic=14436.0
( it also describes setting Squid in transparent proxy mode and changing the web gui port to something other than 80/443, which is obviously necessary as we will be using port 80 for the transparent proxy )
(note: a similar workaround is described here: http://forum.pfsense.org/index.php/topic,731.0.html )
4. Configure Squid for bandwidth throttling through the pfsense gui:
"Services > Proxy server > general"
Custom options: "cache deny all" ( I don't want Squid to cache anything, I just want it to throttle traffic )
"Services > Proxy server > Traffic Mgmt"
Overall bandwidth throttling: "500" (this sets the max. bandwidth for files with extensions as described below to 500 KB/s. The ADSL connection has a max download of about 800 KB/s, so I'm now reserving about 300 KB/s for other traffic like regular web browsing)
Per host throttling: "0"
Throttle only specific extensions: "Yes"
Throttle binary files: "Yes"
Throttle CD images: "Yes"
Throttle multimedia files: "Yes"
Throttle other extensions: "rar,0..,flv,dmg" (I may refine this list later on)5. Start Squid from "Status > Services"
I'm now running this for a day or so and it seems to work OK. Downloading multiple large .zip files simultaneously, and they are indeed nicely throttled to about 500 KB/s altogether while regular web browsing is still comfortable.
System specs:
P3 800 MHz, 256 MB ram
Pfsense 1.2.3-release
(I'm not doing anything fancy with this Pfsense system: about 30 users, just DHCP server, firewall, port forwarding, traffic shaping, only other packages installed: bandwidthd, rate) -
After some more testing I'm sorry to say there still are issues:
Simultaneously downloading multiple 100 MB rar files from Rapidshare results in a lot of incomplete file transfers. The transfers are nicely throttled, but often they don't complete :(
No idea why, it doesn't seem to be a performance issue as CPU and memory are not exactly under stress.