In need of help to solve a bandwidth issue

luke240778

Have looked over and over and there isnt a disable or anything like that in the Proxy Server settings..

I have stopped the service with the widget on the dashboard, and straight away the WAN usage goes back down to match the LAN.. so i now know that it is somehow Squid causing the issue.. no idea why though. But, after a few seconds the bandwidth goes back up as the service restarts itself..

jikjik101

Try the Allow users on interface checkbox on the Proxy Server.

If you have squidguard, try to disable it also.

luke240778

No squidguard.

Wont the Allow users on interface stop the users from being able to access the web? And squid would still be running.. so probably still updating or doing whatever it is that is eating bandwidth

jikjik101

AFAIK, the Allow users on interface (if LAN is the selected interface) will only bypass the proxy if uncheck.

If this field is checked, the users connected to the interface selected in the 'Proxy interface' field will be allowed to use the proxy, i.e., there will be no need to add the interface's subnet to the list of allowed subnets. This is just a shortcut.

luke240778

I have found the issue.. i had some code as an advanced setting that i read on this tutorial on the pfSense forums, that is supposed to assist in caching Windows Updates.. apparently all it does is eat the bandwidth all day long.. i deleted this code, restarted Squid and i have been running all day without a problem.. Would have been nice if i could still use this code without issue.. as caching Windows updates for me would save alot of bandwidth.. do you guys know if there us a way to still use the following code but only allow it to use say 2mb of bandwidth to get updates? or only allow it to run very early in the morning?

Code:

podilarius

I don't think it applies in your case, but that is what a WSUS (Windows Server Update Services) is for.

luke240778

Yes i know, i am experienced as a windows admin, and using WSUS.. Has anyone else got their squid successfuly working as a type of WSUS and its not eating all the bandwidth?

Nachtfalke

Hi,

as said in the thread before. You are using squid. Did you compare the traffic which came in on WAN with the traffic that goes out on LAN ? If there is a big difference than you should optimize your squid cache so that squid stops downloading files when I client aborted a connection. I had a similar problem in the past but I could fix it. There were some hintes posted in the doc pointing to a posting of me.

To compare the traffic I just used the "Interface statistics" widget on the dashboard.

luke240778

Hi yes i was monitoring all traffic on both interfaces and this is how i found the problem at first.. I have fixed it now, but i definately would like the settings back there so that i can cache windows updates, it would save me alot of bandwidth..

Do you recall what you changed the code to to get it to keep working but not eating all the bandwidth?

I read somewhere that if i change the range_offset_limit to 0 instead of -1 then that would help.. what i would like is if i can tell squid to never use more than 2mbps of available bandwidth to get these updates.. The rest of the code, i dont entirely understand anyways..

Nachtfalke

Hi,

you can find tip about that in the pfsense docs.

Iam using squid for caching windows updates, too:

this is for caching all windows updates for 90 days

refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

This is for aborting the download if less than 60% of file has been downloaded. If more than 60% has been downloaded, squid will finish downloading/caching.

quick_abort_pct 60;

This is neccessary that the command above will work:

range_offset_limit 0;

If you use:

range_offset_limit -1;

then squid ignores "quick_abort_" and is downloading the complete file bey default.

This is working without any problems for updates from Windows XP and Windows 2000. I could save much bandwidth in this scenario with these settings.

But I didn't see such an effect for Windows 7 updates.

luke240778

Thanks for that, great news.. Do you at all see any issues when your cache decides to update itself and use up all the bandwidth? That is what mine was doing..

Nachtfalke

@luke240778:

Thanks for that, great news.. Do you at all see any issues when your cache decides to update itself and use up all the bandwidth? That is what mine was doing..

Yes, in the past I had the problem that my cache started downloading files but no client has an open download/connection. So it was just squid which was downloading. I had sometimes a high difference that WAN has downloaded 8GB and LAN only 6GB.

But as I said in my post before I changed the custom options in squid and now it is working as it should: caching files but not downloading "useless" files.

luke240778

Well, i have inserted that code and will see how things go for a couple days.. thanks again!! it will indeed save alot of bandwidth.

Nachtfalke

You could add this for apple updates:

refresh_pattern -i .*apple\.com/.*\.(pkg|dmg) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

and this for ubuntu (an perhaps other linux distris) updates:

refresh_pattern -i .*ubuntu\.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

and perhaps this will save you bandwidth when you want to save some iso files and multimedia:

# refresh_pattern -i /.*\.(iso|wmv|mov|rm|avi|mp4|mpeg|mpg|divx|xvid|swf|flv|x-flv) 2880 100% 2880 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

But please be carful with the caching time.
For caching OS updates I am using 90 days.
For caching ISO files and so on just 2 days.

Perhaps you could play with these settings/times.

With these settings you could avoid that squid is caching wrong DNS entries longer than 5 seconds:

negative_ttl 5 second;
negative_dns_ttl 5 second;

Good luck an dpost back how it is working :o)

Nachtfalke

I did some (successful) changes for caching Windows 7 updates but in a way that squid isn't downloading files without any use.

range_offset_limit 20 MB;

This means:
I f you have a download with 100MB and the download is starting at 35MB till 100MB (what windows 7 seems to do) than this file will NOT be cache by squid with my option above because the range which is not downloades (less than 35MB) is bigger than the size in the range_offset_limit.

If the download of the 100MB file starts at 15MB till 100MB, that the range_offset_limit takes effect and squid is downloading the rest of the file before 15MB.

I hope I could explain it with my small english skills.

That this all will work it is neccessary that your "maximum object size" is high enough to cache the files you are downloading

luke240778

I havent seen my bandwidth issue return since adding this code which is a good thing.. but i do see since that my Disk usage has gone from around 8-10% up to around 70%.. With squid running for over 5 months with all my clients it was only using around 10%.. Nos wince adding this code its way up… What can i do to troubleshoot this?

Nachtfalke

Hi,

of course you disk cache is increasing - that's a good sign because this means that squid is caching many files. And you want to cache many files because you would like to save i-net bandwidth.
In "Cache Management" you find "Low- and High watermark" if more than 80% of the squid cache is used than squid starts to throw out "old" files. but this depends on your replacement algorithms.

For memory replacement I am using: Heap GDSF
For cache replacement I am using: Heap LFUDA

luke240778

Ill take a look at this. Out of interest, is there someway that i can see what files are in my cache? For example, not just websites, but would love to see what large files are in there, like music and movies and updates and stuff.. out of interest to see what my clients interests are to maybe try some other bandwidth saving techniques

Nachtfalke

Hi,

I am not sure where to find these information. But you can use cachemgr.cgi to analyze squid more in detail.

And I found this:
http://wiki.squid-cache.org/SquidFaq/SquidLogs#store.log

But you have to enable this in

/usr/local/pkg/squid.inc

store.log is disabled by default. To enable this I think you just have to modify the line 624:
from

cache_store_log none

to

/var/log/store.log

I am NOT sure if the syntax is correct but it should work. After changing the squid.inc you have to click "Save" in "Services -> Proxy Server -> General" so that the squid.conf will be generated with the new options. But then please check syslog if there are any errors.

luke240778

Thanks for that, i have cachmgr.cgi setup, but where in there shows the files in cache and what they are?

Ill play with that other part now also