In need of help to solve a bandwidth issue
-
What can i use to see what is actually happening as far as squid possibly downloading or something? I am having this issue more often now.. its becoming a problem to my clients.. for an unknown reason for times throughout the day, for hours at a time my WAN usage is right up at my 20mb limit.. Take a look at attachment, i am in need of assistance for sure..
-
Wow! Well if it were my installation I would go back to the basics and remove everything and slowly re-build over time. Try to eliminate potential causes of the issue.
Also, get a capture of that traffic so you can find out exactly what that is. If I had to guess without much insight to what is going on I would not eliminate p2p traffic. -
Thanks for that.. starting again and rebuilding from scratch is hard as i really rely on the build up squid cache that is on there (50gb or so)
Wouldn't ptp traffic show up on my LAN interface (where all my clients are) as well as the LAN? WHen th WAN is like you see in that screenshot, the LAN (clients) is never above say 3-4mb usage..
-
True. I forgot that about your post. Let me run this by some of my colleagues at work and see if I can give you a better hint.
-
In a pfSense console session run pftop to get dynamic display of current firewall states (connections). Type h too get a display of the options. The R option should sort connections on rate and that should give you some clues about who is using the bandwidth.
-
True. I forgot that about your post. Let me run this by some of my colleagues at work and see if I can give you a better hint.
Thanks for this.. looking forward to what you come up with..
-
In a pfSense console session run pftop to get dynamic display of current firewall states (connections). Type h too get a display of the options. The R option should sort connections on rate and that should give you some clues about who is using the bandwidth.
I have done this, at a time when the WAN says 20mb usage, and below is what i see, i can't understand it well enough to see if it actually gives me an answer or not (i see alot of INBOUND traffic with port 127.0.0.1:3128 (squid Proxy Port).. is that my problem? and what uses that port?)
Getting confused with what In and Out refer to here..
-
Squid uses 3128 by default.
-
In and Out are interface related - so a download will generally be In on WAN and Out on LAN (and so on).
-
Right, just saw that in my squid config. So, what i am not understanding is this… if it is possibly ptp traffic, wouldnt that show up on the Traffic Graph also on the LAN interface, where all my clients are connected? This high WAN usage issue to me seems like i am leaking bandwidth somehow, somewhere.. cause the LAN interface is always quite low in usage, but just the WAN is max'd. Does that pftop output make any sense to you?
-
Have you looked at the reverse DNS (rDNS) and WhoIS of the highest volume remote nodes? The top 2 I picked both related to Google services.
What is 192.168.10.240, since it seems associated with some of the highest transfers, through Squid.
-
@Cry:
Have you looked at the reverse DNS (rDNS) and WhoIS of the highest volume remote nodes? The top 2 I picked both related to Google services.
What is 192.168.10.240, since it seems associated with some of the highest transfers, through Squid.
192.168.10.240 is one of my clients.. who according is limited to maximum 2mb/1mb.. as are all my clients
Here is a current snapshot of my pftop… currently my WAN is showing 20mb usage.. I have no idea what is going down... all my clients are limited via captive portal to 2mb, and are also limited on my AP (wireless cliente connecting via antenna).. when this shows 20mb usage, my LAN (192.168.10.0) is showing 4mb usage
-
Three of your top 4 lines refer to 65.54.93.42 (cds39.mia9.msecn.net.). I don't know what that is - does it make sense to you?
-
No.. not at all.. Currently it is chewing up 20mb.. and that IP is no longer there in the pftop output.
-
Also, i should add that this just became more confusing.. i thought it would be some clients doing alot of ptp or torrenting or something, but i just completely turned all the clients off for 10 minutes and the bandwidth never changes, was still max'd at 20mb with no one using the box..
-
Try to change admin passwords and disable outbound ssh connection for testing
-
Ok, just changed passwords and disabled SSH.. will report back in a little while
-
Ok, so its currently back up stuck at 20mb usage.. this just makes no sense.. the LAN traffic is only 2mb usage.. Can i have been hacked or something? Is it even possible that squid or something else is using up all the bandwidth?
-
Ok, so its currently back up stuck at 20mb usage.. this just makes no sense.. the LAN traffic is only 2mb usage.. Can i have been hacked or something? Is it even possible that squid or something else is using up all the bandwidth?
Did you try and disabling squid and see if that cuts the bandwidth down?
-
Thats the only thing i havent already done.. as i am a little worried if this will delete my cache or something? Cache is something that i really need running as i havent got a great deal of bandwidth.. If its ok to disable it and then re-enable it then i'll give it shot
-
It has been a long time since I used squid. I don't think I remember if it clears it or not.
-
Just quickly.. what do you mean by 'disable' squid? I dont see a disable check bo like some other packages have.. do you mean just stop int binding to the interface? Mine currently binds to my LAN and OPT1 interface, and not my WAN interface..
-
Sorry, just stopping it will disable it until reboot or until you start it again.
-
Any idea how to STOP it? i have just done it a few times via the dashboard 'Services" widget, but it keeps just turning itself back on..
-
Stop it and then remove the package. That will certainly keep it stopped ;)
-
Um.. removing it is definately what i want to do.. i need squid running.. i am just trying to stop it so i can see if it is the problem, then i'll have to work out how to fix it..
As far as i know, squid is the only way i can cache alot of my clients usage and save bandwidth, that i desperately need to.. cause the costs of Bandwidth here in Brazil is really high.. need all the help i can get to save on bandwidth
-
Um.. removing it is definately not what i want to do.. i need squid running.. i am just trying to stop it so i can see if it is the problem, then i'll have to work out how to fix it..
As far as i know, squid is the only way i can cache alot of my clients usage and save bandwidth, that i desperately need to.. cause the costs of Bandwidth here in Brazil is really high.. need all the help i can get to save on bandwidth
-
There should be a disable or enable check box in the config. check or uncheck it depending, save, and then stop the service. Once BW has seen to gone down, or stay the same, then reverse the steps.
-
Have looked over and over and there isnt a disable or anything like that in the Proxy Server settings..
I have stopped the service with the widget on the dashboard, and straight away the WAN usage goes back down to match the LAN.. so i now know that it is somehow Squid causing the issue.. no idea why though. But, after a few seconds the bandwidth goes back up as the service restarts itself..
-
Try the Allow users on interface checkbox on the Proxy Server.
If you have squidguard, try to disable it also.
-
No squidguard.
Wont the Allow users on interface stop the users from being able to access the web? And squid would still be running.. so probably still updating or doing whatever it is that is eating bandwidth
-
AFAIK, the Allow users on interface (if LAN is the selected interface) will only bypass the proxy if uncheck.
If this field is checked, the users connected to the interface selected in the 'Proxy interface' field will be allowed to use the proxy, i.e., there will be no need to add the interface's subnet to the list of allowed subnets. This is just a shortcut.
-
I have found the issue.. i had some code as an advanced setting that i read on this tutorial on the pfSense forums, that is supposed to assist in caching Windows Updates.. apparently all it does is eat the bandwidth all day long.. i deleted this code, restarted Squid and i have been running all day without a problem.. Would have been nice if i could still use this code without issue.. as caching Windows updates for me would save alot of bandwidth.. do you guys know if there us a way to still use the following code but only allow it to use say 2mb of bandwidth to get updates? or only allow it to run very early in the morning?
Code:
refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims; range_offset_limit -1;
-
I don't think it applies in your case, but that is what a WSUS (Windows Server Update Services) is for.
-
Yes i know, i am experienced as a windows admin, and using WSUS.. Has anyone else got their squid successfuly working as a type of WSUS and its not eating all the bandwidth?
-
Hi,
as said in the thread before. You are using squid. Did you compare the traffic which came in on WAN with the traffic that goes out on LAN ? If there is a big difference than you should optimize your squid cache so that squid stops downloading files when I client aborted a connection. I had a similar problem in the past but I could fix it. There were some hintes posted in the doc pointing to a posting of me.
To compare the traffic I just used the "Interface statistics" widget on the dashboard.
-
Hi yes i was monitoring all traffic on both interfaces and this is how i found the problem at first.. I have fixed it now, but i definately would like the settings back there so that i can cache windows updates, it would save me alot of bandwidth..
Do you recall what you changed the code to to get it to keep working but not eating all the bandwidth?
I read somewhere that if i change the range_offset_limit to 0 instead of -1 then that would help.. what i would like is if i can tell squid to never use more than 2mbps of available bandwidth to get these updates.. The rest of the code, i dont entirely understand anyways..
refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims; range_offset_limit -1;
-
Hi,
you can find tip about that in the pfsense docs.
Iam using squid for caching windows updates, too:
this is for caching all windows updates for 90 days
refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private; refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
This is for aborting the download if less than 60% of file has been downloaded. If more than 60% has been downloaded, squid will finish downloading/caching.
quick_abort_pct 60;
This is neccessary that the command above will work:
range_offset_limit 0;
If you use:
range_offset_limit -1;
then squid ignores "quick_abort_" and is downloading the complete file bey default.
This is working without any problems for updates from Windows XP and Windows 2000. I could save much bandwidth in this scenario with these settings.
But I didn't see such an effect for Windows 7 updates.
-
Thanks for that, great news.. Do you at all see any issues when your cache decides to update itself and use up all the bandwidth? That is what mine was doing..
-
Thanks for that, great news.. Do you at all see any issues when your cache decides to update itself and use up all the bandwidth? That is what mine was doing..
Yes, in the past I had the problem that my cache started downloading files but no client has an open download/connection. So it was just squid which was downloading. I had sometimes a high difference that WAN has downloaded 8GB and LAN only 6GB.
But as I said in my post before I changed the custom options in squid and now it is working as it should: caching files but not downloading "useless" files.