In need of help to solve a bandwidth issue
-
pfTop via SSH
Type the following commands to get ip's with highest throughput utilization… do not type the commas
7, R (Capital R), s, 1
Another option is install the pflowd package and downloading ManageEngines netflow monitor on a PC or Server on the lan. Point the pfflowd to the PC with manageengine. I used to use it alot!!!!
You get 2 free devices to monitor. Free phone support for setup too!
-
Thanks for that .. i am using both these options now.. great
Just quickly, that RATE in pftop.. is that in bytes or kilobyes? So for example 348015 is what?
-
I think it is in BYTES per sec.
-
If your Squid is set to cache, it is possible that clients have initiated and cancelled downloads which results in Squid still persistently downloading the content for caching purposes though the clients no longer require it.
-
Ah.. interesting… I do indeed have squid setup as a transparent proxy.. I will keep an eye on this.
Whilst on the subject of Squid.. Having the swap.state log rotate daily, doesnt affect the actual cache or clients getting content from cache right? I need all the help i can to conserve bandwidth
-
pfTop via SSH
Type the following commands to get ip's with highest throughput utilization… do not type the commas
7, R (Capital R), s, 1
Another option is install the pflowd package and downloading ManageEngines netflow monitor on a PC or Server on the lan. Point the pfflowd to the PC with manageengine. I used to use it alot!!!!
You get 2 free devices to monitor. Free phone support for setup too!
Where do you get pflowd for pfsense from?
Is nether in the pkg list or the freebsd repo.
TIA!
-
found it.
Thanks!
-
What can i use to see what is actually happening as far as squid possibly downloading or something? I am having this issue more often now.. its becoming a problem to my clients.. for an unknown reason for times throughout the day, for hours at a time my WAN usage is right up at my 20mb limit.. Take a look at attachment, i am in need of assistance for sure..
-
Wow! Well if it were my installation I would go back to the basics and remove everything and slowly re-build over time. Try to eliminate potential causes of the issue.
Also, get a capture of that traffic so you can find out exactly what that is. If I had to guess without much insight to what is going on I would not eliminate p2p traffic. -
Thanks for that.. starting again and rebuilding from scratch is hard as i really rely on the build up squid cache that is on there (50gb or so)
Wouldn't ptp traffic show up on my LAN interface (where all my clients are) as well as the LAN? WHen th WAN is like you see in that screenshot, the LAN (clients) is never above say 3-4mb usage..
-
True. I forgot that about your post. Let me run this by some of my colleagues at work and see if I can give you a better hint.
-
In a pfSense console session run pftop to get dynamic display of current firewall states (connections). Type h too get a display of the options. The R option should sort connections on rate and that should give you some clues about who is using the bandwidth.
-
True. I forgot that about your post. Let me run this by some of my colleagues at work and see if I can give you a better hint.
Thanks for this.. looking forward to what you come up with..
-
In a pfSense console session run pftop to get dynamic display of current firewall states (connections). Type h too get a display of the options. The R option should sort connections on rate and that should give you some clues about who is using the bandwidth.
I have done this, at a time when the WAN says 20mb usage, and below is what i see, i can't understand it well enough to see if it actually gives me an answer or not (i see alot of INBOUND traffic with port 127.0.0.1:3128 (squid Proxy Port).. is that my problem? and what uses that port?)
Getting confused with what In and Out refer to here..
-
Squid uses 3128 by default.
-
In and Out are interface related - so a download will generally be In on WAN and Out on LAN (and so on).
-
Right, just saw that in my squid config. So, what i am not understanding is this… if it is possibly ptp traffic, wouldnt that show up on the Traffic Graph also on the LAN interface, where all my clients are connected? This high WAN usage issue to me seems like i am leaking bandwidth somehow, somewhere.. cause the LAN interface is always quite low in usage, but just the WAN is max'd. Does that pftop output make any sense to you?
-
Have you looked at the reverse DNS (rDNS) and WhoIS of the highest volume remote nodes? The top 2 I picked both related to Google services.
What is 192.168.10.240, since it seems associated with some of the highest transfers, through Squid.
-
@Cry:
Have you looked at the reverse DNS (rDNS) and WhoIS of the highest volume remote nodes? The top 2 I picked both related to Google services.
What is 192.168.10.240, since it seems associated with some of the highest transfers, through Squid.
192.168.10.240 is one of my clients.. who according is limited to maximum 2mb/1mb.. as are all my clients
Here is a current snapshot of my pftop… currently my WAN is showing 20mb usage.. I have no idea what is going down... all my clients are limited via captive portal to 2mb, and are also limited on my AP (wireless cliente connecting via antenna).. when this shows 20mb usage, my LAN (192.168.10.0) is showing 4mb usage
-
Three of your top 4 lines refer to 65.54.93.42 (cds39.mia9.msecn.net.). I don't know what that is - does it make sense to you?
-
No.. not at all.. Currently it is chewing up 20mb.. and that IP is no longer there in the pftop output.
-
Also, i should add that this just became more confusing.. i thought it would be some clients doing alot of ptp or torrenting or something, but i just completely turned all the clients off for 10 minutes and the bandwidth never changes, was still max'd at 20mb with no one using the box..
-
Try to change admin passwords and disable outbound ssh connection for testing
-
Ok, just changed passwords and disabled SSH.. will report back in a little while
-
Ok, so its currently back up stuck at 20mb usage.. this just makes no sense.. the LAN traffic is only 2mb usage.. Can i have been hacked or something? Is it even possible that squid or something else is using up all the bandwidth?
-
Ok, so its currently back up stuck at 20mb usage.. this just makes no sense.. the LAN traffic is only 2mb usage.. Can i have been hacked or something? Is it even possible that squid or something else is using up all the bandwidth?
Did you try and disabling squid and see if that cuts the bandwidth down?
-
Thats the only thing i havent already done.. as i am a little worried if this will delete my cache or something? Cache is something that i really need running as i havent got a great deal of bandwidth.. If its ok to disable it and then re-enable it then i'll give it shot
-
It has been a long time since I used squid. I don't think I remember if it clears it or not.
-
Just quickly.. what do you mean by 'disable' squid? I dont see a disable check bo like some other packages have.. do you mean just stop int binding to the interface? Mine currently binds to my LAN and OPT1 interface, and not my WAN interface..
-
Sorry, just stopping it will disable it until reboot or until you start it again.
-
Any idea how to STOP it? i have just done it a few times via the dashboard 'Services" widget, but it keeps just turning itself back on..
-
Stop it and then remove the package. That will certainly keep it stopped ;)
-
Um.. removing it is definately what i want to do.. i need squid running.. i am just trying to stop it so i can see if it is the problem, then i'll have to work out how to fix it..
As far as i know, squid is the only way i can cache alot of my clients usage and save bandwidth, that i desperately need to.. cause the costs of Bandwidth here in Brazil is really high.. need all the help i can get to save on bandwidth
-
Um.. removing it is definately not what i want to do.. i need squid running.. i am just trying to stop it so i can see if it is the problem, then i'll have to work out how to fix it..
As far as i know, squid is the only way i can cache alot of my clients usage and save bandwidth, that i desperately need to.. cause the costs of Bandwidth here in Brazil is really high.. need all the help i can get to save on bandwidth
-
There should be a disable or enable check box in the config. check or uncheck it depending, save, and then stop the service. Once BW has seen to gone down, or stay the same, then reverse the steps.
-
Have looked over and over and there isnt a disable or anything like that in the Proxy Server settings..
I have stopped the service with the widget on the dashboard, and straight away the WAN usage goes back down to match the LAN.. so i now know that it is somehow Squid causing the issue.. no idea why though. But, after a few seconds the bandwidth goes back up as the service restarts itself..
-
Try the Allow users on interface checkbox on the Proxy Server.
If you have squidguard, try to disable it also.
-
No squidguard.
Wont the Allow users on interface stop the users from being able to access the web? And squid would still be running.. so probably still updating or doing whatever it is that is eating bandwidth
-
AFAIK, the Allow users on interface (if LAN is the selected interface) will only bypass the proxy if uncheck.
If this field is checked, the users connected to the interface selected in the 'Proxy interface' field will be allowed to use the proxy, i.e., there will be no need to add the interface's subnet to the list of allowed subnets. This is just a shortcut.
-
I have found the issue.. i had some code as an advanced setting that i read on this tutorial on the pfSense forums, that is supposed to assist in caching Windows Updates.. apparently all it does is eat the bandwidth all day long.. i deleted this code, restarted Squid and i have been running all day without a problem.. Would have been nice if i could still use this code without issue.. as caching Windows updates for me would save alot of bandwidth.. do you guys know if there us a way to still use the following code but only allow it to use say 2mb of bandwidth to get updates? or only allow it to run very early in the morning?
Code:
refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims; range_offset_limit -1;