High traffic WAN, locate source on LAN
-
Sorry for the bump, this is still an issue for me…
Could someone please help me at least better diagnose this?
Maybe I'm not defining the fw blocking rule as I should? -
its traffic coming from 80.. Open the traffic up in wireshark - what is it?? Is it a website?
-
The one thing that really blows my mind is this:
here's my fw rule setup as block, first rule on WAN
here is the traffic graph on WAN, with the above rule active:
The rule has logging enabled, but System Logs > Firewall STILL SHOWS UP AS EMPTY…
AM I GOING MAD, or is there some problem..?
.. all of this started happening since switching to 2.1.3 RELEASE..
All of the originating IP's (which randomly change to some other subnet) of the traffic that hogs my WAN pipe resolve as part of Akamai CDN.
I have MANUALLY checked ALL of my hosts, virtual and physical, for running WU session, or strange traffic, found none.How can I block all that traffic? It seems to ignore rules on WAN..
I tried setting up packet capture on WAN for one of the source IP's, promiscuous mode, 10 packets, full details.
Here are the contents (source IP changed while writing this post):15:39:33.404897 AF IPv4 (2), length 1496: (tos 0x0, ttl 58, id 60877, offset 0, flags [DF], proto TCP (6), length 1492) 93.186.135.67.http > MY_WAN_IP.62175: Flags [.], cksum 0x53cd (correct), seq 2280624651:2280626091, ack 2637429275, win 8312, options [nop,nop,TS val 3403809280 ecr 313059907], length 1440 15:39:37.243545 AF IPv4 (2), length 1496: (tos 0x0, ttl 58, id 26417, offset 0, flags [DF], proto TCP (6), length 1492) 93.186.135.67.http > MY_WAN_IP.64988: Flags [.], cksum 0xd3ab (correct), seq 571079934:571081374, ack 271564911, win 7776, options [nop,nop,TS val 3403813120 ecr 312943553], length 1440 15:40:32.282698 AF IPv4 (2), length 1496: (tos 0x0, ttl 58, id 53873, offset 0, flags [DF], proto TCP (6), length 1492) 93.186.135.67.http > MY_WAN_IP.35999: Flags [.], cksum 0xcca9 (correct), seq 58513007:58514447, ack 399691569, win 7776, options [nop,nop,TS val 3403868160 ecr 312940868], length 1440 15:40:42.710002 AF IPv4 (2), length 64: (tos 0x0, ttl 64, id 6192, offset 0, flags [DF], proto TCP (6), length 60) MY_WAN_IP.64458 > 93.186.135.67.http: Flags [s], cksum 0x3f8d (correct), seq 4082480794, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 313193393 ecr 0], length 0 15:40:42.739815 AF IPv4 (2), length 64: (tos 0x20, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 93.186.135.67.http > MY_WAN_IP.64458: Flags [S.], cksum 0x32bb (correct), seq 3973581394, ack 4082480795, win 14480, options [mss 1452,sackOK,TS val 3403878645 ecr 313193393,nop,wscale 1], length 0 15:40:42.739874 AF IPv4 (2), length 56: (tos 0x0, ttl 64, id 4328, offset 0, flags [DF], proto TCP (6), length 52) MY_WAN_IP.64458 > 93.186.135.67.http: Flags [.], cksum 0x97e6 (correct), seq 1, ack 1, win 517, options [nop,nop,TS val 313193423 ecr 3403878645], length 0 15:40:42.740980 AF IPv4 (2), length 466: (tos 0x0, ttl 64, id 38914, offset 0, flags [DF], proto TCP (6), length 462) MY_WAN_IP.64458 > 93.186.135.67.http: Flags [P.], cksum 0x810e (correct), seq 1:411, ack 1, win 517, options [nop,nop,TS val 313193424 ecr 3403878645], length 410 15:40:42.775509 AF IPv4 (2), length 56: (tos 0x20, ttl 58, id 6977, offset 0, flags [DF], proto TCP (6), length 52) 93.186.135.67.http > MY_WAN_IP.64458: Flags [.], cksum 0x79cc (correct), seq 1, ack 411, win 7776, options [nop,nop,TS val 3403878681 ecr 313193424], length 0 15:40:42.781819 AF IPv4 (2), length 1496: (tos 0x0, ttl 58, id 6978, offset 0, flags [DF], proto TCP (6), length 1492) 93.186.135.67.http > MY_WAN_IP.64458: Flags [.], cksum 0x9b62 (correct), seq 1:1441, ack 411, win 7776, options [nop,nop,TS val 3403878685 ecr 313193424], length 1440 15:40:42.781856 AF IPv4 (2), length 56: (tos 0x0, ttl 64, id 3486, offset 0, flags [DF], proto TCP (6), length 52) MY_WAN_IP.64458 > 93.186.135.67.http: Flags [.], cksum 0x9065 (correct), seq 411, ack 1441, win 506, options [nop,nop,TS val 313193465 ecr 3403878685], length 0 [font][/font] [/s] -
You can block traffic all day long - doesn't stop traffic from coming down your pipe if requested or sent.
blocking at wan doesn't do much for using up bandwidth. Even if pfsense drops it or sends it on, its still using your pipe. You need to stop the traffic from being sent, ie requested I would assume if your saying its not some form of attack.
Capture the traffic and then load it up in wireshark so you can take a look to what it actually is. It looks to be just http so not encrypted (ie https) so it should all be in the clear and you can see what is being requested to send to your IP.
also look in your state tables for these IPs - you should see the IP on your side that created the state.
Do you have this agent installed on any of your machines
http://www.akamai.com/html/solutions/client_faq.html -
also look in your state tables for these IPs - you should see the IP on your side that created the state.
I basically love you… Being a pfSense newbie, I never thought of filtering the states...
I got to the bottom of this and as usual, Windows Update is the culprit...
Geniuses at Microsoft decided that WU traffic does not show up in Task Manager > Performance tab > Network, so one can have a host hogging all bandwidth via WU, but still show up as 0Kbps on the client...
I knew the Akamai NetSession client works in p2p mode so I never allowed it to be installed on anything.
... so to anyone not knowing where traffic comes from, LOOK AT YOUR STATES...
Sadly I still do no understand why the fw rule (whatever action) doesn't log traffic originating from Akamai's CDN.. Could they be using some weird protocol? I dunno...
Also, Traffic Graph on the LAN interface shows nothing hinting at the client that's downloading WU stuff; perhaps you're right about the acks being negligible so not really visible on the graph.
Thank you johnpoz.
-
.. someone may mark this as SOLVED as far as I'm concerned…
-
Well that wan rule doesn't seem like it would be firing because traffic is return traffic to a state.. if it was syn traffic from that source to your wan IP than that rule would fire and be logged per your setting.
If your clients are requesting something. Its better log the allow or block rule on the lan interface to see what client is generating traffic to where, etc. Can not really think of too many examples when you would need specific deny rules on your wan because of the default deny. You would normally only allow stuff like icmp, or rules to allow your port forwards to work. Now I allow ping but use the pfblocker as a source filter, so you can ping my wan unless your listed in the spammers, bad countries list, etc.
So if your try and ping my wan IP, and your listed in the pfblocker top spammers alias list then you would not trigger that rule that allows and fall through to the default deny.
-
Well that wan rule doesn't seem like it would be firing because traffic is return traffic to a state.. if it was syn traffic from that source to your wan IP than that rule would fire and be logged per your setting.
If your clients are requesting something. Its better log the allow or block rule on the lan interface to see what client is generating traffic to where, etc. Can not really think of too many examples when you would need specific deny rules on your wan because of the default deny. You would normally only allow stuff like icmp, or rules to allow your port forwards to work. Now I allow ping but use the pfblocker as a source filter, so you can ping my wan unless your listed in the spammers, bad countries list, etc.
So if your try and ping my wan IP, and your listed in the pfblocker top spammers alias list then you would not trigger that rule that allows and fall through to the default deny.
I always thought fw rules would also apply to incoming traffic even if tcp session is initiated internally.. You live and learn!
Golden advice, will cherish..
Thank you again.
-
Someone knows why the statistics are not corresponding? This is driving me crazy!!
In my case, I also heavy all the traffic comming from Akamai and I'm sure that this is not an attack…
Look to my interface statistics:
All the traffic is originated from Akamai.
-
what is not matching up? Looks like your not showing some of your interfaces..
Are you talking about the 145GB into wan, but only 14GB out lan? Just because you see traffic to wan, doesn't mean pfsense is going to send that traffic out the lan, etc.. While 145 to 14 seems high.. Do you have that traffic going out a different interface other than lan?
-
My second WAN interface is used only for failover.
However, now I discovered that this issue is related, in some way, to SQUID. After I disabled SQUID, the traffic graphs are immediately working appropriately.
Sorry for my bad english :-[
-
Well sure squid could grab all kinds of stuff, and not send it to something on the lan..
-
To locate source on LAN, you need to look at Squid logs…
