Inordinate Increase in Traffic, Can't figure out what it is.

[tomf]

Hi All,

I have a pfSense running in front of a small exchange server, which has been running fine for 9 months.
It is VMware VM, 4.1, running MS2008R2 server with Exchange 2010.

6 days ago the volume of traffic increased from less than .5mb to averaging 2.5mb, in and out. This happens for a few hours at a time, then will suddenly go back to normal, then back up again. At its highest it can hit 8mb. I am on metered traffic, so it is not fun when this goes over 5mb, which if is doing at times.

I have checked logs but I'm not so great at understanding them and I'm not sure exactly what I'm looking for.
I have also run basic netstat -ant and have minimal remote IP connections, mainly NTP and my own remote login. I am trying to learn more about netstat to understand the other data it gives.

I have had several others check my Exchange installation and they see nothing abnormal at all in that. I don't run anything else on this machine.
I have run 3 different recommended apps to check for root kit or other virus/malware. I do run AVG and Malwarebytes on this system.

Main Question:  What is the best way to determine what this traffic is?

Thanks for any help or pointers to the best place to go.

~ Tommy

[Cry Havok]

Best bet is to start recording netflow data. You can then review that (there are various GUI tools to help) to see if there are any particular ports or IP addresses involved in large volumes of traffic.

[cmb]

Several options.
http://doc.pfsense.org/index.php/How_can_I_monitor_bandwidth_usage%3F

[tomf]

Thanks for the replies. I'll be checking into all of this.

~ Tommy

[tomf]

I'm getting lots of data but still trying to understand it.

The main thing that stands out when the bandwidth goes up is DNS traffic, port 53, udp.

Here is a graph (attached I hope) that shows one pattern. These come and go 2 −3 times during a 24hr period.
I can't figure out what this would be or why, or even how to stop it.

~ Tommy


[jimp]

Try to get a capture of the traffic when it happens. You could be getting used as a reflector for a DNS-based DDoS, if you have something locally that is answering recursive queries from outside your network.

[tomf]

I'm not sure how to do an actual capture of traffic while it is happening. I did grab this which I thought looked abnormal.

I didn't get a screenshot at the time, but the third IP down, 198.144.121.89 (first attachment), was "last seen" at 1 sec, so it was active, and that seemed like a lot of traffic. The screenshot is from a few minutes ago,2 hours + after the fact.

No reverse on 198.144.121.89
Traceroute goes into its "* * *" with no results after:  8  so–0-0-0---0.br01.plal.ca.frontiernet.net (74.40.3.150)  33.295 ms  32.901 ms  33.193 ms

So I got more info on that one IP which is the second attachment.

I don't know what it all means and sadly haven't had a lot of time to dig into all this as much as I think I need to and want to.

What would I look for in the DNS doing recursive queries? I have half a dozen other installations identical to this one which aren't having issues (yet).


[SMuD]

http://ip-address-lookup-v4.com/ip/198.144.121.89

Interesting comment on that address…

If you have no reason to interact with that IP, simply drop all packets coming from it via rule (Action - block, Interface - WAN, Protocol - any, Source - Single host or alias & Address=198.144.121.89, Destination - any, Description - remind yourself why is this blocked).

Note too that it is a good idea to create an alias for rogue IP addresses and use that as the source in this rule.  That way if you come across another IP that needs blocking you only have to add it to the alias instead of creating another rule.

[tomf]

Very interesting. Maybe I'll add a comment.

I blocked that IP, we have no reason to be getting that much traffic from them on our email server. Tks for the how-to.

They actually haven't been back for over 24 hours, but are blocked now anyway.

Things were quiet all night, then increased again today, slowed down a while then back again. Today's winner is:  207.118.126.6, which is CenturyTel. They have reverse DNS at least.

File attached. Very different ports, TCP. I don't even have those ports open in pfSense! I don't even know what those ports are for.

This latest surge has been going on for 3-4 hours now and I finally restarted IIS for the Exchange server and all traffic drops, but then resumes. If I restart it 3-4 times the traffic drops down far less, but still is higher than it should be. The CenturyTel is still connecting, or at least showing up.


[Cry Havok]

Check the logs for your Exchange and IIS servers.

[tomf]

I contacted our upstream provider and discussed the issue and it appears that we have a large amount of UDP port 53 traffic going out to a single server "ripe.net".

This is a pfSense firewall in a VM in front of a Windows Exchange server also in a VM.

I'm wondering if I have the DNS between these two misconfigured in some way. They have run fine for past year until last 2 weeks, which makes me think something else is going on.

Networking is not my forte at all.

How would I identify what is causing it and stop it?

Meanwhile, doing my homework on this!

Thanks,

~ tommy

[Cry Havok]

RIPE.net is the European Internet Registry. In and of itself that means little.

Time to do a capture on port 53 (TCP and UDP) and examine the traffic. You need to work out what host on your network is generating it and what the requests are for (if indeed they are DNS requests). Once you know what the requests are for then you should be able to narrow down the source application (eg if all the lookups are for something starting with www. then you probably need to look at your IIS server).

I assume you've already checked that your Windows Server is fully patched and up to date and that all the AV is fully current too? My first suspicion at this point is that you've either got malware, a mis-configured SMTP server or a mis-configured IIS server (or script).

[tomf]

All Malware and AV has been kept up to date and finds nothing. I ran few root kit apps from Kaspersky and another, nothing showed.

This is what our provider said: "We are seeing high traffic volumes on our DNS server and it exactly mirrors the traffic from your interface. The info from the query is:  Standard query ANY ripe.net".

They also said that they see the traffic coming from 6 different IPs. I have checked all of these, no Malware or AV issues, and I have only seen traffic actually coming from two of them that is anything above normal.

The two that appear to have above normal traffic are an Exchange Server with Domain Controller, DNS, in it, and a Certificate Authority, which also has DNS. They are on different subnets.

If do the capture on the pfSense it will likely only report the pan IP that it is coming from and there is only one IP on that subnet, the Exchange Server. So I'm wondering if the capture has to happen on the Windows machine instead.

[Cry Havok]

That type of query is itself unusual if there's nothing else. Are all the hosts that are sending this traffic DNS servers? Are they accessible from the Internet? What else do those hosts have in common?

[tomf]

#1: Exchange Server with pfSense VM in front of it.  DNS, Full Exchange Server, IIS — Has most noticeable traffic
#2: MS Certificate Authority. DNS, IIS. — Occasionally has noticeable traffic
#3: Single Instance of MailEnable for testing, single mailbox. pfSense VM in front. NO DNS — No noticeable traffic.
#4: FileServer shared via VPN, pfSense in front. DNS and Terminal Services. — No noticeable traffic.
#5: FileServer shared via VPN, pfSense in front. NO DNS, still in 'demo' mode, not being  used.  — No noticeable traffic.
#6: FileServer shared via VPN, pfSense in front. NO DNS, still in 'demo' mode, not being  used.  — No noticeable traffic.

Only #1 and #2 have any noticeable traffic, the other are lightly used or not used at all.
All but one have pfSense in front.
All are VMs (VMware 4.1)
All are Server2008R2
All have up to date Malware/AV and tested.
Some need System Updates/Patches.

[Cry Havok]

I'd go do a full packet capture on #1 and then look to see if there's anything obvious at the times that the DNS queries occur. If the activity is triggered externally then it should show up in the moments before the query.

I suspect, strongly, that somehow you're being used for a DNS amplification attack. See here and here for starters, along with many other Google results. Are you sure that neither of your 2 DNS servers are accessible from the Internet?

[tomf]

For the past few hours all is quiet and totally normal. I'm capturing data on #1 now in windows using Network Monitor.
DNS on #1 is behind pfSense, so unless I've misconfigured it, it should be ok.

#2 is only behind a Server2008R2 firewall. This has occasional noticeable traffic when traffic is really high.

I have also completely shut down #3, #6, #5.

I'm reading up on the links, thanks.

~ tommy

[tomf]

Sorry, this thing has me running circles here.

To clarify, Server #1 is a mail server which is not just for internal email, there is access from anywhere via webmail, pop/imap, Exchange protocols,  but the DNS ports are NOT forwarded.
Internal DNS is on a local subnet x.x.x.45. DNS forwarding is enabled on pfSense.

[tomf]

Update: The traffic problem has not recurred for three days now, with exception of a small rise one time, which occurred in the middle of the night and I missed capturing it. It was not so high and not like previous ones, so may not have been related. Three of the servers have been completely off-line, so we may have a winner, or a loser, in that bunch, which I'll track down one by one this coming week.

Thanks again for the help.

~ Tommy

[tomf]

Was about to declare this thing closed, but today about 2 hours ago traffic went up on two servers.

This is the majority of the traffic on both:
  DNS:QueryId = 0x3B8, QUERY (Standard query), Query  for ripe.net of type ALL on class Internet

The frame details show that the Src varies when the Dest is my server.
  Ipv4: Src = 212.118.0.68, Dest = 208.74.xxx.xxx, Next Protocol = UDP, Packet ID = 21625, Total IP Length = 66

But in some cases the Src is my server and Dest is the name servers I use.
  Ipv4: Src = 208.74.xxx.xxx, Dest = 208.74.xxx.xxx, Next Protocol = UDP, Packet ID = 5459, Total IP Length = 82

#1 is an exchange server with pfSense in front, has DNS server.
#2 is a Microsoft Certificate Authority, which had a DNS server, but I disabled it, turned off the DNS service a month ago.

I have read the Amplified DNS attacks, but I don't grasp how it is happening, or how to stop it.