Inordinate Increase in Traffic, Can't figure out what it is.

Cry Havok

That type of query is itself unusual if there's nothing else. Are all the hosts that are sending this traffic DNS servers? Are they accessible from the Internet? What else do those hosts have in common?

tomf

#1: Exchange Server with pfSense VM in front of it. DNS, Full Exchange Server, IIS — Has most noticeable traffic
#2: MS Certificate Authority. DNS, IIS. — Occasionally has noticeable traffic
#3: Single Instance of MailEnable for testing, single mailbox. pfSense VM in front. NO DNS — No noticeable traffic.
#4: FileServer shared via VPN, pfSense in front. DNS and Terminal Services. — No noticeable traffic.
#5: FileServer shared via VPN, pfSense in front. NO DNS, still in 'demo' mode, not being used. — No noticeable traffic.
#6: FileServer shared via VPN, pfSense in front. NO DNS, still in 'demo' mode, not being used. — No noticeable traffic.

Only #1 and #2 have any noticeable traffic, the other are lightly used or not used at all.
All but one have pfSense in front.
All are VMs (VMware 4.1)
All are Server2008R2
All have up to date Malware/AV and tested.
Some need System Updates/Patches.

Cry Havok

I'd go do a full packet capture on #1 and then look to see if there's anything obvious at the times that the DNS queries occur. If the activity is triggered externally then it should show up in the moments before the query.

I suspect, strongly, that somehow you're being used for a DNS amplification attack. See here and here for starters, along with many other Google results. Are you sure that neither of your 2 DNS servers are accessible from the Internet?

tomf

For the past few hours all is quiet and totally normal. I'm capturing data on #1 now in windows using Network Monitor.
DNS on #1 is behind pfSense, so unless I've misconfigured it, it should be ok.

#2 is only behind a Server2008R2 firewall. This has occasional noticeable traffic when traffic is really high.

I have also completely shut down #3, #6, #5.

I'm reading up on the links, thanks.

~ tommy

tomf

Sorry, this thing has me running circles here.

To clarify, Server #1 is a mail server which is not just for internal email, there is access from anywhere via webmail, pop/imap, Exchange protocols, but the DNS ports are NOT forwarded.
Internal DNS is on a local subnet x.x.x.45. DNS forwarding is enabled on pfSense.

tomf

Update: The traffic problem has not recurred for three days now, with exception of a small rise one time, which occurred in the middle of the night and I missed capturing it. It was not so high and not like previous ones, so may not have been related. Three of the servers have been completely off-line, so we may have a winner, or a loser, in that bunch, which I'll track down one by one this coming week.

Thanks again for the help.

~ Tommy

tomf

Was about to declare this thing closed, but today about 2 hours ago traffic went up on two servers.

This is the majority of the traffic on both:
DNS:QueryId = 0x3B8, QUERY (Standard query), Query for ripe.net of type ALL on class Internet

The frame details show that the Src varies when the Dest is my server.
Ipv4: Src = 212.118.0.68, Dest = 208.74.xxx.xxx, Next Protocol = UDP, Packet ID = 21625, Total IP Length = 66

But in some cases the Src is my server and Dest is the name servers I use.
Ipv4: Src = 208.74.xxx.xxx, Dest = 208.74.xxx.xxx, Next Protocol = UDP, Packet ID = 5459, Total IP Length = 82

#1 is an exchange server with pfSense in front, has DNS server.
#2 is a Microsoft Certificate Authority, which had a DNS server, but I disabled it, turned off the DNS service a month ago.

I have read the Amplified DNS attacks, but I don't grasp how it is happening, or how to stop it.

Cry Havok

You probably need to find a local IT consultant, preferably one with both Microsoft and security backgrounds, to have a look at your systems. You've got something running on both those systems that's performing this activity, but whether it is malware (and the fact that your AV hasn't picked it up doesn't mean it isn't there) or something else isn't clear.

tomf

Thanks, I am in the process. Seems that my pfSense config allows DNS access from external. I'll have to look up how to change that.

~ tommy

Cry Havok

Check your WAN rules and ensure that you don't allow port 53 (TCP and UDP) to any IP address but only to any device you're providing DNS services to external IP addresses from.

tomf

Sorry for the late followup on this. Finally resolved the issue. The DNS was open to public, closed that and after a week it all went back to normal.

Thanks everyone for the input and help. Learning as I go.

~ Tom