Need help with a Security issue

[yodasoda]

We have Recursive-DNS activity in our syslog that might indicate spyware somewhere on the network.
The DNS lookups are not normal activity, but i can't figure out which machine is requesting them.

Is there a way to indicate in the logs which machine has made a particular RDNS request?
As it stands, the log has hundreds of UDP:53 lookups streaming out of it, and no indication if a reply to any request was even received.
I can't really even be sure they are actually DNS lookups and not just spyware using that port directly.

Can anyone help me? I'm not sure where to ask.

[KOM]

What is handling your DNS?  pfSense?  Windows AD?  You're a little light on details.

[johnpoz]

"I can't really even be sure they are actually DNS lookups and not just spyware using that port directly."

Without some details how can we even begin to help?  How about a sample of this log your looking at that has you curious..  Are these logs on pfsense?  You can for sure up the logging to get exact details, or just sniff and look to see what is being asked by who, etc.

[yodasoda]

@KOM:

What is handling your DNS?  pfSense?  Windows AD?  You're a little light on details.

A Pfsense router is connected to a cable modem.
pfsense performs the dns lookups, I assume (they're coming out of interface 'igb0'.
I imagine unbound is performing this?
https://doc.pfsense.org/index.php/Unbound_DNS_Resolver

@johnpoz:

Without some details how can we even begin to help?  How about a sample of this log your looking at that has you curious..  Are these logs on pfsense?  You can for sure up the logging to get exact details, or just sniff and look to see what is being asked by who, etc.

I can't post the log for reasons i don't want to get into.
The logs are from pfsense's syslog logger. They are remotely logged on another machine.
I'm unable to do packet sniffing as the activity in the log is highly infrequent and i have no way to remotely save the data (packet sniffing generates huge files fast). The router has 4 GB of ram and nothing more.
Unless snort supports remote logging I'm unable to do save the data.

Is there anyway to determine which machine is doing a specific DNS lookup?

[johnpoz]

Dude do you want help or not.. Reasons you don't want to get into??

As to sniffing.. Sniffing for dns packets is not going to get going to be big..  You don't have to sniff for everything.  Just limit to your dns queries.

If your using the resolver, pfsense is not asking your router for jack.. The resolver walks down the tree from roots to the authoritative server for the domain your looking for a record in.  It does not forward anything.

So log the queries..

Or install dnstop on pfsense and find the client that way. http://dns.measurement-factory.com/tools/dnstop/dnstop.8.html

[2.3.2-RELEASE][root@pfsense.local.lan]/root: pkg info dnstop
dnstop-20140915
Name           : dnstop
Version        : 20140915
Installed on   : Fri Apr 22 22:53:08 2016 CDT
Origin         : dns/dnstop
Architecture   : freebsd:10:x86:64
Prefix         : /usr/local
Categories     : dns ipv6
Licenses       : BSD2CLAUSE
Maintainer     : mark@foster.cc
WWW            : http://dnstop.measurement-factory.com/
Comment        : Captures and analyzes DNS traffic (or analyzes libpcap dump)
Annotations    :
Flat size      : 55.9KiB
Description    :
dnstop is a libpcap application (ala tcpdump) that displays various
tables of DNS traffic on your network. Currently dnstop displays
tables of:

    * Source IP addresses
    * Destination IP addresses
    * Query types
    * Top level domains
    * Second level domains

WWW: http://dnstop.measurement-factory.com/
[2.3.2-RELEASE][root@pfsense.local.lan]/root:

[KOM]

Dude do you want help or not.. Reasons you don't want to get into??

His DNS lookups are a secret, John  ;D  He doesn't want everyone to know he spends his days here…

[johnpoz]

heheh hey being a brony is not that shameful ;)  While I have an excuse of having a 6 year old grand daughter for why I know most of the main characters names.. There are many people that embrace the brony tag..

edit:
On a side note I just noticed my box did a query for wpad.local.lan - JFC ms how do you turn that nonsense off??  I have tried everything I have found to try and disable that and still the queries come..  I even hand out loopback via dhcp and dns, but still the noise is there…  Anyone know of a sure fire way to make windows stop asking for freaking wpad??

[W4RH34D]

9/10 i see something strange on multicast DNS it's a printer or printer software.

[KOM]

Anyone know of a sure fire way to make windows stop asking for freaking wpad??

Don't hijack the thread, you thread-hijacker!!!

[johnpoz]

heheeh - nothing like a good hijack ;)

[yodasoda]

Thanks johnpoz , that's basically exactly what i needed.
i'm going to try that now and see if it works on my setup.

[yodasoda]

Ok, this is the problem I'm having: (sample log)
I'm seeing a huge number of UDP port 53 packets leaving the WAN address (XX.XX.XX.XX).
It seems they are not going though unbound, or logging the queries isn't working correctly.

I only have the Squid server package installed, and i can't figure out where hundreds of these packets are coming from. I thought they were Unbound activity, but now i'm confused.

Can anyone tell me what these are, what's likely generating them, and how i can figure out where they are coming from?

filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,48170,0,none,17,udp,73,XX.XX.XX.XX,199.212.0.53(tinnie.arin.net),15171,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,64143,0,none,17,udp,73,XX.XX.XX.XX,202.12.29.25(ns1.apnic.net),31831,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,19498,0,none,17,udp,84,XX.XX.XX.XX,168.95.192.3(vns1.hinet.net),25909,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,1265,0,none,17,udp,84,XX.XX.XX.XX,168.95.192.3(vns1.hinet.net),25275,53
unbound,[41964:1] info: 192.168.2.13(PC.localdomain) 42.100.161.218.in-addr.arpa. PTR IN,,,,,,,,,,,,,,,,,,,,,
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,4973,0,none,17,udp,84,XX.XX.XX.XX,168.95.1.15(ans2.hinet.net),41443,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,37586,0,none,17,udp,84,XX.XX.XX.XX,194.146.106.106(apnic1.dnsnode.net),6366,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,16405,0,none,17,udp,84,XX.XX.XX.XX,200.10.60.53(d.in-addr-servers.arpa),39344,53
unbound,[41964:0] info: 192.168.2.13(PC.localdomain) 42.100.161.218.in-addr.arpa. PTR IN,,,,,,,,,,,,,,,,,,,,,
filterlog,59,16777216,,1000001581,igb0,match,block,in,4,0x0,,255,39482,0,none,17,udp,379,10.102.0.1,255.255.255.255,67,68
filterlog,59,16777216,,1000001581,igb0,match,block,in,4,0x0,,255,39479,0,none,17,udp,379,10.102.0.1,255.255.255.255,67,68
filterlog,59,16777216,,1000001581,igb0,match,block,in,4,0x0,,255,39478,0,none,17,udp,379,10.102.0.1,255.255.255.255,67,68
filterlog,59,16777216,,1000001581,igb0,match,block,in,4,0xc0,,1,61583,0,none,2,igmp,36,10.102.0.1,224.0.0.1(all-systems.mcast.net),datalength=12 ,
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,46546,0,none,17,udp,73,XX.XX.XX.XX,193.0.9.11(lacnic.authdns.ripe.net),55023,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,44128,0,none,17,udp,83,XX.XX.XX.XX,200.27.2.7(ns2.telmexchile.cl),37449,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,48708,0,none,17,udp,74,XX.XX.XX.XX,200.27.2.2(ns.telmexchile.cl),21411,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,43427,0,none,17,udp,75,XX.XX.XX.XX,200.27.2.2(ns.telmexchile.cl),30226,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,13797,0,none,17,udp,75,XX.XX.XX.XX,200.27.2.2(ns.telmexchile.cl),38204,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,59747,0,none,17,udp,74,XX.XX.XX.XX,200.7.4.7(b.nic.cl),28141,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,3292,0,none,17,udp,74,XX.XX.XX.XX,200.27.2.7(ns2.telmexchile.cl),26460,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,4802,0,none,17,udp,74,XX.XX.XX.XX,204.61.216.30(cl-ns.anycast.pch.net),47636,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,46755,0,none,17,udp,75,XX.XX.XX.XX,200.7.4.7(b.nic.cl),8157,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,23473,0,none,17,udp,65,XX.XX.XX.XX,200.16.112.16(c.nic.cl),11759,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,35385,0,none,17,udp,75,XX.XX.XX.XX,192.5.4.1(sns-pb.isc.org),59833,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,45357,0,none,17,udp,74,XX.XX.XX.XX,192.5.5.241(f.root-servers.net),44470,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,32832,0,none,17,udp,75,XX.XX.XX.XX,198.41.0.4(a.root-servers.net),4924,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,39827,0,none,17,udp,75,XX.XX.XX.XX,192.36.148.17(i.root-servers.net),50921,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,8729,0,none,17,udp,74,XX.XX.XX.XX,192.203.230.10(e.root-servers.net),42172,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,5690,0,none,17,udp,83,XX.XX.XX.XX,200.160.11.50(a.arpa.dns.br),43916,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,8734,0,none,17,udp,83,XX.XX.XX.XX,199.253.183.183(b.in-addr-servers.arpa),20802,53
filterlog,107,16777216,,1000008011,igb0,match,pass,out,4,0x0,,64,23957,0,none,17,udp,83,XX.XX.XX.XX,203.113.131.2(dns4.vietel.com.vn),25050,53

[johnpoz]

well download that sniff - what are the queries for??

That first one on the list tinne.arin.net is a NS for afrinic.net domain

;; QUESTION SECTION:
;afrinic.net.                  IN      NS

;; ANSWER SECTION:
afrinic.net.            3600    IN      NS      ns1.afrinic.net.
afrinic.net.            3600    IN      NS      ns2.lacnic.net.
afrinic.net.            3600    IN      NS      ns2.afrinic.net.
afrinic.net.            3600    IN      NS      sec1.apnic.net.
afrinic.net.            3600    IN      NS      sec3.apnic.net.
afrinic.net.            3600    IN      NS      tinnie.arin.net.
afrinic.net.            3600    IN      NS      afrinic.authdns.ripe.net.

;; ADDITIONAL SECTION:
ns1.afrinic.net.        3576    IN      A      196.216.2.1
ns1.afrinic.net.        3576    IN      AAAA    2001:42d0::200:2:1
ns2.afrinic.net.        3576    IN      A      196.216.168.10
ns2.afrinic.net.        3576    IN      AAAA    2001:43f8:120::10

So you get how the resolver works right???  You ask for say www.google.com, and then the resolve walks down the tree from roots til it finds the authoritative server for the domain your looking for..

So yeah out your wan your going to see lots of queries to NS on 53..  If you turn up the logging to say 5 in unbound you will see what its doing, etc.  Not going to look up all of those - but from just looking at the names you looked up on them can pretty much be sure they are just NS for domains, that your clients are trying to lookup and then need to be resolved to find the authoritative server for whatever.com, etc.

Do a simple dig +trace and you will get the idea of how resolving works..

here I cleaned it up a bit looking for www.pfsense.org

dig www.pfsense.org +trace

; <<>> DiG 9.11.0-P1 <<>> www.pfsense.org +trace
;; global options: +cmd
.                      498539  IN      NS      a.root-servers.net.
.                      498539  IN      NS      b.root-servers.net.
.                      498539  IN      NS      c.root-servers.net.
.                      498539  IN      NS      d.root-servers.net.
<snipped>;; Received 525 bytes from 192.168.9.253#53(192.168.9.253) in 0 ms

org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
<snipped>;; Received 817 bytes from 193.0.14.129#53(k.root-servers.net) in 93 ms

pfsense.org.            86400  IN      NS      ns1.netgate.com.
pfsense.org.            86400  IN      NS      ns2.netgate.com.
<snipped>;; Received 584 bytes from 199.19.54.1#53(b0.org.afilias-nst.org) in 78 ms

www.pfsense.org.        300    IN      A      208.123.73.69
pfsense.org.            300    IN      NS      ns1.netgate.com.
pfsense.org.            300    IN      NS      ns2.netgate.com.
;; Received 139 bytes from 162.208.119.38#53(ns2.netgate.com) in 46 ms</snipped></snipped></snipped>

[yodasoda]

I ran a sniff on the router's WAN and on each interface.

I'm seeing queries for hordes of Russian and Chinese websites, even for.. unusual websites ending in dot m-i-l
The activity (not just the root servers) seems to occur even when all other machines are turned off at night.
As far as i can tell the queries are receiving what appear to be legit replies.

If you turn up the logging to say 5 in unbound you will see what its doing, etc.

Can you tell me how do i do that? I want to be sure unbound is doing this, and there isn't something else more nefarious going on.

[johnpoz]

in the unbound advanced tab..

[yodasoda]

Thanks