DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution

pftdm007

Hello!

Often (5+ times per week) I end up not being able to resolve most websites from my LAN due to Snort being triggered into blocking one (or more) of the root-servers IP's because of a suspicious resolution of a .top domain.

This is what happens:

I notice most clients cannot reach the internet (actually they cant resolve most websites which mimics loss of internet connection)
I see in Snort's alerts (WAN interface) several instances of the following alert:

INDICATOR-COMPROMISE Suspicious .top dns query

I remove the offending hosts (the X.root-servers) from Snort's blocked list
All goes back to normal.

My understanding is that something (a service, a client, etc) is requesting resolution of a .top domain, the request is sent to the X.root-servers.net network, Snort sees the traffic as "malicious" and blocks the "offending" external IP (X.root-servers.net )

My problem is that I cannot pinpoint the source of these requests on my network. I tried looking in the System Logs > DNS resolver, but often the incident happened a while ago and either the logs are truncated, or have been rotated and the relevant lines are no longer available.

I tried doing a "cat" on /var/log/resolver.log but its empty.

What would be the proper way of tracking the source of these requests?

pfSense is configured to use "1.0.0.1" & "1.1.1.1" as its DNS servers in General config.
I am using pfBLockerNG & DNSBL so Unbound is active.
I am not using DNS forwarder (Services > DNS Forwarder)
I am using DNS resolver (Services > DNS Resolver)

Thanks!

johnpoz

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

pfSense is configured to use "1.0.0.1" & "1.1.1.1" as its DNS servers in General config.

Which has ZERO to do with unbound resolving which is default..

.top is a valid tld.. has been since 2014

This why you should run your IDS/IPS on your lan side interfaces if you want to know which client kicks off an alert.

If you want to have unbound log queries, then you would set that in the custom options box
server:
log-queries: yes
log-replies: yes

then you will get stuff like this in the dns resolver log

Apr 22 09:56:58 unbound 168:0 info: 192.168.9.100 something.testdomainlog.top. A IN

NogBadTheBad

I had a similar issue, due to a block for China via GeoIP on the WAN interface.

I struggled to find what was triggering the alert as it was only ever seen on the WAN interface, turned out it was pfBlocker itself causing the alert when trying to resolve an IP address for the reports page.

https://forum.netgate.com/topic/140634/finding-internal-ip-causing-block

pftdm007

@johnpoz

I am indeed running Snort instances on all interfaces, but when this happens, I see no alerts on the LAN interface, only on WAN with source being my public IP, and destination being the root servers. This is why its difficult to pinpoint the LAN client doing this since no alerts are generated on LAN. To be honest it does not make sense to me...

@NogBadTheBad
By the looks of it, both you and the OP of the thread you sent me to had incoming traffic being blocked by snort on WAN interface. For me, its the opposite, it seems to be outgoing traffic to the blocked IP's (root servers).

Could it be possible the issue be with NAT and running snort on WAN? In that case, I'm aware internal client IP's would show as public IP's on the WAN interface making it difficult to pinpoint the offending LAN client...

I will enable unbound logs as suggested by johnpoz and see what happens...

johnpoz

So you understand it would be quite possible for say domainX.tld that has a cname to other.domain.top right...

So client ask for host.domainX.tld, and now unbound trying to resolve works down from roots and would be doing queries to roots for this other.domain.top

So client never actual asks for something with tld=top

that 199.7.91.13 is root server, and so are the other 3 IPs

NogBadTheBad

Under /var/log/snort/snort_interfaceRANDOMNUMBER there should be a file with u2 in the file name, do a u2spewfoo FILENAME or a u2uboat FILENAME > output.pcap.

bmeeks

More than likely it is actually your firewall itself (via Unbound) that is making the DNS request to the root servers and not a LAN client. So having the WAN public IP address is correct in that case as the firewall DNS would source the request from the WAN's public IP. Unbound could be doing the lookup in order to display information for logging or reporting on behalf of another package like @NogBadTheBad mentioned.

When you use a package like pfBlockerNG, it may cause Unbound to go digging for DNS info about an IP address (or domain) in order to populate info boxes, log entries or reports. The act of "digging" for such info can then trigger these rather hair-trigger Snort rules. You might consider putting all the root DNS servers into an alias and then using that alias as part of a Pass List for Snort so that the IP addresses of the root DNS servers are never blocked. I would generally feel safe trusting the DNS roots.

pftdm007

Hmmm, I had suspicions about the FW itself doing these requests. I am looking into pfblockerNG's ipv4 or DNSBL lists...

I will create an alias and add the root server's IP's to the whitelisted IP's and see if this works.

In the meantime I did what @NogBadTheBad suggested. What''s the purpose of this? Not sure how to get info from that..

bmeeks

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

Hmmm, I had suspicions about the FW itself doing these requests. I am looking into pfblockerNG's ipv4 or DNSBL lists...

I will create an alias and add the root server's IP's to the whitelisted IP's and see if this works.

In the meantime I did what @NogBadTheBad suggested. What''s the purpose of this? Not sure how to get info from that..

The packet trace would let you see the entire conversation between the two hosts, but I doubt it would show much beyond the fact the firewall asked the root server for information about a .top domain.

To be perfectly honest, I would not sweat this at all. There are many rules in the Snort and Suricata rules packages that are really meant for just notification only. The problem Snort has on pfSense is that all alerts block. With an inline IPS solution, you can have rules that just alert and other rules that block. In this instance, the suspicious domain lookup should be just an alert. Like raising a flag saying "hey, you might want to take a look at this". However, the way Snort works today on pfSense that "hey you might just want to take a look" becomes a full-fledged block of traffic to the root server. So either disable that rule, or use the alias trick I suggested to put all the root servers on a Pass List so they never get blocked.

pftdm007

@bmeeks

I have created an alias to whitelist the root servers in Snort, but as time goes, I am adding more and more servers... The alias is now containing 9 IP's and I expect to add more as Snort continues to block these IP's.

I am actually worried about this because until about a month ago, I had never experienced issues like these, and I have been using Snort for years, and nothing on my FW has changed configuration wise except Snort and pfblockerNG rules and databases which of course are being updated regularly by the applications, so I wonder why this is starting now..

Another set of IP's that have just been blocked, and those seems to be from China:

a.zdnscloud.com (203.99.24.1)
b.zdnscloud.com (203.99.25.1)

I have not whitelisted those! Am I being paranoid?

bmeeks

Well, I can certainly tell you that Snort is not doing any DNS lookups on its own. So that leaves other packages you may have on the firewall such as pfBlockerNG and/or the DNSBL setup. If you can't see any LAN clients performing the suspicious lookups, then try temporarily disabling the pfBlockerNG and DNSBL setup to see if the DNS queries continue.

If you have Snort running on the LAN and have the same rules enabled on both the LAN and WAN, then if a client is performing the DNS query you should see the alert on the LAN instance of Snort and that alert would show the local IP of that client and not your WAN IP. The fact you are seeing it only on the WAN indicates to me something on the firewall itself is doing the lookup.

I am still willing to bet the lookups are benign and are coming from a service just trying to get additional domain info to display in a log message.

NogBadTheBad

This post is deleted!

johnpoz

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

a.zdnscloud.com (203.99.24.1)
b.zdnscloud.com (203.99.25.1)

Those are the freaking NS For .top

;; QUESTION SECTION:
;top.                           IN      NS

;; ANSWER SECTION:
top.                    3600    IN      NS      a.zdnscloud.com.
top.                    3600    IN      NS      c.zdnscloud.com.
top.                    3600    IN      NS      d.zdnscloud.com.
top.                    3600    IN      NS      f.zdnscloud.com.
top.                    3600    IN      NS      i.zdnscloud.com.
top.                    3600    IN      NS      g.zdnscloud.com.
top.                    3600    IN      NS      j.zdnscloud.com.
top.                    3600    IN      NS      b.zdnscloud.com.

Yes your tin foil hat is cutting off circulation to your brain.. That you even have those rule enabled seems paranoid!! So what was the domain being asked for if you looked in the packet capture???

You understand that there is more bad shit on .com or .net or .org then .top right! You should block those as well ;)

What is in the packet captures... It will tell you what was being queried for domainxyz.top for example..

The logic of such a rule is nonsense. Lets say a client was asking 8.8.8.8 for dns in your typical forwarding mode... So if the client ask for something.top - snort would block the client from asking for ANY dns from 8.8.8.8??? Just freaking stupid!!!

Such a rule should never be in block mode - only alert..

bmeeks

@johnpoz said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

The logic of such a rule is nonsense. Lets say a client was asking 8.8.8.8 for dns in your typical forwarding mode... So if the client ask for something.top - snort would block the client from asking for ANY dns from 8.8.8.8??? Just freaking stupid!!!

Such a rule should never be in block mode - only alert..

@johnpoz is right on target here. Such rules are really just designed to let you know something possibly suspicious ,but not guaranteed to be suspicious, is happening. That's all. Unfortunately, Snort's blocking mechanism uses a vary large hammer with no ability to finess things. It puts one or both of IP SRC and IP DST in the snort2c table of pf and that blocks all further traffic from the IP addresses until the block is removed. An inline IPS such as Suricata's Inline IPS Mode can selectively block traffic for just a single IP packet exchange, so it has much greater finess when it comes to blocking traffic. I'm working on getting the same technology into the Snort package, but it's going to require me to rewrite some portions of the DAQ packet capture utility used by Snort and then get my changes accepted into upstream. DAQ can use netmap today on FreeBSD, but the implementation is very wasteful of physical interfaces as you have to use two physical interfaces for each Snort instance (you literally have to run a patch cable between two physical ports in order to have two-way traffic over the tunnel).

So I would disable that rule or else use the Pass List method I mentioned. My preference would be to disable that rule and any others like it.

NogBadTheBad

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

INDICATOR-COMPROMISE Suspicious .top dns query

I wonder if the OP is using Security or higher for his IPS Policy Selection, maybe it needs toning down a bit, I can't trigger the rule using Balanced.

bmeeks

@NogBadTheBad said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

INDICATOR-COMPROMISE Suspicious .top dns query

I wonder if the OP is using Security or higher for his IPS Policy Selection, maybe it needs toning down a bit, I can't trigger the rule using Balanced.

That's my guess. I advise folks to run Balanced as their top policy because Security is really just too sensitive. A lot of users think the best thing is the most secure thing, but with an IDS/IPS that's not necessarily true when you factor in "useability" of the network. When you make it so secure that it doesn't work any more, that's not good ... .

pftdm007

@bmeeks
I am in agreement that these alerts are probably benign and can be either disabled which I did. Didnt have time yet to do packet capture or any other fiddling with the setup but for the sake of learning a bit more... Will do!

@NogBadTheBad
I didnt know the command you recommended dumped the contents of the packet that triggered the block action in a more or less readable way... Learning each day.

BTW I was using Security but learnt the hard way ;) To be honest, the tooltip in pfSense is kinda misleading. To me, the "connectivity" ruleset meant you get very basic coverage. Then the word "starter" for "Balanced" meant for limited systems or if "connectivity" wasnt enough. Clearly, I did OVERKILL!

Connectivity blocks most major threats with few or no false positives. Balanced is a good starter policy. It is speedy, has good base coverage level, and covers most threats of the day. It includes all rules in Connectivity. Security is a stringent policy. It contains everything in the first two plus policy-type rules such as a Flash object in an Excel file.

@johnpoz
Haha thanks for the health advice! I understand what you're saying but from a guy who probably spends 20x more time working with this technology its easy to reach that conclusion. Not so much for an entry level user like me, but always listening to experienced guys here ;)

bmeeks

@pftdm007 said in DNS request from LAN client triggers Snort which blocks X.root-servers.net causing loss of DNS resolution:

@bmeeks
BTW I was using Security but learnt the hard way ;) To be honest, the tooltip in pfSense is kinda misleading. To me, the "connectivity" ruleset meant you get very basic coverage. Then the word "starter" for "Balanced" meant for limited systems or if "connectivity" wasnt enough. Clearly, I did OVERKILL!

Connectivity blocks most major threats with few or no false positives. Balanced is a good starter policy. It is speedy, has good base coverage level, and covers most threats of the day. It includes all rules in Connectivity. Security is a stringent policy. It contains everything in the first two plus policy-type rules such as a Flash object in an Excel file.

I copied that text almost verbatim from some old Snort documentation a long time ago. I admit to not reading it too closely since. Next time I make a change in the GUI package I will try to remember to update that text a bit. "Connectivity" could probably be best described as a good starter policy and "Balanced" as more suited for admins with experience running an IDS/IPS. "Security" is really for the truly paranoid or for folks who run in IDS (non-blocking) mode only. Running with the "IPS - Security" policy is going to result in a number of nuisance blocks when blocking mode is enabled.

johnpoz

To finish this off how bat shit insane these rules are in blocking mode.. Some AD could be hosted off a .top site - so now user minding their own business not doing anything other than reading the news on bbc or cnn.com and ad pops up for something.top and next thing you know their whole internet is broken because access to their dns server is blocked.