(SOLVED) Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN



  • Hi.

    I run a small smb network for a business. The topology is WAN LAN and DMZ. This morning I found about 4 of these entries in the snort alerts, and they are sourced from the WAN IP.

    "INDICATOR-COMPROMISE suspicious .null dns query"

    I have snort's IPS policy set to security without blocking offenders, running on the WAN and DMZ interfaces. My pfsense version is 2.4.4-RELEASE-p2 (amd64)

    The server in the DMZ is accessible from the Internet. So it concerns me that I may have a compromised server on my hands, that may be a pivot into the into the rest of the network and I need advice on how to proceed.

    Thanks for any help and have a nice day.


  • LAYER 8 Global Moderator

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    INDICATOR-COMPROMISE suspicious .null dns query

    And what was the specific query for? .null is a valid OpenNIC tld - just because snort flags/blocks it doesn't mean its actually bad.. Something asks pfsense for host.domain.tld, then yeah its going to try and resolve or forward that query (based on your settings in pfsense) out its wan IP.

    btw - moved your thread to IDS/IPS section.



  • I don't know the contents of the query, but all of the queries are not going to dns servers on have set on the computers or pfsense. I looked some of them up and some go to japan others to the netherlands.


  • LAYER 8 Global Moderator

    And what are some of these IPs... Resolving which is pfsense default setting, then yeah the queries would go all over the globe as it "resolves" and actually talks to the authoritative NS for domain its resolving.

    OpenNIC is not normal public domains from ICANN, etc. pfsense would not be able to actually resolve those, unless setup for OpenNIC.



  • 202.12.27.33
    193.0.14.129
    198.41.0.4
    199.9.14.201
    192.36.148.17

    also I didn't setup anything with opennic. I wasn't even aware of that.


  • LAYER 8 Global Moderator

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    202.12.27.33

    Those are all roots..
    ;; QUESTION SECTION:
    ;33.27.12.202.in-addr.arpa. IN PTR

    ;; ANSWER SECTION:
    33.27.12.202.in-addr.arpa. 86400 IN PTR M.ROOT-SERVERS.NET.

    ;; ANSWER SECTION:
    17.148.36.192.in-addr.arpa. 86400 IN PTR i.root-servers.net.

    ;; ANSWER SECTION:
    129.14.0.193.in-addr.arpa. 21599 IN PTR k.root-servers.net.

    So yeah - when unbound on pfsense is asked for host.something.NULL as the tld, it would ask roots - hey roots who is the NS for this tld .null...

    You need to track down on your lan side network what is asking for whatever.NULL to see if legit or not... Could be billy bob user trying to access some p0rn site ;)



  • ok I'll start looking.


  • LAYER 8 Global Moderator

    but those are never going to resolve unless you setup your DNS to understand how to talk to OpenNIC

    You can turn on full query logging in unbound, but has to be done in the custom options box

    server:
    log-queries: yes

    edit, example I just did a query for

    Apr 21 05:36:10 unbound 65356:0 info: 192.168.9.100 test.domain.null. A IN NXDOMAIN 0.013277 0 120

    So you can see that my pc at 192.168.9.100 asked for that .null domain. If you turn on full query logging, you should be able to figure out who is asking for it, and exactly what real quick.



  • quick tcpdump for udp port 53 shows odd things. The server is resolving stackoverflow.com and some yandex stuff zen.spamhaus.org


  • LAYER 8 Global Moderator

    Those don't seem too odd to me... Those are very legit domains.

    See my edit above - you prob have easier time just turning on logging in unbound and then looking through the log for .null



  • I turned that on, but it isn't sending queries at the moment. I'll let it settle for a while and let you know. Thank you for the help btw.


  • Galactic Empire

    Do you run pfBlocker?

    It could be some dubious hosts hitting your WAN interface and pfBlocker doing a lookup for the host.

    Is the alert just in the WAN interface or on the WAN and DMZ.

    Also have a poke about in the /var/log/snort directory, there may be a file that you can do a u2spew or u2boat command.


  • LAYER 8 Global Moderator

    And how would pfblocker know to look for something.null? It could maybe do a PTR for an IP..

    Oh you mean in one of its block lists? Having a null tld in a block list could be very problematic, since there is no way to resolve those without setting up use of OpenNIC



  • @tman904 As @bmeeks has recommended several times in this forum, you should turn off monitoring WAN in Snort and instead run it on the internal interfaces such as LAN and in this case DMZ. This will make it easy to see which device on your network is causing these alerts by looking at your internal IP addresses. Since everything is blocked by default on pFsense, running Snort on the WAN just duplicates what pFsense already does.



  • NogBadTheBad
    The alert is only on my WAN but sourced from the WAN's IP.

    Johnpoz I turned log level up to 5 on the dns resolver, but I still havn't found anything.


  • Galactic Empire

    @jdeloach said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    @tman904 As @bmeeks has recommended several times in this forum, you should turn off monitoring WAN in Snort and instead run it on the internal interfaces such as LAN and in this case DMZ. This will make it easy to see which device on your network is causing these alerts by looking at your internal IP addresses. Since everything is blocked by default on pFsense, running Snort on the WAN just duplicates what pFsense already does.

    @bmeeks also states that if you have open ports on the WAN interface to enable Snort on the WAN interface.


  • Galactic Empire

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    NogBadTheBad
    The alert is only on my WAN but sourced from the WAN's IP.

    Johnpoz I turned log level up to 5 on the dns resolver, but I still havn't found anything.

    Is there anything in the Snort log directory?



  • I thought you want to run snort on WAN if you have open ports which I do.

    Yes there is a directory for each interface. As well as an update log file. The gui did state the most recent update was successful.


  • LAYER 8 Global Moderator

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    Johnpoz I turned log level up to 5 on the dns resolver, but I still havn't found anything.

    Doesn't actually log queries... You have to set the query as stated in the option box..

    logqueries.png



  • at the moment I have two ssh connections to the pfsense and I'm running.
    tcpdump -ni em0 udp port 53 |grep .null -LAN
    tcpdump -ni em1 udp port 53 |grep .null -DMZ

    Hopefully that will grab it at some point.


  • LAYER 8 Global Moderator

    That not going to catch anything.. Test it!!! Do a query with that in the fqdn..



  • -LAN and -DMZ aren't in the command.


  • LAYER 8 Global Moderator

    And again TEST it...

    where do you think in that tcpdump the info of the query would be listed? So how could you grep for it.



  • well can you explain that to me?

    There they are
    07:33:13.204333 IP 192.168.0.141.57062 > 192.168.0.2.53: 57715+ [1au] A? test.null. (38)
    07:33:13.241310 IP 192.168.0.141.57062 > 192.168.0.2.53: 57715+ A? test.null. (27)
    07:33:54.634610 IP 192.168.0.141.59863 > 4.2.2.2.53: 5400+ [1au] A? test.null. (50)

    guess I didn't need the -A

    This is what I used
    "tcpdump -ni em0 udp port 53 |grep .null"
    "tcpdump -ni em1 udp port 53 |grep .null"


  • LAYER 8 Global Moderator

    Ok so now that you have validate your method can find an example - yeah lets see if you find anything at the time you see the snort log.

    When I try that grep doesn't find my test

    But without grep I see it
    [2.4.4-RELEASE][admin@sg4860.local.lan]/root: tcpdump -ni igb0 udp port 53
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
    06:39:44.111865 IP 192.168.9.100.61488 > 192.168.9.253.53: 11023+ [1au] A? test.domain.null. (57)



  • The same thing that the gui says lines up with the logs. But it says it's from the WAN IP. with grep .null on my dmz interface log file in /var/log/snort doesn't seem to have .null in it.

    It took a few seconds after I ran the command. Also if you change the snap length it should get to grep quicker.



  • I just enabled snort on my LAN interface. If it's sourced from there it should line up in the logs for LAN and WAN.


  • Galactic Empire

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    I just enabled snort on my LAN interface. If it's sourced from there it should line up in the logs for LAN and WAN.

    Are your LAN and DMZ VLANS, if they are and both share the same parent interface you can just run Snort on the parent interface as it puts the interface into promiscuous mode.



  • In my case LAN and DMZ are two physical interfaces. Hopefully by tonight the queries will be sent again. Maybe it's nothing but it doesn't seem normal to me.

    Thanks again for everyone's help with this issue. I'll report back if it's finds anything by tonight or tomorrow.


  • LAYER 8 Global Moderator

    I would be curious as well to what they are looking up.. but domain.null could be almost anything..

    Allowing for unbound to resolve opennic tlds is a simple as adding a few stub zones pointing to the opennic NSs..

    Here I added stub zones for geek and null so could get to search engine and fine a null site, etc..

    geekandnull.png

    That snort marks them as suspicious could be seen a few different ways... For starters anyone trying to do something bad, writing code so it could resolve opennic tlds would be "extra" work ;)

    But maybe its someone trying to circumvent something??

    It could be just honest sort of mistake where a search suffix is adding .null to something and causing the query to roots for whatever.com.null for example..

    You have my curiosity cat meowing ;) So please report back what you find...


  • Galactic Empire

    Under /var/log/snort/snort_interfaceRANDOMNUMBER there should be a file with u2 in the file name, do a u2spewfoo FILENAME or a u2uboat FILENAME > output.pcap.

    The first command will dump the output to screen the second will create a file that can be read by Wireshark.

    Also I'd be tempted to use Balanced rather than Security.



  • I'll run those commands and see if I spot anything in wireshark.

    As of today/tonight I haven't seen incidents on DMZ or LAN that line up with the WAN incidents. I wonder if it's simply because it hasn't happened again since I starting logging on LAN? That would make since since I was only logging on WAN and DMZ when it happened yesterday.

    What do you guys think? Could it have came from pfsense directly or is it more likely something on the LAN caused it when I wasn't logging there?


  • LAYER 8 Global Moderator

    pfsense would not have been trying to to some domain.null it only checks for updates, the rss feed if you have that setup and package updates. all would be pfsense/netgate domains.

    Did you install any unofficial packages? Here is the thing trying to access any of the opennic tlds from some application would just be stupid - UNLESS!!! the documentation of said software gave you info on how to make sure you can resolve opennic tlds.. Its a pretty small use base in the big picture to be honest.. Other than say the browser addon users..

    Guess the software could be hard coded to query an opennic NS... But you stated the queries were to root servers looking for whatever.null - which would never work.. Your just going to get back NX from roots for any of the opennic tlds



  • Here is the course of action I took, note this is after I enabled snort on the LAN interface.

    Here's my thought process step by step. Hopefully it's understandable and will help someone in the future facing something similar.

    ##########################################################################################################
    cause found LAN snort compromise report in snort alerts - coming from 192.168.0.132 on the LAN interface

    first course of action was to look at status->arp table - didn't yeld anything so client hasn't sent in roughly five minutes or so

    second course of action was to look at status->dhcp leases - didn't have an active lease but found the IP in the expired leases

    third course of action was to set the mac address from that lease for 192.168.0.132 to a static dhcp mapping to IP 192.168.0.250

    fourth course of action was to create a rule under firewall->rules->LAN that passes any traffic sourced from 192.168.0.250. I also added a pass rule for 192.168.0.132 just in case the host doesn't request a lease for some reason. Finally I added a description saying this is the compromised host, and set all packets matching the rules to be logged for either of those two hosts. The reason I'm passing the traffic is to see a working example of what they are doing.

    note - 192.168.0.250 finally sent packets see below.

    LAN tcp 192.168.0.250:53028 -> 209.85.200.189:443 ESTABLISHED:ESTABLISHED 101 / 146 15 KiB / 25 KiB
    WAN tcp X.X.X.X:XXXX (192.168.0.250:53028) -> 209.85.200.189:443 ESTABLISHED:ESTABLISHED 101 / 146 15 KiB / 25 KiB
    ##########################################################################################################

    To be honest I'm still not sure exactly what's going on yet. I think that it's just an https connection, when I do a whois on the destination IP it returns google.

    Anyway for now that is how the situation stands.


  • LAYER 8 Global Moderator

    Yeah that is google, and where is doing dns queries for opennic?



  • Well I definitely found the culprit. Yesterday a snort alert showed up from the existing server for .win. I'm 99.9% sure it originated from the server and somehow got to the other segment. In any case I need to rebuild the server. Hopefully I can get by with some hot standby/swap kind of thing. Can't have much downtime. Wish me luck I'll need it lol.

    Thanks for everyone's help I really appreciate it.


  • LAYER 8 Global Moderator

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    I'm 99.9% sure it originated from the server and somehow got to the other segment. In any case I need to rebuild the server.

    What?

    So you have a packet capture of this query? "somehow" is a not good RCA for what is happening in your network.. Your saying its now doing queries for something.win, that is not a opennic tld? That is a valid normal new tld that you can register anywhere..

    Blocking queries to domains because you don't like the tld and its not .com or .org is sure fire way to break the internet for your users..

    Maybe its some legit software installed on the server checking to see if there update available.. And that company just happen to use .win for their tld because its cheap and hip ;)



  • I don't plan on blocking them. I found them in the snort alerts again, just .win this time. After this cropped up I discovered the Web mail panel for the server was exposed to the internet. Unfortunately it runs a product that also has a management ui for the mail server part on 443 also exposed. I blocked that port along with the Web mail. smtp/Imap is ok atm.

    I dug deeper and the server hasn't been patched for a few years. In the servers admin logs, it shows the firewalls lan ip successfully logged in at a time and date I didn't login. Thats when i got the permission to build a new firewall with pfsense. Also when we started using snort. This is why I think it's hacked. It was installed before I came on board. So I don't know the history, and almost no documentation of anything.

    Tldr version
    These dns queries, along with the web mail and admin panel facing the Internet. make me think the server may have been compromised.


  • LAYER 8 Global Moderator

    @tman904 said in Snort detecting INDICATOR-COMPROMISE suspicious .null dns query on my WAN:

    These dns queries,

    What was the query for? Can you not get this info from snort? just saying something.win is not very useful logging.. I can see the alert could be called that, but there should be a log of the actual query done.



  • Is there a way to filter under status dns resolver? Also if the dns request wasn't answered will it not show up in the log?


Log in to reply