Unable to illegal DNS record from pfsense (DNS-resolver corruption)

asadz

@asadz also i checked DC2 points to PFsense infra interface (don't know why) should it not point to DC1 server IP address?

johnpoz

@asadz and what is this 192.168.4.4?

And yes you need to check both of them.

If they are not set to forward to pfsense.. And you still resolve your whatever to 100, then you prove to yourself its your AD dns issue..

Forward them to say google dns - these are the name servers used for resolving stuff not in your name space - they should not point to your AD for dns.. These are used to resolve the internet stuff, not stuff you are authoritative for forwarding to AD.. When then does what with it when asking for say www.google.com - forwards it back to the other one?

asadz

@johnpoz great makes 100% sense let me try this variation now.

asadz

@asadz
test done on DC1, DC2 i removed all pfsense IP put 8.8.8.8 in DC2,
in fwd section of DC1 i have put IP address of DC2, and I also tried put 8.8.8.8 with no reference of DC2, still resolves to 100.* :(

johnpoz

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

still resolves to 100.* :(

Because you have an issue with your AD, that was what I have been saying since the start..

in fwd section of DC1 i have put IP address of DC2

Pointless.. If client asks dc1 for say www.google.com which you are not authoritative for, what good is it to forward that to dc2? It is not authoritative for google.com either - so it would just have to forward that query as well.

The forwarders in your AD dns should point to what can resolve stuff you are not authoritative for.. The dns set on the server itself should point to itself, and the other AD dns running in your network..

My guess to your issue is you have a wildcard in your AD dns, do you have a * as a record.. this would resolve anything.otherthing.whatever.something.local to that IP.. Anything that does not have a specific record say host.local or sb.whatever.local to what that wildcard entry.

So if a client happens to ask for sb.scorecard.com.local - then it would return the wildcard record.

But now that you are not forwarding to pfsense, and your client directly asks your AD dns for whatever - and it returns that 100.x address you know for sure its an issue in your AD dns.. Or where your AD forwards.. but .local should never return anything on the public internet - it is not a valid tld.. And the 2 examples you gave of sb.domain.tld sure do not resolve 100.x on the public internet - one is not a valid domain, and the other resolves to public IP addresses that do not start with 100.

asadz

Because you have an issue with your AD, that was what I have been saying since the start..

but it was the pfsense according to dns-reply.log that return the bad ip address, on 14th dec the AD 192.168.3.6 did request for sb.scorecardsearch.com and it was returned that address. So, it behave as expected for ip/domains not under its control it fwd as expected.

My guess to your issue is you have a wildcard in your AD dns, do you have a * as a record.. this would resolve anything.otherthing.whatever.something.local to that IP.. Anything that does not have a specific record say host.local or sb.whatever.local to what that wildcard entry.

any specific place to check (please note im not an AD guy), i checked in forward and reverse lookup section found none with * in it

But now that you are not forwarding to pfsense, and your client directly asks your AD dns for whatever - and it returns that 100.x address you know for sure its an issue in your AD dns.. Or where your AD forwards.. but .local should never return anything on the public internet - it is not a valid tld.. And the 2 examples you gave of sb.domain.tld sure do not resolve 100.x on the public internet - one is not a valid domain, and the other resolves to public IP addresses that do not start with 100.

Yes, i get your point AD is not forgetting this mapping , it is holding on to something also the TTL value 0 was a indicator as well, the request is not leaving the network (and going to internet)
what about stale records? also i'm able to resolve hostnames correctly with local nslookup/ping from DC, but that shouldn't matter as its local.

asadz

@asadz on DC this is output of nslookup

Default Server:  VM-DC1.dummy.local
Address:  192.168.3.6

> sb.scorecardsearch.com
Server:  VM-DC1.dummy.local
Address:  192.168.3.6

Non-authoritative answer:
Name:    sb.scorecardsearch.com
Address:  100.2.3.4

johnpoz

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

but it was the pfsense according to dns-reply.log that return the bad ip address

well yeah if pfsense asks your AD, and it returned that 100 then it would be cached.. This is why separation now allows you to figure out where the problem is.

Clear your AD dns cache.. if it still returns that address then you need to figure out why.. Sniff to see if your AD forwards that fqdn and gets back that answer. Or if when you client asks it just gets returned..

And again I would highly suggest you turn on debug when you do your nslookup - so you can validate exactly what is getting asked.

That "Non-authoritative answer" points to that being in cache.

asadz

@johnpoz I will run wireshark then, but i don't know if its install winpcap drivers on DC idk.

> sb.scorecardsearch.com
Server:  VM-DC1.dummy.local
Address:  192.168.3.6

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 10, rcode = NXDOMAIN
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        sb.scorecardsearch.com.dummy.local, type = A, class = IN
    AUTHORITY RECORDS:
    ->  dummy.local
        ttl = 3600 (1 hour)
        primary name server = vm-dc1.dummy.local
        responsible mail addr = hostmaster.dummy.local
        serial  = 15316
        refresh = 900 (15 mins)
        retry   = 600 (10 mins)
        expire  = 86400 (1 day)
        default TTL = 3600 (1 hour)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 11, rcode = NXDOMAIN
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        sb.scorecardsearch.com.dummy.local, type = AAAA, class = IN
    AUTHORITY RECORDS:
    ->  dummy.local
        ttl = 3600 (1 hour)
        primary name server = vm-dc1.dummy.local
        responsible mail addr = hostmaster.dummy.local
        serial  = 15316
        refresh = 900 (15 mins)
        retry   = 600 (10 mins)
        expire  = 86400 (1 day)
        default TTL = 3600 (1 hour)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 12, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        sb.scorecardsearch.com, type = A, class = IN
    ANSWERS:
    ->  sb.scorecardsearch.com
        internet address = 100.2.3.4
        ttl = 0 (0 secs)

------------
Non-authoritative answer:
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 13, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        sb.scorecardsearch.com, type = AAAA, class = IN
    AUTHORITY RECORDS:
    ->  scorecardsearch.com
        ttl = 363 (6 mins 3 secs)
        primary name server = ns1.bodis.com
        responsible mail addr = dnsadmin.bodis.com
        serial  = 2017062202
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 1209600 (14 days)
        default TTL = 3600 (1 hour)

------------
Name:    sb.scorecardsearch.com
Address:  100.2.3.4

johnpoz

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

    QUESTIONS:
        sb.scorecardsearch.com, type = A, class = IN
    ANSWERS:
    ->  sb.scorecardsearch.com
        internet address = 100.2.3.4
        ttl = 0 (0 secs)

Well that shows your AD returned that answer..

Did you sniff on traffic on your AD dns when that was asked? Did you flush your cache? Sniff will validate to you that your AD answered that on its own, or got that answer after asking its forwarder..

Or look to your AD dns logs - don't recall the level of logging, but you would hope you could log if answered on its own or got that from one of its forwarders..

asadz

@johnpoz

Well that shows your AD returned that answer..
from the TTL value i guess?

I'm doing a trace using netsh, and using message analyzer to analyse it, will share outcome soon.

johnpoz

@asadz you could also turn on logging

simple google found this
http://woshub.com/enable-dns-query-logging-parse-logfile/

What your wanting to validate is if your AD dns answered it from something it has a record for - or did it get that answer from where your forwarding.. I still say you have that record setup. Or you did not flush the cache?

ttl=0 is for sure not the ttl value of that record..

This is the TTL from the authoritative ns for that domain

;; QUESTION SECTION:
;sb.scorecardsearch.com.                IN      A

;; ANSWER SECTION:
sb.scorecardsearch.com. 10800   IN      A       199.59.243.222

;; Query time: 15 msec
;; SERVER: 185.85.196.36#53(185.85.196.36)

If you got it from cache it would be something less than that, but a TTL 0 points to either that ttl is set in your dns, or you have your dns set to return ttl of 0 on stuff that has timed out, this is a special setting.. I have not worked with MS dns in such a long time I am not aware of that being a feature or not. I don't ever recall that feature back when I was dealing with ms dns on a daily basis - but that was many years ago.

see when I ask my dns for that record, clearly I get back a cached item since the ttl value is less than original value, etc. And it was resolved some time ago because the cach is couple 1000 seconds lower than the original

;; QUESTION SECTION:
;sb.scorecardsearch.com.                IN      A

;; ANSWER SECTION:
sb.scorecardsearch.com. 8791    IN      A       199.59.243.222

;; Query time: 7 msec
;; SERVER: 192.168.3.10#53(192.168.3.10)

When you do your sniff - you need to be sniffing on the server, on the interface(s) that the server would use to talk to its forwarders.. If you ask it and you see no query to the forwarders for that, then the dns returned it via its own record, or it returned it from cache. That you see that it was non authoritative points to it having it in its cache or getting it from where it forwarded to - but ttl=0 makes really no sense..

asadz

@johnpoz here is the result of etl files analysis.

johnpoz

@asadz so from that its saying that 89.36.170.207 sent you your server 192.168.3.6 that 100 answer for your query?

What is that address?

;; QUESTION SECTION:
;207.170.36.89.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
207.170.36.89.in-addr.arpa. 7200 IN     PTR     zoho-170-207.dub3.computerline.net.

You forward your dns queries to them?

edit: that makes no sense that traffic is to 443..

How about a simple wireshark - so can actually see what is on the wire, and what is received on the wire vs some nonsense application trying to make it easier for you to read.. But that info shows us nothing.. Really - where is your client asking, did the dns forward anywhere?

Can really tell what that is - is that the server 3.7 asking what? Or is that the server answering a client - what IP was the client..

Do a simple wireshark for gosh sake.

asadz

@johnpoz no 89 is separate flow, from analysis of etl it means no traffic went outside the box.

johnpoz

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

t means no traffic went outside the box.

well answering a client would be outside the box.. you just had the server ask itself for that query?

But if no traffic went outside - then the server has a record for that.. You need to find out where.. Be it a cache or a actual record you setup..

asadz

@johnpoz its a production DC, i dnt want to install wireshark as it normally requires a reboot, plus the above etl analysis shows zero bytes was exchanged for this IP, so it remain passive, no for entry i need to do some forensic to see where this IP is

johnpoz

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

wireshark as it normally requires a reboo

nope never seen it require that.. Have installed it hundreds of time - I have never seen it require a reboot that I can ever remember.

asadz

@johnpoz
6.8 is my IP i get after connecting to VPN. The reboot is for npcap driver in case of wireshark. Anyhow, here is the debug logs from the link you shared. I can't make much of it. Seems its resolving requests on its own.

18.12.2022 20:24:02 1774 PACKET  000002A1016A7D50 UDP Rcv 192.168.6.8     0006   Q [0001   D   NOERROR] A      (2)sb(15)scorecardsearch(3)com(0)
UDP question info at 000002A1016A7D50
  Socket = 868
  Remote addr 192.168.6.8, port 61364
  Time Query=171623, Queued=0, Expire=0
  Buf length = 0x0fa0 (4000)
  Msg length = 0x0028 (40)
  Message:
    XID       0x0006
    Flags     0x0100
      QR        0 (QUESTION)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        0
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    0
    NSCOUNT   0
    ARCOUNT   0
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   A (1)
      QCLASS  1
    ANSWER SECTION:
      empty
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
      empty

18.12.2022 20:24:02 1774 PACKET  000002A1007E2960 UDP Snd 192.168.3.7     a4c2   Q [0001   D   NOERROR] A      (2)sb(15)scorecardsearch(3)com(0)
UDP question info at 000002A1007E2960
  Socket = 13744
  Remote addr 192.168.3.7, port 53
  Time Query=0, Queued=0, Expire=0
  Buf length = 0x0fa0 (4000)
  Msg length = 0x0033 (51)
  Message:
    XID       0xa4c2
    Flags     0x0100
      QR        0 (QUESTION)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        0
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    0
    NSCOUNT   0
    ARCOUNT   1
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   A (1)
      QCLASS  1
    ANSWER SECTION:
      empty
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
    Offset = 0x0028, RR count = 0
    Name = (0)

18.12.2022 20:24:02 1774 PACKET  000002A10025D0B0 UDP Rcv 192.168.3.7     a4c2 R Q [8081   DR  NOERROR] A      (2)sb(15)scorecardsearch(3)com(0)
UDP response info at 000002A10025D0B0
  Socket = 13744
  Remote addr 192.168.3.7, port 53
  Time Query=171623, Queued=0, Expire=0
  Buf length = 0x0fa0 (4000)
  Msg length = 0x0043 (67)
  Message:
    XID       0xa4c2
    Flags     0x8180
      QR        1 (RESPONSE)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        1
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    1
    NSCOUNT   0
    ARCOUNT   1
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   A (1)
      QCLASS  1
    ANSWER SECTION:
    Offset = 0x0028, RR count = 0
    Name = [C00C](2)sb(15)scorecardsearch(3)com(0)
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
    Offset = 0x0038, RR count = 0
    Name = (0)

18.12.2022 20:24:02 1774 PACKET  000002A1016A7D50 UDP Snd 192.168.6.8     0006 R Q [8081   DR  NOERROR] A      (2)sb(15)scorecardsearch(3)com(0)
UDP response info at 000002A1016A7D50
  Socket = 868
  Remote addr 192.168.6.8, port 61364
  Time Query=171623, Queued=171623, Expire=171626
  Buf length = 0x0200 (512)
  Msg length = 0x0038 (56)
  Message:
    XID       0x0006
    Flags     0x8180
      QR        1 (RESPONSE)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        1
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    1
    NSCOUNT   0
    ARCOUNT   0
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   A (1)
      QCLASS  1
    ANSWER SECTION:
    Offset = 0x0028, RR count = 0
    Name = [C00C](2)sb(15)scorecardsearch(3)com(0)
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
      empty

18.12.2022 20:24:03 1774 PACKET  000002A100165910 UDP Rcv 192.168.6.8     0007   Q [0001   D   NOERROR] AAAA   (2)sb(15)scorecardsearch(3)com(0)
UDP question info at 000002A100165910
  Socket = 868
  Remote addr 192.168.6.8, port 61365
  Time Query=171624, Queued=0, Expire=0
  Buf length = 0x0fa0 (4000)
  Msg length = 0x0028 (40)
  Message:
    XID       0x0007
    Flags     0x0100
      QR        0 (QUESTION)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        0
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    0
    NSCOUNT   0
    ARCOUNT   0
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   AAAA (28)
      QCLASS  1
    ANSWER SECTION:
      empty
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
      empty

18.12.2022 20:24:03 1774 PACKET  000002A100165910 UDP Snd 192.168.6.8     0007 R Q [8081   DR  NOERROR] AAAA   (2)sb(15)scorecardsearch(3)com(0)
UDP response info at 000002A100165910
  Socket = 868
  Remote addr 192.168.6.8, port 61365
  Time Query=171624, Queued=0, Expire=0
  Buf length = 0x0200 (512)
  Msg length = 0x005f (95)
  Message:
    XID       0x0007
    Flags     0x8180
      QR        1 (RESPONSE)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        1
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    0
    NSCOUNT   1
    ARCOUNT   0
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name = (2)sb(15)scorecardsearch(3)com(0)
      QTYPE   AAAA (28)
      QCLASS  1
    ANSWER SECTION:
      empty
    AUTHORITY SECTION:
    Offset = 0x0028, RR count = 0
    Name = [C00F](15)scorecardsearch(3)com(0)
    ADDITIONAL SECTION:
      empty

asadz

@johnpoz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

@asadz said in Unable to illegal DNS record from pfsense (DNS-resolver corruption):

wireshark as it normally requires a reboo

nope never seen it require that.. Have installed it hundreds of time - I have never seen it require a reboot that I can ever remember.

I found interesting thing , when i parsed log using
https://github.com/AleksandrReznik/DNSdebugLogParser

for correctly resolved local domains I get ldap as

Name = (5)_ldap(4)_tcp(6)VM-DC2(10)dummy(5)local(0)

where as for scorecardsearch it is
```
Name = (2)sb(15)scorecardsearch(3)com(0)

it doesn't include the _ldap, meaning it doesn't look at local directory ldap for