Unknown hosts always resolving to particular IP

rsaanon

ENV: pf: 2.4.4 Release, Resolver/Unbound: 1.8.1, Local Domain: rsahome.com
Problem Descripton: Bogus hostnames are always resolved to a particular IP (160.124.44.70)

I have a handful of host overides defined in the resolver (eg: h1.rsahome.come, h2.rsahome.com, etc.). However, anytime I manually try to resolve (using nslookup) an invalid internal or external host, the resolution always returns the same IP of 160.124.44.70. Here are a few of examples of the queries performed:

QUERY                                                 RESULT/RESPONSE
nslookup xyz                    Name: xyz.rsahome.com, Address: 160.124.44.70 //invalid query/results
nslookup xyz.dklafj.com         Name: xyz.dklafj.com.rsahome.com //invalid query/results
nslookup xxx                    Name: xxx.rsahome.com, Address: 160.124.44.70 //invalid query/results
nslookup xxx.dklafj.com         Name: xxx.dklafj.com.rsahome.com //invalid query/results
nslookup host1                  Name: host1.rsahome.com, Address: 192.168.1.100  //valid query/result

Where is this IP of 160.124.44.70 coming from and Why are all invalid queries returning this IP?

On a side note:

I see unbound restarting every few seconds (see below): notice: Restart of unbound 1.8.1.
Why is there a "NULL IN" in the log (see below): info: generate keytag query _ta-4f66. NULL IN

Something peculiar I see in the unbound log:

May 13 09:26:55 fw254 unbound: [34032:0] info: server stats for thread 3: requestlist max 25 avg 4.05 exceeded 0 jostled 0
May 13 09:26:55 fw254 unbound: [34032:0] info: average recursion processing time 0.227232 sec
May 13 09:26:55 fw254 unbound: [34032:0] info: histogram of recursion processing times
May 13 09:26:55 fw254 unbound: [34032:0] info: [25%]=0.16384 median[50%]=0.24576 [75%]=0.378652
May 13 09:26:55 fw254 unbound: [34032:0] info: lower(secs) upper(secs) recursions
May 13 09:26:55 fw254 unbound: [34032:0] info:    0.016384    0.032768 2
May 13 09:26:55 fw254 unbound: [34032:0] info:    0.065536    0.131072 1
May 13 09:26:55 fw254 unbound: [34032:0] info:    0.131072    0.262144 8
May 13 09:26:55 fw254 unbound: [34032:0] info:    0.262144    0.524288 9
May 13 09:26:55 fw254 unbound: [34032:0] notice: Restart of unbound 1.8.1.
May 13 09:26:57 fw254 unbound: [34032:0] notice: init module 0: validator
May 13 09:26:57 fw254 unbound: [34032:0] notice: init module 1: iterator
May 13 09:26:57 fw254 unbound: [34032:0] info: start of service (unbound 1.8.1).
May 13 09:26:57 fw254 unbound: [34032:3] info: generate keytag query _ta-4f66. NULL IN
May 13 09:29:11 fw254 unbound: [34032:0] info: service stopped (unbound 1.8.1).
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 3: 121 queries, 77 answers from cache, 44 recursions, 0 prefetch, 0 rejected by ip ratelimiting
May 13 09:29:11 fw254 unbound: [34032:0] info: server stats for thread 3: requestlist max 16 avg 1.47727 exceeded 0 jostled 0
May 13 09:29:11 fw254 unbound: [34032:0] info: average recursion processing time 0.145298 sec
May 13 09:29:11 fw254 unbound: [34032:0] info: histogram of recursion processing times
May 13 09:29:11 fw254 unbound: [34032:0] info: [25%]=0.0428505 median[50%]=0.0786432 [75%]=0.209715

johnpoz

Couple of things off the bat - if unbound is restarting every few seconds you for sure have an issue.. Do you have that many dhcp clients? Turn off registering dhcp clients - see if unbound settings down a bit..

Also rsahome.com is a valid domain on the internet.. Do you own it? registered with godaddy
user@uc:~$ whois rsahome.com
Domain Name: RSAHOME.COM
Registry Domain ID: 324787846_DOMAIN_COM-VRSN

Out of the box unbound is set to transparent for domains, ie if there is nothing local for host it will try and resolve it public.. It is normally a bad idea to use local a domain athat is same as public domain - especially if you do not actually own said domain and control it public..

You can stop unbound trying to resolve stuff in domainX.com that you are using locally from setting the zone type to static vs transparent..

But big problem is if restarting every few seconds its really going to be not very usable at all.

rsaanon

@johnpoz Thanks John for always quickly chiming in and responding. It's much appreciated!

I don't think we have too many DHCP clients. I would say about 15-20 DHCP clients. I will turn off the DHCP server to see if that affects the unbound.

Actually, rsahome.com is something I had just made up for this post. The actual internal domain I am using is rsaanon.com which is not a valid/registered Internet domain. Since my internal network is L2 segmented via VLANs, I also have the following internal subdomains: iot.rsaanon.com and kids.rsaanon.com. Any queries made by the clients on LAN to the pfSense/Unbound, results in pfsense box doing the name resolution. Also, any queries made to any non-local domains/subdomains (e.g.: pbs.org, etc.) that have not been previously queried/cached are forwarded to 1.1.1.1

~~BTW, anyway to start the unbound from the GUI in debug mode? The unbound logs do not show any detailed messages regarding what's causing it to continuously retart.~~ Nevermind. Figured out how to change loglevel.

johnpoz

I personally wouldn't recommend using any domainname.tld where the tld is public - while the domainname might not be currently used today, somebody could register it tmrw.

Use something like .lan as your tld - this is not a valid public tld, and prob will not become one.. Guess it could be currently its not.

Common practice for larger org's that like to use their own domain name internally and externally is use say domain.com externally and say domain.net internally. Where they own both.

Where that 160.124 address coming from is odd.. I show that netblock as
route: 160.124.0.0/16
descr: Posix Systems, South Africa

And I show no PTR for that IP.. so yes that indeed is strange.. So your forwarding to 1.1.1.1.. Are you in the South Africa region of the world? Is possible they are handing back some sort of block address or parking page for stuff that does not resolve... I get back NX when I try and resolve that, or even directly ask 1.1.1.1 about it.. Possible your ISP is doing some hijacking and returning that 160 IP???

Did you alter that IP in your post as well and its actually returning something else?

You don't need to turn off your dhcpd - just turn off the option in unbound to register it.. Which could restart unbound everytime say a client renews its dhcp lease - which if you have a really short dhcp lease time x20 clients could be a lot of restarts, etc.

edit: Back to that 160 IP... it is serving up http..

Very strange what could be serving you that IP... Can you do a sniff on your wan, flush your unbound cache and then query for something that returns that IP... Or again did you alter the IP you are getting back to some random IP you made up?

rsaanon

@johnpoz Back in the good 'ole days, we had a handful of TLDs (e.g: .net, .com, gov, .edu, etc.) . Now a days, there are crazy number of TLDs. So, while .lan is not registered as the TLD today, it might be tomorrow :) Also, since I'm using name resolution internally on LAN only, I don't want to pay $$ to registrars.

As for 160.24.0.0/16, not only am I perplexed but I also have security concern as the geo. loc. for this block is in South Africa. I am in the eastern part of the U.S. Also, any LAN traffic destined for port #53 is intercepted and handled by the pfsense box. So, for the case where a LAN client changes the default DNS from the pfsense IP to some external DNS would be handled/intercepted by the pfSense. So again, all DNS queries on the internal network is handled by the pfsense. Any queries that the unbound doesn't have a record for in its cache is forwarded to the quad 1s.

I had thought about what you had mentioned (i.e.: ISP hijacking the query being forwarded to 1.1.1.1). However, I don't think that's the case because I recently changed my ISP and I still see the same 160.x IP when a LAN client makes a DNS query to some bogus host internally or externally.

No, I did not alter the 160.x IP. 160.124.44.70 is the actual IP that's returned from the unbound process for any host names that are bogus. Valid hostname/domains resolution has no issues. Regarding, DHCP restarts, the dhcp lease time is left at default to 7200 secs (2hrs).

Wow!! You're right 160.124.44.70 is in fact serving HTTP. Now, I'm starting to wonder if my system has been compromised. I will flush the unbound cache and sniff on the WAN & report back.

BTW, here's a trace route to the 160.x destination starting at hop 5 (for security reasons):

Thank you, John!

johnpoz

@rsaanon said in Unknown hosts always resolving to particular IP:

lan is not registered as the TLD today, it might be tomorrow

I use local.lan for my public... but any non public tld is going to be better than using some domain with a public tld.. I am betting .lan not registered if so years out...l but could just switch to .lanlocal then ;) hehehe or .locallan

Agree it could be - but what I can tell you is more likely is name you choose is registered in com ;) Lan would make a HORRIBLE public tld choice.. But sure anyone with the money could register that if OK'd etc..

Lets see this sniff to what is returning that 160 address - very odd for sure..

also odd with that IP is its own by AFRINIC, but the lang is Chinese? very strange..

You can see from your trace though that your going through HK.. so either someone hijacked the routing for that address space.. Or RIR not updated their info..

rsaanon

@johnpoz Not sure how to clear the unbound cache. Restarting the unbound doesn't clear cache.

johnpoz

yeah it would.. you sure it actually restarted.

Ok looks like maybe the RIR has just not been updated, or their whois info is outdated.. I show this on radb

route:      160.124.0.0/16
descr:      CMI  (Customer Route)
origin:     AS40065
mnt-by:     MAINT-AS58453
changed:    qas_support@cmi.chinamobile.com 20181018
source:     RADB

route:      160.124.0.0/16
descr:      Powerline-Internal
origin:     AS132839
mnt-by:     MAINT-AS132839
changed:    noc@powerline.hk 20171115  #07:43:26Z
source:     RADB

route:      160.124.0.0/16
descr:      Olivetti Africa (care of Internet Solutions)
origin:     AS6083
mnt-by:     MAINT-AS3741
changed:    andre@is.co.za 20031024
source:     RADB

My guess is was moved - and RIR AFRINIC never updated their info ;)

rsaanon

@johnpoz Back from a long break.

I think I can explain what's happening in terms of 160.x resolution. I used to own a domain that I let expire. I bet some entitiy in HK bought that domain for reselling or nefarious purposes ;). Now, since I no longer own that domain, but some of my internal lan hosts still references to that FQDN, the name resolution requests get sent out to the forwarder as configured on the pfSense which return 160x address. So, there we have it..

Thanks for your help, @johnpoz !