DHCP contains a search domain, causing incorrect 404…

bradley.wagner

We're running into problems right now (seemingly only on Linux and OS X) machines because domains like alksdjflkajsdflk.com are resolving to our apps server when we do "nslookup <domain>". It seems to be a problem with the "search ourdomain.com" that ends up in my /etc/resolv.conf after I acquire a lease from the DHCP server.

"dig <domain>" does not return an answer, but "nslookup <domain>" returns apps.

These domains need to return 404.

Any ideas?

Thanks!</domain></domain></domain>

ktims

A 404 error has nothing to do with DNS. By the time you get a 404, DNS has necessarily already resolved to an IP running a web server. I think what you're looking for is an NXDOMAIN reply from the DNS server.

This is happening because you're not 'terminating' the domain when you pass it to nslookup. Some versions of nslookup will add the search domain to the end of your request (Windows nslookup works this way - it adds the search domain to every request whether it exists or not, unless you terminate the domain name). Try your query using 'nslookup alksdjflkajsdflk.com.' instead (note the trailing '.'). You should see the results you expect from nslookup. Applications may still query using the search domain though, depending on their configuration, so you may still want to remove it (and I can't really help you there).

That said, if you fixed the DNS server at alksdjflkajsdflk.com to not respond to every request with the IP address of your app server you wouldn't have problems either. I really hate wildcarded DNS entries, they cause far more problems than they solve and are only useful in very specific instances.

bradley.wagner

Thank you for your help, but we're seeing (in the browser) where we type in http://garbageurl.com and get redirected to our internal Apache instance's 404 page.

I believe this is happening in some way because garbage.com is ending up garbage.com.hannonhill.com

Does this make sense? Any help at all would be much appreciated.

Thanks!

Cry Havok

Sounds like an issue with your DNS server. Who manages your internal DNS server?

ktims

Yes, the main problem here is that the DNS server at hannonhill.com is providing a response when asked about 'garbage.com.hannonhill.com' that points to your web server. These sorts of issues are why wildcard DNS entries are almost always a bad idea. I would fix that first.

Are you actually having this issue in a web browser or other application, or is it just nslookup?

bradley.wagner

We manage it, or, I do ;)

Currently we do:

* IN A 10.0.1.110 (110 being our internal DNS server)

I always thought that was saying that all *.hannonhill.com goes to the proxy, sounds like you guys are saying that's not the case..

We were under the impression that it would only try to process *.hannonhill.com and not things like asdfasdfasdfsdf.com

I believe we're onto something with the wildcard DNS but I still don't understand why domains like adasdfasdf.com would become asdfasdf.com.hannonhill.com

Could you elaborate a bit more? I'm quite confused here, any help is much appreciated ;)

Thanks!

Cry Havok

Err, no, that says that <anything>resolves to 10.0.1.110. Wikipedia has a good article for detail.

Very simply, if the name you are looking up isn't qualified (doesn't end in .) then it is assumed to be in the current domain. This means that if your doman is hannonhill.com. and somebody does a DNS lookup for www.google.com, the first attempt will be for www.google.com.hannonhill.com. If you're using wildcard records then that will resolve.

There are 2 solutions. The first is to stop using wildcard records. The other is to educate people to always fully qualify hostnames. Guess which one will work ;)</anything>

bradley.wagner

Should the same be true in the browser though? For example, why if I type http://google.com does it correctly resolve to google (not http://google.com.ourdomain.com) but if I type http://asdfasdfasdfasdf.com it goes to our Apache's 404 page. It seems like if it's going to catch EVERYTHING we shouldn't be able to hit anything outside not just domains that don't exist.

Also, how would you go about taking an entire domain and pointing it to a single IP? For example, we also host a bunch of sub-domains on the whatever1.com and I don't want to have to add an entry to the DNS every time we setup a new sub-domain.

Thanks!

ktims

As far as I understand most resolvers will assume a fully-qualified query even if you don't specify one, and if they don't get a response will then check the search domain. nslookup, at least some implementations, doesn't do this.

I don't use wildcard DNS so I'm not sure if there's a way to specify that the wildcard should only go one level deep, but I suspect not.

bradley.wagner

@ktims:

As far as I understand most resolvers will assume a fully-qualified query even if you don't specify one, and if they don't get a response will then check the search domain. nslookup, at least some implementations, doesn't do this.

I don't use wildcard DNS so I'm not sure if there's a way to specify that the wildcard should only go one level deep, but I suspect not.

Does that mean that if I had an entry in my company's DNS for www.google.com in the mycompany.com origin:


$ORIGIN mydomain.com.
@     IN        SOA     apps.mydomain.com. (
                        2001062501 ; serial
                        21600      ; refresh after 6 hours
                        3600       ; retry after 1 hour
                        604800     ; expire after 1 week
                        86400 )    ; minimum TTL of 1 day

        IN      NS      apps.mydomain.com.

www.google.com     IN        A        10.0.1.110

it would actually result in www.google.com never making it out of the office? Also, if that's the case why isn't the wildcard entry doing that now? If it's resolving everything to an internal server, I should never be able to get out.

I think ultimately I would like it to STOP appending our domain to unqualified queries and stop sending:


search mydomain.com

in the DHCP leases and this is where PFSense comes back into it.

bradley.wagner

Another bizarre happening: when I remove the wildcard, things like jira.mydomain.com which used to resolve to our internally hosted JIRA instance no longer work even though there is definitely an entry for this in our public DNS that should route it back to our office and through our Apache instance.

I tried a "dig jira.mydomain.com" and got no answer from within the office even though I do get an answer from outside.

What gives?

ktims

Does that mean that if I had an entry in my company's DNS for www.google.com in the mycompany.com origin:

I'm pretty sure this is resolver implementation dependent. It depends if the resolver first searches the search domain, or first checks for a matching FQDN. I think most implementations do the latter, and only fall back to the search domain if no FQDN is found - in which case that record wouldn't normally be hit and www.google.com would resolve normally - but as you've seen with nslookup, this isn't always the case.

I tried a "dig jira.mydomain.com" and got no answer from within the office even though I do get an answer from outside.

I'm not totally sure about this one, the pfSense box could be acting as an authoritative server, or it could be a cache issue, or it could be that your zone isn't configured properly. Post your zone file and the results of dig +trace internally and we'll see.

I think ultimately I would like it to STOP appending our domain to unqualified queries and stop sending:

If you really want to override this, you'll need to modify /etc/inc/services.inc and change the default config. While you can override the domain-name option in config.xml, from a quick glance at the code, it looks like there's no way to disable it entirely. Removing it from the default config block should work though, look for the line that reads

option domain-name "{$syscfg['domain']}";
```and remove it, then restart dhcpd and re-request all your leases.

bradley.wagner

@ktims:

I'm not totally sure about this one, the pfSense box could be acting as an authoritative server, or it could be a cache issue, or it could be that your zone isn't configured properly. Post your zone file and the results of dig +trace internally and we'll see.

Here is our db.office file:


$ORIGIN hannonhill.com.

@     IN        SOA     apps.hannonhill.com. office.hannonhill.com. (
                        2001062501 ; serial
                        21600      ; refresh after 6 hours
                        3600       ; retry after 1 hour
                        604800     ; expire after 1 week
                        86400 )    ; minimum TTL of 1 day

        IN      NS      apps.hannonhill.com.

; external

          IN      A       64.13.223.43
www   IN      A       64.13.223.43

; internal

mysql  IN      A       10.0.1.103
proxy  IN      A        10.0.1.110

; used to have the following wildcard
; *      IN       A       10.0.1.110

The proxy is our front-end Apache server that handles all the proxying for things like support.hannonhill.com (our JIRA instance).

The results of the "dig +trace" are:


; <<>> DiG 9.4.2-P2 <<>> +trace
;; global options:  printcmd
.			518256	IN	NS	C.ROOT-SERVERS.NET.
.			518256	IN	NS	D.ROOT-SERVERS.NET.
.			518256	IN	NS	E.ROOT-SERVERS.NET.
.			518256	IN	NS	F.ROOT-SERVERS.NET.
.			518256	IN	NS	G.ROOT-SERVERS.NET.
.			518256	IN	NS	H.ROOT-SERVERS.NET.
.			518256	IN	NS	I.ROOT-SERVERS.NET.
.			518256	IN	NS	J.ROOT-SERVERS.NET.
.			518256	IN	NS	K.ROOT-SERVERS.NET.
.			518256	IN	NS	L.ROOT-SERVERS.NET.
.			518256	IN	NS	M.ROOT-SERVERS.NET.
.			518256	IN	NS	A.ROOT-SERVERS.NET.
.			518256	IN	NS	B.ROOT-SERVERS.NET.
;; Received 500 bytes from 10.0.1.110#53(10.0.1.110) in 1 ms

.			518400	IN	NS	a.root-servers.net.
.			518400	IN	NS	b.root-servers.net.
.			518400	IN	NS	c.root-servers.net.
.			518400	IN	NS	d.root-servers.net.
.			518400	IN	NS	e.root-servers.net.
.			518400	IN	NS	f.root-servers.net.
.			518400	IN	NS	g.root-servers.net.
.			518400	IN	NS	h.root-servers.net.
.			518400	IN	NS	i.root-servers.net.
.			518400	IN	NS	j.root-servers.net.
.			518400	IN	NS	k.root-servers.net.
.			518400	IN	NS	l.root-servers.net.
.			518400	IN	NS	m.root-servers.net.
;; Received 492 bytes from 128.63.2.53#53(H.ROOT-SERVERS.NET) in 100 ms

If I do a "dig support.hannonhill.com" from inside the office, I get:


; <<>> DiG 9.4.2-P2 <<>> support.hannonhill.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 47773
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;support.hannonhill.com.		IN	A

;; AUTHORITY SECTION:
hannonhill.com.		86400	IN	SOA	apps.hannonhill.com. office.hannonhill.com. 2001062501 21600 3600 604800 86400

;; Query time: 0 msec
;; SERVER: 10.0.1.110#53(10.0.1.110)
;; WHEN: Tue Mar  3 13:36:37 2009
;; MSG SIZE  rcvd: 88

But when I login to a server outside the office and try the same dig it correctly gets routed to our office:


; <<>> DiG 9.2.4 <<>> support.hannonhill.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28217
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;support.hannonhill.com.		IN	A

;; ANSWER SECTION:
support.hannonhill.com.	86400	IN	CNAME	office.hannonhill.com.
office.hannonhill.com.	86400	IN	A	69.15.214.217

;; Query time: 146 msec
;; SERVER: 66.33.216.208#53(66.33.216.208)
;; WHEN: Tue Mar  3 10:35:23 2009
;; MSG SIZE  rcvd: 77

I would think that since the support.hannonhill.com is not listed in our internal DNS it should be consulting the external DNS servers.

ktims

Okay, well your zone file doesn't include records for support or jira, so they won't be resolved. You also don't have a record for apps.hannonhill.com, which is specified as your NS server, and that could cause problems as well. This configuration will be even more problematic if the internal zone file is separate and different than what's published to the Internet DNS, which is probably why you're having these issues. If you provide a zone file, the server is authoritative for that domain, and it won't refer queries to another server for that domain.

If you want to have special internal DNS records for your LAN servers, I think the best way to do this is to create a separate subdomain, say office.hannonhill.com and use that for all internal services, and refer all other queries to the Internet NS for hannonhill.com. You should be able accomplish this with a zone file similar to the following, on your internal DNS server - the Internet DNS won't need to change:


$ORIGIN hannonhill.com.

@     IN        SOA     dns129.a.register.com. root.register.com. (
                        200012300; serial
                        10800; refresh after 6 hours
                        3600       ; retry after 1 hour
                        604800     ; expire after 1 week
                        86400 )    ; minimum TTL of 1 day

        IN      NS      dns129.a.register.com.
        IN      NS      dns249.d.register.com.
        IN      NS      dns025.b.register.com.
        IN      NS      dns243.c.register.com.

$ORIGIN office.hannonhill.com.

        IN      NS      ns1.office.hannonhill.com.

ns1     IN      A      10.0.1.1      ; your internal nameserver with this zone file

mysql   IN      A       10.0.1.103
proxy   IN      A       10.0.1.110

Basically this should refer any queries for hannonhill.com to the registered Internet name servers (I took these from what's currently set in DNS), but point resolvers at ns1.office.hannonhill.com. for that domain. You'd then set the internal pfSense domain to office.hannonhill.com. This a) won't expose any of your internal configuration to the internet since the Internet DNS doesn't contain the NS record for office.hannonhill.com, or any of the records under that domain, and b) should solve your wildcard DNS problem since you don't have any wildcards on the internal domain, which the local server is authoritative for. You can still use wildcards on the Internet DNS if you wish, but as long as the pfSense domain is office.hannonhill.com and you've got it set up this way, you shouldn't run into any issues.

bradley.wagner

@ktims:

Okay, well your zone file doesn't include records for support or jira, so they won't be resolved. You also don't have a record for apps.hannonhill.com, which is specified as your NS server, and that could cause problems as well. This configuration will be even more problematic if the internal zone file is separate and different than what's published to the Internet DNS, which is probably why you're having these issues. If you provide a zone file, the server is authoritative for that domain, and it won't refer queries to another server for that domain.

Sorry, I truncated everything but what I thought was the relevant part of the file. We do have an entry for the nameserver apps.hannonhill.com that points to its internal IP. I did not realize that with the zone file the server would be authoritative for that domain; that helps!

We'll look into creating another subdomain to be used internally. Still sounds like it won't solve the problem of having to put DNS entries in our zone file for all of our internal application servers that need to point to our Apache instance and then proxied to the appropriate place. The wildcard took care of this for us before.

ktims

I think your only alternative is to remove the search domain entirely, which also means your users will have to enter the FQDN in their browser/client. And it means you have to make a small change to the pfSense code.

I still think the best route is to actually maintain a zone file instead of using a catchall. Where wildcards are useful is sites like deviantart and sourceforge that use subdomains for user accounts. If you've got a relatively static set of hostnames it makes more sense to maintain a proper zone.