DNS Domain Override Problem

netgate_user_2024

Hello.

I have gone through many forum posts concerning this issue and have tweaked many settings trying to find a solution.
I will try to give a clear picture of our configuration and the issue we are facing.

We have 3 locations using pfSense firewalls. These are all connected with ipsec VPNs to each other.
2 of the locations are offices and 1 is just for offsite backups.
Not that it is needed so much for our simple setup but I also have OSPF configured on all 3 firewalls.
DHCP is being handled by pfSense at all 3 locations.
Our AD domain was set up long ago using our public base domain name.
Like this:
ourdomain.com (instead of something sensible like internal.ourdomain.com)
I have domain overrides on both offsite pfSense firewalls that point to our AD domain controllers at the main office. (So all lookups for ourdomain.com should be forwarded to them)

Our main office uses 2 domain controllers for DNS.
The other 2 locations use pfSense for DNS.

Now for the issues. I've been having trouble with backups in 2 ways which stem from DNS problems at our second office.
Sometimes the backup software is unable to find the offsite backup server. It's address is stored in our AD DNS.
Let's call it:
offsite-backup.ourdomain.com

Also sometimes the backup software cannot find the email proxy (address also stored in AD DNS):
emailproxy.ourdomain.com

These 2 things do not necessarily happen at the same time either. I think it just relates to timing and DNS caching.

Now what I have determined is if I go to the pfSense GUI and restart DNS Resolver and then go to Diagnostics/DNS Lookup I can lookup both of those addresses.

Then the issue will happen again after some time. Maybe a day or two.
The server will stop backing up or stop sending alert emails or both.

I've tried setting a scheduled task in pfSense to run this command (I also just tried running this from the GUI command prompt):

unbound-control -c /var/unbound/unbound.conf reload

But this command does not fix the problem like using the restart button for the DNS Resolver in the GUI does.

I found a post that mentioned how if a bad address was looked up by unbound for a domain (example: bad.mydomain.com) then unbound would stop looking up addresses for that entire domain.
I could be wrong about the details but I think that description is close.

Is there another command I can use to get the DNS resolver to fully restart?
Or does anyone have another suggestion for fixing this issue?

I could just point all of the devices at the offsite locations to use the AD DNS and be done with this but I wanted to allow them to still get access to the internet in case the VPN was down for some reason.

I could also deploy offsite AD controllers but this is overkill for our smallish network.

Thanks for any suggestions.

SteveITS

@netgate_user_2024 As long as there isn’t an Exchange server an AD domain can be renamed.

Is it only the backup software that has this problem?

Does the backup PC have any other DNS configured? Windows uses (short version) last known good DNS, not an order.

johnpoz

@netgate_user_2024 said in DNS Domain Override Problem:

I have domain overrides on both offsite pfSense firewalls that point to our AD domain controllers at the main office. (So all lookups for ourdomain.com should be forwarded to them)

And did set these as private domains - because if pfsense forwards, ie a domain override to somewhere and the answer is rfc1918 then that would be rebind.

https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html

netgate_user_2024

Hello and thanks for the replies.

Yes there is an on premises Exchange server. We are in the process of migrating to 365 but I do not know when or if the Exchange server will be retired.
No, all devices on the offsite networks have only their respective pfSense firewalls set as their DNS servers. Note that when there is a problem I run DNS lookups from pfSense's GUI and it also has the same issue. Unbound is either blocking lookups to the overridden domain or forgetting it somehow.
Regarding the DNS Rebinding Protections. Are you suggesting I exclude our domain from the rebinding protection?

Thanks again.

SteveITS

@netgate_user_2024 said in DNS Domain Override Problem:

suggesting I exclude our domain from the rebinding protection

It won't hurt, but per the note on that page, "This behavior is automatically overridden for domains in the DNS Resolver and DNS Forwarder domain override lists as the most common usage of that functionality is to resolve internal DNS hostnames." Plus if that was the problem it would always fail and not sometimes work.

@netgate_user_2024 said in DNS Domain Override Problem:

do not know when or if the Exchange server will be retired

That's another, OT, discussion but AFAIK going forward on premises Exchange is going to become a subscription rather than having a new version every 3 years. Plus last I knew the requirements for Exchange were oddly high, e.g. 128 GB minimum recommended for a mailbox server.

netgate_user_2024

@SteveITS

Thanks.

So I just implemented:

private-domain: "ourdomain.com"

I put it after the entry for pfBlocker that already existed in the custom options field.

If I'm understanding your suggestion and the documentation correctly, a lookup to our "local" domain which is technically our public domain resulting in both public and private space IP addresses is what is causing our problem due to the rebinding protection.

I will report back with the results once I've determined this fixes the issue.

And I do see the section that says it's automatically overridden but perhaps it's not working as expected. We do have IP addresses for our subdomains in both public and private IP spaces.

netgate_user_2024

Well that didn't take long.
I tested the 2 DNS entries on the offsite office pfSense just now and 1 of them could not be found.

Restarting the DNS resolver service in the GUI fixed the issue again as usual.

I'm not sure where to go from here.
Could this be a bug?

I could change my device's DNS servers, or maybe set DNS host overrides.
I would like for this just to work as expected though.

johnpoz

@netgate_user_2024 if you forward and the answer is a private it will not give that answer to the client asking.

If unbound asks a public for your fqdn that is not in public you won't get an answer. If you setup a domain override - all queries for that domain would go to where you forward.

If you use a public domain internally your going to have a bad day. When different records exist on public vs local.

netgate_user_2024

@johnpoz

Thank you.
Yes we have for example:
www.ourdomain.com externally hosted.
I have an A record for it in our AD DNS.

So all queries for ourdomain.com should be resolvable from our internal DNS whether they are private or public IP addresses.

I have done a domain migration to fix this type of a situation in the past and it is not an easy task.
(another company running Exchange on an AD with a single label domain lol)

If the rebinding protection exclusion does not work I will have to look at some work around.

Thanks again for the replies.