Internal Hosts Resolving to External WAN IP

maravilla

I am experiencing an issue where I cannot resolve, but I believe I know what the issue is (but, unsure why it's happening or how to fix it).

BACKGROUND:
(a lot of it is probably irrelevant, but I wanted to give the full history the issue(s) I am experiencing)

System: Super Micro 1537
Version: 23.09.1-RELEASE (amd64)
DNS: DNS Resolver (Unbound)
DHCP: ISC DHCP

This part may not matter, but my original domain name for pfsense was example.io (the FQDN for my pfsense was pfsense.example.io). I ended up changing the domain to internal.example.io (FQDN for pfsense is pfsense.internal.example.io) so that its would be the default domain for all of my internal hosts to differentiate the hosts I would want to expose to the internet later, which would use the domain <host>.example.io.

I setup HAProxy and generated a wildcard Let's Encrypt certificate (*.internal.example.io) so that I could secure some internal applications, access them through the proxy and use the cert on pfsense itself, instead of the self-signed GUI certificate. Everything seemed to work okay, as far as accessing my applications - following the How To Guide For HAProxy and Let's Encrypt on pfSense: Detailed Steps for Setting Up Reverse Proxy.

I ran into some issues when attempting to deploy VMs via a newly set up rancher (rancher.internal.example.io) to a newly set up harvester (harvester.internal.example.io) and removed the host overrides needed that pointed the hosts to HAProxy to troubleshoot. Thats where I realized that any host with the domain of .internal.example.io and any hostname (whether it exists or not) without the domain pings to my WAN's IP.

I ran into some other issues with old hosts that I planned on migrating to the new internal domain, but currently has the domain of example.io and host overrides in the DNS Resolver.

Not sure what was going on at the time and facing other issues with pfsense, I just decided to re-image the system and start from scratch. When setting up the bare minimum to get the network back online (VLANS and ANY/ANY firewall rules), I noticed the same exact issue persisted.

ISSUE:

Pinging any hostname or FQDN pings the WAN external IP address
Dig shows any FQDN has an A record of my public IP address and NS records of public registrars
Dig shows any valid internal hostname (without the domain name) has an SOA of a.root-servers.net. nstld.verisign-grs.com., but SERVER: returns the interface's IP of the VLAN of which the host's resides (ex. 192.168.2.1)

I suspect that the Let's Encrypt wildcard I generated (as explained in the background) is the culprit here. Also my limited knowledge of how DNS works... I'm sure everything I have said so far is most likely the expected behavior based off of my configuration(s).

GOALS:

Use my purchased domain for external access (<host>.example.io) and also use it for internal hosts/applications, but differentiate internal hosts with a subdomain of my purchased domain (<host>.internal.example.io). As an example, my website should be reached internally via hostname "app1", by it's FQDN "app1.internal.example.io" and (if it has one) by it's external name "app1.example.io". Externally, it should only be reached by "app1.example.io".

I'd also like to use HAProxy to access all internal and external applications, while maintaining the aforementioned DNS setup.

Any help on resolving this issue and the correct setup/configuration would be much appreciated!

maravilla

Update:

I had wildcard A entries within my external DNS provider for example.io and internal.example.io and I removed them. That helped with most of my issues. However, I still have one A record that points to a host host.example.io and I also have it listed within the DNS Resolver host overrides. My original issue persists for this host, internally. If there is a host override, when I ping the FQDN (which happens to be the same as the external host and domain name) shouldn't it resolve to the local IP?

SteveITS

Yes, a wildcard DNS entry overrides all others.