Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Split DNS not working

    Scheduled Pinned Locked Moved
    DHCP and DNS
    3
    18
    10.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      awesomekyle
      last edited by

      I'd like to access a web server from both inside and outside my home network. I then have my firewall (pfSense) named as example.com. From within my network, I can access every machine by it's fully resolved hostname (webserver.example.com, jenkins.example.com, nas.example.com, etc). This works great on all my machines.

      The issue comes into play when I try to add external access, specifically to my gitlab server. I own a domain and use dynamic DNS to update it to always point to my residential IP address. I've then opened port 443 to point to my internal gitlab IP and added a CNAME entry for gitlab.example.com to point to example.com. This works as expected, I can now access gitlab.example.com from both inside and outside my network.

      The issue is that OS X now has a problem resolving the proper IP address. gitlab now resolves to both 192.168.1.XXX and whatever my external IP address is. OS X inconsistently chooses the wrong IP to access. When I ping gitlab.example.com, sometimes I'll see my internal IP, sometimes my external. nslookup always seems to show the internal IP, while Network Utility's lookup function seems to show either both IPs (76.xx.xxx.xxx, 192.168.1.xxx) or just the external IP. This only seems to be a problem when accessing my server from the command line via git. The connection times out because it can't seem to find the server.

      Using Chrome, I seem to always be able to access the web client, regardless of which IP get's resolved. Safari can only access the server if the internal IP is resolved. When the external IP is used, Safari notifies me that it can't open the page "because the server where this page is located isn't responding." I can "force" the OS to forget the external IP by flushing the DNS cache (sudo killall -HUP mDNSResponder), which works for a few minutes, but then it goes right back to the external IP.

      Only OS X seems to have this problem, and I can reproduce it reliably with multiple Macs. Neither Linux or Windows seem to have this problem. Each of those always resolve the internal IP when inside the network, and the external IP outside the network. I posted this question on askdifferent.stackexchange.com, and someone mentioned I'm looking at Split DNS. After some investigation, I found this page, which seems to be exactly what I want. However, I already had Method 2 (Split DNS) set up, and I still have the same problem. Network Utility's lookup function will either respond with both the internal and external IP of my server, or just the external IP. Is there another setting I should look into to prevent pfSense from returning the external IP? My OS X client shows only 192.168.1.1 as the DNS server, and pfSense's DNS Server list has 127.0.0.1 as the first server. Why would it still be returning the external IP?

      1 Reply Last reply Reply Quote 0
      • KOMK
        KOM
        last edited by

        Do you have multiple DNS (internal & external) setup on the Macs?  Are they set to resolve in parallel?  I know nothing baout Mac networking, but perhaps you need to set a resolve order for your servers so that on LAN it goes to the LAN DNS first.

        1 Reply Last reply Reply Quote 0
        • A
          awesomekyle
          last edited by

          I don't. All my computers, including my Mac, only have my router listed as the DNS server.

          For reference, here are pfSense's DNS servers: http://i.imgur.com/dKbC5Xj.png

          DNS Forwarder settings: http://i.imgur.com/sbSaqxZ.png

          DNS Forwarder Split DNS settings: http://i.imgur.com/zgAByBc.png

          I've tried also using 127.0.0.1 as the authoritative DNS server for my domain, that doesn't work either.

          1 Reply Last reply Reply Quote 0
          • KOMK
            KOM
            last edited by

            Hmm, everything looks good to me at a glance.  Does your Mac have any DNS diagnostic tools so that you can see what's going on during a resolve?

            1 Reply Last reply Reply Quote 0
            • A
              awesomekyle
              last edited by

              None of the tools I've tried have helped, but they might be able to narrow it down. Here's the output from some of the diagnosis tools I do have:

              ➜  ~  host gitlab.example.com
gitlab.example.com has address 192.168.1.21
gitlab.example.com is an alias for example.com.
gitlab.example.com is an alias for example.com.
➜  ~  ping gitlab.example.com
PING example.com (76.XX.XXX.XXX): 56 data bytes
64 bytes from 76.XX.XXX.XXX: icmp_seq=0 ttl=64 time=1.869 ms
➜  ~  nslookup gitlab.example.com
Server:		192.168.1.1
Address:	192.168.1.1#53

Name:	gitlab.example.com
Address: 192.168.1.21

              As soon as I kill/flush the DNS cache, ping resolves the correct hostname again:

              ➜  ~  sudo killall -HUP mDNSResponder
➜  ~  ping gitlab.example.com
PING gitlab.example.com (192.168.1.21): 56 data bytes
64 bytes from 192.168.1.21: icmp_seq=0 ttl=64 time=3.395 ms

              And after some time (no idea how long or what triggers it), it'll go back to the external IP:

              ➜  ~  ping gitlab.example.com
PING example.com (76.XX.XXX.XXX): 56 data bytes
64 bytes from 76.XX.XXX.XXX: icmp_seq=0 ttl=64 time=3.127 ms

              I have no idea what would make this happen specifically on OS X. I found this link which describes a similar issue, but it looks like the issue there was specifically with the domain attached. I'm always trying to ping the FQDN.

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                dig is your friend.  Don't use nslookup.  Don't use host.  Don't use ping.

                Ask specific nameservers to resolve your name with:

                dig @nameserver_ip fqdn a

                I'd run some tests for you now but you didn't include the real fqdn in question.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • A
                  awesomekyle
                  last edited by

                  @Derelict:

                  dig @nameserver_ip fqdn a

                  Here are the responses:

                  ➜  ~  dig @8.8.8.8 gitlab.example.com a

; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 gitlab.example.com a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51009
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	A

;; ANSWER SECTION:
gitlab.example.com.	135	IN	CNAME	example.com.
example.com.	59	IN	A	76.XX.XXX.XXX

;; Query time: 191 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Aug 11 22:31:03 2014
;; MSG SIZE  rcvd: 70

➜  ~  dig @192.168.1.1 gitlab.example.com a

; <<>> DiG 9.8.3-P1 <<>> @192.168.1.1 gitlab.example.com a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46736
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	A

;; ANSWER SECTION:
gitlab.example.com.	1	IN	A	192.168.1.21

;; Query time: 30 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Aug 11 22:31:08 2014
;; MSG SIZE  rcvd: 56

                  Does the "AUTHORITY: 0" for the second option imply that pfSense is somehow not the authoritative DNS server for my domain? I have "Domain Overrides" set so 192.168.1.1 is the authoritative DNS server for my domain. I've also tried 127.0.0.1 to no avail as well.

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    Yes.  The dnsmasq overrides are non-authoritative.

                    What is the output of cat /etc/resolv.conf in an OS X terminal window?

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • A
                      awesomekyle
                      last edited by 

                      #
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain example.com
nameserver 192.168.1.1
nameserver 2601:7:2000:34:20c:xxxx:xxxx:xxxx

                      The IPv6 nameserver is my router.

                      The same settings are listed in System Preferences under DNS Servers and Search Domains.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        OK.  So we also have IPv6 in the mix.

                        All your external DNS looks good on dyn.

                        dig @192.168.1.1 fqdn a
                        dig @192.168.1.1 fqdn aaaa
                        dig -6 @2601:7:2000:34:20c:xxxx:xxxx:xxxx fqdn a
                        dig -6 @2601:7:2000:34:20c:xxxx:xxxx:xxxx fqdn aaaa

                        Does any of that turn up any clues as to where the problem is?

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • A
                          awesomekyle
                          last edited by 

                          ➜  ~  dig @192.168.1.1 gitlab.example.com a

; <<>> DiG 9.8.3-P1 <<>> @192.168.1.1 gitlab.example.com a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9719
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	A

;; ANSWER SECTION:
gitlab.example.com.	1	IN	A	192.168.1.21

;; Query time: 30 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Aug 12 06:59:45 2014
;; MSG SIZE  rcvd: 56

➜  ~  dig @192.168.1.1 gitlab.example.com aaaa

; <<>> DiG 9.8.3-P1 <<>> @192.168.1.1 gitlab.example.com aaaa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57959
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	AAAA

;; ANSWER SECTION:
gitlab.example.com.	572	IN	CNAME	example.com.

;; AUTHORITY SECTION:
example.com.	1772	IN	SOA	ns1.mydyndns.org. zone-admin.dyndns.com. 2011111880 10800 1800 604800 1800

;; Query time: 30 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Aug 12 06:59:48 2014
;; MSG SIZE  rcvd: 124

➜  ~  dig -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx gitlab.example.com a

; <<>> DiG 9.8.3-P1 <<>> -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx gitlab.example.com a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33894
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	A

;; ANSWER SECTION:
gitlab.example.com.	1	IN	A	192.168.1.21

;; Query time: 29 msec
;; SERVER: 2601:7:2000:34:xxxx:xxxx:xxxx:xxxx#53(2601:7:2000:34:xxxx:xxxx:xxxx:xxxx)
;; WHEN: Tue Aug 12 07:00:49 2014
;; MSG SIZE  rcvd: 56

➜  ~  dig -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx gitlab.example.com aaaa

; <<>> DiG 9.8.3-P1 <<>> -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx gitlab.example.com aaaa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48424
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;gitlab.example.com.		IN	AAAA

;; ANSWER SECTION:
gitlab.example.com.	329	IN	CNAME	example.com.

;; AUTHORITY SECTION:
example.com.	148	IN	SOA	ns1.mydyndns.org. zone-admin.dyndns.com. 2011111880 10800 1800 604800 1800

;; Query time: 31 msec
;; SERVER: 2601:7:2000:34:xxxx:xxxx:xxxx:xxxx#53(2601:7:2000:34:xxxx:xxxx:xxxx:xxxx)
;; WHEN: Tue Aug 12 07:00:51 2014
;; MSG SIZE  rcvd: 124

                          So it looks like Dyn is the authoritative DNS server. Is that likely what the problem is?

                          1 Reply Last reply Reply Quote 0
                          • DerelictD
                            Derelict LAYER 8 Netgate
                            last edited by

                            Do you have DNS overrides in pfSense for both gitlab.example.com and example.com?  I just looked at your images.  I would get rid of the domain override for .com and set host overrides for example  .com  and gitlab  example.com.

                            I have personally never used the dnsmasq overrides to set an A record on a 2nd level domain name.  I can see how that might be problematic.

                            dyn is going to be authoritative.  That's how it's set up.  I wouldn't worry too much about that.  From the mac you could also:

                            dig @192.168.1.1 example.com ns
                            dig -6 @@2601:7:2000:34:20c:xxxx:xxxx:xxxx example.com ns

                            To see what name servers are being returned.

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • A
                              awesomekyle
                              last edited by

                              @Derelict:

                              I would get rid of the domain override for .com and set host overrides for example  .com  and gitlab  example.com.

                              Do you mean set an override for example.com to point to 192.168.1.1 and gitlab.example.com to 192.168.1.21? I tried that and every *.example.com hostname resolved to my router, even gitlab.example.com.

                              @Derelict:

                              dig @192.168.1.1 example.com ns
                              dig -6 @@2601:7:2000:34:20c:xxxx:xxxx:xxxx example.com ns

                              
➜  ~  dig @192.168.1.1 example.com ns

; <<>> DiG 9.8.3-P1 <<>> @192.168.1.1 example.com ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19718
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.		IN	NS

;; ANSWER SECTION:
example.com.	21599	IN	NS	ns2200.dns.dyn.com.
example.com.	21599	IN	NS	ns3131.dns.dyn.com.
example.com.	21599	IN	NS	ns4191.dns.dyn.com.
example.com.	21599	IN	NS	ns1171.dns.dyn.com.

;; Query time: 231 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Aug 12 18:50:28 2014
;; MSG SIZE  rcvd: 125

➜  ~  dig -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx example.com ns

; <<>> DiG 9.8.3-P1 <<>> -6 @2601:7:2000:34:xxxx:xxxx:xxxx:xxxx example.com ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36010
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.		IN	NS

;; ANSWER SECTION:
example.com.	21599	IN	NS	ns2200.dns.dyn.com.
example.com.	21599	IN	NS	ns1171.dns.dyn.com.
example.com.	21599	IN	NS	ns3131.dns.dyn.com.
example.com.	21599	IN	NS	ns4191.dns.dyn.com.

;; Query time: 183 msec
;; SERVER: 2601:7:2000:34:xxxx:xxxx:xxxx:xxxx#53(2601:7:2000:34:xxxx:xxxx:xxxx:xxxx)
;; WHEN: Tue Aug 12 18:50:48 2014
;; MSG SIZE  rcvd: 125
                              1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by

                                No.  You have a domain override sending all .com queries to 192.168.1.1 (http://i.imgur.com/zgAByBc.png)  Delete that.

                                And the record for gitlab has .com as the domain name.  That should be gitlab with example.com as the domain.

                                Is this name server the pfSense DNS forwarder?  nameserver 2601:7:2000:34:20c:xxxx:xxxx:xxxx

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  I think this will fix your problem as long as that IPv6 name server is NOT external but is the local pfSense dns forwarder.  When you're doing split dns like this, you cannot have internal hosts querying both internal and external nameservers.

                                  1.  Delete all host overrides and domain overrides.

                                  2.  Create a host override for:

                                  host gitlab
                                  domain example.com
                                  ip 192.168.1.21

                                  3.  Create a host override for:

                                  host example
                                  domain com
                                  ip 192.168.1.21

                                  Try again.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    awesomekyle
                                    last edited by

                                    @Derelict:

                                    And the record for gitlab has .com as the domain name.  That should be gitlab with example.com as the domain.

                                    I should've made it more clear, it's domain is correct, I was just trying to obfuscate it.

                                    @Derelict:

                                    I think this will fix your problem as long as that IPv6 name server is NOT external but is the local pfSense dns forwarder.  When you're doing split dns like this, you cannot have internal hosts querying both internal and external nameservers.

                                    1.  Delete all host overrides and domain overrides.

                                    2.  Create a host override for:

                                    host gitlab
                                    domain example.com
                                    ip 192.168.1.21

                                    3.  Create a host override for:

                                    host example
                                    domain com
                                    ip 192.168.1.21

                                    Try again.

                                    That appears to have worked, though I'm not sure why. I had already tried removing the domain override, which didn't fix the issue. Adding the host override for example with the .com domain seems to have resolved it.

                                    Why is that? Is OS X resolving both gitlab.example.com and example.com simultaneously when I try to connect? Shouldn't the host override for gitlab be enough?

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      Is the IPv6 nameserver 2601:7:2000:34:20c:xxxx:xxxx:xxxx any server other than the pfSense DNS forwarder?

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • A
                                        awesomekyle
                                        last edited by

                                        @Derelict:

                                        Is the IPv6 nameserver 2601:7:2000:34:20c:xxxx:xxxx:xxxx any server other than the pfSense DNS forwarder?

                                        Nope, it's the pfSense LAN IPv6 address.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.