2.4.1: local DNS not working

JKnott

Here's the relevant lines from /etc/resolv.conf

nameserver 2607:fea8:4cdf216:17ff:fea7:xyz
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

The first is my firewall, with address changed to protect the guilty and the other 2 are Google.

With resolver enabled.

To pfSense DNS

$ dig @2607:fea8:4cdf216:17ff:fea7:xyz google.com A

; <<>> DiG 9.9.9-P1 <<>> @2607:fea8:4cdf216:17ff:fea7:xyz google.com A
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

$ dig @2607:fea8:4cdf216:17ff:fea7:xyz google.com AAAA

; <<>> DiG 9.9.9-P1 <<>> @2607:fea8:4cdf216:17ff:fea7:xyz google.com AAAA
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

To Google DNS
$ dig @2001:4860:4860::8888 google.com A

; <<>> DiG 9.9.9-P1 <<>> @2001:4860:4860::8888 google.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65367
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN A

;; ANSWER SECTION:
google.com. 299 IN A 172.217.0.238

;; Query time: 48 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Sun Nov 05 22:19:49 EST 2017
;; MSG SIZE rcvd: 55

$ dig @2001:4860:4860::8888 google.com AAAA

; <<>> DiG 9.9.9-P1 <<>> @2001:4860:4860::8888 google.com AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 990
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN AAAA

;; ANSWER SECTION:
google.com. 299 IN AAAA 2607:f8b0:400b:808::200e

;; Query time: 84 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Sun Nov 05 22:20:34 EST 2017
;; MSG SIZE rcvd: 67

As you can see in the above, pfSense fails and Google works. When I switch pfSense to forwarder, it works fine.

BTW, I run openSUSE Leap 42.3.

Derelict

Are you passing IPv6 DNS into that interface?

Are you listening for DNS on that interface? Meaning does the resolver have that interface or All interfaces selected?

What is the output of this command run on the firewall?

netstat -an | grep LISTEN | grep 53

Does the DNS Resolver log show anything interesting?

When I switch pfSense to forwarder, it works fine.

And the forwarder is probably configured to forward to IPv4 name servers. So there might be a problem with IPv6 traffic from the firewall itself or maybe something else. Really hard to say with the information that has been provided. It is generally pretty difficult when someone has it set in their head that pfSense is the broken component and not a misconfiguration of the same..

Gertjan

@JKnott:

…..
To pfSense DNS

$ dig @2607:fea8:4cdf216:17ff:fea7:xyz google.com A
=> connection timed out; no servers could be reached

$ dig @2607:fea8:4cdf216:17ff:fea7:xyz google.com AAAA
=> connection timed out; no servers could be reached

Repeat - and force to use IPv4 and IPv6 :
dig -4 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A
and
dig -6 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A

hda

Anyway, my setup does work as expected ;)

With a simple Resolver DNSSEC config:

Network Interfaces:
LAN
OPT1
OPT2
Localhost

Outgoing Network Interfaces:
Localhost


[2.4.2-DEVELOPMENT][root@apu2b2.thisplaced]/root: netstat -an | grep LISTEN | grep 53
tcp4       0      0 127.0.0.1.953          *.*                    LISTEN
tcp6       0      0 ::1.53                 *.*                    LISTEN
tcp4       0      0 127.0.0.1.53           *.*                    LISTEN
tcp6       0      0 2001:beaf:babe:3:.53   *.*                    LISTEN
tcp4       0      0 192.168.22.1.53        *.*                    LISTEN
tcp4       0      0 10.8.4.1.53            *.*                    LISTEN
tcp6       0      0 2001:beaf:babe:1:.53   *.*                    LISTEN
tcp4       0      0 192.168.1.1.53         *.*                    LISTEN
[2.4.2-DEVELOPMENT][root@apu2b2.thisplaced]/root: cat /etc/resolv.conf
nameserver 127.0.0.1
search thisplaced


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Nov  5 23:57:37 2017 from 192.168.1.115
pi@Pi-df-RED:~ $ cat /etc/resolv.conf
# Generated by resolvconf
domain thisplaced
nameserver 192.168.22.1
nameserver 2001:beaf:babe:3::1
nameserver 2001:beaf:babe:1::1
pi@Pi-df-RED:~ $ dig @2001:beaf:babe:3::1 google.com

; <<>> DiG 9.9.5-9+deb8u13-Raspbian <<>> @2001:beaf:babe:3::1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30509
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       172.217.17.46

;; Query time: 34 msec
;; SERVER: 2001:beaf:babe:3::1#53(2001:beaf:babe:3::1)
;; WHEN: Mon Nov 06 13:26:35 UTC 2017
;; MSG SIZE  rcvd: 55

pi@Pi-df-RED:~ $

JKnott

Repeat - and force to use IPv4 and IPv6 :
dig -4 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A
and
dig -6 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A

$ dig -4 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A
dig: couldn't get address for '2607:fea8:4cdf:ef00:216:17ff:fea7:f2d3': address family not supported

As expected, forcing an IPv4 query to an IPv6 address won't work.

$ dig -6 @2607:fea8:4cdf216:17ff:fea7:xyz google.com A

; <<>> DiG 9.9.9-P1 <<>> -6 @2607:fea8:4cdf:ef00:216:17ff:fea7:f2d3 google.com A
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Same result as before.

Are you passing IPv6 DNS into that interface?

Are you listening for DNS on that interface? Meaning does the resolver have that interface or All interfaces selected?

My desktop computer is configured only with IPv6 DNS addresses, so yes IPv6 is enabled. Also, as mentioned earlier, forwarder works fine, so that would rule out any address issues. I did not make any changes from 2.4.0, where resolver worked to 2.4.1, where it fails.

root: netstat -an | grep LISTEN | grep 53
tcp4 0 0 127.0.0.1.953 . LISTEN
tcp6 0 0 ::1.53 . LISTEN
tcp4 0 0 127.0.0.1.53 . LISTEN

It's listening only on the loopback. Prior to the update, I had resolver network interfaces configured to all local networks and outgoing interface to WAN. However, after the update, that failed, giving errors as described in the other thread.

However, I think I just found the problem. When I was trying to resolve the problem initially, someone mentioned to select All & All for the interfaces. That didn't work and the config wouldn't let me save just the local networks. I'd get the error "This system is configured to use the DNS Resolver as its DNS server, so Localhost or All must be selected in Network Interfaces". I had been running with WAN for the outgoing interface and localhost for network, as that was one of the 2 allowed in the configuration. I had previously tried ALL and it failed, but appears to be working now. Why is it now necessary to choose ALL, when previously it worked with selected interfaces?

hda

@JKnott:

Why is it now necessary to choose ALL…

Well, it isn't mandatory. See my post #29.

Derelict

It is listening only on loopback.

Then you need to port forward DNS requests to the loopback interface or make it listen on the interfaces you are trying to use as name servers by selecting those interfaces in the resolver configuration.

Like I said MANY TIMES! Select "All" for the interfaces in the resolver config and you'll get this:

tcp4 0 0 *.53 . LISTEN
*tcp6 0 0 .53 . LISTEN

Who knows what weird configuration you had that was working and now isn't.

This is all I see for unbound from 2.4.0 to 2.4.1.

https://redmine.pfsense.org/issues/7884

https://redmine.pfsense.org/issues/7814

JKnott

@hda:

@JKnott:

Why is it now necessary to choose ALL…

Well, it isn't mandatory. See my post #29.

I had something similar to that and it work well, until 2.4.1.

Derelict

Sigh.

johnpoz

@JKnott:

My main computer uses static configuration for DNS, with IPv6 addresses for pfSense and Google DNS servers. Devices that connect via DHCP get the IPv4 address for pfSense DNS for the 1st DNS server and 8.8.8.8 & 4.4.4.4 for 2nd & 3rd.

This is so freaking BORKED!

Derelict is right on the money when he says that is broken..

When you state " first DNS to try and Google as the 2nd, should the first fail."

Doesn't work that way!! Your logic is that client always asks pfsense… What does pfsense return for what you queried? NX, Refused, Error? Timeout.. What? So then you believe client asks google for whatever it is you asked. Now the next thing it wants to lookup up you believe it goes back to ask your 1st listed dns?

Sorry but it does NOT work that way!! You have no control over which dns is going to get asked what from a client.. Especially with windows... When you point a client to more than 1 nameserver.. These multiple name server need to be able to return the same info... You do not point a ns that resolve local, and then also point to a ns that can not resolve your local.. If for some reason your client asks google for a local record.. its going to send back NX... Once a dns client gets NX why would it go ask a different NS.. It was told that record doesn't exist.. Not that ns timed out.. Or sorry go ask someone else I don't know.. It got told that record does not exist - period.. So why should ask some other NS hoping the answer is different..

dig @8.8.8.8 pfsense.local.lan

; <<>> DiG 9.11.2 <<>> @8.8.8.8 pfsense.local.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 52361
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

My client will not ask another NS in its list after getting that.. There is no point to it.. So pointing clients to different ns that can not resolve the same info is asking for problems plain and simple..

From MS
https://technet.microsoft.com/en-us/library/cc961411.aspx
"If at any point the DNS Client service receives a negative response from a server, it removes every server on that adapter from consideration during this search. For example, if in step 2, the first server on Alternate Adapter A gave a negative response, the DNS Client service would not send the query to any other server on the list for Alternate Adapter A."

JKnott

@johnpoz:

@JKnott:

My main computer uses static configuration for DNS, with IPv6 addresses for pfSense and Google DNS servers. Devices that connect via DHCP get the IPv4 address for pfSense DNS for the 1st DNS server and 8.8.8.8 & 4.4.4.4 for 2nd & 3rd.

This is so freaking BORKED!

Derelict is right on the money when he says that is broken..

When you state " first DNS to try and Google as the 2nd, should the first fail."

Doesn't work that way!! Your logic is that client always asks pfsense… What does pfsense return for what you queried? NX, Refused, Error? Timeout.. What? So then you believe client asks google for whatever it is you asked. Now the next thing it wants to lookup up you believe it goes back to ask your 1st listed dns?

Sorry but it does NOT work that way!! You have no control over which dns is going to get asked what from a client.. Especially with windows... When you point a client to more than 1 nameserver.. These multiple name server need to be able to return the same info... You do not point a ns that resolve local, and then also point to a ns that can not resolve your local.. If for some reason your client asks google for a local record.. its going to send back NX... Once a dns client gets NX why would it go ask a different NS.. It was told that record doesn't exist.. Not that ns timed out.. Or sorry go ask someone else I don't know.. It got told that record does not exist - period.. So why should ask some other NS hoping the answer is different..

dig @8.8.8.8 pfsense.local.lan

; <<>> DiG 9.11.2 <<>> @8.8.8.8 pfsense.local.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 52361
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

My client will not ask another NS in its list after getting that.. There is no point to it.. So pointing clients to different ns that can not resolve the same info is asking for problems plain and simple..

From MS
https://technet.microsoft.com/en-us/library/cc961411.aspx
"If at any point the DNS Client service receives a negative response from a server, it removes every server on that adapter from consideration during this search. For example, if in step 2, the first server on Alternate Adapter A gave a negative response, the DNS Client service would not send the query to any other server on the list for Alternate Adapter A."

Who said anything about running Windows? I try to avoid using it. My computers run mainly Linux and I provided a link to a Linux man page about resolv.conf that showed it worked exactly as I said. Also, what's borked about a device that uses DHCP to get it's IPv4 config also using IPv4 DNS addresses? Last I checked IPv4 DHCP can't provide IPv6 addresses and I don't run DHCPv6. So, what exactly is "borked"?

johnpoz

Pointing to local NS and public NS that can not resolve your local is BORKED - plain and simple.. This has been this way since the start of DNS..

Please show me your sniff of your dns queries coming from your machine showing it asking another NS after you got a NX from some public NS..

Unless the 1st server ask returns SERVFAIL or Timesouts - why should the clients dns move to the second NS in the list.. NX stops all queries.. Since it got an answer that the record in question does not exist, why bother asking anything else.

And even using say dnsmasq as local dns client that sends to all NS in parallel, it uses the first one to answer.. And doesn't look at any other responses, etc..

So how exactly is your linux box setup.. And what did it query and what did it get in response… Saying that you have pfsense, google for your NS 1, 2, 3 doesn't tell us what the problem is or isn't.

Do a simple query from the client.. Does it resolve or not.. If you query pfsense unbound on whatever address its listening on.. What is the response... This is a simple dig or nslookup or host command. Please post the output on why you think dns is not working... Saying its broke is just nonsense..

You don't tell you mech the car is broke.. You tell him or show him what is not working.. Unless you show us what is not working.. I resolve in this order staying in the dns theme - unless I can duplicate the error its PEBKAC.. Unless you can show me what its going on or not happening, PEBKAC..

Once I get back PEBKAC, I don't bother looking for other answers until actually get shown some info to work with..

How exactly do you expect unbound to work with

Outgoing Network Interfaces:
Localhost

Come on dude - really!!

kncar77

I concur. My DNS resolver broke after updating to 2.4.1
Have been using the same setup and have run every update since 2.2 and never had any problem.
I'm using a VPN client and am forcing the DNS resolver to query the root requests through the VPN tunnel rather than WAN, otherwise my middle eastern ISP will hijack the requests (verified with DNS leaktest). It has worked like a charm until 30 minutes ago after updating to 2.4.1.
When I manually add a DNS server (8.8.8.8 for example) on the clients it works but using the pfsense DNS resolver, then it times out.
All solved. Turned out to be a freak issue with my VPN provider rather than pfSense

Derelict

Please enumerate, completely and thoroughly, the steps you have taken in the DNS resolver, The OpenVPN client connection, and policy routing to effect such a configuration.

Saying "resolver broke" doesn't give anyone anything to go on.

JKnott

@johnpoz:

Pointing to local NS and public NS that can not resolve your local is BORKED - plain and simple.. This has been this way since the start of DNS..

Please show me your sniff of your dns queries coming from your machine showing it asking another NS after you got a NX from some public NS..

Unless the 1st server ask returns SERVFAIL or Timesouts - why should the clients dns move to the second NS in the list.. NX stops all queries.. Since it got an answer that the record in question does not exist, why bother asking anything else.

And even using say dnsmasq as local dns client that sends to all NS in parallel, it uses the first one to answer.. And doesn't look at any other responses, etc..

So how exactly is your linux box setup.. And what did it query and what did it get in response… Saying that you have pfsense, google for your NS 1, 2, 3 doesn't tell us what the problem is or isn't.

Do a simple query from the client.. Does it resolve or not.. If you query pfsense unbound on whatever address its listening on.. What is the response... This is a simple dig or nslookup or host command. Please post the output on why you think dns is not working... Saying its broke is just nonsense..

You don't tell you mech the car is broke.. You tell him or show him what is not working.. Unless you show us what is not working.. I resolve in this order staying in the dns theme - unless I can duplicate the error its PEBKAC.. Unless you can show me what its going on or not happening, PEBKAC..

Once I get back PEBKAC, I don't bother looking for other answers until actually get shown some info to work with..

How exactly do you expect unbound to work with

Outgoing Network Interfaces:
Localhost

Come on dude - really!!

Actually, my local IPv6 addresses are available via public DNS. I use a DNS service to do that. So, having an external DNS is just a back up. I have attached a capture showing the fall back in action. I modified the local DNS address in resolv.conf, to force queries to the Google DNS. Take a look at the transactions for google.ca. You'll first see an attempt to the phony local address. It fails, so the Google DNS is tried next. There are 3 separate examples in the capture for A, AAAA and MX records. In each case, the local lookup fails and then Google is tried. This is exactly the behavior I said would happen and also as described in the resolv.conf man page. Please do not try to confuse the issue by bringing in what Windows does, as I am not using Windows when I can avoid it. Also, who said they had localhost for the outgoing interface?

johnpoz, you know more than I do about pfSense and I assume you also know a lot about networking. However, here I'm seeing what I referred to before, where someone takes what's considered common knowledge and assumes it's gospel, without actually exploring the evidence and facts. In this thread, I earlier demonstrated with dig that the Google DNS was being used, when pfSense failed. I then posted a link to the resolv.conf man page, which describes how things work on Linux. Yet, despite that, you still insisted I was wrong and told me how it works in Windows, even though that is entirely irrelevant to my situation. Now, please go back and look at what I said, what the man page says and what the capture shows and show me if I'm wrong. Based on what I've provided, I am not.

BTW, I demonstrated to myself, years ago, that the DNS list works exactly as I described. That is the first DNS server is tried, then the 2nd, etc.. Please note, this is on Linux, not Windows.

DNS_Capture.pcap

Flamethrower

Hi to all!

I want to share a little observation about issue with DNS Resolver. I updated my PFSENSE couple of weeks ago without any issues, but today DNS Resolver plays dead for me. After some tests I seen Resolver service is stopped and tried to put it up, with no effect, it just not starting, no errors, nothing. So I tried to switch off DNSSEC for a test, and when I clicked "Save", I've got this:

–----
The following input errors were detected:

The generated config file cannot be parsed by unbound. Please correct the following errors:
/var/unbound/test/unbound_server.pem: No such file or directory
[1510244677] unbound-checkconf[89948:0] fatal error: auto-trust-anchor-file: "/var/unbound/test/root.key" does not exist in chrootdir /var/unbound
–---

And - what a surpise! - there is no such folder /var/unbound/test/
So, I think we nailed the root of a problem.

P.S. Sorry for my bad English

JKnott

^^^^
I have also seen that error message and other issues. 2.4.1 took a working 2.4.0 installation and caused it to fail. Why should that happen?

kejianshi

I can verify that directory is also non-existent for me. Not sure why or if it is the cause of the breakage, but it is still broken. (1 VM Only is like this)

However, the file also seems to not exist on any of my machines that work just fine.

Flamethrower

a little update:

after another attempt to start Reslover I've got in log this one:

–--
Nov 10 12:14:31 unbound 31406:0 notice: init module 0: validator
Nov 10 12:14:31 unbound 31406:0 error: failed to read /root.key
Nov 10 12:14:31 unbound 31406:0 error: error reading auto-trust-anchor-file: /var/unbound/root.key
Nov 10 12:14:31 unbound 31406:0 error: validator: error in trustanchors config
Nov 10 12:14:31 unbound 31406:0 error: validator: could not apply configuration settings.
Nov 10 12:14:31 unbound 31406:0 error: module init for module validator failed
Nov 10 12:14:31 unbound 31406:0 fatal error: failed to setup modules

After that I checked /var/unbound/root.key and found it zero sized. I tried to rebuild the /var/unbound/root.key but on each startup attempt unbound truncates it again to 0 than fails to start.

Gertjan

@Flamethrower:

After that I checked /var/unbound/root.key and found it zero sized. …..

From what I make of it, this file is related to the DNSSEC unbound housekeeping.
The file is auto re-created rather often it seems.

unbound is unable to write to this pace ? A problem with the file system ?

Btw: this my file :

; autotrust trust anchor file
;;id: . 1
;;last_queried: 1510324020 ;;Fri Nov 10 15:27:00 2017
;;last_success: 1510324020 ;;Fri Nov 10 15:27:00 2017
;;next_probe_time: 1510363838 ;;Sat Nov 11 02:30:38 2017
;;query_failed: 0
;;query_interval: 43200
;;retry_time: 8640
.	172800	IN	DNSKEY	257 3 8 AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjFFVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoXbfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaDX6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpzW5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relSQageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulqQxA+Uk1ihz0= ;{id = 19036 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1422533205 ;;Thu Jan 29 13:06:45 2015
.	172800	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1502688003 ;;Mon Aug 14 07:20:03 2017

Also : DNSSEC is checked in my unbound setup (some of my sites are already DNSSEC comptabile - all of them will be seen as soon as I fully understand how this all works AND how to automatize the maintenance of it (which is rather daunting).
Example : http://dnsviz.net/d/test-domaine.fr/dnssec/

edit : or may be this : you checked DNSSEC, but your unbound can not connect to DNS servers to establish DNSSEC traffic (if that even exists - other then basic port 53 UDP and TCP streams)